Electric power multi-mode corpus construction query method and system based on sliding window

Abstract

The invention discloses an electric power multi-modal corpus construction query method and system based on a sliding window, which is applied to the field of electric power data query, and comprises the following steps: obtaining a structured document according to electric power multi-modal data, segmenting the structured document to obtain a plurality of segmented text blocks, and storing the segmented text blocks into a database; inputting each segmented text block into a large language model to generate a to-be-stored text vector and construct an electric power multi-mode corpus, when a user query request is received, generating a plurality of query variants according to query data, performing nearest neighbor search on each query variant in the electric power multi-mode corpus to obtain a corresponding nearest neighbor search result, and storing the nearest neighbor search result in the electric power multi-mode corpus. And fusing each nearest neighbor search result to generate an electric power related document set comprising a multi-modal association mark. According to the method, the semantic units can be accurately captured, semantic breakage is avoided, the retrieval continuity and coverage rate are improved, the comprehensiveness and context adaptability of retrieval results are improved, and the data retrieval requirement under the complex scene of the power industry is met.

Description

Technical Field

[0001] This invention relates to the field of power data query technology, and in particular to a power multimodal corpus construction and query system based on a sliding window. Background Technology

[0002] With the accelerated digital and intelligent transformation of new power systems, the scale of power data is growing exponentially, covering multiple modal forms such as text, images, and time-series signals. The demand for efficient processing and analysis of multimodal data is becoming increasingly urgent.

[0003] Existing power data processing methods have several limitations, such as insufficient structured processing and integration capabilities during the construction of existing databases, difficulty in effectively mining the correlation between different types of data, and the fact that existing methods often fail to fully consider the semantic relationships between multimodal data when retrieving and querying data, resulting in limited accuracy of query results.

[0004] Therefore, improving the efficiency and accuracy of power data retrieval has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides a method and system for constructing and querying a multimodal corpus of power data based on a sliding window, thereby providing comprehensive and efficient query results for power data.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for constructing and querying a multimodal corpus of electricity based on a sliding window, comprising: The acquired multimodal data from the power industry is processed into structured documents, wherein the structured processing includes multimodal association tagging of the multimodal data.

[0007] The structured document is traversed and segmented based on a dynamic sliding window strategy to obtain several segmented text blocks. The traversal and segmentation process is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association tags during the segmentation process.

[0008] Each segmented text block is input into a pre-trained large language model to obtain the corresponding text vector to be stored, and an electric power multimodal corpus is constructed based on each of the text vectors to be stored.

[0009] In response to a query request from a target user, the obtained query data of the target user is input into the large language model to obtain several query variants corresponding to the query data.

[0010] For each of the query variants, a nearest neighbor search is performed on the power multimodal corpus to obtain the corresponding nearest neighbor search results. The nearest neighbor search results are then merged to generate a power-related document set including the multimodal association tags.

[0011] Furthermore, the process of structuring the acquired multimodal data from the power industry to obtain structured documents includes: The acquired multimodal data from the power industry is formatted to obtain multimodal text data.

[0012] Entity recognition is performed on the multimodal text data based on the power knowledge graph to obtain the data entities in the multimodal text data.

[0013] Each data entity is associated with a corresponding modal identifier to form a multimodal association tag for each data entity.

[0014] The multimodal text data, after associating the data entities, is divided into several semantic units. Each semantic unit is semantically labeled, and the semantically labeled semantic units are integrated to obtain a structured document.

[0015] Furthermore, the multimodal data includes at least image data, PDF data, CSV data, and text data.

[0016] The process of unifying the format of the acquired multimodal data from the power industry to obtain multimodal text data includes: The image data is feature aligned using a CLIP encoder, and the aligned image data is classified and recognized using a trained image classification model to obtain the first text data.

[0017] The table content in the PDF data is extracted and converted into a text format with row and column markers to obtain the second text data.

[0018] The key-value pairs in the CSV data are mapped to natural language descriptive text to obtain the third text data.

[0019] By integrating the first text data, the second text data, the third text data, and the text data, multimodal text data is obtained.

[0020] Furthermore, the structured document is traversed and segmented based on a dynamic sliding window strategy to obtain several segmented text blocks, including: The sliding window is determined based on the average information density of the structured document.

[0021] The structured document is traversed using an initial sliding window. During the traversal, the information density and multimodal association markers corresponding to each window are identified and analyzed. The size of the sliding window is dynamically adjusted based on the analysis results.

[0022] The structured document is segmented based on the adjusted sliding window to obtain several segmented text blocks.

[0023] Further, the step of inputting each of the segmented text blocks into a pre-trained large language model to obtain the corresponding text vector to be stored includes: Each of the segmented text blocks is encoded to generate a corresponding input sequence.

[0024] Each of the input sequences is input into a pre-trained large language model to generate the corresponding text vector.

[0025] The text vectors are dimensionality reduced, and the dimensionality-reduced text vectors are normalized to obtain a fixed-length text vector to be stored.

[0026] Furthermore, the construction of the power multimodal corpus based on each of the text vectors to be stored includes: Vector indexes are constructed for each of the text vectors to be stored, resulting in a vector index library.

[0027] By integrating the vector index library with the multimodal association tag, a power multimodal corpus is constructed.

[0028] Further, the obtained query data of the target user is input into the large language model to obtain several query variants corresponding to the query data, including: The obtained query data of the target user is semantically parsed to obtain query semantic data.

[0029] The parsed query semantic data is input into a pre-trained large language model to generate several first query variants.

[0030] Calculate the similarity value between each of the first query variants and the query semantic data, and filter out the first query variants with similarity values ​​higher than a preset similarity value as query variants.

[0031] Further, the step of performing a nearest neighbor search on each of the query variants in the power multimodal corpus to obtain the corresponding nearest neighbor search result includes: Each of the query variants is vectorized to obtain a query variant vector.

[0032] The similarity between the query variant vector and each of the text vectors to be stored in the power multimodal corpus is calculated, and a number of text vectors to be stored are selected as nearest neighbor vectors based on the calculation results.

[0033] The nearest neighbor vectors are sorted according to their similarity to obtain the nearest neighbor search results.

[0034] Further, the fusion of the nearest neighbor search results to generate a power-related document set including the multimodal association tags includes: The nearest neighbor search results are subjected to semantic deduplication to obtain a set of semantic units.

[0035] The semantic unit sets are integrated according to the semantic relationships between them to obtain a power-related document set including multimodal association tags.

[0036] Another embodiment of the present invention provides a sliding window-based query system for constructing a multimodal corpus of electrical power data, comprising: The structuring module is used to perform structuring processing on the acquired multimodal data from the power industry to obtain structured documents. The structuring processing includes multimodal association tagging of the multimodal data.

[0037] The traversal segmentation module is used to perform traversal segmentation processing on the structured document based on a dynamic sliding window strategy to obtain several segmented text blocks. The traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association tags during the segmentation process.

[0038] The corpus construction module is used to input each of the segmented text blocks into a pre-trained large language model to obtain the corresponding text vectors to be stored, and to construct an electric multimodal corpus based on each of the text vectors to be stored.

[0039] The data query module is used to respond to the query request of the target user, input the obtained query data of the target user into the large language model, and obtain several query variants corresponding to the query data.

[0040] The data generation module is used to perform nearest neighbor search on each of the query variants in the power multimodal corpus to obtain the corresponding nearest neighbor search results, and to merge the nearest neighbor search results to generate a power-related document set including the multimodal association tags.

[0041] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following: By employing a dynamic sliding window strategy, the window size is flexibly adjusted based on information density and multimodal association markers to accurately capture semantic units, avoid semantic breaks, and improve retrieval coherence and coverage. Query variants are generated using a large language model and the results are fused to enhance the comprehensiveness and contextual adaptability of retrieval results. Furthermore, by using a large language model to generate query variants and perform multi-dimensional searches, the fused results generate a power-related document set, mitigating the semantic bias of single queries, improving the comprehensiveness and contextual adaptability of retrieval results, and better meeting the data retrieval needs of complex scenarios in the power industry. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the steps of a sliding window-based query method for constructing a multimodal power corpus in one embodiment of the present invention. Figure 2 This is a structural block diagram of a query system for constructing a multimodal power corpus based on a sliding window, according to one embodiment of the present invention. Figure labels: 21, Structured module; 22, Traversal and segmentation module; 23, Corpus construction module; 24, Data query module; 25, Data generation module. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0044] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0045] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0046] One embodiment of the present invention provides a method for constructing and querying a multimodal corpus of electricity based on a sliding window. For details, please refer to [link to documentation]. Figure 1 , Figure 1 The diagram shows a flowchart of the steps for constructing a query method for a multimodal power corpus based on a sliding window, as shown in one embodiment of the present invention, including: S11. Perform structured processing on the acquired multimodal data from the power industry to obtain structured documents. The structured processing includes multimodal association tagging of the multimodal data.

[0047] In data analysis within the power industry, data sources are diverse and varied, including but not limited to equipment operation data, sensor-collected data, maintenance records, fault reports, and related multimedia information such as images and videos. This data typically exists in different formats, such as tabular data, text files, and image files, posing significant challenges to subsequent data processing and analysis. Therefore, to efficiently process and analyze this multimodal data, it is first necessary to unify the format of the acquired data to obtain multimodal text data.

[0048] Specifically, the data to be formatted in this embodiment includes at least image data, PDF data, CSV data, and text data. The process of formatting different types of data in this embodiment includes: The CLIP (Contrastive Language–Image Pre-training) encoder is an advanced multimodal model that maps images and text to the same feature space. By processing image data through the CLIP encoder, feature representations of the image can be extracted and aligned with text features. This allows image data and text data to be compared and matched in the same feature space. Based on feature alignment, a pre-trained image classification model is used to classify and recognize the image data, and the classification result is converted into a text description, i.e., the first text data. For example, if the image classification model identifies the image as a "transformer," the first text data could be "The image contains a transformer."

[0049] PDF files are also a common data carrier in the power industry, often containing a large amount of tabular information, such as equipment parameter tables and operation record tables. PDF parsing tools are used to parse the PDF data, extracting the table structure, including rows, columns, and cells, and then converting it into text format with row and column markers to obtain the second text data.

[0050] CSV (Comma-Separated Values) files are a common data storage format, typically used to store key-value pairs. In the power industry, CSV files may contain information such as equipment parameters and operational data. To integrate CSV data with other modal data, it is necessary to map their key-value pairs to natural language descriptive text. For example, if a row of data in a CSV file is "Equipment Name, Transformer; Equipment Model, T100", it can be mapped to the natural language descriptive text: "Equipment Name is Transformer, Equipment Model is T100". Timestamp annotations (such as "Load Peak @ 2023-01-01 14:30") are added to time-series data columns with timestamps. After mapping the key-value pairs in the CSV data to natural language descriptive text, the resulting text is the third-party text data.

[0051] After processing the image data, PDF table data, and CSV data, three types of text data from different sources were obtained: first text data (text descriptions of image classification results), second text data (text format of PDF table content), and third text data (natural language descriptions of CSV key-value pairs). In addition, raw text data also exists. To achieve unified processing and analysis of multimodal data, it is necessary to integrate these text data from different sources to obtain multimodal text data.

[0052] When integrating text data from different sources, it is necessary to align the text data from different sources. The alignment can be based on the data's timestamp, device name, event identifier, etc. For example, if the first text data describes the image classification results of a device at a certain time, the second text data records the table information of the same device at the same time, and the third text data describes the parameter information of the device, then these text data can be aligned according to the device name and timestamp.

[0053] In data analysis within the power industry, multimodal text data integrates information from various modalities such as images, tables, and CSV files. This data can point to specific entity information, such as equipment names, fault types, and parameter values.

[0054] A power knowledge graph is a structured form of knowledge representation that organizes various entities in the power field (such as equipment, parameters, fault types, etc.) and their relationships in the form of a graph. Based on the power knowledge graph, entity recognition is performed on multimodal text data to obtain the data entities within the multimodal text data.

[0055] In multimodal text data, each data entity may originate from different modalities. For example, a device name may come from image classification results, while device parameter values ​​may come from tables or CSV files. To fully leverage the advantages of multimodal data, it is necessary to associate each data entity with its corresponding modality identifier, thereby creating an additional multimodal association tag for each data entity.

[0056] Based on the semantic structure and logical relationships of the text, multimodal text data is divided into several semantic units. The division of semantic units can be based on sentences, paragraphs, or specific semantic topics. For example, equipment operation records can be divided into one semantic unit, and fault reports into another. Then, using natural language processing techniques, such as text generation algorithms, the labeled semantic units are integrated into a structured document according to semantic logic. Semantic units such as equipment operation records and fault reports can be integrated into the same equipment operation status structure.

[0057] S12. The structured document is traversed and segmented based on a dynamic sliding window strategy to obtain several segmented text blocks. The traversal segmentation process is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association tags during the segmentation process.

[0058] Structured documents typically contain a wealth of information, but this information is not evenly distributed. Some sections may contain a large amount of key data, while others may be relatively sparse. To process these documents efficiently while ensuring that no key information is missed, a sliding window-based approach can be used to analyze and segment the documents.

[0059] First, the initial sliding window size needs to be determined based on the average information density of the structured document. Information density refers to the amount of effective information contained within a unit of text length, which can be calculated by statistically analyzing the distribution frequency of keywords, data entities, etc., in the document. Specifically, if a document contains 10 key data points per 100 characters, its information density can be considered to be 0.1. Based on the overall average information density of the document, an appropriate sliding window size can be initially set to ensure that sufficient key information is covered in subsequent analysis.

[0060] Next, the structured document is traversed using an initial sliding window. During the traversal, the content within each window is analyzed in detail. This includes two main aspects: first, identifying the information density within the window, i.e., calculating the concentration of effective information within the current window; and second, analyzing the multimodal association markers within the window to understand the source and modal characteristics of the information. If the information density within a window is significantly higher than the average and contains data entities of multiple modalities (such as the association information between image and text modalities), it indicates that the information within that window is relatively important and may require more detailed processing.

[0061] Specifically, based on the above analysis results, the size of the sliding window is dynamically adjusted. If the information density in the current window is high and the multimodal association markers indicate that the information is relatively complex, the window size can be appropriately reduced to allow for more detailed analysis of this key information. Conversely, if the information density in the window is low and the multimodal association markers show relatively simple information, the window size can be appropriately increased to improve processing efficiency. This dynamic adjustment mechanism ensures that key information can be captured while processing documents, and that important details are not missed due to an excessively large window.

[0062] The structured document is segmented based on the adjusted sliding window to obtain several segmented text blocks. During the segmentation process, it is also necessary to ensure that the segmented text blocks contain complete semantics and retain multimodal association tags.

[0063] S13. Input each segmented text block into the pre-trained large language model to obtain the corresponding text vector to be stored, and construct an electric multimodal corpus based on each text vector to be stored.

[0064] Each segmented text block is encoded and converted into an input sequence that a computer can process. These input sequences are then fed into a pre-trained large language model to capture the semantic information of the text and obtain a high-dimensional text vector.

[0065] Since text vectors generated by large language models typically have high dimensionality, this can negatively impact the efficiency and effectiveness of subsequent processing. Therefore, dimensionality reduction is necessary to improve processing efficiency and reduce storage costs. During dimensionality reduction, it's crucial to reduce vector dimensionality while preserving as much of the original vector's semantic information as possible.

[0066] After dimensionality reduction, the text vectors are normalized to adjust the values ​​of each text vector to a fixed range, making the vectors of different text blocks comparable, and obtaining a fixed-length text vector to be stored.

[0067] After obtaining a fixed-length text vector to be stored, a vector index is constructed on it to obtain a vector index library, which enables fast retrieval and matching of text vectors, thereby achieving fast querying and retrieval of text vectors and improving the efficiency of data processing.

[0068] Multimodal association tags contain source modal information of text blocks, such as image modality and table modality. By integrating the constructed vector index library and multimodal association tags, a power multimodal corpus is built. The power multimodal corpus not only contains the semantic information of the text, but also retains the multimodal features of the text.

[0069] S14. In response to the target user's query request, input the obtained query data of the target user into the large language model to obtain several query variants corresponding to the query data.

[0070] After the power multimodal corpus is constructed, target users can send query requests to retrieve the content they need from the multimodal corpus. The query request contains various information, such as the target user's account information, login status information, network link information, and query data. Upon receiving the target user's query request, the system responds by performing semantic parsing on the query data contained in the query request to obtain the query semantic data.

[0071] The query semantic data is input into a pre-trained large language model, which can generate multiple possible expressions, i.e., several first query variants, based on the input semantic data. These variants are semantically similar to the original query semantic data, but may use different vocabulary, sentence structures, or expressions. For example, for the query semantic data "fault records of device A", the large language model may generate multiple first query variants such as "fault logs of device A" and "what are the fault conditions of device A".

[0072] Since the number of first query variants generated by the large language model may exceed the required number of variants, and not all variants fully match the user's original intent, further filtering of these variants is necessary. Specifically, the system calculates the similarity value between each first query variant and the query semantic data. Similarity calculation can be implemented using various methods, such as cosine similarity based on semantic vectors and Jaccard similarity based on keyword matching.

[0073] By calculating similarity scores, the system can assess the semantic similarity between each variant and the original query. Based on a preset similarity threshold, the first query variant with a similarity score higher than the threshold is selected as the query variant. In actual querying, the preset similarity threshold needs to be adjusted according to actual needs. If the number of first query variants generated from a certain query's semantic data is too small, the threshold value can be appropriately reduced; conversely, the threshold value can be increased to ensure that the selected variants can both cover the target user's query intent and not contain too much irrelevant or redundant content.

[0074] S15. Perform nearest neighbor search on each query variant in the power multimodal corpus to obtain the corresponding nearest neighbor search results. Merge the nearest neighbor search results to generate a power-related document set including multimodal association tags.

[0075] When processing user query requests, the system first needs to convert each query variant into a vector form to obtain the query variant vector, in order to perform subsequent similarity calculations.

[0076] After obtaining the query variant vector, the system needs to calculate its similarity with each text vector to be stored in the power multimodal corpus. In this embodiment, the similarity calculation is achieved through cosine similarity.

[0077] Specifically, in this embodiment, each text vector to be stored in the power multimodal corpus is obtained by storing it in a vector database after vectorization processing. The vector database can be FAISS or Milvus. In this embodiment, the vector database used to obtain the power multimodal corpus is FAISS. Through the efficient retrieval capability of the vector database, the system can quickly find several text vectors to be stored that are most similar to the query variant vector. These vectors are used as nearest neighbor vectors.

[0078] Based on the similarity calculation results, the nearest neighbor vectors are sorted from high to low similarity to ensure that the most relevant documents are listed first.

[0079] To improve the quality of search results, semantic deduplication is required for nearest neighbor search results. The purpose of semantic deduplication is to remove documents that are semantically repetitive or highly similar, and retain documents with different semantic information. If the similarity between two documents is higher than the set maximum value, they are considered to be duplicates and are removed. After removing duplicate content, a set of semantic units is obtained.

[0080] The data in the semantic unit set are integrated according to their semantic relationships. During the integration process, multimodal association tags need to be considered, that is, the text content is associated with the corresponding multimodal information such as images and tables. Through integration, a power-related document set containing multimodal information is generated and fed back to the target user to provide the target user with query results.

[0081] The sliding window-based query method for constructing a multimodal corpus in the power industry, as described in this invention, employs a dynamic sliding window strategy. This strategy flexibly adjusts the window size based on information density and multimodal association markers, accurately capturing semantic units, avoiding semantic breaks, and improving retrieval coherence and coverage. It utilizes a large language model to generate query variants and fuses the results, enhancing the comprehensiveness and contextual adaptability of the retrieval results. By generating query variants using a large language model and performing multi-dimensional searches, the fused results generate a power-related document set, mitigating the semantic bias of single queries, improving the comprehensiveness and contextual adaptability of the retrieval results, and better meeting the data retrieval needs of complex scenarios in the power industry.

[0082] This invention also provides a sliding window-based power multimodal corpus construction and query system, used to execute the sliding window-based power multimodal corpus construction and query method described above. Figure 2 This is a structural block diagram of a sliding window-based query system for constructing a multimodal corpus of electricity data according to an embodiment of the present invention. The system includes: The structuring module 21 is used to perform structuring processing on the acquired multimodal data from the power industry to obtain a structured document. The structuring processing includes multimodal association tagging of the multimodal data.

[0083] The traversal segmentation module 22 is used to perform traversal segmentation processing on the structured document based on a dynamic sliding window strategy to obtain several segmented text blocks. The traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association tags during the segmentation process.

[0084] The corpus construction module 23 is used to input each of the segmented text blocks into a pre-trained large language model to obtain the corresponding text vectors to be stored, and to construct an electric multimodal corpus based on each of the text vectors to be stored.

[0085] The data query module 24 is used to respond to the query request of the target user, input the obtained query data of the target user into the large language model, and obtain several query variants corresponding to the query data.

[0086] The data generation module 25 is used to perform nearest neighbor search on each of the query variants in the power multimodal corpus to obtain the corresponding nearest neighbor search results, and to merge the nearest neighbor search results to generate a power-related document set including the multimodal association tags.

[0087] The technical features and effects of the system proposed in the embodiments of the present invention are the same as those of the method proposed in the embodiments of the present invention, and will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0088] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for constructing and querying a multimodal corpus of electricity based on a sliding window, characterized in that, include: The acquired multimodal data from the power industry is processed in a structured manner to obtain a structured document, wherein the structured processing includes multimodal association tagging of the multimodal data; The structured document is traversed and segmented based on a dynamic sliding window strategy to obtain several segmented text blocks. The traversal and segmentation process is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association tags during the segmentation process. Each of the segmented text blocks is input into a pre-trained large language model to obtain the corresponding text vector to be stored, and an electric multimodal corpus is constructed based on each of the text vectors to be stored. In response to a query request from a target user, the obtained query data of the target user is input into the large language model to obtain several query variants corresponding to the query data; For each of the query variants, a nearest neighbor search is performed on the power multimodal corpus to obtain the corresponding nearest neighbor search results. The nearest neighbor search results are then merged to generate a power-related document set including the multimodal association tags.

2. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The acquired multimodal data from the power industry is processed into structured documents, including: The acquired multimodal data from the power industry is formatted to obtain multimodal text data; Entity recognition is performed on the multimodal text data based on the power knowledge graph to obtain the data entities in the multimodal text data; Each data entity is associated with a corresponding modal identifier to form a multimodal association tag for each data entity; The multimodal text data, after associating the data entities, is divided into several semantic units. Each semantic unit is semantically labeled, and the semantically labeled semantic units are integrated to obtain a structured document.

3. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 2, characterized in that, The multimodal data includes at least image data, PDF data, CSV data, and text data; The process of unifying the format of the acquired multimodal data from the power industry to obtain multimodal text data includes: The image data is feature-aligned using a CLIP encoder, and the aligned image data is classified and identified using a trained image classification model to obtain the first text data. The table content in the PDF data is extracted and converted into a text format with row and column markers to obtain the second text data; The key-value pairs in the CSV data are mapped to natural language descriptive text to obtain the third text data; By integrating the first text data, the second text data, the third text data, and the text data, multimodal text data is obtained.

4. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The structured document is traversed and segmented using a dynamic sliding window strategy to obtain several segmented text blocks, including: The sliding window is determined based on the average information density of the structured document; The structured document is traversed using an initial sliding window. During the traversal, the information density and multimodal association markers corresponding to each window are identified and analyzed. The size of the sliding window is dynamically adjusted based on the analysis results. The structured document is segmented based on the adjusted sliding window to obtain several segmented text blocks.

5. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The step of inputting each of the segmented text blocks into a pre-trained large language model to obtain the corresponding text vector to be stored includes: Each of the segmented text blocks is encoded to generate a corresponding input sequence; Each of the input sequences is input into a pre-trained large language model to generate corresponding text vectors; The text vectors are dimensionality reduced, and the dimensionality-reduced text vectors are normalized to obtain a fixed-length text vector to be stored.

6. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The construction of the power multimodal corpus based on each of the text vectors to be stored includes: A vector index library is obtained by constructing a vector index for each of the text vectors to be stored. By integrating the vector index library with the multimodal association tag, a power multimodal corpus is constructed.

7. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The process involves inputting the obtained query data of the target user into the large language model to obtain several query variants corresponding to the query data, including: Semantic parsing is performed on the obtained query data of the target user to obtain query semantic data; The parsed query semantic data is input into a pre-trained large language model to generate several first query variants; Calculate the similarity value between each of the first query variants and the query semantic data, and filter out the first query variants with similarity values ​​higher than a preset similarity value as query variants.

8. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The step of performing a nearest neighbor search on each of the query variants in the power multimodal corpus to obtain the corresponding nearest neighbor search result includes: Each of the query variants is vectorized to obtain a query variant vector; The similarity between the query variant vector and each of the text vectors to be stored in the power multimodal corpus is calculated, and a number of text vectors to be stored are selected as nearest neighbor vectors based on the calculation results. The nearest neighbor vectors are sorted according to their similarity to obtain the nearest neighbor search results.

9. The method for constructing and querying a multimodal power corpus based on a sliding window as described in claim 1, characterized in that, The fusion of the nearest neighbor search results generates a power-related document set including the multimodal association tags, including: The nearest neighbor search results are subjected to semantic deduplication to obtain a set of semantic units; The semantic unit sets are integrated according to the semantic relationships between them to obtain a power-related document set including multimodal association tags.

10. A sliding window-based query system for constructing a multimodal corpus of electrical data, characterized in that, include: The structuring module is used to perform structuring processing on the acquired multimodal data from the power industry to obtain a structured document. The structuring processing includes multimodal association tagging of the multimodal data. The traversal segmentation module is used to perform traversal segmentation processing on the structured document based on a dynamic sliding window strategy to obtain several segmented text blocks. The traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association tags during the segmentation process. The corpus construction module is used to input each of the segmented text blocks into a pre-trained large language model to obtain the corresponding text vectors to be stored, and to construct an electric multimodal corpus based on each of the text vectors to be stored. The data query module is used to respond to the query request of the target user, input the obtained query data of the target user into the large language model, and obtain several query variants corresponding to the query data; The data generation module is used to perform nearest neighbor search on each of the query variants in the power multimodal corpus to obtain the corresponding nearest neighbor search results, and to merge the nearest neighbor search results to generate a power-related document set including the multimodal association tags.

Images