Power multi-modal corpus construction query method and system based on sliding window

Abstract

The application discloses a kind of based on sliding window's electric power multi-modal corpus construction query method and system, applied to electric power data query field, including according to electric power multi-modal data obtains structured document, structured document is split and handled to obtain multiple split text blocks, each split text block is input into large language model to generate to be stored text vector and constructs electric power multi-modal corpus, when receiving user query request, according to query data generates multiple query variants, carries out nearest neighbor search in electric power multi-modal corpus for each query variant, obtains corresponding nearest neighbor search result, fuses each nearest neighbor search result, generates including multi-modal association mark electric power relevant document set.The application can accurately capture semantic unit, avoid semantic break, improve retrieval coherence and coverage, improve the comprehensiveness and context adaptability of search result, meet the data retrieval demand under the complex scene of electric power industry.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric power data query, and in particular to an electric power multi-modal corpus construction query system based on a sliding window. BACKGROUND

[0002] With the acceleration of the digitalization and intelligentization transformation of new power systems, the scale of electric power data grows exponentially, covering multi-modal forms such as text, images, and time series signals. The demand for efficient processing and analysis of multi-modal data is increasingly urgent.

[0003] The existing processing of electric power data has many limitations. For example, the existing database construction lacks structured processing and integration capabilities, and the correlation between different types of data is difficult to effectively mine. Moreover, the existing methods often fail to fully consider the semantic association between multi-modal data when searching and querying data, resulting in limited accuracy of the query results.

[0004] Therefore, how to improve the efficiency and accuracy of electric power data query has become a technical problem to be solved by those skilled in the art. SUMMARY

[0005] The present application provides an electric power multi-modal corpus construction query method and system based on a sliding window, thereby providing comprehensive and efficient query results for electric power data.

[0006] To solve the above technical problems, the present application provides an electric power multi-modal corpus construction query method based on a sliding window, comprising:

[0007] Structurally processing the obtained multi-modal data of the electric power industry to obtain a structured document, wherein the structured processing includes multi-modal association labeling of the multi-modal data.

[0008] Based on a dynamic sliding window strategy, the structured document is processed by traversal segmentation to obtain a plurality of segmented text blocks, wherein the traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multi-modal association label during the segmentation process.

[0009] Each of the segmented text blocks is input into a pre-trained large language model to obtain a corresponding to-be-stored text vector, and an electric power multi-modal corpus is constructed based on each of the to-be-stored text vectors.

[0010] In response to a query request of a target user, the obtained query data of the target user is input into the large language model to obtain a plurality of query variants corresponding to the query data.

[0011] Respectively, each of the query variants is subjected to a nearest neighbor search on the power multimodal corpus to obtain a corresponding nearest neighbor search result, and the nearest neighbor search results are fused to generate a power-related document set including the multimodal association labels.

[0012] Further, the obtained multimodal data of the power industry is subjected to structured processing to obtain a structured document, including:

[0013] The obtained multimodal data of the power industry is subjected to format unification to obtain multimodal text data.

[0014] Based on the power knowledge graph, entity recognition is performed on the multimodal text data to obtain data entities in the multimodal text data.

[0015] Each of the data entities is associated with a corresponding modality identifier to form a multimodal association label for each of the data entities.

[0016] The multimodal text data after association of each of the data entities is divided into a plurality of semantic units, each of the semantic units is subjected to semantic annotation, and the semantic units after semantic annotation are integrated to obtain a structured document.

[0017] Further, the multimodal data at least includes image data, PDF data, CSV data and text data.

[0018] The obtained multimodal data of the power industry is subjected to format unification to obtain multimodal text data, including:

[0019] The image data is subjected to feature alignment using a CLIP encoder, and the aligned image data is subjected to classification recognition using a trained image classification model to obtain first text data.

[0020] The table content in the PDF data is extracted and converted into a text format with row and column labels to obtain second text data.

[0021] The key-value pairs in the CSV data are mapped to natural language description text to obtain third text data.

[0022] The first text data, the second text data, the third text data and the text data are integrated to obtain multimodal text data.

[0023] Further, the structured document is subjected to traversal segmentation processing based on a dynamic sliding window strategy to obtain a plurality of segmented text blocks, including:

[0024] The sliding window is determined according to the average information density of the structured document.

[0025] The structured document is traversed using an initial sliding window. During the traversal, the information density and multimodal association markers corresponding to each window are identified and analyzed. The size of the sliding window is dynamically adjusted based on the analysis results.

[0026] The structured document is segmented based on the adjusted sliding window to obtain several segmented text blocks.

[0027] Further, the step of inputting each of the segmented text blocks into a pre-trained large language model to obtain the corresponding text vector to be stored includes:

[0028] Each of the segmented text blocks is encoded to generate a corresponding input sequence.

[0029] Each of the input sequences is input into a pre-trained large language model to generate the corresponding text vector.

[0030] The text vectors are dimensionality reduced, and the dimensionality-reduced text vectors are normalized to obtain a fixed-length text vector to be stored.

[0031] Furthermore, the construction of the power multimodal corpus based on each of the text vectors to be stored includes:

[0032] Vector indexes are constructed for each of the text vectors to be stored, resulting in a vector index library.

[0033] By integrating the vector index library with the multimodal association tag, a power multimodal corpus is constructed.

[0034] Furthermore, the process of inputting the obtained query data of the target user into the large language model yields several query variants corresponding to the query data, including:

[0035] The obtained query data of the target user is semantically parsed to obtain query semantic data.

[0036] The parsed query semantic data is input into a pre-trained large language model to generate several first query variants.

[0037] Calculate the similarity value between each of the first query variants and the query semantic data, and filter out the first query variants with similarity values ​​higher than a preset similarity value as query variants.

[0038] Further, the step of performing a nearest neighbor search on each of the query variants in the power multimodal corpus to obtain the corresponding nearest neighbor search result includes:

[0039] vectorize each of the query variants to obtain a query variant vector.

[0040] perform similarity calculation on the query variant vector and each of the to-be-stored text vectors in the power multimodal corpus, and select a number of to-be-stored text vectors as nearest neighbor vectors according to a calculation result.

[0041] sort each of the nearest neighbor vectors according to similarity to obtain a nearest neighbor search result.

[0042] Further, the fusion of each of the nearest neighbor search results generates a power-related document set including the multimodal association mark, including:

[0043] perform semantic deduplication processing on the nearest neighbor search result to obtain a semantic unit set.

[0044] integrate each of the semantic unit sets according to semantic association between the semantic unit sets to obtain a power-related document set including a multimodal association mark.

[0045] Another embodiment of the application provides a sliding window-based power multimodal corpus construction query system, including:

[0046] a structured module configured to perform structured processing on the obtained multimodal data of the power industry to obtain a structured document, wherein the structured processing includes multimodal association marking on the multimodal data.

[0047] a traversal segmentation module configured to perform traversal segmentation processing on the structured document based on a dynamic sliding window strategy to obtain a plurality of segmented text blocks, wherein the traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association mark during the segmentation process.

[0048] a corpus construction module configured to input each of the segmented text blocks into a pre-trained large language model to obtain a corresponding to-be-stored text vector, and construct a power multimodal corpus based on each of the to-be-stored text vectors.

[0049] a data query module configured to input the obtained query data of a target user into the large language model in response to a query request of the target user to obtain a plurality of query variants corresponding to the query data.

[0050] a data generation module configured to perform nearest neighbor search on each of the query variants in the power multimodal corpus to obtain a corresponding nearest neighbor search result, and fuse each of the nearest neighbor search results to generate a power-related document set including the multimodal association mark.

[0051] Compared with the prior art, the beneficial effects of the embodiment of the present application are at least one of the following:

[0052] Through the dynamic sliding window strategy, the window size is flexibly adjusted according to the information density and the multi-modal association label, the semantic unit is accurately captured, the semantic break is avoided, and the retrieval continuity and coverage are improved. The large language model is used to generate query variants and fuse the results, and the comprehensiveness and context adaptability of the retrieval results are improved. The large language model is used to generate query variants and perform multi-dimensional retrieval, and the power-related document set is generated after fusing the results, which alleviates the semantic deviation of a single query, improves the comprehensiveness and context adaptability of the retrieval results, and better meets the data retrieval needs in complex scenarios in the power industry. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A step flow chart of the power multi-modal corpus construction query method based on a sliding window in one of the embodiments of the present application;

[0054] Figure 2 A structural block diagram of the power multi-modal corpus construction query system in one of the embodiments of the present application;

[0055] The reference signs: 21, structured module; 22, traversal segmentation module; 23, corpus construction module; 24, data query module; 25, data generation module. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0057] In the description of the present application, the terms "first", "second", "third" and the like are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0058] In the description of the present application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The specific meanings of the above terms in the present application can be understood by those skilled in the art on a case-by-case basis.

[0059] An embodiment of the present application provides a sliding window-based power multimodal corpus construction query method, and specifically, please refer to Figure 1 , Figure 1 A step flowchart of the sliding window-based power multimodal corpus construction query method in one embodiment of the present application is shown, which includes:

[0060] S11, structuring the obtained multimodal data of the power industry to obtain structured documents, wherein the structuring includes multimodal association marking of the multimodal data.

[0061] In data analysis of the power industry, data sources are extensive and diverse in form, including but not limited to equipment operation data, sensor collection data, operation and maintenance records, fault reports, and related multimedia information such as images and videos. These data usually exist in different formats, such as table data, text files, image files, etc., which brings great challenges to subsequent data processing and analysis. Therefore, in order to efficiently process and analyze these multimodal data, it is necessary to first unify the format of the obtained data to obtain multimodal text data.

[0062] Specifically, the data to be unified in format by the present embodiment at least includes image data, PDF data, CSV data and text data. The process of unifying the format of different data by the present embodiment includes:

[0063] The CLIP (Contrastive Language-Image Pre-training) encoder is an advanced multimodal model that can map images and text into the same feature space. By processing image data through the CLIP encoder, the feature representation of the image can be extracted and aligned with the text feature. Image data can be compared and matched with text data in the same feature space. On the basis of feature alignment, a pre-trained image classification model is used to classify and identify the image data, and the classification result is converted into a text description, i.e. the first text data. For example, if the image classification model identifies the image as a "transformer", the first text data can be "the image contains a transformer".

[0064] PDF files are also common data carriers in the power industry, often containing a large amount of table information, such as equipment parameter tables, operation record tables, etc. Using PDF parsing tools to parse PDF data, extract the table structure, including table rows, columns, cells, etc. information, and then convert it to a text format with row and column labels to obtain the second text data.

[0065] CSV (Comma-Separated Values) files are a common data storage format, often used to store key-value pair data. In the power industry, CSV files may contain equipment parameters, operation data, etc. In order to integrate CSV data with other modal data, it is necessary to map its key-value pairs to natural language description text, for example, if a row of data in the CSV file is "device name, transformer; device model, T100", it can be mapped to natural language description text: "device name is transformer, device model is T100". Add timestamp annotations to time series data columns (such as "peak load @ 2023-01-01 14:30"), and after mapping the key-value pairs in the CSV data to natural language description text, the resulting text is the third text data.

[0066] After processing image data, PDF table data and CSV data, three different sources of text data are obtained: first text data (text description of image classification results), second text data (text format of PDF table content), and third text data (natural language description of CSV key-value pairs). In addition, there is also original text data. In order to realize the unified processing and analysis of multi-modal data, it is necessary to integrate these different sources of text data to obtain multi-modal text data.

[0067] When integrating text data from different sources, it is necessary to align different sources of text data. The basis for alignment can be the timestamp, device name, event identifier, etc. For example, if the first text data describes the image classification results of a certain device at a certain time, the second text data records the table information of the device at the same time, and the third text data describes the parameter information of the device, then these text data can be aligned according to the device name and timestamp.

[0068] In the data analysis of the power industry, multi-modal text data integrates information from multiple modalities such as images, tables, CSV files, etc. These data can point to specific entity information, such as device name, fault type, parameter value, etc.

[0069] An electric power knowledge graph is a structured knowledge representation form that organizes various entities (such as devices, parameters, fault types, etc.) in the electric power field and their relationships in the form of a graph. Based on the electric power knowledge graph, entity recognition is performed on multi-modal text data to obtain data entities in the multi-modal text data.

[0070] In multi-modal text data, each data entity may come from different modalities, such as device names that may come from image classification results, and device parameter values that may come from tables or CSV files. In order to fully utilize the advantages of multi-modal data, each data entity needs to be associated with its corresponding modality identifier, so that additional multi-modal association labels are formed for each data entity.

[0071] According to the semantic structure and logical relationship of the text, the multi-modal text data is divided into several semantic units. The division of semantic units can be based on sentences, paragraphs or specific semantic topics. For example, device operation records are divided into a semantic unit, and fault reports are divided into another semantic unit. Then, through natural language processing techniques such as text generation algorithms, the labeled semantic units are integrated into a structured document according to semantic logic. Device operation records and fault reports can be integrated into the running state structure of the same device.

[0072] S12, based on a dynamic sliding window strategy, the structured document is processed by traversal segmentation to obtain several segmented text blocks, wherein the traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density and multi-modal association labels of the structured document during the segmentation process.

[0073] Structured documents usually contain a wealth of information, but the distribution of these information is not uniform. Some parts may contain a large amount of key data, while some parts may be relatively sparse. In order to efficiently process these documents while ensuring that key information is not missed, a sliding window-based method can be used to analyze and segment the documents.

[0074] First, the size of the initial sliding window needs to be determined based on the average information density of the structured document. Information density refers to the amount of effective information contained in a unit of text length, which can be calculated by counting the distribution frequency of keywords, data entities, etc. in the document. Specifically, if a document contains 10 key data points per 100 characters, it can be considered to have an information density of 0.1. According to the overall average information density of the document, a suitable sliding window size can be initially set to cover enough key information in subsequent analysis.

[0075] Next, the initial sliding window is used to traverse the structured document. During the traversal, the content in each window is analyzed in detail. This includes two main aspects: one is to identify the information density in the window, that is, to calculate the concentration of effective information in the current window; the second is to analyze the multi-modal association marks in the window to understand the source and modal characteristics of the information in the window. If the information density in a window is much higher than the average value, and contains data entities of multiple modalities (such as associated information of image modalities and text modalities), it means that the information in the window is more important and may need more detailed processing.

[0076] Specifically, according to the above analysis results, the size of the sliding window is dynamically adjusted. If the information density in the current window is high, and the multi-modal association marks show that the information is complex, the window size can be appropriately reduced to analyze these key information more finely. On the contrary, if the information density in the window is low, and the multi-modal association marks show that the information is relatively single, the window size can be appropriately enlarged to improve the processing efficiency. This dynamic adjustment mechanism can ensure that both key information and important details are captured when processing the document without missing important details due to the large window size.

[0077] Based on the adjusted sliding window, the structured document is segmented to obtain a plurality of segmented text blocks. In the segmentation process, it is also necessary to ensure that the segmented text blocks contain complete semantics and retain multi-modal association marks.

[0078] S13, input each segmented text block into a pre-trained large language model to obtain a corresponding to-be-stored text vector, and construct a power multi-modal corpus based on each to-be-stored text vector.

[0079] Each segmented text block is encoded and converted into an input sequence that can be processed by a computer. These input sequences are input into a pre-trained large language model to capture semantic information of the text and obtain high-dimensional text vectors.

[0080] Since the text vectors generated by the large language model usually have a high dimension, it may affect the efficiency and effectiveness of subsequent processing. Therefore, it is necessary to reduce the dimension of these text vectors to improve the processing efficiency and reduce the storage cost. In the dimension reduction process, the dimension of the vector is reduced while trying to preserve the semantic information of the original vector as much as possible.

[0081] After dimension reduction, the text vectors are normalized to adjust the values of the text vectors to a fixed range, so that the vectors of different text blocks are comparable, and fixed-length to-be-stored text vectors are obtained.

[0082] After obtaining the fixed-length text vector to be stored, vector indexing is constructed to obtain a vector index library for fast retrieval and matching of the text vector, thereby realizing fast query and retrieval of the text vector and improving the efficiency of data processing.

[0083] The multi-modal association label contains the source modal information of the text block, such as image modal, table modal, etc. The vector index library and the multi-modal association label are integrated and constructed to build a power multi-modal corpus. The power multi-modal corpus not only contains the semantic information of the text, but also retains the multi-modal characteristics of the text.

[0084] S14, in response to the query request of the target user, input the obtained query data of the target user into the large language model to obtain a plurality of query variants corresponding to the query data.

[0085] After the power multi-modal corpus is constructed, the target user can query the required content from the multi-modal corpus by sending a query request. The query request contains various information, such as account information, login state information, network link information, and query data of the target user. After receiving the query request of the target user, the query data contained in the query request is semantically analyzed to obtain query semantic data in response to the query request of the target user.

[0086] The query semantic data is input into the pre-trained large language model. The large language model can generate a plurality of possible expression modes, i.e., a plurality of first query variants, according to the input semantic data. These variants are similar in semantics to the original query semantic data, but may use different words, sentence patterns or expression modes. For example, for the query semantic data "fault record of device A", the large language model may generate a plurality of first query variants such as "fault log of device A", "what are the fault conditions of device A", etc.

[0087] Since the number of first query variants generated by the large language model may exceed the required number of variants, and not all variants completely meet the original intention of the user, it is necessary to further screen these variants. Specifically, the system calculates the similarity value between each first query variant and the query semantic data. Similarity calculation can be realized by various methods, such as cosine similarity based on semantic vectors, Jaccard similarity based on keyword matching, etc.

[0088] By calculating the similarity value, the system can evaluate the semantic closeness of each variant to the original query. According to the preset similarity threshold, the first query variant with a similarity value higher than the threshold is selected as the query variant. In the actual query process, the setting of the preset similarity threshold needs to be adjusted according to the actual demand. If the number of first query variants generated by a certain query semantic data is too small, the threshold value can be appropriately reduced. Otherwise, the threshold value is increased to ensure that the selected variants can cover the query intention of the target user and do not contain too much irrelevant or redundant content.

[0089] S15, respectively, each query variant in the power multi-modal corpus is searched for the nearest neighbor, and the corresponding nearest neighbor search result is obtained. The multi-modal association marks are fused to generate a power-related document set.

[0090] When processing the query request of the user, the system first needs to convert each query variant into a vector form to obtain a query variant vector for subsequent similarity calculation.

[0091] After obtaining the query variant vector, the system needs to calculate the similarity between the query variant vector and each to-be-stored text vector in the power multi-modal corpus. The similarity calculation in this embodiment is realized by cosine similarity.

[0092] Specifically, each to-be-stored text vector in the power multi-modal corpus in this embodiment is obtained by storing the vectorization processing in the vector database. The vector database can select FAISS or Milvus. The vector database used by the power multi-modal corpus in this embodiment is FAISS. Through the efficient retrieval capability of the vector database, the system can quickly find several to-be-stored text vectors most similar to the query variant vector. These vectors are the nearest neighbor vectors.

[0093] According to the result of the similarity calculation, the nearest neighbor vectors are sorted in descending order of similarity to ensure that the most relevant documents are placed in the front.

[0094] In order to improve the quality of the retrieval result, the nearest neighbor search result needs to be processed for semantic deduplication. The purpose of semantic deduplication is to remove documents with repeated or highly similar semantics and retain documents with different semantic information. If the similarity of two documents is higher than a set maximum value, they are considered to be repeated and are removed. After removing the repeated content, a semantic unit set is obtained.

[0095] The data in the semantic unit set is integrated according to the semantic correlation between each other, and in the integration process, the multi-modal correlation mark needs to be considered, that is, the text content is associated with the corresponding multi-modal information such as images and tables, a power-related document set containing multi-modal information is generated through integration, and the power-related document set is fed back to the target user to provide query results for the target user.

[0096] The sliding window-based power multi-modal corpus construction query method of the application flexibly adjusts the window size according to the information density and multi-modal correlation mark through the dynamic sliding window strategy, accurately captures semantic units, avoids semantic breakage, and improves retrieval coherence and coverage. Utilizing a large language model to generate query variants and fuse results improves the comprehensiveness and context adaptability of the retrieval results. Utilizing a large language model to generate query variants and performing multi-dimensional retrieval, generating a power-related document set after fusing the results, alleviates the semantic deviation of a single query, improves the comprehensiveness and context adaptability of the retrieval results, and better meets the data retrieval needs in complex scenarios in the power industry.

[0097] The embodiment of the application also provides a sliding window-based power multi-modal corpus construction query system for executing the sliding window-based power multi-modal corpus construction query method as described above, Figure 2 The structure block diagram of the sliding window-based power multi-modal corpus construction query system of the embodiment of the application, the system comprises:

[0098] The structured module 21 is configured to perform structured processing on the acquired multi-modal data of the power industry to obtain structured documents, wherein the structured processing comprises multi-modal correlation marking on the multi-modal data.

[0099] The traversal segmentation module 22 is configured to perform traversal segmentation processing on the structured documents based on a dynamic sliding window strategy to obtain a plurality of segmented text blocks, wherein the traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured documents and the multi-modal correlation mark during the segmentation process.

[0100] The corpus construction module 23 is configured to input each of the segmented text blocks into a pre-trained large language model to obtain corresponding to-be-stored text vectors, and construct a power multi-modal corpus based on each of the to-be-stored text vectors.

[0101] The data query module 24 is configured to input the acquired query data of the target user into the large language model in response to a query request of the target user to obtain a plurality of query variants corresponding to the query data.

[0102] The data generation module 25 is configured to perform a nearest neighbor search on the power multimodal corpus for each query variant respectively to obtain a corresponding nearest neighbor search result, fuse the nearest neighbor search results to generate a power-related document set comprising the multimodal association mark.

[0103] The technical features and effects of the system are the same as those of the method, and are not described here. Each module in the system can be realized by software, hardware, or a combination thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0104] The above-described embodiments only express several implementation manners of the present application, which are described in detail, but should not be construed as limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for constructing a power multimodal corpus based on a sliding window, characterized in that, The method comprises the following steps: performing structured processing on the obtained multi-modal data of the power industry to obtain a structured document, wherein the structured processing comprises multi-modal association marking on the multi-modal data; performing traversal segmentation processing on the structured document based on a dynamic sliding window strategy to obtain a plurality of segmented text blocks, wherein the traversal segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multi-modal association marking during the segmentation process; inputting each of the segmented text blocks into a pre-trained large language model to obtain a corresponding to-be-stored text vector, and constructing a power multi-modal corpus based on each of the to-be-stored text vectors; in response to a query request of a target user, inputting obtained query data of the target user into the large language model to obtain a plurality of query variants corresponding to the query data; performing nearest neighbor search on each of the query variants in the power multi-modal corpus to obtain a corresponding nearest neighbor search result, fusing each of the nearest neighbor search results to generate a power-related document set comprising the multi-modal association marking; the structured processing on the obtained multi-modal data of the power industry to obtain a structured document comprises: performing format unification on the obtained multi-modal data of the power industry to obtain multi-modal text data; performing entity recognition on the multi-modal text data based on a power knowledge graph to obtain data entities in the multi-modal text data; associating each of the data entities with a corresponding modal identifier to form a multi-modal association marking for each of the data entities; dividing the multi-modal text data after association of each of the data entities into a plurality of semantic units, performing semantic annotation on each of the semantic units, and integrating the semantic units after semantic annotation to obtain a structured document.

2. The sliding window based power multimodal corpus construction query method of claim 1, wherein, The multi-modal data at least comprises image data, PDF data, CSV data and text data; the format unification on the obtained multi-modal data of the power industry to obtain multi-modal text data comprises: aligning features of the image data using a CLIP encoder, and classifying the aligned image data using a trained image classification model to obtain first text data; extracting table contents in the PDF data and converting them into a text format with row and column markers to obtain second text data; mapping key-value pairs in the CSV data into natural language description text to obtain third text data; integrating the first text data, the second text data, the third text data and the text data to obtain multi-modal text data.

3. The sliding window based power multimodal corpus construction query method as claimed in claim 1, wherein, The traversal segmentation processing on the structured document based on the dynamic sliding window strategy to obtain a plurality of segmented text blocks comprises: determining a sliding window according to the average information density of the structured document; using an initial sliding window to traverse the structured document, identifying and analyzing the information density and multi-modal association marking of each window during the traversal, and dynamically adjusting the size of the sliding window according to the analysis result; Segment the structured document based on the adjusted sliding window to obtain a plurality of segmented text blocks.

4. The sliding window based power multimodal corpus construction query method of claim 1, wherein, The inputting each of the segmented text blocks into the pre-trained large language model comprises: encoding each of the segmented text blocks to generate a corresponding input sequence; inputting each of the input sequences into the pre-trained large language model to generate a corresponding text vector; dimensionality reduction processing each of the text vectors, and normalizing each of the dimensionally reduced text vectors to obtain a fixed-length to-be-stored text vector.

5. The sliding window based power multimodal corpus construction query method as claimed in claim 1, wherein, The constructing a power multimodal corpus based on each of the to-be-stored text vectors comprises: vector index construction of each of the to-be-stored text vectors to obtain a vector index library; integrating the vector index library and the multimodal association mark to construct a power multimodal corpus.

6. The sliding window based power multimodal corpus construction query method as claimed in claim 1, wherein, The inputting the obtained query data of the target user into the large language model to obtain a plurality of query variants corresponding to the query data comprises: semantic analysis of the obtained query data of the target user to obtain query semantic data; inputting the parsed query semantic data into the pre-trained large language model to generate a plurality of first query variants; calculating the similarity value between each of the first query variants and the query semantic data, and selecting the first query variant with a similarity value higher than a preset similarity as a query variant.

7. The sliding window based power multimodal corpus construction query method as claimed in claim 1, wherein, The nearest neighbor search of each of the query variants in the power multimodal corpus comprises: vectorization processing each of the query variants to obtain a query variant vector; similarity calculation between the query variant vector and each of the to-be-stored text vectors in the power multimodal corpus, and selecting a plurality of to-be-stored text vectors as nearest neighbor vectors according to the calculation result; sorting each of the nearest neighbor vectors according to the similarity to obtain a nearest neighbor search result.

8. The sliding window based power multimodal corpus construction query method as claimed in claim 1, wherein, The fusion of each of the nearest neighbor search results to generate a power-related document set comprising the multimodal association mark comprises: semantic deduplication processing of the nearest neighbor search results to obtain a semantic unit set; integrating each of the semantic unit sets according to the semantic association between them to obtain a power-related document set comprising a multimodal association mark. 9.A sliding window based power multimodal corpus construction query system, characterized in that, comprises: a structured module for structuring the obtained multimodal data of the power industry to obtain a structured document, wherein the structured processing comprises multimodal association marking of the multimodal data; an iterative segmentation module for iterative segmentation processing of the structured document based on a dynamic sliding window strategy to obtain a plurality of segmented text blocks, wherein the iterative segmentation processing is designed to dynamically adjust the size of the sliding window based on the information density of the structured document and the multimodal association mark during segmentation; a corpus construction module for inputting each of the segmented text blocks into a pre-trained large language model to obtain a corresponding to-be-stored text vector, and constructing a power multimodal corpus based on each of the to-be-stored text vectors; a data query module, configured to input the obtained query data of the target user into the large language model in response to a query request of the target user, to obtain a plurality of query variants corresponding to the query data; a data generation module, configured to perform nearest neighbor search on each of the query variants in the power multi-modal corpus respectively, to obtain corresponding nearest neighbor search results, to fuse the nearest neighbor search results, and to generate a power-related document set including the multi-modal association mark; the structured module is further configured to: perform format unification on the obtained multi-modal data of the power industry, to obtain multi-modal text data; perform entity recognition on the multi-modal text data based on a power knowledge graph, to obtain data entities in the multi-modal text data; associate each of the data entities with a corresponding modal identifier, to form a multi-modal association mark for each of the data entities; divide the multi-modal text data after the association of the data entities into a plurality of semantic units, perform semantic labeling on each of the semantic units, integrate the semantic units after the semantic labeling, and obtain a structured document.

Images