Knowledge extraction and multi-dimensional index optimization method for power document

Abstract

A knowledge extraction and multi-dimensional index optimization method for an electric power document relates to the technical field of information retrieval, and comprises the following steps: preprocessing an original electric power document to obtain a standardized electric power document segment; performing multi-dimensional professional feature extraction on the standardized power document fragment, wherein the multi-dimensional professional features comprise power field core keywords, technical guidance abstracts and semantic vectors; the extracted multi-dimensional professional features are stored in a mixed storage architecture matched with the power business features, the mixed storage architecture comprises a relational database used for storing structured metadata and a vector database used for storing semantic vectors, and the relational database is used for storing the structured metadata; the structured metadata comprises the core keyword and the technical guidance abstract; the method is used for solving the problems of single text and vector storage mode and low retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically to a method for knowledge extraction and multi-dimensional index optimization of power documents. Background Technology

[0002] Against the backdrop of digital transformation in the power industry, core business scenarios such as power grid and substation engineering design continuously generate massive amounts of professional text data, covering key knowledge resources such as case data, regulations and standards, and typical designs. These documents are an important foundation for supporting power engineering design and decision-making, and their storage and retrieval efficiency directly affects the effectiveness of business operations. However, with the continuous expansion of data scale, traditional power document storage methods face significant challenges, mainly manifested in the limitations of the following three models: First, the single text storage method saves the document content in plain text format. Although it preserves the integrity of the original text, it cannot effectively identify and extract core features of the power industry (such as professional terms, technical parameters and regulation numbers such as "electrical main wiring", "short circuit current" and "transformer rated capacity"). This method requires full text traversal during retrieval, resulting in high computational overhead and low retrieval efficiency.

[0003] Second, there is the single vector storage method, which uses a general pre-trained language model to convert text into vectors and store them in a vector database. Although this method can capture the overall semantics, it is not good at representing the identification information with precise business meaning in the power field (such as equipment model, procedure version number, etc.), which is prone to semantic confusion. At the same time, vectors are difficult to reflect the technical logic relationship inside the document, resulting in a lack of structure in the stored content, which cannot meet the needs of power industry retrieval.

[0004] Third, while traditional relational database storage can manage some structured information, it has limited ability to process a large amount of unstructured text in power documents (such as operation descriptions and fault analyses). When performing multi-dimensional combined retrieval, it relies on complex multi-table join queries, resulting in a complex system architecture, low query efficiency, and difficulty in maintaining data consistency.

[0005] In existing technologies, such as Chinese Patent Publication No. CN120764494A, a method for generating power document data based on entity relationship extraction is proposed. This method obtains multi-source information such as power grid operation, construction, and regulations, extracts entity relationship structures, and performs iterative processing in conjunction with argumentation and reasoning to improve data processing efficiency. However, this type of method still has problems such as weak ability to represent professional core features, lack of power business-oriented screening logic, and insufficient adaptability of storage architecture to power document characteristics. As a result, in practical applications, the retrieval accuracy is low, the response speed is slow, and the system scalability is limited.

[0006] Therefore, we propose a method that enables multi-dimensional and high-precision knowledge representation of power documents and improves retrieval efficiency. Summary of the Invention

[0007] The purpose of this invention is to provide a method for knowledge extraction and multi-dimensional indexing optimization of power documents, which solves the problems of single text and vector storage methods and low retrieval efficiency.

[0008] This invention is achieved through the following technical solution: A method for knowledge extraction and multi-dimensional indexing optimization of power documents, specifically including: The original power documents are preprocessed to obtain standardized power document fragments; The standardized power document fragments are subjected to multi-dimensional professional feature extraction, which includes core keywords in the power field, technology-oriented summaries, and semantic vectors. The extracted multi-dimensional professional features are stored in a hybrid storage architecture adapted to the characteristics of power business. The hybrid storage architecture includes a relational database for storing structured metadata and a vector database for storing semantic vectors. The structured metadata includes the core keywords and the technology-oriented summary.

[0009] Furthermore, the preprocessing of the original power documents specifically includes: The original power documents are classified according to the preset power document classification system, and formatting marks and meaningless comments are removed. Based on device type and chapter information, the categorized documents are fragmented to obtain an initial set of document fragments; The initial set of document fragments is segmented using a word segmentation model trained on a power industry terminology database, and stop words in the power field are removed to obtain a standardized set of power document fragments.

[0010] Furthermore, the steps for extracting the core keywords in the power sector include: Calculate the underlying TF-IDF values ​​of words in a standardized document fragment; Construct a power-specific terminology database, and enhance the TF-IDF values ​​of terms belonging to the database based on the power industry terminology database. The keywords are sorted according to their enhanced scores, and a predetermined number of the top-ranking words are selected to form a core keyword set.

[0011] Furthermore, the mathematical formula for calculating the basic TF-IDF values ​​of words in the standardized document fragment is as follows:

[0012]

[0013] in, Words In the current standardized text fragments Frequency of occurrence in; Indicates the current standardized text fragment The total frequency of all words in the text; Indicates the total number of document fragments related to electricity; Indicates included words The number of power document fragments.

[0014] Furthermore, the construction of a power-specific terminology database, and the weighting of the TF-IDF values ​​of terms belonging to the database based on the power industry terminology database, are described by the following mathematical formula:

[0015] In the formula, Scoring for electricity-specific keywords Weighting coefficients for electricity-specific terms. A terminology database specifically for the power industry.

[0016] Furthermore, the steps for generating the technology-oriented summary include: Standardized document fragments are segmented into sets of sentences, and the sentences are categorized according to power business logic; Calculate the semantic similarity score between each sentence and the document fragment as a whole; By combining the business priority of sentence classification with the semantic similarity score, a preset number of sentences are selected, concatenated to generate a structured technical summary, and a corresponding summary vector is generated.

[0017] Furthermore, the steps for obtaining the semantic vector include: A pre-trained embedding model is used to convert standardized document fragments into initial semantic vectors; The initial semantic vector is normalized to obtain a unit semantic vector, which is used for subsequent similarity calculation and storage.

[0018] Furthermore, the construction steps of the hybrid storage architecture specifically include: A power-specific metadata table is constructed in the relational database to store the unique identifier of document fragments, business type, equipment type, core keyword set, and technology-oriented summary; In the vector database, semantic vectors are partitioned and stored according to the business type, and secondary indexes are established in each partition according to device type or scenario type; The unique identifier is used to establish an association between the metadata in the relational database and the corresponding semantic vector in the vector database.

[0019] Furthermore, the business types include: engineering design cases, equipment technical specifications, industry regulations and standards, and typical design schemes.

[0020] Furthermore, when the business type is an engineering design case, the scenario type includes: hydrological conditions, geological conditions, and meteorological conditions.

[0021] The technical solution of the present invention has at least the following advantages and beneficial effects: This invention discloses a knowledge extraction and multi-dimensional indexing optimization method for power documents. By adopting multi-dimensional feature extraction and combining it with a hybrid storage architecture, it can simultaneously capture accurate identification information, key business logic, and overall semantic content in power documents. This overcomes the shortcomings of one-sided representation and poor adaptability, providing a solid data foundation for subsequent high-precision and high-efficiency retrieval, and realizing multi-dimensional and high-precision knowledge representation and efficient storage of power documents. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the knowledge extraction and multi-dimensional indexing optimization method for power documents according to the present invention; Figure 2 This is a schematic diagram of the knowledge extraction and multi-dimensional indexing optimization system structure for power documents in this invention; Figure 3 This is a schematic diagram of the electronic device in this invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Example 1 like Figure 1 The method for knowledge extraction and multi-dimensional indexing optimization of power documents, as shown, specifically includes: Preprocessing the original power documents to obtain standardized power document fragments can transform the messy original documents into standardized, high-quality, business-oriented text fragments through structured operations guided by domain knowledge. This essentially builds a high-quality "input interface" for subsequent intelligent algorithms. The preprocessing of the original power documents specifically includes: Based on the pre-defined power document classification system: Cases / Specifications / Equipment / Classic Designs, the original power documents are classified and formatting symbols (table characters, formula placeholders) and meaningless comments are removed. The power document classification system here is for the initial categorization of documents. This is not simply labeling, but rather mapping documents to their respective business scenarios and workflows. For example, a "transformer fault analysis report" is classified as a "case study," with the implicit business intention of "providing fault diagnosis and design reference," thus laying the logical foundation for subsequent differentiated processing and business diversion of retrieval. Removing formatting marks and meaningless comments eliminates interference from non-semantic information, ensuring that the algorithm focuses on the text content itself.

[0025] Based on equipment type and chapter information (such as "110kV Transformer - Standard Chapter"), the categorized documents are fragmented to obtain an initial set of document fragments. ,in This represents the number of document fragments. Through classification and device type-based fragmentation, the power business logic, such as what type of document and what device it relates to, is encoded into the data structure before the data flows into the algorithm. This allows subsequent storage and retrieval to directly utilize this structure, doubling efficiency and realizing business-driven technology, rather than technology barely adapting to business.

[0026] Since power engineering documents are typically lengthy and cover multiple devices or topics, fragmentation based on "equipment type - chapter information" is essentially cutting them according to the natural logical units of power engineering (chaps centered around specific devices). This breaks down lengthy documents into highly condensed "micro-documents," such as the 110kV transformer - insulation design chapter. This makes each segment have a single theme and focused content, greatly improving the accuracy and representativeness of subsequent feature extraction (such as keywords and abstracts) and avoiding cross-contamination of information from different devices. In addition, each segment focuses on a single device / topic, and the keywords extracted from it and the generated summary naturally have a high degree of business relevance and interpretability. For example, keywords such as "winding hot spot temperature" and "load guidelines" extracted from the "110kV transformer - temperature rise calculation chapter" segment can accurately reflect the core of the segment, and the summary can also be developed around a single theme with clearer logic. This directly solves the problem of "lack of power industry screening logic" in the background technology.

[0027] The initial set of document fragments was segmented using a word segmentation model trained on a power industry terminology database, and stop words in the power field (such as "according to relevant regulations") were removed to obtain a standardized set of power document fragments. At the same time, some words in the text will be replaced, such as abbreviations of equipment names; This word segmentation model ensures that such professional compound terms are identified as complete semantic primitives. At the same time, it removes stop words that are high-frequency but have very low information content in the power field, such as "according to relevant regulations" and "should meet the following requirements", to further purify the text. The result of this operation is a set of text fragments composed of standardized power semantic units (correctly segmented professional terms and effective vocabulary), which provides accurate basic data for subsequent mathematical processing such as TF-IDF calculation and vectorization.

[0028] The standardized power document fragments are subjected to multi-dimensional professional feature extraction, which includes core keywords in the power field, technology-oriented summaries, and semantic vectors. This step can abandon the single and one-sided document representation method, and instead deeply characterize the same standardized document fragment from three orthogonal and complementary dimensions, and jointly construct a three-dimensional and computable knowledge model. The steps for extracting the core keywords in the power sector include: The mathematical formula for calculating the basic TF-IDF values ​​of words in a standardized document fragment is as follows:

[0029]

[0030] in, Words In the current standardized text fragments Frequency of occurrence in; Indicates the current standardized text fragment The total frequency of all words in the text; Indicates the total number of document fragments related to electricity; Indicates included words The number of power document fragments; in addition, adding 1 to both the numerator and denominator in the formula is to avoid the case where the denominator is 0 (i.e. a certain word does not appear in all document fragments), and to ensure the validity of the calculation.

[0031] calculate The formula is used to measure vocabulary. Local importance in a single document fragment: The more frequently a word appears in the current document fragment (e.g., "110kV" appears frequently in a case document fragment), the larger the TF value, indicating that the word contributes more to the information within the document fragment; conversely, the smaller the TF value, the lower the local importance. calculate The formula is used to measure vocabulary. Domain-specific differentiation: If a term appears in only a few document fragments (such as the power-specific term "GIS combined electrical appliances"), then A smaller IDF value and a larger IDF value indicate that the term has a strong ability to distinguish documents in the power industry; if a term appears in most document fragments (such as the general term "equipment"), then... A high IDF value and a low IDF value indicate a weak ability to differentiate between different domains. IDF calculation can help filter out keywords that are more representative of the power industry.

[0032] A power industry-specific terminology database is constructed, and the TF-IDF values ​​of terms belonging to the database are weighted and enhanced based on the database. The mathematical formula is as follows:

[0033] In the formula, Scoring for electricity-specific keywords Weighting coefficients for electricity-specific terms. This is a dedicated terminology database for the power industry, containing core identifiers such as standard terms, equipment models, regulation numbers, and technical parameters, including terms like "GIS," "technical guidelines," and "50MVA." The purpose of this formula is to highlight the importance of core terms in the power sector: because the basic TF-IDF may not adequately allocate weight to highly business-related terms such as "equipment models" and "regulation numbers," this formula is introduced to... The coefficient allows power-specific terms to receive higher priority in keyword filtering, ensuring that the final extracted keywords accurately target the core business attributes of power documents and reduce the interference of general terms on search accuracy. in, and Both are indicator functions, with the following meanings: when the condition is met... At the time of its establishment, It is 1 if the condition is met, otherwise it is 0; At the time of its establishment, It is 1 if it is true, otherwise it is 0. By combining TF-IDF with domain terminology weighting algorithms, the set of words that best represents the core entities and precise parameters of the fragment is extracted. This is equivalent to labeling the document fragment with a set of precise, discrete, and indexable tags. Based on the enhanced keyword scores, the top-ranked keywords are selected to form a core keyword set. . Furthermore, the core of the technology-oriented summary generation process is to accurately extract key business information such as "operation procedures, parameter standards, and fault diagnosis" from document fragments, specifically including: Standardized document fragments are segmented into sets of sentences, and the sentences are categorized according to power business logic; That is to Segmented into a set of sentences The sentences are categorized into "operation step sentences / parameter standard sentences / fault diagnosis sentences / background explanation sentences"; through an embedding model, the sentences are encoded into vectors, resulting in 1024-dimensional vectors. ; Calculate the semantic similarity score between each sentence and the document fragment as a whole; then, based on power demand, sort different types of sentences according to "business value" (voltage level sentences > background explanation sentences > parameter standard sentences) to ensure that the summary prioritizes retaining core decision-making information in the design scenario. The mathematical formula involved is:

[0034] in, For document fragments The overall vector is calculated by the formula, which selects the sentence that best represents the core content of the document by calculating the semantic similarity between the sentence and the document as a whole.

[0035] By combining the business priority of sentence classification with the semantic similarity score, a preset number of sentences are selected, concatenated to generate a structured technical summary, and a corresponding summary vector is generated. When filtering a preset number of sentences in this step, sentences are first filtered according to business types such as "operation process / parameter standards," and then combined with semantic similarity scores. Take first in descending order These sentences are pieced together to form a logically coherent technical summary. Simultaneously, a summary vector is generated through the embedding model. This is used for subsequent retrieval and matching.

[0036] As needed, the steps for obtaining the semantic vector include: A pre-trained embedding model is used to convert standardized document fragments into initial semantic vectors; Specifically, the preprocessed document fragments (e.g., a text fragment from the "110kV Transformer - Design Chapter") Input the embedded model and output a 1024-dimensional initial vector. ;in, This is a preprocessed fragment of the original power document. As the initial numerical representation of the semantics of document fragments by the model, 1024 dimensions are the conventional output dimension of this type of pre-trained model, which can balance semantic representation ability and computational efficiency. The initial semantic vector is normalized to obtain a unit semantic vector, which is used for subsequent similarity calculation and storage. The mathematical formula involved in normalization is:

[0037] in, This is the final text vector after normalization. This is the initial vector output by the model. This represents the L2 norm of the initial vector; the purpose of this formula is to convert the initial vector into a unit vector, eliminating the influence of vector length differences on subsequent similarity calculations. Without normalization, long vectors may cause similarity calculation biases due to their numerical scale advantage. Normalization ensures that semantic similarity is measured only by the angle between vectors, improving the accuracy of retrieval and matching, and ultimately yielding... Used for subsequent vector database storage and retrieval matching.

[0038] The extracted multi-dimensional professional features are stored in a hybrid storage architecture adapted to the characteristics of power business. The hybrid storage architecture includes a relational database for storing structured metadata and a vector database for storing semantic vectors. The structured metadata includes the core keywords and the technology-oriented summary. The hybrid storage architecture is responsible for classifying and storing the extracted multi-dimensional professional features according to the characteristics of the power business, ensuring the orderliness, efficiency and retrieval of data storage; that is, storing the core keywords and the technology-oriented summary in a relational database, and storing the semantic vector in a vector database.

[0039] In addition, the construction steps of the hybrid storage architecture specifically include: A power-specific metadata table is constructed in the relational database to store the unique identifier of document fragments, business type, equipment type, core keyword set, and technology-oriented summary; In addition, the metadata table is built using a relational database (such as MySQL), and indexes are created for fields such as keywords, business types, and equipment types to improve the query efficiency of subsequent keyword matching and condition filtering. For example, the equipment type index can quickly locate all document fragments related to "110kV transformer"; the specification version number index can quickly filter out specification documents of a specific version. In the vector database, semantic vectors are partitioned and stored according to the business type, and secondary indexes are established in each partition according to device type or scenario type; Vector databases use high-performance vector databases (such as Milvus) to store normalized text vectors. To improve subsequent retrieval efficiency, the vector storage adopts a strategy of partitioning by business type. That is, the vectors of document fragments of the "case" type are stored in the case partition, the "specification" type is stored in the specification partition, and so on. Within each partition, a secondary index is built based on the equipment type of the document to further narrow the search scope. For example, the secondary index within the case partition is divided according to scenario types such as "hydrology, geology, and meteorology". When searching for content related to "what transformer should be selected when designing a 110kV substation at an altitude of 1500m", only vector matching needs to be performed in the transformer secondary index of the case partition, which greatly reduces the amount of computation. The unique identifier establishes a connection between the metadata in the relational database and the corresponding semantic vectors in the vector database. Specifically, the keywords, technical summaries, and summary vectors of each document fragment in the metadata database are bound to the corresponding text vectors in the vector database using the same unique document ID. Simultaneously, a data synchronization and update mechanism is established: when the original power documents are modified, updated, or deleted, a preprocessing and feature extraction process is triggered to regenerate multi-dimensional feature data and synchronously update the corresponding records in the metadata and vector databases, ensuring the consistency between the stored data and the original documents. Furthermore, the stored data is periodically validated and optimized to remove duplicate and invalid data (such as document fragments of expired regulations), improving the performance of the storage system.

[0040] Specifically, the business types include: engineering design cases, equipment technical specifications, industry regulations and standards, and typical design schemes.

[0041] It should be noted that when the business type is an engineering design case, the scenario type includes: hydrological conditions, geological conditions, and meteorological conditions.

[0042] Example 2 like Figure 2 The system shown is a knowledge extraction and multi-dimensional indexing optimization system for power documents, comprising: The preprocessing module is used to preprocess the original power documents to obtain standardized power document fragments; A multi-dimensional professional feature extraction module is used to extract multi-dimensional professional features from the standardized power document fragments. The multi-dimensional professional features include core keywords in the power field, technology-oriented summaries, and semantic vectors. The power-adaptive storage module is used to store the extracted multi-dimensional professional features into a hybrid storage architecture adapted to the characteristics of power business. The hybrid storage architecture includes a relational database for storing structured metadata and a vector database for storing semantic vectors. The structured metadata includes the core keywords and the technology-oriented summary.

[0043] Example 3 As attached Figure 3 An electronic device shown is characterized in that it comprises: Processor, memory, communication interface; The memory is used to store the executable instructions of the processor; The processor is configured to execute the aforementioned knowledge extraction and multi-dimensional indexing optimization method for power documents by executing the executable instructions.

[0044] A readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the aforementioned knowledge extraction and multi-dimensional indexing optimization method for power documents.

[0045] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for knowledge extraction and multi-dimensional indexing optimization of power documents, characterized in that, Specifically, it includes: The original power documents are preprocessed to obtain standardized power document fragments; The standardized power document fragments are subjected to multi-dimensional professional feature extraction, which includes core keywords in the power field, technology-oriented summaries, and semantic vectors. The extracted multi-dimensional professional features are stored in a hybrid storage architecture adapted to the characteristics of power business. The hybrid storage architecture includes a relational database for storing structured metadata and a vector database for storing semantic vectors. The structured metadata includes the core keywords and the technology-oriented summary.

2. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 1, characterized in that: The preprocessing of the original power documents specifically includes: The original power documents are classified according to the preset power document classification system, and formatting marks and meaningless comments are removed. Based on device type and chapter information, the categorized documents are fragmented to obtain an initial set of document fragments; The initial set of document fragments is segmented using a word segmentation model trained on a power industry terminology database, and stop words in the power field are removed to obtain a standardized set of power document fragments.

3. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 2, characterized in that: The steps for extracting the core keywords in the power sector include: Calculate the underlying TF-IDF values ​​of words in a standardized document fragment; Construct a power-specific terminology database, and enhance the TF-IDF values ​​of terms belonging to the database based on the power industry terminology database. The keywords are sorted according to their enhanced scores, and a predetermined number of the top-ranking words are selected to form a core keyword set.

4. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 3, characterized in that: The mathematical formula for calculating the basic TF-IDF values ​​of words in a standardized document fragment is as follows: in, Words In the current standardized text fragments Frequency of occurrence in; Indicates the current standardized text fragment The total frequency of all words in the text; Indicates the total number of document fragments related to electricity; Indicates included words The number of power document fragments.

5. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 4, characterized in that: The construction of a power-specific terminology database, and the weighting of the TF-IDF values ​​of terms belonging to the database based on the power industry terminology database, are described by the following mathematical formula: In the formula, Scoring for electricity-specific keywords Weighting coefficients for electricity-specific terms. A terminology database specifically for the power industry.

6. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 1, characterized in that: The steps for generating the technology-oriented summary include: Standardized document fragments are segmented into sets of sentences, and the sentences are categorized according to power business logic; Calculate the semantic similarity score between each sentence and the document fragment as a whole; By combining the business priority of sentence classification with the semantic similarity score, a preset number of sentences are selected, concatenated to generate a structured technical summary, and a corresponding summary vector is generated.

7. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 1, characterized in that: The steps for obtaining the semantic vector include: A pre-trained embedding model is used to convert standardized document fragments into initial semantic vectors; The initial semantic vector is normalized to obtain a unit semantic vector, which is used for subsequent similarity calculation and storage.

8. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 1, characterized in that: The construction steps of the hybrid storage architecture specifically include: A power-specific metadata table is constructed in the relational database to store the unique identifier of document fragments, business type, equipment type, core keyword set, and technology-oriented summary; In the vector database, semantic vectors are partitioned and stored according to the business type, and secondary indexes are established in each partition according to device type or scenario type; The unique identifier is used to establish an association between the metadata in the relational database and the corresponding semantic vector in the vector database.

9. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 8, characterized in that: The business types include: engineering design cases, equipment technical specifications, industry regulations and standards, and typical design schemes.

10. The knowledge extraction and multi-dimensional indexing optimization method for power documents according to claim 8, characterized in that: When the business type is an engineering design case, the scenario type includes: hydrological conditions, geological conditions, and meteorological conditions.

Images