Automatic marking method for power dispatching document based on semantic analysis

Abstract

The invention discloses a semantic analysis-based power dispatching document automatic marking method, which comprises the following steps of: converting a power dispatching document into a structured text and extracting key information through document data preprocessing; carrying out keyword extraction and weighting by utilizing a TF-IWF algorithm to generate a document vector; matching with a predefined template through cosine similarity calculation, and selecting the most suitable template for automatic filling; the extracted key information is automatically checked and then filled into a template field, and finally, a generated standardized document is stored in a document management system, so that subsequent retrieval, query and decision support are facilitated. According to the method, the key information in the document can be automatically extracted and subjected to standardized marking, the document processing efficiency and accuracy are remarkably improved, manual intervention is reduced through document automatic matching and template filling, manual omission is avoided, and the management requirement of the large-scale power dispatching document can be quickly responded.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to an automated tagging method for power dispatch documents based on semantic analysis. Background Technology

[0002] The power dispatching field generates a wide variety of documents, including dispatch orders, official documents, management regulations, and dispatch reports. These documents are mostly unstructured data, in diverse formats (such as PDF, Word, and images), and their content is complex, making manual processing cumbersome and hindering efficient information extraction and management. Chinese patent CN117521662A discloses a method based on deep learning technology to annotate fault information in power dispatching documents and extract relational information through a model to build a knowledge base to assist dispatchers in decision-making. While this can improve system intelligence, process fault information, and provide decision support, it relies on existing fault logs and document information, limiting its ability to handle unknown or novel dispatching documents. Chinese patent CN118469199A discloses a power dispatching strategy generation system based on natural language understanding. This system uses natural language processing technology to parse user voice commands and interacts with the power dispatching system. Through voice recognition, the system can accurately identify commands and convert them into operations, simplifying the operation process and improving dispatching system efficiency. However, this patent mainly focuses on voice command recognition and basic operations, relying on real-time user voice input. Therefore, there is an urgent need for an efficient and accurate automated tagging method to automatically identify, analyze, and extract key information from power dispatch documents. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide an automated marking method for power dispatch documents based on semantic analysis, in order to solve the problems of excessive manual intervention and low efficiency in existing power dispatch document processing.

[0004] Technical solution: The automatic marking method for power dispatch documents based on semantic analysis described in this invention includes the following steps:

[0005] (1) Through document data preprocessing, power dispatch documents are converted into structured text and key information is extracted;

[0006] (2) Use the TF-IWF algorithm to extract and weight keywords, and generate document vectors;

[0007] (3) The most suitable template is selected for automatic filling by matching the predefined template with the cosine similarity calculation;

[0008] (4) The extracted key information is automatically verified and then filled into the template fields. Finally, the generated standardized document is stored in the document management system for easy retrieval, query and decision support.

[0009] Furthermore, step (1) includes format conversion, text cleaning, and keyword extraction.

[0010] Furthermore, the keyword extraction includes:

[0011] Term frequency calculation: This is the ratio of the number of times a word appears in a document to the total number of words in the document. The formula is as follows:

[0012]

[0013] Where count(w,D) is the number of times word w appears in document D, and N is the total number of words in document D;

[0014] Inverse document frequency (IWF) calculation: Used to assess the rarity of a word in the entire document set. The formula for calculating IWF(w) is:

[0015]

[0016] Where N is the total number of documents in the document set, and df(w) is the number of documents containing the word w;

[0017] TF-IWF weighting: The term frequency and inverse document frequency of each word are multiplied to obtain the weighted term frequency value. The calculation formula is as follows:

[0018]

[0019] This weighting takes into account both the frequency of a word in a document and its rarity in the entire document set.

[0020] Furthermore, step (2) includes text vector construction and cosine similarity calculation.

[0021] Furthermore, the text vector construction includes:

[0022] (2.1) Assume document D i It contains m keywords, namely w 1, w2,…,w m ;

[0023] (2.2) Each keyword w i Corresponding TF-IWF value This reflects the importance of the word in the document, taking into account both the frequency of the word in the document and its rarity in the entire document set.

[0024] (2.3) By calculating the TF-IWF value, a weighted value is assigned to each keyword in the document, and finally a vector is generated to represent the document.

[0025] document The vector representation is:

[0026]

[0027] Each document vector's dimension corresponds to the number of all unique keywords in the document, and the value of each dimension is the TF-IWF weighted value of that keyword, representing the importance of that keyword in the document.

[0028] Furthermore, the formula for the cosine similarity is as follows:

[0029]

[0030] in, This is the document to be analyzed. The vector representation of , It is a standard template document The vector representation of , where ⋅ represents the dot product of vectors. and These are documents and templates The norm of a vector represents the "size" of each vector.

[0031] Furthermore, step (3) includes template matching and automatic tagging and text generation.

[0032] Furthermore, the template matching includes:

[0033] (3.1) Vector representation of documents and templates: Through the text vectorization steps mentioned above, each document and template will be converted into a vector representation, and each dimension of the vector corresponds to the TF-IWF weighted value of the keywords in the document or template;

[0034] (3.2) Similarity calculation: The cosine similarity between the document vector and the standard template vector is used to calculate their matching degree. By calculating the angle between the document and the template, a similarity score is obtained, which represents the degree of similarity between the document content and the template.

[0035] (3.3) Select the best matching template: After calculating the similarity between multiple templates and the document, the system selects the template with the highest similarity, which is considered to be the most suitable for the content structure of the document. The system automatically selects the most suitable template based on the specific content of the document.

[0036] Furthermore, the automatic tagging and text generation includes automatically extracting key information from the document according to the template structure and filling it into the corresponding fields in the template. The extracted information is automatically verified. If the field information is incomplete or does not meet the requirements, the system automatically reminds the user to supplement it. After tagging is completed, the generated standardized document is saved in the document management system for subsequent retrieval, query and decision support.

[0037] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention overcomes the problems of reliance on manual processing, low efficiency, and susceptibility to errors in existing technologies by using a semantic analysis-based automated marking method for power dispatch documents. Compared with traditional methods, this invention can automatically extract key information from documents and perform standardized marking, significantly improving the efficiency and accuracy of document processing. Through automatic document matching and template filling, it reduces manual intervention, avoids human oversight, and can quickly respond to the management needs of large-scale power dispatch documents. Furthermore, the system also has document retrieval and data analysis functions, providing effective support for decision support and intelligent management of power dispatch, further improving the overall efficiency and accuracy of power dispatch work. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method described in this invention. Detailed Implementation

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0040] The semantic analysis-based automated marking method for power dispatch documents described in this invention solves the problems of low processing efficiency and excessive manual intervention in existing power dispatch document technologies. Through document data preprocessing, power dispatch documents are transformed into structured text and key information is extracted. The TF-IWF algorithm is used for keyword extraction and weighting to generate document vectors. Cosine similarity calculation is used to match with predefined templates, and the most suitable template is automatically filled in. Extracted key information, such as equipment number, dispatch instructions, and operation time, is automatically verified and then filled into template fields to ensure data integrity and accuracy. Finally, the generated standardized documents are stored in a document management system for easy retrieval, querying, and decision support. This solution improves document processing efficiency, reduces manual intervention, and ensures efficient and accurate management of power dispatch documents. A detailed flowchart is shown below. Figure 1 As shown:

[0041] (1) Document data preprocessing:

[0042] The main objective of document data preprocessing is to convert power dispatching documents into structured standard texts, remove irrelevant information, extract key information and conduct analysis. This step ensures that the document data can efficiently enter the subsequent analysis and processing processes, avoiding affecting the results due to inconsistent formats or cluttered information. The following are the detailed steps of document data preprocessing:

[0043] (1.1)Format conversion: Power dispatching documents usually exist in formats such as PDF, Word, and images. To ensure the consistency of document data, all documents need to be first converted into an operable structured text format.

[0044] PDF and Word document parsing: Extract the text content in PDF and Word format documents through existing document parsing tools (such as Apache Tika) or PDF extraction libraries. This tool can identify the text in the document and convert it into a processable text format, ensuring that the extraction of document information is not restricted by the format.

[0045] Processing of image format documents: For power dispatching documents in image format (such as scanned copies), OCR (Optical Character Recognition) technology (such as Tesseract OCR) can be used to extract the text in the image and convert it into structured text. OCR technology can recognize and extract text information from images and convert it into an editable text format, ensuring that the information in image files can also be utilized.

[0046] (1.2)Text cleaning: Text cleaning aims to remove noise and ensure that the key information in the document is retained. Specific operations include:

[0047] Remove irrelevant characters, numbers, punctuation marks, special symbols, and extra spaces, etc., to reduce unnecessary interference.

[0048] To ensure consistency, unify the character format in the document (usually converted to lowercase) to avoid analysis errors caused by different cases.

[0049] Delete common stop words (such as "of", "already") and irrelevant content. These words do not bring substantial information gain to subsequent analysis. The removal of stop words can effectively reduce the redundancy of the text and ensure that the subsequent keyword extraction is more targeted.

[0050] (1.3)Keyword extraction

[0051] Identify the key information in the document through keyword extraction algorithms (such as TF-IWF). The TF-IWF algorithm calculates the weighted term frequency and inverse document frequency of vocabulary to evaluate the importance of each vocabulary.

[0052] Term frequency (TF) calculation: This is the ratio of the number of times a particular word appears in a document to the total number of words in the document. The formula is:

[0053]

[0054] Here, count(w,D) is the number of times word w appears in document D, and N is the total number of words in document D. A higher word frequency indicates that the word is more important in the document.

[0055] Inverse document frequency (IWF) calculation: Used to evaluate the rarity of a word in the entire document set. Words with lower frequencies are more important and can distinguish core information in different documents. The formula for calculating Inverse Document Frequency (IWF)(w) is:

[0056]

[0057] Where N is the total number of documents in the document set, and df(w) is the number of documents containing the word w. Lower word frequency (i.e., fewer documents containing the word) is given higher weight.

[0058] TF-IWF weighting: The term frequency (TF-IWF) of each word is multiplied by its inverse document frequency (INV), resulting in a weighted TF-IWF value. The formula is as follows:

[0059]

[0060] This weighted value reflects the importance of a word in a document, taking into account both the frequency of the word in the document and its rarity in the entire document set.

[0061] By combining the TF-IWF algorithm, precise keyword extraction can be performed on power dispatch documents, helping the system to accurately identify key information in the documents and further promoting the intelligent management of power dispatch documents.

[0062] (2) Text vectorization:

[0063] The purpose of text vectorization is to transform power dispatch documents into a numerical form for subsequent similarity calculations and template matching. A vector representation is constructed for each document using keyword weights calculated using the TF-IWF method. This vector represents the core information of the document, enabling the computer to evaluate the similarity between documents, thereby achieving automated labeling and classification.

[0064] (2.1) Text Vector Construction

[0065] Each power dispatch document is used to calculate a set of weighted keywords using the TF-IWF algorithm. These weights reflect the importance of each keyword in the document, providing a basis for the numerical representation of the document. The specific steps are as follows:

[0066] Assume document D i It contains m keywords, namely w 1, w2,…,w m .

[0067] Each keyword w i Corresponding TF-IWF value This reflects the importance of the word in the document, taking into account both the frequency of the word in the document and its rarity in the entire document set.

[0068] By calculating TF-IWF values, a weighted value can be assigned to each keyword in the document, and a vector is ultimately generated to represent the document.

[0069] document The vector representation is:

[0070]

[0071] Each document vector's dimension corresponds to the number of all unique keywords in the document. The value of each dimension is the TF-IWF weighted value of that keyword, representing the keyword's importance in the document.

[0072] This vectorized representation provides the foundation for subsequent similarity calculations and template matching, ensuring that documents can accurately represent their content and semantics in a multidimensional space.

[0073] (2) Cosine similarity calculation

[0074] To compare the similarity between power dispatch documents and standard template documents, cosine similarity is used to measure the similarity between the two document vectors. Cosine similarity measures the degree of similarity by calculating the angle between the document vectors. The formula for cosine similarity is as follows:

[0075]

[0076] in, This is the document to be analyzed. The vector representation contains the TF-IWF weighted values ​​of each keyword in the document. It is a standard template document The vector representation of also includes the TF-IWF weighted values ​​of each keyword in the template. ⋅ represents the dot product of the vectors (i.e., multiplying and summing the TF-IWF values ​​of corresponding dimensions). and These are documents and templates The norm of a vector represents the "size" of each vector (i.e., the square root of the sum of the squares of all the dimension values ​​in the vector).

[0077] By calculating cosine similarity, the similarity between a document and a template can be determined. If the similarity is close to 1, it means the document and template are very similar; if the similarity is close to 0, it means the document and template are quite different. The higher the similarity, the better the match between the document and the template, and the more likely it is to automatically fill in the relevant fields in the template.

[0078] (3) Automatic document tagging and template matching:

[0079] The matching between documents and standard templates is achieved through similarity calculations. The system selects the standard template that best matches the document content based on the similarity score, and then automatically fills in the key information in the document according to the template's structure. This process, aided by an automated tagging system, greatly reduces manual operations and improves the efficiency and accuracy of document processing.

[0080] (3.1) Template matching

[0081] Template matching determines the most suitable template based on the similarity between a document and a predefined standard template. Standard templates are typically document formats designed according to the actual needs of the power dispatching field, such as dispatching command templates, management system templates, and operating procedure templates. These templates have fixed fields and structures, containing key information commonly used in power dispatching, such as grid equipment, dispatching tasks, operation time, and operators.

[0082] (3.1.1) Vector representation of documents and templates: Through the text vectorization steps mentioned above, each document and template will be converted into a vector representation, and each dimension of the vector corresponds to the TF-IWF weighted value of the keywords in the document or template.

[0083] (3.1.2) Similarity Calculation: The cosine similarity between the document vector and the standard template vector is used to calculate their matching degree. By calculating the angle (cosine value) between the document and the template, a similarity score is obtained, which represents the degree of similarity between the document content and the template.

[0084] (3.1.3) Selecting the best matching template: After calculating the similarity between multiple templates and the document, the system will select the template with the highest similarity, considering it to best match the content structure of the document. In this way, the system can automatically select the most suitable template based on the specific content of the document.

[0085] Through this template matching process, the system can automatically identify document content and find the most suitable standard template. This automated matching process significantly reduces manual intervention and improves the speed and accuracy of document processing.

[0086] (3.2) Automatic Tagging and Text Generation: Once a document successfully matches the standard template, the system automatically extracts key information from the document based on the template structure and fills it into the corresponding fields in the template, such as equipment number, dispatch instructions, and operation time. The extracted information is automatically verified to ensure the integrity and accuracy of the data. If the field information is incomplete or does not meet the requirements, the system will automatically remind the user to supplement it. After tagging is completed, the generated standardized document is saved in the document management system for subsequent retrieval, querying, and decision support. This process not only improves document processing efficiency but also provides support for intelligent management and data analysis of power dispatching.

[0087] This invention implements an automated marking and information extraction method for power dispatch documents. First, the system converts collected power dispatch documents (such as dispatch orders or management regulations in PDF or Word formats) into structured text using a document parsing tool (such as Apache Tika). Next, OCR technology is used to perform text recognition on the scanned image format documents. After preprocessing the document data, the system extracts keywords from the documents using the TF-IWF algorithm and calculates document vectors based on these keywords. Then, the document vectors are matched against predefined standard templates, cosine similarity is calculated, and the best-matching template is selected. Finally, the system automatically extracts key information from the documents (such as equipment number, dispatch task type, operation time, etc.), fills the extracted information into the corresponding fields in the template, and performs automatic verification to ensure data accuracy and integrity. The generated standardized dispatch documents are stored in a document management system for subsequent retrieval, analysis, and decision support. Through this implementation process, the system can efficiently and accurately process a large number of power dispatch documents, significantly improving work efficiency and reducing manual operations.

Claims

1. A method for automatically marking power dispatch documents based on semantic analysis, characterized in that, Includes the following steps: (1) Through document data preprocessing, power dispatch documents are converted into structured text and key information is extracted; (2) Use the TF-IWF algorithm to extract and weight keywords, and generate document vectors; (3) The most suitable template is selected for automatic filling by matching the predefined template with the cosine similarity calculation; (4) The extracted key information is automatically verified and then filled into the template fields. Finally, the generated standardized document is stored in the document management system for easy retrieval, query and decision support.

2. The method for automated marking of power dispatch documents based on semantic analysis according to claim 1, characterized in that, Step (1) includes format conversion, text cleaning, and keyword extraction.

3. The method for automated marking of power dispatch documents based on semantic analysis according to claim 2, characterized in that, The keyword extraction includes: Term frequency calculation: This is the ratio of the number of times a word appears in a document to the total number of words in the document. The formula is as follows: ， Where count(w,D) is the number of times word w appears in document D, and N is the total number of words in document D; Inverse document frequency (IWF) calculation: Used to assess the rarity of a word in the entire document set. The formula for calculating IWF(w) is: ， Where N is the total number of documents in the document set, and df(w) is the number of documents containing the word w; TF-IWF weighting: The term frequency and inverse document frequency of each word are multiplied to obtain the weighted term frequency value. The calculation formula is as follows: ， This weighting takes into account both the frequency of a word in a document and its rarity in the entire document set.

4. The method for automated marking of power dispatch documents based on semantic analysis according to claim 1, characterized in that, Step (2) includes text vector construction and cosine similarity calculation.

5. The method for automated marking of power dispatch documents based on semantic analysis according to claim 4, characterized in that, The text vector construction includes: (2.1) Assume document D i It contains m keywords, namely w 1, w2,…,w m ; (2.2) Each keyword w i Corresponding TF-IWF value This reflects the importance of the word in the document, taking into account both the frequency of the word in the document and its rarity in the entire document set. (2.3) By calculating the TF-IWF value, a weighted value is assigned to each keyword in the document, and finally a vector is generated to represent the document. document The vector representation is: ， Each document vector's dimension corresponds to the number of all unique keywords in the document, and the value of each dimension is the TF-IWF weighted value of that keyword, representing the importance of that keyword in the document.

6. The method for automated marking of power dispatch documents based on semantic analysis according to claim 4, characterized in that, The formula for cosine similarity is as follows: in, This is the document to be analyzed. The vector representation of , It is a standard template document The vector representation of , where ⋅ represents the dot product of vectors. and These are documents and templates The norm of a vector represents the "size" of each vector.

7. The method for automated marking of power dispatch documents based on semantic analysis according to claim 1, characterized in that, Step (3) includes template matching and automatic tagging and text generation.

8. The method for automated marking of power dispatch documents based on semantic analysis according to claim 7, characterized in that, The template matching includes: (3.1) Vector representation of documents and templates: Through the text vectorization steps mentioned above, each document and template will be converted into a vector representation, and each dimension of the vector corresponds to the TF-IWF weighted value of the keywords in the document or template; (3.2) Similarity calculation: The cosine similarity between the document vector and the standard template vector is used to calculate their matching degree. By calculating the angle between the document and the template, a similarity score is obtained, which represents the degree of similarity between the document content and the template. (3.3) Select the best matching template: After calculating the similarity between multiple templates and the document, the system selects the template with the highest similarity, which is considered to be the most suitable for the content structure of the document. The system automatically selects the most suitable template based on the specific content of the document.

9. The method for automated marking of power dispatch documents based on semantic analysis according to claim 7, characterized in that, The automatic tagging and text generation includes automatically extracting key information from the document based on the template structure and filling it into the corresponding fields in the template. The extracted information is automatically verified. If the field information is incomplete or does not meet the requirements, the system will automatically remind the user to supplement it. After tagging is completed, the generated standardized document is saved in the document management system for subsequent retrieval, query and decision support.

Images