Intelligent error correction method and system for power text

Abstract

The invention relates to a power text intelligent error correction method, which comprises the following steps of: 1, collecting power-related text data, including a report, a technical document, an operation manual and a user manual of the power industry; 2, establishing a data label system, marking a proper label on each piece of data according to a source, a type, a theme and the like, supporting multi-dimensional data retrieval, and performing language detection and word segmentation processing on a text by using a natural language processing tool; 3, performing error detection on the text data subjected to detection and word segmentation processing, and constructing a training data set; step 4, based on a sequence-to-sequence seq2seq model, constructing a power text intelligent error correction model for generating a corrected text, and enhancing training data by adding an error / correct text pair; and 5, performing text intelligent error correction according to the trained power text intelligent error correction model. According to the method, the wrong text can be effectively identified, and the high-quality correct text can be generated.

Description

Technical Field

[0001] The present invention relates to the field of text detection, and in particular to an intelligent error correction method and system for electric power text. Background Art

[0002] With the rapid development of the power industry, the number of various technical documents, operating manuals and user manuals is also increasing. These texts often contain complex professional terms and technical descriptions. Due to negligence in the manual writing and editing process, errors are inevitable. These errors may include spelling errors, grammatical errors and semantic inconsistencies, which in turn affect the readability of the document and the user's understanding, and may even lead to technical errors in operation. Summary of the invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide an intelligent error correction method and system for electric power text, which can effectively identify erroneous text and generate high-quality correct text.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] An intelligent error correction method for electric power text comprises the following steps:

[0006] Step 1: Collect text data related to electricity, including reports, technical documents, operation manuals, and user manuals of the power industry to ensure the diversity and richness of the data;

[0007] Step 2: Establish a data labeling system to label each piece of data appropriately according to source, type, subject, etc., support multi-dimensional data retrieval, and use natural language processing tools to perform language detection and word segmentation on the text;

[0008] Step 3: Perform error detection on the text data after detection and word segmentation processing to build a training data set;

[0009] Step 4: Build an intelligent error correction model for power text based on the sequence-to-sequence (seq2seq) model to generate corrected text. The training data is enhanced by adding error / correct text pairs.

[0010] Step 5: Perform text intelligent error correction based on the trained power text intelligent error correction model.

[0011] Further, step 1 is specifically as follows:

[0012] Use Python's Scrapy to create a web crawler to batch download text from public web pages, and set the crawler to only crawl content that matches the topic or keyword;

[0013] Leverage open APIs to access content, ensuring efficient and consistent data collection; use REST API interfaces to retrieve data from commercial databases;

[0014] For the collected PDF, DOCX and other file types, the language processing library is used to extract the text content; the NoSQL database is used to store structured and semi-structured data.

[0015] Further, step 2 is specifically as follows:

[0016] Build a multi-level labeling architecture, including

[0017] Source label: Classify data according to its source (such as industry reports, technical documents, user manuals);

[0018] Type tags: tag based on content format (such as PDF, HTML, DOCX);

[0019] Topic tags: Thematic analysis is performed based on the text content, and topic tags such as "grid stability", "new energy", and "safe operation" are marked;

[0020] Time tag: collect timestamp information for each data record to facilitate historical data query;

[0021] The NoSQL database MongoDB is used to handle dynamic and complex tag combinations, and tags are stored as nested fields for flexible query and modification.

[0022] Assign a unique identifier to each piece of data to associate tag information, and use JSON format to store data and its corresponding tags for easy expansion and parsing;

[0023] Use Spacy’s langdetect library to automatically detect text language, mark non-Chinese or English data, and ensure appropriate handling of multiple languages.

[0024] Use Spacy for word segmentation to obtain the basic unit vocabulary of the text. For Chinese text, use the jieba tool for accurate word segmentation;

[0025] Use the TF-IDF model to analyze text topics and automatically generate and assign topic tags based on word frequency and semantic information;

[0026] Build a rule engine to identify specific tags through keyword matching and regular expressions, and combine keywords extracted by NLP to achieve automatic tag assignment.

[0027] Furthermore, a rule engine is built to identify specific tags through keyword matching and regular expressions, combined with keywords extracted by NLP to achieve automatic tag assignment, as follows:

[0028] Create a dedicated keyword library covering key terms, equipment names, and technical terms in the power industry, extract high-frequency and professional terms from existing power literature, and form a preliminary keyword list;

[0029] Construct regular expressions for specific information patterns;

[0030] Set regular rules for common text patterns;

[0031] To avoid conflicts during fuzzy matching, establish the rule priority: literal exact matching > regular pattern matching > wildcard matching;

[0032] Use Apache Camel open source rule engine framework, support flexible rule configuration and efficient execution, and save rule configuration in YAML or JSON format;

[0033] Input text data, extract core words, perform keyword matching, and directly assign corresponding tags according to the hit keywords; perform regular matching, detect specific patterns, and tag the text that meets the conditions; select the most suitable tag or combination of tags according to the priority rules;

[0034] Set a keyword weight threshold, only assign tags to keywords that exceed the set value, and determine the importance of keywords based on TF-IDF score and frequency of occurrence;

[0035] The preliminary labels assigned by the rule engine are cross-validated with the keywords extracted by NLP, and the inconsistent parts are adjusted. A feedback mechanism is introduced to allow the user's label selection to adjust and optimize the system.

[0036] Furthermore, step 3 is as follows:

[0037] By building a professional vocabulary for the power industry and implementing Levenshtein distance calculation, each word in the input text is compared with the professional vocabulary to identify spelling errors. Double spelling / initial abbreviation matching is introduced to further shorten the calculation time, and the search efficiency of the vocabulary is optimized based on the Trie tree structure.

[0038] Configure the parser to analyze sentence structure and catch grammatical errors, using custom grammar rule configurations to include special structures and syntax used by power terminology;

[0039] Fine-tune the BERT model based on a large amount of corpus data in the power field, use its contextual semantic understanding ability to identify errors, and judge the semantic rationality of sentences by comparing the input text with the expected output of the model.

[0040] Furthermore, an intelligent error correction model for power text is constructed based on the sequence-to-sequence seq2seq model, as follows:

[0041] Collect real power industry texts, such as technical documents, reports, emails, and field records, and obtain structured and correct texts, including user manuals and product specifications;

[0042] Generate incorrect text by cutting, adding, deleting, conjugating, misplacing, spelling errors and grammatical errors from random words in correct text; collect typical errors from real data and extract error patterns by analyzing logs or user feedback;

[0043] Construct one or more incorrect versions for each correct text to form a training pair of (incorrect text, correct text); ensure that the training set covers a variety of error types and text structures;

[0044] The seq2seq model using the Transformer architecture uses an encoder to convert the input erroneous text into a hidden layer representation, and then generates the corrected text through a decoder;

[0045] Use pre-trained word embeddings to initialize word vectors, combine domain-specific fine-tuning to enhance the model's understanding of terminology, and use a multi-head attention mechanism to ensure that the model captures important features in the text, especially focusing on global text coherence when correcting errors;

[0046] Batch processing and fixed step size options are used for text data to ensure the controllability and efficiency of training, and the cross entropy loss function is used to evaluate the difference between the predicted word and the target word.

[0047] Gradually increase the complexity of error types, starting with simple spelling and grammatical errors, then adding more complex semantic errors, and conduct phased training.

[0048] An intelligent error correction system for electric power texts includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the intelligent error correction method for electric power texts as described above.

[0049] The present invention has the following beneficial effects:

[0050] The present invention can automatically detect and correct various errors in texts. By collecting electricity-related texts and combining them with industry-specific terminology libraries, a deep learning model is constructed, which not only improves the accuracy of error correction, but also enhances the applicability and scalability of the system. In addition, the automated data labeling and error detection process provides a basis for training efficient machine learning models, ensuring the robustness of the system when facing different types of texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the method of the present invention;

[0052] Figure 2 Schematic diagram of error correction in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0054] refer to Figure 1 In this embodiment, a method for intelligent error correction of electric power text is provided, comprising the following steps:

[0055] Step 1: Collect text data related to electricity, including reports, technical documents, operation manuals, and user manuals of the power industry to ensure the diversity and richness of the data;

[0056] Step 2: Establish a data labeling system to label each piece of data appropriately according to source, type, subject, etc., support multi-dimensional data retrieval, and use natural language processing tools to perform language detection and word segmentation on the text;

[0057] Step 3: Perform error detection on the text data after detection and word segmentation processing to build a training data set;

[0058] Step 4: Build an intelligent error correction model for power text based on the sequence-to-sequence (seq2seq) model to generate corrected text. The training data is enhanced by adding error / correct text pairs.

[0059] Step 5: Perform text intelligent error correction based on the trained power text intelligent error correction model.

[0060] In this embodiment, step 1 is specifically as follows:

[0061] Use Python's Scrapy to create a web crawler to batch download text from public web pages, and set the crawler to only crawl content that matches the topic or keyword (such as "power system" and "grid stability");

[0062] Leverage open APIs to access content, ensuring efficient and consistent data collection; use REST API interfaces to retrieve data from commercial databases;

[0063] For the collected PDF, DOCX and other file types, the language processing library is used to extract the text content; the NoSQL database is used to store structured and semi-structured data.

[0064] In this embodiment, step 2 is specifically as follows:

[0065] Build a multi-level labeling architecture, including

[0066] Source label: Classify data according to its source (such as industry reports, technical documents, user manuals);

[0067] Type tags: tag based on content format (such as PDF, HTML, DOCX);

[0068] Topic tags: Thematic analysis is performed based on the text content, and topic tags such as "grid stability", "new energy", and "safe operation" are marked;

[0069] Time tag: collect timestamp information for each data record to facilitate historical data query;

[0070] The NoSQL database MongoDB is used to handle dynamic and complex tag combinations, and tags are stored as nested fields for flexible query and modification.

[0071] Assign a unique identifier to each piece of data to associate tag information, and use JSON format to store data and its corresponding tags for easy expansion and parsing;

[0072] Use Spacy’s langdetect library to automatically detect text language, mark non-Chinese or English data, and ensure appropriate handling of multiple languages.

[0073] Use Spacy for word segmentation to obtain the basic unit vocabulary of the text. For Chinese text, use the jieba tool for accurate word segmentation;

[0074] Use the TF-IDF model to analyze text topics and automatically generate and assign topic tags based on word frequency and semantic information;

[0075] Build a rule engine to identify specific tags through keyword matching and regular expressions, and combine keywords extracted by NLP to achieve automatic tag assignment.

[0076] In this embodiment, a rule engine is constructed to identify specific tags through keyword matching and regular expressions, and to realize automatic tag assignment in combination with keywords extracted by NLP, as follows:

[0077] Create a dedicated keyword library covering key terms, equipment names, and technical terms in the power industry, extract high-frequency and professional terms from existing power literature, and form a preliminary keyword list;

[0078] Build regular expressions for specific information patterns (such as date formats, data specifications, standard numbers); set regular rules for common text patterns (such as "voltage volts");

[0079] To avoid conflicts during fuzzy matching, establish the rule priority: literal exact matching > regular pattern matching > wildcard matching;

[0080] Use Apache Camel open source rule engine framework, support flexible rule configuration and efficient execution, and save rule configuration in YAML or JSON format;

[0081] Input text data, extract core vocabulary (such as obtained through word segmentation and named entity recognition), perform keyword matching, and directly assign corresponding tags based on the hit keywords; perform regular matching, detect specific patterns, and tag the text that meets the conditions; select the most suitable tag or combination of tags according to the priority rules;

[0082] Set a keyword weight threshold, only assign tags to keywords that exceed the set value, and determine the importance of keywords based on TF-IDF score and frequency of occurrence;

[0083] The preliminary labels assigned by the rule engine are cross-validated with the keywords extracted by NLP, and the inconsistent parts are adjusted. A feedback mechanism is introduced to allow the user's label selection to adjust and optimize the system.

[0084] In this embodiment, step 3 is specifically as follows:

[0085] By building a professional vocabulary for the power industry and implementing Levenshtein distance calculation, each word in the input text is compared with the professional vocabulary to identify spelling errors. A threshold is set, for example, a word with a distance less than or equal to 2 is considered a candidate for spelling error. Double spelling / initial abbreviation matching is introduced to further shorten the calculation time, and the search efficiency of the vocabulary is optimized based on the Trie tree structure.

[0086] Configure a parser (such as the PCFG parser in NLTK) to analyze sentence structure and catch grammatical errors, using custom grammar rule configurations to include special structures and syntax used by power terminology;

[0087] Fine-tune the BERT model based on a large amount of corpus data in the power field, use its contextual semantic understanding ability to identify errors, and judge the semantic rationality of sentences by comparing the input text with the expected output of the model.

[0088] In this embodiment, an intelligent error correction model for electric power text is constructed based on a sequence-to-sequence seq2seq model, as follows:

[0089] Collect real power industry texts, such as technical documents, reports, emails, and field records, and obtain structured and correct texts, including user manuals and product specifications;

[0090] Generate incorrect text by cutting, adding, deleting, conjugating, misplacing, spelling errors and grammatical errors from random words in correct text; collect typical errors from real data and extract error patterns by analyzing logs or user feedback;

[0091] Construct one or more incorrect versions for each correct text to form a training pair of (incorrect text, correct text); ensure that the training set covers a variety of error types and text structures;

[0092] The seq2seq model using the Transformer architecture uses an encoder to convert the input erroneous text into a hidden layer representation, and then generates the corrected text through a decoder;

[0093] Use pre-trained word embeddings to initialize word vectors, combine domain-specific fine-tuning to enhance the model's understanding of terminology, and use a multi-head attention mechanism to ensure that the model captures important features in the text, especially focusing on global text coherence when correcting errors;

[0094] Batch processing and fixed step size options are used for text data to ensure the controllability and efficiency of training, and the cross entropy loss function is used to evaluate the difference between the predicted word and the target word.

[0095] Gradually increase the complexity of error types, starting with simple spelling and grammatical errors, then adding more complex semantic errors, and conduct phased training.

[0096] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0100] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. An intelligent error correction method for electric power text, characterized in that: The following steps are involved: Step 1: Collect text data related to electricity, including reports, technical documents, operation manuals, and user manuals of the power industry; Step 2: Establish a data labeling system to label each piece of data appropriately according to source, type, subject, etc., support multi-dimensional data retrieval, and use natural language processing tools to perform language detection and word segmentation on the text; Step 3: Perform error detection on the text data after detection and word segmentation processing to build a training data set; Step 4: Build an intelligent error correction model for power text based on the sequence-to-sequence (seq2seq) model to generate corrected text. The training data is enhanced by adding error / correct text pairs. Step 5: Perform text intelligent error correction based on the trained power text intelligent error correction model.

2. According to claim 1, a method for intelligent error correction of electric power text is characterized in that: The step 1 is specifically as follows: Use Python's Scrapy to create a web crawler to batch download text from public web pages, and set the crawler to only crawl content that matches the topic or keyword; Leverage open APIs to access content, ensuring efficient and consistent data collection; use REST API interfaces to retrieve data from commercial databases; For the collected PDF, DOCX and other file types, the language processing library is used to extract the text content; the NoSQL database is used to store structured and semi-structured data.

3. According to claim 1, the intelligent error correction method for electric power text is characterized in that: The step 2 is specifically as follows: Build a multi-level labeling architecture, including Source label: classify data according to its source; Type tag: tag based on content format; Topic tags: Thematic analysis is performed based on the text content, marking the topic tags of grid stability, new energy, and safe operation; Time tag: collect timestamp information for each data record to facilitate historical data query; The NoSQL database MongoDB is used to handle dynamic and complex tag combinations, and tags are stored as nested fields for flexible query and modification. Assign a unique identifier to each piece of data to associate tag information, and use JSON format to store data and its corresponding tags for easy expansion and parsing; Use Spacy’s langdetect library to automatically detect text language, mark non-Chinese or English data, and ensure appropriate handling of multiple languages. Use Spacy for word segmentation to obtain the basic unit vocabulary of the text. For Chinese text, use the jieba tool for accurate word segmentation; Use the TF-IDF model to analyze text topics and automatically generate and assign topic tags based on word frequency and semantic information; Build a rule engine to identify specific tags through keyword matching and regular expressions, and combine keywords extracted by NLP to achieve automatic tag assignment.

4. According to claim 3, a method for intelligent error correction of electric power text is characterized in that: The rule engine is constructed to identify specific tags through keyword matching and regular expressions, combined with keywords extracted by NLP, to achieve automatic tag assignment, as follows: Create a dedicated keyword library covering key terms, equipment names, and technical terms in the power industry. Extract high-frequency and specialized terms from existing power literature to form a preliminary keyword list; Construct regular expressions for specific information patterns; Set regular rules for common text patterns; To avoid conflicts during fuzzy matching, establish the rule priority: literal exact matching > regular pattern matching > wildcard matching; Use Apache Camel open source rule engine framework, support flexible rule configuration and efficient execution, and save rule configuration in YAML or JSON format; Input text data, extract core words, perform keyword matching, and directly assign corresponding tags according to the hit keywords; perform regular matching, detect specific patterns, and tag the text that meets the conditions; select the most suitable tag or combination of tags according to the priority rules; Set a keyword weight threshold, only assign tags to keywords that exceed the set value, and determine the importance of keywords based on TF-IDF score and frequency of occurrence; The preliminary labels assigned by the rule engine are cross-validated with the keywords extracted by NLP, and the inconsistent parts are adjusted. A feedback mechanism is introduced to allow the user's label selection to adjust and optimize the system.

5. According to claim 1, the intelligent error correction method for electric power text is characterized in that: The step 3 is as follows: By building a professional vocabulary for the power industry and implementing Levenshtein distance calculation, each word in the input text is compared with the professional vocabulary to identify spelling errors; Double-spelling / initial abbreviation matching is introduced to further shorten the calculation time, and the search efficiency of the vocabulary is optimized based on the Trie tree structure. Configure the parser to analyze sentence structure and catch grammatical errors, using custom grammar rule configurations to include special structures and syntax used by power terminology; Fine-tune the BERT model based on a large amount of corpus data in the power field, use its contextual semantic understanding ability to identify errors, and judge the semantic rationality of sentences by comparing the input text with the expected output of the model.

6. The intelligent error correction method for electric power text according to claim 1 is characterized in that: The electric power text intelligent error correction model is constructed based on the sequence-to-sequence seq2seq model, as follows: Collect real power industry texts, such as technical documents, reports, emails, and field records, and obtain structured and correct texts, including user manuals and product specifications; Generate incorrect text by cutting, adding, deleting, conjugating, misplacing, spelling errors and grammatical errors from random words in correct text; collect typical errors from real data and extract error patterns by analyzing logs or user feedback; Construct one or more incorrect versions for each correct text to form a training pair of (incorrect text, correct text); ensure that the training set covers a variety of error types and text structures; The seq2seq model using the Transformer architecture uses an encoder to convert the input erroneous text into a hidden layer representation, and then generates the corrected text through a decoder; Use pre-trained word embeddings to initialize word vectors, combine domain-specific fine-tuning to enhance the model's understanding of terminology, and use a multi-head attention mechanism to ensure that the model captures important features in the text, especially focusing on global text coherence when correcting errors; Batch processing and fixed step size options are used for text data to ensure the controllability and efficiency of training, and the cross entropy loss function is used to evaluate the difference between the predicted word and the target word. Gradually increase the complexity of error types, starting with simple spelling and grammatical errors, then adding more complex semantic errors, and conduct phased training.

7. An intelligent error correction system for electric power text, characterized in that: It includes a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps in the intelligent error correction method for power text as described in any one of claims 1-6.

Images