Document auditing method based on artificial intelligence

Abstract

The invention relates to an artificial intelligence-based official document auditing method, which comprises the following steps of S1, collecting Chinese corpora to construct training data; s2, constructing training data on the basis of the Chinese corpus collected in the step S1, calling an online model to generate the corpus, and further constructing the training data; s3, performing data scoring, and correcting low-score data to improve data quality; s4, establishing a data quality improvement framework, and adjusting and optimizing the framework; s5, fine adjustment of the model; s6, evaluating and testing the combined model after fine tuning in the step S5; and S7, displaying the auditing result in the system. The method has the advantages that automatic, efficient and precise auditing is achieved, the official document processing capacity is improved, the correctness, normalization and legality of official document content are ensured, the labor cost is reduced, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to an official document review method based on artificial intelligence. Background Art

[0002] In today's era of information explosion, facing the growing demand for official document processing, official document review, as an important part of administrative management, its efficiency and accuracy are directly related to the efficiency and accuracy of policy implementation; the traditional official document review method mainly relies on manual review, which has many problems; first, manual review is slow and difficult to cope with a large number of official document processing needs, resulting in a longer review cycle and affecting work efficiency; second, manual review is easily limited by the auditor's personal experience and knowledge level, resulting in subjectivity and inconsistency in the review results. In addition, manual review is inefficient and prone to errors when dealing with a large number of repetitive and regular errors. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by this patent application is how to provide an artificial intelligence-based document review method that can perform automated, efficient and precise review, improve the ability to handle official documents, ensure the correctness, standardization and legality of the content of official documents, reduce labor costs and improve work efficiency.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A document review method based on artificial intelligence includes the following steps: S1: collecting Chinese corpus to construct training data; S2: constructing training data based on the Chinese corpus collected in step S1, and calling an online model to generate corpus to further construct training data; S3: scoring data and correcting low-scoring data to improve data quality; S4: establishing a data quality improvement framework, and adjusting and optimizing the framework; S5: fine-tuning the model; S6: evaluating and testing the model merged after fine-tuning in step S5; S7: displaying the review results in the system.

[0006] As an optimization, in step S1, when collecting Chinese corpus, first determine the authoritative and official data source, then select the corresponding data collection method for collection, and finally pre-process the collected original corpus.

[0007] As an optimization, in step S2, when further constructing training data, first call the online model to generate a large number of erroneous sentence corpora to supplement the error types not covered by the existing corpus, then generate error analysis of erroneous sentences for all corpora, and generate correct sentences, and finally generate error types of erroneous sentences according to the error correction field, and collect or generate correct sentences.

[0008] As an optimization, in step S3, when improving data quality, first establish and improve the scoring rules, then score typical data cases, give scoring reasons and optimization suggestions, and then use an external strong business model to check and score the data quality by adjusting the prompt, and make corrections to low-scoring data, and finally use typical data cases to verify and evaluate the results.

[0009] As an optimization, in step S4, during the specific implementation, firstly, the error type concepts, error correction knowledge, and quality standards are systematically sorted out; then, the actor-critic framework is constructed based on the data generation agent and the scoring agent to form a data generation and evaluation model; then, the model is preliminarily tested to verify the quality and feasibility of the data, and the framework is adjusted and optimized; finally, typical scenarios and edge scenarios are sorted out, and the data pipeline is started.

[0010] As an optimization, in step S5, an error correction model and an embedding model are constructed respectively; the error correction model is mainly used to identify and correct incorrect sentences, irregular use of numbers and punctuation marks, and common errors in official document writing; while the embedding model is used to identify and correct errors in professional terminology in official documents.

[0011] As an optimization, in step S5, the error correction model and the embedding model are fine-tuned respectively; when fine-tuning the error correction model, a large model with a suitable parameter scale is selected as the base model of the error correction model, and a large model with a parameter size of about 10B can be selected; then the large model is fine-tuned using the LoRA fine-tuning technology to improve its adaptability to the official document review task; when fine-tuning the embedding model, the BGE-M3 model is selected as the base model of the embedding model, and the BGE-M3 model is fine-tuned using the deepspeed framework, so that the embedding model can better adapt to the professional terminology error review task in the official document; model merging: the LM-Cocktail tool can be used to merge the fine-tuned model and the base model to improve the performance on the target task and maintain the performance on other tasks.

[0012] As an optimization, in step S6, the evaluation test includes the following steps: A1: Prepare test cases: Prepare test cases for the error correction model and the embedding model; A2: Evaluate the error correction model: The evaluation of the error correction model focuses on the overall recognition and correction capabilities of different types of errors. The evaluation indicators are mainly accuracy and recall. In order to analyze the performance of the model more carefully, the accuracy and recall are calculated for different types of errors respectively, so as to identify which aspects of the model need further optimization; A3: Evaluate the embedding model: The evaluation of the embedding model focuses on the performance of the model in semantic understanding and text similarity evaluation. In addition to accuracy and recall, the evaluation indicators also need to focus on MRR, NDCG, MAP and other indicators; A4: Actual official document test: Simulate the actual scenario of official document review, review real-life official document samples, and evaluate the model's effective recognition and correction capabilities for potential errors.

[0013] As an optimization, in step S7, the display scheme is as follows: the overall page displays all historical audit records, and uses different colors to distinguish between documents with successful audits and documents with unsuccessful audits; the detailed page displays the detailed content and audit results of the document, and at the same time marks the corresponding erroneous content in the original document with bright colors; convenient functions such as quick search or conditional filtering of documents, and exporting audit results as reports or other forms are added; the system records user error correction feedback as an important basis for continuous optimization of the model.

[0014] The present invention has the following advantages:

[0015] (1) Improve audit accuracy: By constructing a corpus containing a variety of common faulty sentences and error types and fine-tuning the model, this patent enables the official document audit system to more accurately identify and correct errors, reducing human omissions and subjective misjudgments;

[0016] (2) Improve audit efficiency: By combining the large model with the small model, this method can realize a fast and automated audit process, reducing the time and labor required for manual audit, especially when processing a large number of documents, which can better reflect the improvement of efficiency;

[0017] (3) Intelligent audit experience: This system can provide intelligent audit services, adapt to the different characteristics of official documents through fine-tuning technology, make the audit process more intelligent and personalized, and improve user experience;

[0018] (4) Ensure audit quality: By establishing scoring criteria and a high-quality data framework, this method not only improves the quality of training data, but also ensures the reliability of audit results, providing users with high-quality audit services;

[0019] (5) Wide applicability: The AI-based document review system can be applied to various governments and enterprises and institutions, whether small organizations or large institutions, and can achieve standardization and automation of the review process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is a flowchart of an official document review method based on artificial intelligence. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "upper, lower" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description. Unless otherwise stated, these directional words do not indicate or imply that the devices or components referred to must have a specific direction or be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present invention; the directional words "inside and outside" refer to the inside and outside relative to the contours of each component itself.

[0022] like Figure 1 As shown, an official document review method based on artificial intelligence includes the following steps: S1: collecting Chinese corpus to construct training data; S2: constructing training data based on the Chinese corpus collected in step S1, and calling the online model to generate corpus to further construct training data; S3: scoring the data and correcting the low-scoring data to improve the data quality; S4: establishing a data quality improvement framework, and adjusting and optimizing the framework; S5: fine-tuning the model; S6: evaluating and testing the merged model after fine-tuning in step S5; S7: displaying the review results in the system.

[0023] In this embodiment, in step S1, when collecting Chinese corpus, first determine the authoritative and official data source, then select the corresponding data collection method for collection, and finally pre-process the collected original corpus.

[0024] Specifically, when determining the data source, you can obtain Chinese corpus from various channels such as open source websites, competition websites, books, journal articles, and social media. The corpus includes but is not limited to text correction, standardized use of numbers and punctuation, precautions for official document writing, and standard terminology. In order to ensure the quality and diversity of the corpus, select authoritative and official information sources.

[0025] Specifically, when collecting data, choose an appropriate data collection method, including but not limited to crawler technology, database, and character recognition, to ensure that the collected data is diverse and representative to avoid model overfitting or bias.

[0026] Specifically, data preprocessing includes removing non-text information, unifying formats, desensitizing data, removing sensitive words and illegal words, and processing a lot of noise information in the collected original Chinese corpus.

[0027] In this embodiment, in step S2, when further constructing training data, first call the online model to generate a large number of erroneous sentence corpora to supplement the error types not covered by the existing corpora, then generate error analysis of erroneous sentences for all corpora, and generate correct sentences, and finally generate error types of erroneous sentences according to the error correction field, and collect or generate correct sentences.

[0028] Specifically, the detailed steps are as follows: B1: Supplementary corpus: Call online models such as GPT-4 to generate a large number of erroneous sentence corpora, supplement the error types not covered by the existing corpus, and ensure that various types of data such as incorrect sentence errors, errors in the use of numbers and punctuation marks, and common official document writing errors meet the quantity requirements; B2: Generate error correction fields: Generate error analysis of incorrect sentences for the collected corpus and the supplementary corpus, and generate correct sentences. The use of numbers and punctuation marks can refer to the provisions of national standards; B3: Generate error types: Generate error types of incorrect sentences according to the error correction fields, among which the errors of various types of incorrect sentence errors The types should be further refined, so that a typical training data entry should include an erroneous sentence, error analysis, error type and the corresponding correct sentence; B4: Collect or generate correct sentences: In addition to erroneous sentences, it is also necessary to collect or construct a large number of correct sentences as benchmark data for the model to learn the correct way of expression. Correct sentences can be extracted from standardized texts or texts proofread by professional reviewers; B5: Construct embedded data: Construct embedded data using professional terms in official documents, which are specifically divided into positive data and negative data. Positive data is obtained by expanding sentences with standard terms, and negative data is obtained by expanding sentences with corresponding non-standard terms.

[0029] In this embodiment, in step S3, when improving data quality, first establish and improve the scoring rules, then score typical data cases, give scoring reasons and optimization suggestions, and then use the external strong business model to check and score the data quality by adjusting the prompt, and make corrections to low-scoring data, and finally use typical data cases to verify and evaluate the results.

[0030] Specifically, improving data quality includes the following detailed steps: C1: Establishing and improving scoring rules: Establishing data quality scoring rules, combining the results of some data quality checks, and improving the scoring rules during the sorting and checking process; C2: Scoring typical data: Scoring typical data cases according to the improved rules, giving scoring reasons, and finally giving optimization suggestions for result verification; C3: Checking, correcting and scoring data quality: Using external strong business models to check, correct and score the remaining data, you can adjust the prompt multiple times to correct the effect of the model output, and complete the checking, correction and scoring of data quality, specifically: Check the error type, correct sentence, and error description corresponding to each piece of data, score it according to the updated scoring rules, and give scoring reasons. For data with scores below a certain value, give corrections to the error type, correct text, and error description. C4: Result verification and evaluation: Use typical data cases to verify and evaluate the results, including whether the case is correct, whether the case errors are common, and whether all error types are covered, so that the data quality meets the requirements.

[0031] In this embodiment, in step S4, during the specific implementation, firstly, the error type concepts, error correction knowledge, and quality standards are systematically sorted out; then, an actor-critic framework is constructed based on the data generation agent and the scoring agent to form a data generation and evaluation model; then, the model is preliminarily tested to verify the quality and feasibility of the data, and the framework is adjusted and optimized; finally, typical scenarios and edge scenarios are sorted out, and the data pipeline is started.

[0032] Specifically, on the basis of improving data quality, a high-quality data framework is established, so that when adding corpus, this framework can be directly used to improve data quality. The specific steps are as follows: D1: Re-sorting out the concepts of error types: sort out the existing classification concepts, clarify the definitions and specific circumstances of each error type, so that errors can be accurately classified in the subsequent processing process to ensure the accuracy and completeness of the data; D2: Error correction knowledge sorting: compile a detailed error correction knowledge base, including common grammatical errors and official document errors, error analysis and correct writing methods. This knowledge base provides a basis for subsequent data error correction; D3: Quality standard sorting: formulate a complete data quality evaluation system, Including requirements for grammatical semantics, language expression, and standardization, which will help maintain consistent evaluation standards during data processing; D4: Framework implementation and preliminary testing: Build an actor-critic framework based on the data generation agent and the scoring agent to form a data generation and evaluation model. In the preliminary testing phase, verify the quality and feasibility of the data by simulating real scenarios. According to the test results, adjust and optimize the framework to ensure the quality and diversity of the data; D5: Start the data pipeline: Sort out typical scenarios and edge scenarios, use these scenarios as the basis for data generation, ensure that the data covers various situations, start the data pipeline, and generate high-quality data sets.

[0033] In this embodiment, in step S5, an error correction model and an embedding model are constructed respectively; the error correction model is mainly used to identify and correct incorrect sentences, irregular use of numbers and punctuation marks, and common errors in official document writing; while the embedding model is used to identify and correct errors in professional terminology in official documents.

[0034] In this embodiment, in step S5, the error correction model and the embedding model are fine-tuned respectively; when fine-tuning the error correction model, a large model with a suitable parameter scale is selected as the base model of the error correction model, and a large model with a parameter size of about 10B can be selected; then the large model is fine-tuned using the LoRA fine-tuning technology to improve its adaptability to the official document review task; when fine-tuning the embedding model, the BGE-M3 model is selected as the base model of the embedding model, and the BGE-M3 model is fine-tuned using the deepspeed framework, so that the embedding model can better adapt to the professional terminology error review task in the official document; model merging: the LM-Cocktail tool can be used to merge the fine-tuned model and the base model to improve the performance on the target task and maintain the performance on other tasks.

[0035] In this embodiment, in step S6, the evaluation test includes the following steps: A1: preparing test cases: preparing test cases for the error correction model and the embedding model; A2: evaluating the error correction model: the evaluation of the error correction model focuses on the overall recognition and correction capabilities of different types of errors. The evaluation indicators are mainly accuracy and recall. In order to analyze the performance of the model more carefully, the accuracy and recall are calculated for different types of errors respectively, so as to identify which aspects of the model need further optimization; A3: evaluating the embedding model: the evaluation of the embedding model focuses on the performance of the model in semantic understanding and text similarity evaluation. In addition to accuracy and recall, the evaluation indicators also need to focus on MRR, NDCG, MAP and other indicators; A4: actual official document test: simulating the actual scenario of official document review, reviewing real-life official document samples, and evaluating the model's effective recognition and correction capabilities for potential errors.

[0036] In this embodiment, in step S7, the display scheme is: the overall page displays all historical audit records, and distinguishes between documents with successful audit and documents with unsuccessful audit with different colors; the detailed page displays the detailed content and audit results of the document, and at the same time marks the corresponding erroneous content in the original document with bright colors; adds convenient functions such as quick search or conditional screening of documents, and exporting audit results as reports or other forms; the system records user correction feedback as an important basis for continuous optimization of the model.

[0037] In this solution, the system uses artificial intelligence technology to automatically review official documents, including intelligent review of the content, grammar, terminology, etc. of official documents to ensure that the official documents comply with relevant standards and legal and regulatory requirements. By using the method and system of this patent, the efficiency and accuracy of official document review can be improved, the workload of manual review can be reduced, and missed or misreviewed situations can be avoided. At the same time, the standardization and legality of the content of official documents can be ensured, and the quality of official documents can be improved.

[0038] Finally, it should be noted that those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A document review method based on artificial intelligence, characterized in that: The following steps are involved: S1: Collect Chinese corpus to construct training data; S2: construct training data based on the Chinese corpus collected in step S1, and call the online model to generate corpus to further construct training data; S3: Scoring data and correcting low-scoring data to improve data quality; S4: Establish a data quality improvement framework and adjust and optimize the framework; S5: Model fine-tuning; S6: Perform evaluation test on the model merged after fine-tuning in step S5; S7: Display the audit results in the system.

2. According to the method of claim 1, the method is characterized in that: In step S1, when collecting Chinese corpus, first determine the authoritative and official data source, then select the corresponding data collection method for collection, and finally pre-process the collected original corpus.

3. According to the artificial intelligence-based document review method of claim 2, it is characterized in that: In step S2, when further constructing training data, first call the online model to generate a large number of erroneous sentence corpora to supplement the error types not covered by the existing corpora, then generate error analysis of erroneous sentences for all corpora, and generate correct sentences, and finally generate error types of erroneous sentences according to the error correction field, and collect or generate correct sentences.

4. According to the artificial intelligence-based document review method of claim 3, it is characterized in that: In step S3, when improving data quality, first establish and improve the scoring rules, then score typical data cases, give scoring reasons and optimization suggestions, and then use an external strong business model to check and score the data quality by adjusting the prompt, and make corrections to low-scoring data. Finally, use typical data cases to verify and evaluate the results.

5. According to the artificial intelligence-based document review method of claim 4, it is characterized in that: In step S4, during the specific implementation, firstly, the concepts of error types, error correction knowledge, and quality standards are systematically sorted out; then, the actor-critic framework is constructed based on the data generation agent and the scoring agent to form a data generation and evaluation model; then, the model is preliminarily tested to verify the quality and feasibility of the data, and the framework is adjusted and optimized; finally, typical scenarios and edge scenarios are sorted out, and the data pipeline is started.

6. The method for reviewing official documents based on artificial intelligence according to claim 5, characterized in that: In step S5, an error correction model and an embedding model are constructed respectively; the error correction model is mainly used to identify and correct incorrect sentences, irregular use of numbers and punctuation marks, and common errors in official document writing; while the embedding model is used to identify and correct errors in professional terminology in official documents.

7. The method for reviewing official documents based on artificial intelligence according to claim 6, characterized in that: In step S5, the error correction model and the embedding model are fine-tuned respectively; When fine-tuning the error correction model, a large model with a suitable parameter scale is selected as the base model of the error correction model. A large model with a parameter scale of about 10B can be selected. Then, the LoRA fine-tuning technology is used to fine-tune the large model to improve its adaptability to the document review task. When fine-tuning the embedding model, the BGE-M3 model is selected as the base model of the embedding model, and the BGE-M3 model is fine-tuned using the deepspeed framework, so that the embedding model can better adapt to the task of reviewing professional terminology errors in official documents; Model merging: You can use the LM-Cocktail tool to merge the fine-tuned model with the base model to improve the performance on the target task and maintain the performance on other tasks.

8. The method for reviewing official documents based on artificial intelligence according to claim 7, characterized in that: In step S6, performing the evaluation test includes the following steps: A1: Prepare test cases: Prepare test cases for the error correction model and the embedding model; A2: Evaluate the error correction model: The evaluation of the error correction model focuses on the overall ability to identify and correct different types of errors. The evaluation indicators are mainly accuracy and recall. In order to analyze the performance of the model more carefully, the accuracy and recall are calculated for different types of errors to identify which aspects of the model need further optimization. A3: Evaluate the embedding model: The evaluation of the embedding model focuses on the performance of the model in semantic understanding and text similarity evaluation. In addition to accuracy and recall, other evaluation indicators include MRR, NDCG, MAP, etc. A4: Actual official document test: simulate the actual scenario of official document review, review real-life official document samples, and evaluate the model's ability to effectively identify and correct potential errors.

9. The method for reviewing official documents based on artificial intelligence according to claim 8, characterized in that: In step S7, the display scheme is: the whole page displays all historical audit records, and the documents with successful audit and unsuccessful audit are distinguished by different colors; The detailed page displays the detailed content and review results of the document, and highlights the corresponding errors in the original document; add quick search or conditional filtering of documents, Convenient functions such as exporting audit results into reports or other forms; the system records user correction feedback, As an important basis for continuous optimization of the model.

Images