Document data processing method under large model scene

Abstract

The invention discloses an official document data processing method in a large model scene, and relates to the technical field of natural language processing, and the method comprises the steps: automatically collecting official document data from a multi-source channel, and carrying out format conversion and preprocessing optimization to obtain a data set; performing intelligent sampling on the data set by adopting a dynamic sampling algorithm, performing intelligent review on the sampled data, and making a cleaning strategy template library according to a review result; calling a cleaning strategy template library through a strategy configuration and execution engine to perform data cleaning, and executing data deduplication and similarity management; evaluating and reviewing the cleaned data; executing a set cleaning strategy and de-duplication on a data set in a large scale by utilizing a distributed computing framework; scientifically dividing the data set subjected to cleaning and duplicate removal, and converting the data set into an efficient storage format; the whole cleaning process is monitored in real time, and key steps and performance indexes in the data cleaning process are comprehensively recorded. The document data processing efficiency and response speed can be improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for processing official document data in a large model scenario. Background Art

[0002] With the development of artificial intelligence technology, especially the rise of large language models, the demand for high-quality datasets is increasing.

[0003] As an important information resource, official document data has been widely used in government agencies, enterprises, institutions and other fields. As an important carrier of formal communication, the data quality of official documents directly affects the learning effect of the model and the final application performance. However, during the collection, storage and transmission process, official document data often contains a large amount of noise, redundancy and irregular information. The original official document data often contains typos, inconsistent formats, sensitive information leakage and other problems. A systematic cleaning process is urgently needed to improve data quality. Summary of the invention

[0004] In response to the needs and shortcomings of current technological development, the present invention provides a method for processing official document data in a large model scenario, aiming to improve the quality of official document data and ensure the accuracy of model training and reasoning. It is suitable for big data processing needs of governments, enterprises and other institutions in scenarios such as automated office, document analysis, compliance review, and official document writing.

[0005] The document data processing method in a large model scenario of the present invention adopts the following technical solutions to solve the above technical problems:

[0006] A method for processing official document data in a large model scenario comprises the following steps:

[0007] S1. Source data collection and preprocessing: Use web crawler technology to automatically collect official document data from multiple source channels, convert the format of the collected official document data and perform preprocessing optimization to obtain a data set;

[0008] S2, Intelligent Sampling, Problem Mining and Strategy Formulation: Use dynamic sampling algorithms based on data diversity and complexity to intelligently sample data sets, review sampled data through trained machine learning models and manual review, and formulate corresponding cleaning strategy template libraries based on the review results;

[0009] S3, Data cleaning and intelligent deduplication: Based on the sampled data and its review results, the policy configuration and execution engine calls the cleaning policy template library to perform data cleaning, and perform data deduplication and similarity management;

[0010] S4. Document data quality review and verification: Conduct a comprehensive assessment and business rule review of the cleaned document data to ensure data quality;

[0011] S5. Data segmentation and formatted storage: The cleaned and deduplicated data set is divided according to the sampling ratio to train and evaluate the machine learning model. At the same time, the cleaned and deduplicated data set is converted into an efficient storage format and archived in a designated cloud storage system or data warehouse in an orderly manner.

[0012] S6. Cleaning process monitoring and continuous optimization closed loop: Through the real-time monitoring system, the key steps and performance indicators in the data cleaning process are fully recorded, and the monitoring data is analyzed using machine learning algorithms. The data change trends and cleaning requirements are independently learned to achieve dynamic adjustment and intelligent iterative updates of the cleaning strategy.

[0013] Optionally, the step S1 specifically includes the following operations:

[0014] S1.1. Use web crawler technology to automatically collect official document data from government public websites and internal information systems, convert official document data in different formats into a unified structured or semi-structured format, then remove irrelevant information in the official document data, extract valid content, and standardize the text;

[0015] S1.2. Encrypt and transmit the standardized official document data, and implement a breakpoint-resume transmission mechanism during the transmission process;

[0016] S1.3. Design and develop a data parsing module based on the specific format and structural characteristics of official document data to identify and collect key information in the official document data. At the same time, apply regular expressions or pattern recognition technology to automatically identify and standardize the specified information in the official document data.

[0017] S1.4. Use natural language processing technology to automatically identify and annotate key entities in official document data, and understand the relationships between key entities and the semantic roles they play in official document data through semantic role annotation and dependency syntax analysis;

[0018] S1.5. Introduce a semantic understanding model based on deep learning to conduct in-depth analysis of official document data, understand complex sentences and professional terms, and accurately identify the type of official document based on the semantic analysis results combined with predefined rules or trained classification models;

[0019] S1.6. Conduct preliminary statistical analysis on the collected official document data, construct a multi-dimensional data overview chart, and intuitively display the multi-dimensional characteristics of the official document data;

[0020] S1.7. Evaluate the content integrity of official document data, and identify and mark incomplete documents that may be caused by document damage or format conversion through text summarization technology and paragraph structure analysis.

[0021] Further optionally, the step S2 specifically includes the following operations:

[0022] S2.1. Adopt a dynamic sampling algorithm based on data diversity and complexity. First, conduct a preliminary analysis of the data set to identify the heterogeneous characteristics of the official document data in the data set, calculate the number of samples of different types of official documents, and then apply stratified random sampling technology to extract representative samples of different types of official documents from the data set according to the proportion of the calculated results;

[0023] S2.2. Pre-train a machine learning model that can identify document data and mark potential problems, automatically review the representative samples extracted through the trained machine learning model, and randomly select representative samples after automatic review for manual review;

[0024] S2.3. Classify the potential problems identified and marked after manual review, and develop a corresponding cleaning strategy template library based on the problem classification results.

[0025] Preferably, step S2.2 is performed to randomly select representative samples after automatic review for manual review, compare the results of manual review with those of automatic review, find representative samples corresponding to different review results, and feed the found representative samples and their manual review results back to the machine learning model;

[0026] The above operations are executed repeatedly to achieve iterative optimization of the machine learning model.

[0027] Preferably, step S2.3 is performed, and the formulated cleaning strategy template library covers relevant strategies of multiple data processing dimensions, specifically including:

[0028] Paragraph optimization and format standardization: Use natural language processing algorithms to identify and adjust improper paragraph divisions, and intelligently remove redundant blank characters;

[0029] Content integration: Using document parsing technology, table content, image content and reference content are automatically converted into a unified Markdown format;

[0030] Date processing and filtering: Use composite pattern matching and context analysis to identify and standardize date formats, while screening and excluding records with illogical dates;

[0031] Sensitive information desensitization: Pattern recognition technology based on deep learning can identify and process variant and obscure sensitive information;

[0032] Encoding repair and invisible character removal: Use the tool libraries ftfy and charset_normalizer in the Python ecosystem to automatically repair incorrect encodings and remove invisible characters.

[0033] Further optionally, the step S3 specifically includes the following operations:

[0034] S3.1. Design a policy configuration and execution engine, which has a built-in extensible cleaning policy template library and supports policy execution order optimization;

[0035] S3.2. Based on the review results of the data set and the sampled data in the data set, the policy configuration and execution engine intelligently selects an adaptive cleaning strategy from the cleaning strategy template library. At the same time, the user fine-tunes the policy parameters of the selected cleaning strategy or adds cleaning logic under a specified scenario according to actual needs to meet the specified data processing requirements;

[0036] S3.3, based on the selected cleaning strategy or the cleaning strategy parameters fine-tuned, the strategy configuration and execution engine simulates and analyzes the influence relationship between the cleaning steps in the cleaning strategy, generates an optimal strategy execution sequence, and cleans the corresponding data in the data set based on the generated optimal strategy execution sequence;

[0037] S3.4. Generate a unique hash identifier for each piece of official document data, use the hash table to quickly locate suspected duplicates, and in the event of hash conflicts, use open addressing or chain addressing to ensure high efficiency and low false alarm rate; for records of hash conflicts, perform secondary confirmation based on cosine similarity and Jaccard coefficient to ensure the accuracy of deduplication decisions.

[0038] Preferably, after each cleaning iteration, the strategy parameters and cleaning logic are dynamically adjusted according to the deduplication results and cleaning effect feedback, and the overall quality and consistency of the data set are continuously improved through continuous optimization cycles.

[0039] Further optionally, the step S4 specifically includes the following operations:

[0040] Use statistical methods to comprehensively evaluate the cleaned sample data to ensure that the data integrity, consistency and accuracy meet the preset quality thresholds;

[0041] Combined with business rules, the preset review strategy is used to automatically review whether the data meets the requirements of the specified business scenario, and combined with manual review, the accuracy of the automatic review is verified.

[0042] Further optionally, when executing step S5, first refer to the intelligent sampling process of step S2 to find all samples corresponding to the extracted representative samples, then call the cleaning strategy template library corresponding to the representative samples for all samples, and finally use the distributed computing framework to execute the corresponding cleaning strategy template library on different samples of the data set, and perform data deduplication and similarity management to achieve streaming cleaning of all official document data in the data set.

[0043] Further optionally, when executing step S7, an automatic archiving mechanism for data cleaning reports is established to record the detailed process and results of each cleaning to facilitate subsequent audits and reviews;

[0044] Build a strategic knowledge base, accumulate and integrate data cleaning experience and strategies, form a mechanism for knowledge sharing and inheritance, and promote the long-term efficient operation and sustainable development of the data cleaning process.

[0045] Compared with the prior art, the document data processing method in a large model scenario of the present invention has the following beneficial effects:

[0046] 1. The present invention covers data collection, preprocessing, intelligent sampling and problem mining, formulation of cleaning strategies, cleaning of sampled data, deduplication, quality review, and cleaning, processing and storage of full-scale data. The entire process includes monitoring and continuous optimization, aiming to improve the quality of official document data and ensure the accuracy of model training and reasoning. It is suitable for big data processing needs of government, enterprises and other institutions in scenarios such as automated office, document analysis, compliance review, and official document writing.

[0047] 2. The present invention significantly reduces manual operations through automated data collection, preprocessing, cleaning, deduplication and other steps, improves the efficiency and response speed of official document data processing, and reduces labor costs; through encrypted transmission, intelligent monitoring, sensitive information processing and other measures, ensures the security and compliance of data during transmission, storage and processing, and maintains the accuracy and integrity of data through high-quality cleaning strategies; through in-depth analysis, intelligent classification, standardized storage and other means, it makes official document data easier to analyze and use, provides a reliable data foundation for government decision support, policy research, information service innovation, etc., and enhances the business value and insight of data; designs dynamic sampling strategies, adaptive selection cleaning strategies and other mechanisms, so that the technical solution can flexibly respond to official document data of different types, sources and formats, and adapt to the ever-changing data processing needs and environment; through continuous monitoring, feedback, and optimization closed loops, ensure the continuous improvement and self-evolution of the data processing process, adapt to the future growth of data volume and the increase in complexity, and maintain the long-term efficiency and stability of the data processing system;

[0048] 3. The present invention can quickly locate and solve quality problems in large-scale data sets through intelligent data capture, dynamic sampling strategies and optimized cleaning processes, reducing the need for human intervention and greatly improving the speed and accuracy of data processing; it can flexibly adapt to various complex document data formats and contents, improving the versatility and applicability of the system; the full monitoring and optimization closed loop of the cleaning process ensures the continuity and stability of data processing, while providing transparency of data processing status for easy management and decision support; by providing complete, accurate and timely data support, it promotes the scientific decision-making of governments and enterprises, as well as the intelligent upgrading of public services; through feedback loops and continuous iterations, it can optimize cleaning strategies and execution logic according to changes in data characteristics and business needs, ensuring long-term data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Attached Figure 1 It is a flow chart of the method of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0050] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0051] Embodiment 1:

[0052] Reference Figure 1 This embodiment proposes a method for processing official document data in a large model scenario, which includes the following steps:

[0053] S1. Source data collection and preprocessing to obtain data sets.

[0054] S1.1. Use web crawler technology to automatically collect official document data from multiple source channels such as government public websites, e-government platforms, and internal document management systems, convert official document data in different formats into a unified structured or semi-structured format, then remove irrelevant information in the official document data, extract valid content, and standardize the text;

[0055] S1.2. Encrypt and transmit the standardized official document data, and implement a breakpoint-resume transmission mechanism during the transmission process to ensure that the data is complete and without omission;

[0056] S1.3. Design and develop data parsing modules based on the specific format and structural characteristics of official document data to identify and collect key information in official document data, and apply regular expressions or pattern recognition technology to automatically identify and standardize specified information in official document data, including but not limited to extracting pure text content from multiple document formats such as HTML, PDF, Word, etc., while retaining the original structured information such as title, text, attachments, etc.;

[0057] S1.4. Use natural language processing technology, such as named entity recognition (NER), to automatically identify and annotate key entities in official document data. Through semantic role labeling and dependency syntax analysis, understand the relationship between key entities and the semantic roles they play in official document data. Key entities such as issuing agencies, time, document numbers, etc. provide precise field guidance for subsequent cleaning;

[0058] S1.5. Introduce a semantic understanding model based on deep learning to conduct in-depth analysis of official document data, understand complex sentences and professional terms, and accurately identify the type of official document based on the semantic analysis results combined with predefined rules or trained classification models;

[0059] S1.6. Conduct preliminary statistical analysis on the collected official document data, construct a multi-dimensional data overview chart, and intuitively display the multi-dimensional characteristics of the official document data, including but not limited to the distribution frequency of various official documents, the average and median length range, and the heat map display of the field missing rate. In addition, special attention should be paid to the consistency of the time format, and an adaptive algorithm should be used to identify and standardize the date expression. Whether it is "XXXX year XX month X day", "YYYY / MM / DD" and other formats, it can be automatically converted to the unified ISO date standard to improve the standardization level of data.

[0060] S1.7. Evaluate the content integrity of official document data, identify and mark incomplete official documents that may be caused by document damage or format conversion through text summarization technology and paragraph structure analysis, and provide targets for subsequent manual review or intelligent text completion. The optimization work at this stage lays a solid foundation for subsequent data cleaning and ensures the efficiency and accuracy of the data cleaning process.

[0061] S2, intelligent sampling, problem mining and strategy formulation.

[0062] A dynamic sampling algorithm based on data diversity and complexity is used to intelligently sample data sets. The sampled data is reviewed through trained machine learning models and manual review. A corresponding cleaning strategy template library is formulated based on the review results, aiming to efficiently locate and solve data quality challenges and ensure the targeted and effective cleaning work.

[0063] The step S2 involved specifically includes the following operations:

[0064] S2.1. Adopt a dynamic sampling algorithm based on data diversity and complexity. First, conduct a preliminary analysis of the data set to identify the heterogeneous characteristics of the official document data in the data set (such as the proportion of different types of official documents, length distribution, field completeness, etc.), calculate the number of samples of different types of official documents, and then apply stratified random sampling technology to consider the balance of multiple dimensions such as document type, length, and publication source. Extract representative samples of different types of official documents from the data set according to the proportion of the calculated results to ensure that the extracted samples fully cover the diversity of the data set;

[0065] S2.2. Pre-train a machine learning model that can identify document data and mark potential problems, automatically review the representative samples extracted through the trained machine learning model, and randomly select representative samples after automatic review for manual review;

[0066] Compare the results of manual review and automatic review, find representative samples corresponding to different review results, and feed the representative samples and their manual review results back to the machine learning model;

[0067] By repeating the above operations, iterative optimization of the machine learning model can be achieved;

[0068] S2.3. Classify the potential problems identified and marked after manual review, and develop a corresponding cleaning strategy template library based on the problem classification results.

[0069] Potential issues identified and flagged include: irregular paragraph divisions, leakage of sensitive information, and identification of irrelevant characters.

[0070] The developed cleaning strategy template library covers relevant strategies in multiple data processing dimensions, including:

[0071] Paragraph optimization and format standardization: The natural language processing algorithm is used to identify and adjust improper paragraph divisions, while redundant blank characters are intelligently removed. The adaptive learning algorithm is used to continuously update the irrelevant character feature library, effectively filter out the messy information unique to the network environment, and maintain the purity of the official document content;

[0072] Content integration: Using document parsing technology, table content, image content and reference content are automatically converted into a unified Markdown format;

[0073] Date processing and filtering: Use composite pattern matching and context analysis to identify and standardize date formats, while screening and excluding records with illogical dates;

[0074] Sensitive information desensitization: Pattern recognition technology based on deep learning can identify and process variant and obscure sensitive information;

[0075] Encoding repair and invisible character removal: Use the tool libraries ftfy and charset_normalizer in the Python ecosystem to automatically repair incorrect encodings and remove invisible characters.

[0076] S3, data cleaning and intelligent deduplication: Based on the sampled data and its review results, the policy configuration and execution engine calls the cleaning policy template library to perform data cleaning, and perform data deduplication and similarity management.

[0077] The step S3 involved specifically includes the following operations:

[0078] S3.1. Design a policy configuration and execution engine, which has a built-in extensible cleaning policy template library and supports policy execution order optimization;

[0079] S3.2. Based on the review results of the data set and the sampled data in the data set, the policy configuration and execution engine intelligently selects an adaptive cleaning strategy from the cleaning strategy template library. At the same time, the user fine-tunes the policy parameters of the selected cleaning strategy or adds cleaning logic under a specified scenario according to actual needs to meet the specified data processing requirements;

[0080] S3.3, based on the selected cleaning strategy or the cleaning strategy parameters fine-tuned, the strategy configuration and execution engine simulates and analyzes the influence relationship between the cleaning steps in the cleaning strategy, generates an optimal strategy execution sequence, and cleans the corresponding data in the data set based on the generated optimal strategy execution sequence;

[0081] S3.4. Generate a unique hash identifier for each document data, use the hash table to quickly locate suspected duplicates, and use open addressing or chain addressing to ensure high efficiency and low false positive rate in the event of hash conflicts. For records of hash conflicts, perform secondary confirmation based on cosine similarity and Jaccard coefficient to ensure the accuracy of deduplication decisions.

[0082] S3.5. After each cleaning iteration, dynamically adjust the strategy parameters and cleaning logic based on the deduplication results and cleaning effect feedback, and continuously improve the overall quality and consistency of the data set through continuous optimization cycles.

[0083] S4. Review and verification of official document data quality: Conduct a comprehensive assessment and business rule review of the cleaned official document data to ensure data quality.

[0084] The step S4 involved specifically includes the following operations:

[0085] Use statistical methods to comprehensively evaluate the cleaned sample data to ensure that the data integrity, consistency and accuracy meet the preset quality thresholds;

[0086] Combined with business rules, preset review strategies are used to automatically review whether the data meets the requirements of specified business scenarios, such as outlier detection and compliance verification. Then, combined with manual review, the accuracy of the automatic review is verified to ensure data quality.

[0087] During manual review, on the one hand, the sampled data that has been cleaned is carefully reviewed to verify the effectiveness of the cleaning strategy; on the other hand, special attention is paid to the data that has been intervened by the strategy, and targeted sampling inspections are conducted to ensure that no new errors are introduced during the cleaning process, while verifying the universality and robustness of the cleaning logic. If new problems are found during the review or the standards are not met, a backtracking mechanism will be implemented to feedback to the data cleaning execution process, and the cleaning strategy will be dynamically adjusted and the cleaning process will be re-executed until the data quality meets the standards.

[0088] S5. Full-scale data stream cleaning: Utilize a distributed computing framework to implement established cleaning strategies and deduplication on a large scale for the official document data in the data set.

[0089] When executing step S5, first refer to the intelligent sampling process of step S2 to find all samples corresponding to the extracted representative samples, then call the cleaning strategy template library corresponding to the representative samples for all samples, and finally use the distributed computing framework to execute the corresponding cleaning strategy template library on different samples of the data set, and perform data deduplication and similarity management to achieve streaming cleaning of all official document data in the data set.

[0090] It should be noted that the quality of sampling data during the cleaning process needs to be monitored in real time to ensure the consistency and stability of the cleaning effect.

[0091] S6. Data segmentation and formatted storage: The cleaned and deduplicated data set is divided according to the sampling ratio for training and evaluation of machine learning models. At the same time, the cleaned and deduplicated data set is converted into an efficient storage format, such as Parquet, and then archived in an orderly manner to the designated cloud storage system or data warehouse, providing a solid data foundation for subsequent data mining and machine learning projects.

[0092] S7. Cleaning process monitoring and continuous optimization closed loop: Through the real-time monitoring system, the key steps and performance indicators in the data cleaning process are fully recorded, including processing time, data quality changes, resource usage, etc.

[0093] Based on the collected monitoring data and business feedback, the monitoring data is analyzed using machine learning algorithms, and data change trends and cleaning requirements are learned autonomously to achieve dynamic adjustment and intelligent iterative update of cleaning strategies.

[0094] Establish an automated filing mechanism for data cleaning reports to record the detailed process and results of each cleaning process, facilitating subsequent audits and reviews;

[0095] Build a strategic knowledge base, accumulate and integrate data cleaning experience and strategies, form a mechanism for knowledge sharing and inheritance, and promote the long-term efficient operation and sustainable development of the data cleaning process.

[0096] In summary, the document data processing method in a large model scenario of the present invention can significantly improve the efficiency and quality of document data cleaning, reduce the cost and error of manual intervention, enhance the purity and representativeness of model training data, and at the same time, ensure the legality and compliance of data processing, laying a solid data foundation for the application of large models in the field of document processing.

[0097] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for processing official document data in a large model scenario, characterized in that: The steps include: S1. Source data collection and preprocessing: Use web crawler technology to automatically collect official document data from multiple source channels, convert the format of the collected official document data and perform preprocessing optimization to obtain a data set; S2, Intelligent Sampling, Problem Mining and Strategy Formulation: Use dynamic sampling algorithms based on data diversity and complexity to intelligently sample data sets, review sampled data through trained machine learning models and manual review, and formulate corresponding cleaning strategy template libraries based on the review results; S3, Data cleaning and intelligent deduplication: Based on the sampled data and its review results, the policy configuration and execution engine calls the cleaning policy template library to perform data cleaning, and perform data deduplication and similarity management; S4. Document data quality review and verification: Conduct a comprehensive assessment and business rule review of the cleaned document data to ensure data quality; S5. Full-scale data stream cleaning: Using a distributed computing framework, the established cleaning strategy and deduplication are implemented on a large scale for the official document data in the data set; S6. Data segmentation and formatted storage: The cleaned and deduplicated data set is divided according to the sampling ratio to train and evaluate the machine learning model. At the same time, the cleaned and deduplicated data set is converted into an efficient storage format and archived in a designated cloud storage system or data warehouse in an orderly manner. S7. Cleaning process monitoring and continuous optimization closed loop: Through the real-time monitoring system, the key steps and performance indicators in the data cleaning process are fully recorded, and the monitoring data is analyzed using machine learning algorithms. The data change trends and cleaning requirements are independently learned to achieve dynamic adjustment and intelligent iterative updates of the cleaning strategy.

2. The document data processing method in a large model scenario according to claim 1 is characterized in that: The step S1 specifically includes the following operations: S1.

1. Use web crawler technology to automatically collect official document data from government public websites and internal information systems, convert official document data in different formats into a unified structured or semi-structured format, then remove irrelevant information in the official document data, extract valid content, and standardize the text; S1.

2. Encrypt and transmit the standardized official document data, and implement a breakpoint-resume transmission mechanism during the transmission process; S1.

3. Design and develop a data parsing module based on the specific format and structural characteristics of official document data to identify and collect key information in the official document data. At the same time, apply regular expressions or pattern recognition technology to automatically identify and standardize the specified information in the official document data. S1.

4. Use natural language processing technology to automatically identify and annotate key entities in official document data, and understand the relationships between key entities and the semantic roles they play in official document data through semantic role annotation and dependency syntax analysis; S1.

5. Introduce a semantic understanding model based on deep learning to conduct in-depth analysis of official document data, understand complex sentences and professional terms, and accurately identify the type of official document based on the semantic analysis results combined with predefined rules or trained classification models; S1.

6. Conduct preliminary statistical analysis on the collected official document data, construct a multi-dimensional data overview chart, and intuitively display the multi-dimensional characteristics of the official document data; S1.

7. Evaluate the content integrity of official document data, and identify and mark incomplete documents that may be caused by document damage or format conversion through text summarization technology and paragraph structure analysis.

3. The document data processing method in a large model scenario according to claim 2 is characterized in that: The step S2 specifically includes the following operations: S2.

1. Adopt a dynamic sampling algorithm based on data diversity and complexity. First, conduct a preliminary analysis of the data set to identify the heterogeneous characteristics of the official document data in the data set, calculate the number of samples of different types of official documents, and then apply stratified random sampling technology to extract representative samples of different types of official documents from the data set according to the proportion of the calculated results; S2.

2. Pre-train a machine learning model that can identify document data and mark potential problems, automatically review the representative samples extracted through the trained machine learning model, and randomly select representative samples after automatic review for manual review; S2.

3. Classify the potential problems identified and marked after manual review, and develop a corresponding cleaning strategy template library based on the problem classification results.

4. The document data processing method in a large model scenario according to claim 3 is characterized in that: Execute step S2.2, randomly select representative samples after automatic review for manual review, compare the results of manual review with those of automatic review, find representative samples corresponding to different review results, and feed the found representative samples and their manual review results back to the machine learning model; The above operations are executed repeatedly to achieve iterative optimization of the machine learning model.

5. The method for processing official document data in a large model scenario according to claim 3 is characterized in that: Execute step S2.3, and the developed cleaning strategy template library covers relevant strategies of multiple data processing dimensions, including: Paragraph optimization and format standardization: Use natural language processing algorithms to identify and adjust improper paragraph divisions, and intelligently remove redundant blank characters; Content integration: Using document parsing technology, table content, image content and reference content are automatically converted into a unified Markdown format; Date processing and filtering: Use composite pattern matching and context analysis to identify and standardize date formats, while screening and excluding records with illogical dates; Sensitive information desensitization: Pattern recognition technology based on deep learning can identify and process variant and obscure sensitive information; Encoding repair and invisible character removal: Use the tool libraries ftfy and charset_normalizer in the Python ecosystem to automatically repair incorrect encodings and remove invisible characters.

6. The document data processing method in a large model scenario according to claim 3 is characterized in that: The step S3 specifically includes the following operations: S3.

1. Design a policy configuration and execution engine, which has a built-in extensible cleaning policy template library and supports policy execution order optimization; S3.

2. Based on the review results of the data set and the sampled data in the data set, the policy configuration and execution engine intelligently selects an adaptive cleaning strategy from the cleaning strategy template library. At the same time, the user fine-tunes the policy parameters of the selected cleaning strategy or adds cleaning logic under a specified scenario according to actual needs to meet the specified data processing requirements; S3.3, based on the selected cleaning strategy or the cleaning strategy parameters fine-tuned, the strategy configuration and execution engine simulates and analyzes the influence relationship between the cleaning steps in the cleaning strategy, generates an optimal strategy execution sequence, and cleans the corresponding data in the data set based on the generated optimal strategy execution sequence; S3.

4. Generate a unique hash identifier for each piece of official document data, use the hash table to quickly locate suspected duplicates, and in the event of hash conflicts, use open addressing or chain addressing to ensure high efficiency and low false alarm rate; for records of hash conflicts, perform secondary confirmation based on cosine similarity and Jaccard coefficient to ensure the accuracy of deduplication decisions.

7. The method for processing official document data in a large model scenario according to claim 6, characterized in that: After each cleaning iteration, the strategy parameters and cleaning logic are dynamically adjusted based on the deduplication results and cleaning effect feedback, and the overall quality and consistency of the data set are continuously improved through continuous optimization cycles.

8. The method for processing official document data in a large model scenario according to claim 3 is characterized in that: The step S4 specifically includes the following operations: Use statistical methods to comprehensively evaluate the cleaned sample data to ensure that the data integrity, consistency and accuracy meet the preset quality thresholds; Combined with business rules, the preset review strategy is used to automatically review whether the data meets the requirements of the specified business scenario, and combined with manual review, the accuracy of the automatic review is verified.

9. The document data processing method in a large model scenario according to claim 3 is characterized in that: When executing step S5, first refer to the intelligent sampling process of step S2 to find all samples corresponding to the extracted representative samples, then call the cleaning strategy template library corresponding to the representative samples for all samples, and finally use the distributed computing framework to execute the corresponding cleaning strategy template library on different samples of the data set, and perform data deduplication and similarity management to achieve streaming cleaning of all official document data in the data set.

10. The document data processing method in a large model scenario according to claim 1 is characterized in that: When executing step S7, an automatic archiving mechanism for data cleaning reports is established to record the detailed process and results of each cleaning, so as to facilitate subsequent audits and reviews; Build a strategic knowledge base, accumulate and integrate data cleaning experience and strategies, form a mechanism for knowledge sharing and inheritance, and promote the long-term efficient operation and sustainable development of the data cleaning process.

Images