Large model corpus processing method and system

Abstract

The invention relates to a large model corpus processing method and system. The method comprises the following steps: performing data acquisition on language data; value data identification is carried out on the collected data, and a valuable area is reserved; extracting the reserved area in the form of a paragraph document; the paragraph document is converted into a unified text format, and invalid content deletion, format and grammar error correction, content value screening and format unified processing are carried out; performing duplicate removal processing on the data; carrying out overall quality inspection on the data subjected to the duplicate removal processing so as to optimize the processing process; performing feature extraction on the data through a deep learning model to obtain corpus features; according to corpus features, semantic information is extracted through a semantic understanding model and used for training and application of a large model; and optimizing the data processing process according to the training and application results and feedback information of the large model. Compared with the prior art, the method has the advantages that unified processing of various large model training data is realized, and the accuracy and efficiency of data processing are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of corpus processing, and in particular to a large model corpus processing method and system. Background Art

[0002] At present, the general artificial intelligence large model represented by ChatGPT has set off a technological wave around the world, promoting progress in many fields such as natural language processing, image recognition, and intelligent recommendation. And from the scale of the large model training corpus, it can be seen that the corpus scale is growing rapidly, following the development of large models, showing a trend of synchronous expansion with SOTA models, and the growth and update of various types of corpora in the future will be closely related to the demand for improving various model capabilities.

[0003] The current development of corpus has the following problems:

[0004] Scarcity of high-quality corpus: Currently, high-quality corpus data is relatively scarce, and large model companies conceal training corpus information. There is a lack of high-quality open source corpus endorsed by large models, which makes it difficult to meet the needs of large model training. With the consumption of data sets, the problem of "high-quality language data" may be exhausted in the future, which restricts the development of large model data research.

[0005] Diversity: Open source corpora are mainly English texts, lacking large-scale high-quality, multimodal Chinese and English training corpora covering different fields and topics.

[0006] Difficult to process: Real-world documents have complex layout structures and diverse layout types, which increases the difficulty of corpus processing. Advanced document parsing technology is needed to accurately identify elements in the document and understand their logical relationships.

[0007] Computing power and cost: Large model training requires huge computing resources and energy consumption, which places higher demands on computing power. At the same time, the high cost of training, including computing chip costs, server costs, electricity costs, etc., increases the economic burden on enterprises.

[0008] Security ethics and legal issues: Large models may learn biases and false information when generating content, leading to misleading and inaccurate results. Therefore, data security, such as personal privacy protection and national security, needs to be considered, and in line with China's mainstream values, privacy and data security issues also need to be paid attention to. How to ensure the fairness, transparency and security of large models is an urgent problem to be solved.

[0009] In summary, the current development of artificial intelligence, especially large model technology, is in a period of rapid growth, and corpus processing, as a key link, is of great significance. However, current technology faces challenges such as scarcity of high-quality corpus, difficulty in processing, high computing power and cost, as well as ethical and legal issues. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a large model corpus processing method and system to achieve unified processing of data from multiple sources and improve data processing efficiency and accuracy.

[0011] The purpose of the present invention can be achieved by the following technical solutions:

[0012] A large model corpus processing method comprises the following steps:

[0013] Acquisition steps: data collection of language data;

[0014] Identification step: Identify the valuable data in the collected data, retain the valuable areas, and remove the worthless areas;

[0015] Extraction step: extract the retained area after recognition into the form of paragraph document;

[0016] Cleaning step: convert the extracted paragraph documents into a unified text format, delete invalid content, correct format and grammatical errors, screen content value, and unify the format;

[0017] Fuzzy deduplication step: deduplication the data processed in the cleaning step;

[0018] Data quality inspection step: perform overall quality inspection on the deduplicated data and provide feedback on defect information, so as to iteratively optimize the cleaning step based on the defect information;

[0019] Feature extraction step: extracting features from the data that has passed the data quality inspection step through a deep learning model to obtain corpus features;

[0020] Semantic understanding step: according to the corpus features extracted in the feature extraction step, semantic information is extracted through the trained semantic understanding model for the training and application of the large model;

[0021] Result optimization step: Optimize the data processing process according to the results and feedback information of the training and application of the big model based on the semantic information.

[0022] Furthermore, the data collected in the acquisition step includes web page data, PDF data and e-book format data, and the web page data is acquired from the website by a crawler program;

[0023] The extraction step uses a general extraction method to extract the web page data, PDF data and e-book format data processed by the identification step, and the obtained paragraph document contains text, pictures, videos and tables.

[0024] Further, in the cleaning step, the invalid content deletion includes removing empty documents, meaningless texts and coded error contents;

[0025] The format and grammatical error correction includes correcting md / latex format errors, punctuation usage errors, and grammatical spelling errors;

[0026] The content value screening includes screening out unsafe content and advertising content, and screening out professional knowledge articles and high-value data. The unsafe content includes pornographic, political, toxic and gambling content. The advertising content includes advertisements and product lists. The professional knowledge articles include popular science articles, professional blogs and encyclopedia entries. The high-value data is screened according to the number of readings and likes.

[0027] The format unification processing includes encoding and paragraph division processing of the cleaned data according to a unified format standard.

[0028] Furthermore, the deduplication process in the fuzzy deduplication step includes:

[0029] Deduplication algorithms are used to deduplicate paragraphs within documents, paragraphs between documents, and the entire document after the cleaning step, and then all data is uniformly deduplicated using minhash.

[0030] If the data size exceeds the optimal data size of the deduplication algorithm, the data processed in the cleaning step is divided into a Chinese part and an English part and deduplication is performed separately.

[0031] Furthermore, the defect information fed back in the data quality inspection step includes defect definition, discovery method, scope of involvement, drop issues and retained data quality issues.

[0032] Furthermore, the corpus features obtained in the feature extraction step include vocabulary, phrases, sentence structures and grammatical relations.

[0033] Furthermore, the processing process of the semantic understanding step is specifically as follows:

[0034] Performing semantic analysis on the corpus features extracted in the feature extraction step using natural language processing techniques such as named entity recognition, sentiment analysis, and semantic role labeling;

[0035] The semantic information is extracted based on the semantic analysis results through the trained semantic understanding model, and the semantic information includes the relationship between entities and the tendency of emotions.

[0036] Furthermore, the process of optimizing the data processing process in the result optimization step is specifically as follows:

[0037] Optimize the processing rules and parameters in the cleaning step and the deep learning model structure and training method in the feature extraction step.

[0038] Furthermore, the method also includes classifying the data collected in the acquisition step according to content genre, language type and / or text format, and performing data processing in the identification step and the extraction step on different categories of data respectively.

[0039] The present invention also provides a large model corpus processing system based on the large model corpus processing method as described above, comprising:

[0040] An acquisition module, used for collecting language data;

[0041] The recognition module is used to identify the valuable data in the collected data, retain the valuable areas, and remove the worthless areas;

[0042] An extraction module is used to extract the region retained after recognition into the form of a paragraph document;

[0043] The cleaning module is used to convert the extracted paragraph documents into a unified text format, delete invalid content, correct format and grammatical errors, screen content value, and unify the format;

[0044] The fuzzy deduplication module is used to perform deduplication processing on the data processed by the cleaning module;

[0045] A data quality inspection module is used to perform overall quality inspection on the data after deduplication processing and feedback defect information, so as to iteratively optimize the cleaning module according to the defect information;

[0046] A feature extraction module is used to extract features of the data that passes through the data quality inspection module through a deep learning model to obtain corpus features;

[0047] A semantic understanding module is used to extract semantic information based on the corpus features extracted by the feature extraction module through a trained semantic understanding model for use in the training and application of a large model;

[0048] The result optimization module is used to optimize the data processing process according to the results and feedback information of the training and application of the large model based on the semantic information.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] (1) The present invention realizes the unified processing of various types of large model training data through a method and system for efficient large model corpus processing, improves the accuracy and efficiency of data processing, helps to solve the current challenges such as scarcity of high-quality corpus, difficulty in processing, high computing power and cost, and ethical and legal issues, and promotes the development of the field of artificial intelligence.

[0051] (2) Through innovative algorithms and technical means, the present invention effectively solves the data quality problem in large model corpus processing. From the screening of invalid documents, the correction of format and grammatical errors, to the identification and removal of duplicate content, this series of cleaning steps significantly improves the integrity and consistency of the data set. High-quality data is the basis for training efficient AI models. Therefore, the application of the present invention can directly enhance the performance of large models, enabling them to more accurately understand and generate natural language text.

[0052] (3) The present invention decouples the development of data extraction and cleaning modules from the data delivery task, enabling developers to deeply iterate and optimize module quality. This design allows developers to focus on improving module performance and fixing potential errors without being affected by the urgency of data delivery. This not only improves the efficiency and accuracy of data cleaning, but also lays a solid foundation for subsequent module upgrades and maintenance.

[0053] (4) In addition to the basic data cleaning function, the present invention further explores the potential value of data through algorithms such as semantic clustering and quality classification. Through intelligent classification and screening, the present invention can identify high-quality and valuable data content, providing strong support for subsequent data analysis and application. In the field of AI, these data can be used to train more accurate professional models, such as medical diagnosis models, legal document analysis models, etc., thereby promoting the development of intelligent applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a process flow of a large model corpus processing method provided in an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a quality inspection process of a large model corpus processing method provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a first process of quality inspection problem processing of a large model corpus processing method provided in an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a second process for processing quality inspection problems in a large model corpus processing method provided in an embodiment of the present invention;

[0058] Figure 5Schematic diagram of main modules of a large model corpus processing system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0062] Example 1

[0063] This embodiment provides a large model corpus processing method, the core idea of ​​which is to optimize the large model corpus processing process through innovative algorithms and technical means. Specifically, it transforms the original common way of checking and processing data according to the source of a single large model data set into checking and processing as a whole, so as to more effectively grasp the coverage and impact of the problem, improve the input-output ratio on the corresponding problem, and form effective accumulation.

[0064] like Figure 1 As shown, the specific steps include:

[0065] Acquisition step S1: collecting language data;

[0066] The collected data includes web page data, PDF data and e-book format data;

[0067] Among them, web page data is obtained from the website by a crawler program, a general crawler is performed on the Chinese Internet, and an in-site crawler is performed on static and dynamic websites to obtain CC's warc format, self-acquired and purchased html format data, and the data is stored in html form as raw material for subsequent processing.

[0068] The electronic book format data may be in the form of an ePUB file.

[0069] Identification step S2: identifying valuable data on the collected data, retaining valuable areas and removing worthless areas;

[0070] The data in HTML format is processed and stored in slim_html format to prepare for the subsequent extraction stage.

[0071] Extraction step S3: extracting the identified and retained area into the form of a paragraph document;

[0072] Specifically, the regions retained after recognition are extracted into the form of paragraph documents, each of which is a sequence of paragraphs;

[0073] Paragraph documents may contain a variety of content such as text, pictures / videos, tables, etc.

[0074] Data from all sources are extracted through the general extraction method after the extraction step. For data from a specific source, if it does not meet the general extraction requirements, the extraction logic is customized.

[0075] Cleaning step S4: converting the extracted paragraph documents into a unified text format, deleting invalid content, correcting format and grammatical errors, screening content value and unifying the format;

[0076] The specific processes of the above processing methods are as follows:

[0077] Convert from original format to unified format: For example, convert content in formats such as html, text, pdf, and epub into markdown text to ensure that all input data has a unified format, which facilitates subsequent unified filtering and cleaning of all data, including format rules, heuristic cleaning rules, content security filtering, etc. Repair or filter out incomplete data to improve the data integrity of the data set.

[0078] Invalid content deletion: including removing empty documents, meaningless text, encoding errors, etc. to ensure the validity of the data.

[0079] Format and grammatical error correction: Check and correct MD / latex format errors, punctuation errors, grammatical spelling errors, etc. to improve the quality of the data.

[0080] Content value screening: Screen out unsafe content (such as pornography, politics, toxicity, gambling, etc.) and advertising content (partial advertisements, entire advertisements, product lists), and at the same time screen out professional knowledge articles (popular science articles, professional blogs, encyclopedia entries, etc.) and high-value data (screened by number of readings, number of likes, etc.), thereby improving the data value of the data set.

[0081] Format unification: The cleaned corpus data is encoded and segmented according to a unified format standard to facilitate subsequent feature extraction and index construction. At the end of this step, multiple fields such as "ContentStyle", "Language", and "Format" will be added to the data.

[0082] Fuzzy deduplication step S5: deduplication processing is performed on the data processed in the cleaning step;

[0083] The specific deduplication process is as follows: Deduplication algorithms such as Local Sensitive Hash (LSH) are used to deduplicate paragraphs within a document, paragraphs between documents, and the entire document to ensure data uniqueness. All data is uniformly deduplicated using minhash. This ensures that data in the entire corpus is unique, avoiding the impact of duplicate data on model training.

[0084] Preferably, considering the size of the data, the data can be divided into two parts, Chinese and English, and duplicate removal is performed separately.

[0085] The remaining data after filtering is clean_data, and data with text quality below the standard line and unsafe content should be excluded.

[0086] Data quality inspection step S6: Perform overall quality inspection on the deduplicated data and feedback defect information, so as to iteratively optimize the cleaning steps according to the defect information;

[0087] The defect information provided by the overall quality inspection includes defect definition, discovery method, scope, drop issues, and retained data quality issues.

[0088] Optionally, the defects are collected and sent to the product manager for evaluation and priority sorting, and the content of the next version upgrade is sorted out to form a product requirement document (divided according to the new pipeline module), and reviewed with the development team to determine the feature functions and data scope of the next version. R&D personnel continue to iterate solutions to various problems based on the results of the requirement review. This process requires that all data that needs to be updated be produced once in each production cycle, including the promotion of full data and the separate processing of new data. The quality and efficiency of data processing can be improved by continuously optimizing and iterating the cleaning process and completing semantic tasks according to requirements.

[0089] The quality inspection process can be referred to Figure 2 As shown, the process of iterating solutions to various problems can be referred to Figure 3 and Figure 4 shown.

[0090] Feature extraction step S7: extracting features from the data that has passed the data quality inspection step through a deep learning model to obtain corpus features;

[0091] Deep learning models can select appropriate models according to task requirements, such as BERT, Transformer, etc.; these models can automatically learn and extract key features such as vocabulary, phrases, sentence structure, grammatical relations, etc. from corpus data.

[0092] Preferably, a specific feature extraction model is designed and trained for a specific task to obtain more targeted corpus features. At the same time, the accuracy and efficiency of feature extraction are improved by optimizing the algorithm and model parameters.

[0093] Semantic understanding step S8: According to the corpus features extracted in the feature extraction step, semantic information is extracted through the trained semantic understanding model for the training and application of the large model;

[0094] This is equivalent to applying natural language processing technology to enhance the ability to understand the semantics of the corpus, so that the model can more accurately grasp the deep meaning of the corpus data. The specific process can be divided into the following parts:

[0095] Selection of semantic analysis technology: Use natural language processing technologies such as named entity recognition, sentiment analysis, and semantic role labeling to conduct in-depth semantic analysis of corpus data.

[0096] Semantic model training: According to specific task requirements, semantic understanding models are trained. These models can extract semantic information from corpus data, such as the relationship between entities, sentiment tendencies, etc.

[0097] Application of semantic information: Apply the extracted semantic information to the training and application of large models to improve the model's ability to understand and utilize corpus data.

[0098] Result optimization step S9: According to the results and feedback information of the training and application of the big model based on semantic information, the data processing process is optimized, that is, the accuracy and reliability of the processing results are improved through algorithm optimization and feedback mechanism, so as to continuously improve and optimize the big model corpus processing process.

[0099] The specific process of optimizing the data processing process is as follows:

[0100] Optimize the processing rules and parameters in the cleaning step and the deep learning model structure and training methods in the feature extraction step.

[0101] This is equivalent to the following process:

[0102] Algorithm optimization: Optimize the algorithms in the corpus processing flow based on the processing results and feedback information. For example, adjust the rules and parameters of data cleaning, improve the structure and training methods of feature extraction models, etc.

[0103] Establishment of feedback mechanism: Establish an effective feedback mechanism to collect evaluations and suggestions of downstream tasks on corpus processing results. Improve the accuracy and reliability of processing results through continuous iteration and optimization.

[0104] Continuous monitoring and improvement: Continuously monitor and evaluate the entire corpus processing process to promptly identify and resolve problems. At the same time, keep paying attention to and researching new technologies and methods, and constantly explore and optimize new ideas and methods for large model corpus processing processes.

[0105] Preferably, the method further comprises classifying the data collected in the acquisition step according to content genre, language type and / or text format, and performing data processing of the identification step and the extraction step respectively on the data of different categories.

[0106] Specifically, content genre: from the perspective of cleaning, the content genre is classified. Currently, it includes the following categories:

[0107]

[0108] The expected behavior is to assign large model data to book, paper, pedia, forum, and qna categories first, and to artical otherwise. misc indicates a genre that cannot be processed. If a misc genre is encountered, it should be dropped during data processing.

[0109] Language:

[0110] Language classification needs to be identified during the data cleaning phase;

[0111] Articles can be single-language or mixed-language. For single-language documents, two- to three-letter abbreviations are used to mark the language according to the ISO_639 standard (actually in accordance with the results of fasttext 176). For documents that are judged to be mixed-language, "mix" is used;

[0112] In the early data processing stage, you can only focus on "en" and "zh" data. However, you need to correctly separate the mix and other language types.

[0113] Text format:

[0114] Currently, the text formats transmitted by the upstream system of large model data are only md, text, and latex. For other formats, they need to be correctly diverted;

[0115] Judgment priority during data processing:

[0116] i. If any item except the formula is latex, then it is latex

[0117] ii. If latex is not hit, and any item except the formula is md, then it is md

[0118] iii. Otherwise, it is text.

[0119] The data processing flow after classification can be summarized as follows:

[0120] Data cleaning: including converting from the original format to json format, assigning a globally unique ID to each piece of data, formatting the document content (such as converting html, text, pdf, etc. to markdown), performing ngram filtering, removing advertising words, deduplicating paragraphs, and performing statistics and deduplication on the cleaned content.

[0121] Semantic clustering and quality classification: Perform semantic clustering and quality classification on the cleaned data to further screen and optimize data quality.

[0122] Global deduplication: Use mature algorithms such as LSH to perform global paragraph deduplication to ensure data uniqueness.

[0123] Generate Tokens: Call the corresponding tokenizer to generate tokens according to different model requirements.

[0124] The data quality inspection and result optimization process after data processing can be summarized as follows:

[0125] Processing speed: By optimizing data processing algorithms and improving quality inspection efficiency, the problem of mismatch between quality inspection manpower and data set size can be solved and processing speed can be improved.

[0126] Corpus analysis accuracy: Solve the problem of inaccurate corpus analysis by improving data quality, optimizing model selection and parameter adjustment, and enhancing semantic understanding capabilities.

[0127] Security issues: Strengthen data security and personal privacy protection in the data processing process to ensure compliance with legal and ethical requirements.

[0128] like Figure 5 As shown, this embodiment provides a large model corpus processing system based on the above large model corpus processing method, including:

[0129] An acquisition module, used for collecting language data;

[0130] The recognition module is used to identify the valuable data in the collected data, retain the valuable areas, and remove the worthless areas;

[0131] An extraction module is used to extract the region retained after recognition into the form of a paragraph document;

[0132] The cleaning module is used to convert the extracted paragraph documents into a unified text format, delete invalid content, correct format and grammatical errors, screen content value, and unify the format;

[0133] The fuzzy deduplication module is used to perform deduplication processing on the data processed by the cleaning module;

[0134] The data quality inspection module is used to perform overall quality inspection on the deduplicated data and provide feedback on defect information, so as to iteratively optimize the cleaning module based on the defect information;

[0135] The feature extraction module is used to extract features from the data that passes the data quality inspection module through a deep learning model to obtain corpus features;

[0136] The semantic understanding module is used to extract semantic information based on the corpus features extracted by the feature extraction module through the trained semantic understanding model for the training and application of large models;

[0137] The result optimization module is used to optimize the data processing process according to the results and feedback information of the training and application of the big model based on semantic information.

[0138] It should be noted that the specific content and beneficial effects of the system of the present application can be found in the above-mentioned method embodiment, which will not be repeated here.

[0139] Overall, this solution has achieved a number of important innovations in the field of data cleaning and processing. These innovations are significantly reflected in algorithm design, technical means and system architecture, showing distinct advantages over existing technologies:

[0140] Module decoupling and independent iteration

[0141] First, the present invention decouples the development of data extraction and cleaning modules from data delivery tasks, enabling developers to deeply iterate and optimize module quality. This design allows developers to focus on improving module performance and fixing potential errors without being affected by the urgency of data delivery. This not only improves the efficiency and accuracy of data cleaning, but also lays a solid foundation for subsequent module upgrades and maintenance.

[0142] General data processing flow

[0143] Secondly, the present invention abandons the cleaning scheme centered on specific source data and adopts a more general data processing process. This change breaks the limitations of data sources, so that whether it is web page data, document data or knowledge data, it can be effectively processed through a unified cleaning process. This versatility not only improves the efficiency of data cleaning, but also enhances the consistency of cleaning results, providing more reliable data support for subsequent data analysis and application.

[0144] Automation and reduction of manpower input

[0145] In addition, the present invention significantly reduces manpower input in the data processing process and achieves a higher degree of automation. By introducing advanced algorithms and technical means, such as natural language processing and machine learning, the present invention can automatically perform complex cleaning tasks, such as removing invalid content, correcting format errors, identifying duplicate content, etc. This not only reduces labor costs, but also significantly reduces the probability of human errors and improves the accuracy and reliability of data processing.

[0146] Technical means and algorithm innovation

[0147] At the technical implementation level, the present invention makes full use of advanced natural language processing and machine learning algorithms to achieve deep cleaning and intelligent classification of data. By executing a refined set of cleaning rules and an efficient semantic clustering algorithm, the present invention can accurately identify and remove noise and redundant information in the data, retaining high-quality, valuable content. At the same time, through the application of quality classification algorithms, the present invention can further screen out data sets that meet specific needs, providing precise support for subsequent data analysis and applications.

[0148] System architecture and modular design

[0149] Finally, the present invention adopts a modular design concept in the system architecture. By dividing the data processing flow into different modules and interacting through a unified interface, the present invention realizes independent operation and upgrading between modules. This modular design not only reduces the complexity and maintenance cost of the system, but also improves the scalability and flexibility of the system. At the same time, through the application of a unified data processing framework and a global deduplication mechanism, the present invention further improves the efficiency and quality of data processing, providing a strong guarantee for large-scale data processing tasks.

[0150] The practicality of the present invention in the field of large model data and AI is mainly reflected in the following aspects:

[0151] 1. Improve data quality and enhance model performance

[0152] Through innovative algorithms and technical means, the present invention effectively solves the data quality problem in large model corpus processing. From the screening of invalid documents, the correction of format and grammatical errors, to the identification and removal of duplicate content, this series of cleaning steps significantly improves the integrity and consistency of the data set. High-quality data is the basis for training efficient AI models. Therefore, the application of the present invention can directly enhance the performance of large models, enabling them to understand and generate natural language text more accurately.

[0153] 2. Improve data processing efficiency and reduce labor costs

[0154] The present invention greatly reduces the manpower input in the data cleaning process through automated processing procedures. Automated cleaning not only reduces labor costs, but also significantly improves the speed and efficiency of data processing. This is especially important for large model training that requires processing massive amounts of data, because it can shorten the data processing cycle and accelerate model iteration and update, thereby applying AI technology to actual scenarios more quickly.

[0155] 3. Enhance data versatility and scalability

[0156] The universal data processing process adopted by the present invention breaks the limitation of data sources, so that data from different sources can be effectively processed through a unified cleaning process. This versatility not only improves the efficiency of data cleaning, but also enhances the diversity and coverage of data sets. In the process of large model training, diversified data sets can enhance the generalization ability of the model, making it better adapted to various practical application scenarios. At the same time, the modular system architecture makes the present invention have good scalability, and can easily integrate new cleaning modules or processing flows to cope with the ever-changing data processing needs.

[0157] 4. Support data value mining and intelligent application

[0158] In addition to the basic data cleaning function, the present invention further explores the potential value of data through algorithms such as semantic clustering and quality classification. Through intelligent classification and screening, the present invention can identify high-quality and valuable data content, providing strong support for subsequent data analysis and application. In the field of AI, these data can be used to train more accurate professional models, such as medical diagnosis models, legal document analysis models, etc., thereby promoting the development of intelligent applications.

[0159] In summary, the present invention has wide practicability in the field of large model data and AI. It not only improves data quality and model performance, but also reduces labor costs, enhances data versatility and scalability, and supports the development of data value mining and intelligent applications. These advantages make the present invention of great significance in promoting the advancement of AI technology and industrial application.

[0160] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A large model corpus processing method, characterized in that: The following steps are involved: Acquisition steps: data collection of language data; Identification step: Identify the valuable data in the collected data, retain the valuable areas, and remove the worthless areas; Extraction step: extract the retained area after recognition into the form of paragraph document; Cleaning step: convert the extracted paragraph documents into a unified text format, delete invalid content, correct format and grammatical errors, screen content value, and unify the format; Fuzzy deduplication step: deduplication the data processed in the cleaning step; Data quality inspection step: perform overall quality inspection on the deduplicated data and provide feedback on defect information, so as to iteratively optimize the cleaning step based on the defect information; Feature extraction step: extracting features from the data that has passed the data quality inspection step through a deep learning model to obtain corpus features; Semantic understanding step: according to the corpus features extracted in the feature extraction step, semantic information is extracted through the trained semantic understanding model for the training and application of the large model; Result optimization step: Optimize the data processing process according to the results and feedback information of the training and application of the big model based on the semantic information.

2. A large model corpus processing method according to claim 1, characterized in that: The data collected in the acquisition step includes web page data, PDF data and e-book format data, and the web page data is acquired from the website by a crawler program; The extraction step uses a general extraction method to extract the web page data, PDF data and e-book format data processed by the identification step, and the obtained paragraph document contains text, pictures, videos and tables.

3. A large model corpus processing method according to claim 1, characterized in that: In the cleaning step, the invalid content deletion includes removing empty documents, meaningless texts and coded error contents; The format and grammatical error correction includes correcting md / latex format errors, punctuation usage errors, and grammatical spelling errors; The content value screening includes screening out unsafe content and advertising content, and screening out professional knowledge articles and high-value data. The unsafe content includes pornographic, political, toxic and gambling content. The advertising content includes advertisements and product lists. The professional knowledge articles include popular science articles, professional blogs and encyclopedia entries. The high-value data is screened according to the number of readings and likes. The format unification processing includes encoding and paragraph division processing of the cleaned data according to a unified format standard.

4. A large model corpus processing method according to claim 1, characterized in that: The deduplication process in the fuzzy deduplication step includes: Deduplication algorithms are used to deduplicate paragraphs within documents, paragraphs between documents, and the entire document after the cleaning step, and then all data is uniformly deduplicated using minhash. If the data size exceeds the optimal data size of the deduplication algorithm, the data processed in the cleaning step is divided into a Chinese part and an English part and deduplication is performed separately.

5. A large model corpus processing method according to claim 1, characterized in that: The defect information fed back in the data quality inspection step includes defect definition, discovery method, scope, drop problem and retained data quality problem.

6. A large model corpus processing method according to claim 1, characterized in that: The corpus features obtained in the feature extraction step include vocabulary, phrases, sentence structures and grammatical relations.

7. A large model corpus processing method according to claim 1, characterized in that: The processing process of the semantic understanding step is specifically as follows: Performing semantic analysis on the corpus features extracted in the feature extraction step using natural language processing techniques such as named entity recognition, sentiment analysis, and semantic role labeling; The semantic information is extracted based on the semantic analysis results through the trained semantic understanding model, and the semantic information includes the relationship between entities and the tendency of emotions.

8. A large model corpus processing method according to claim 1, characterized in that: The process of optimizing the data processing process in the result optimization step is specifically as follows: Optimize the processing rules and parameters in the cleaning step and the deep learning model structure and training method in the feature extraction step.

9. A large model corpus processing method according to claim 1, characterized in that: The method further includes classifying the data collected in the acquisition step according to content genre, language type and / or text format, and performing the data processing of the identification step and the extraction step on the data of different categories respectively.

10. A large model corpus processing system based on a large model corpus processing method as claimed in any one of claims 1 to 9, characterized in that: include: An acquisition module, used for collecting language data; The recognition module is used to identify the valuable data in the collected data, retain the valuable areas, and remove the worthless areas; An extraction module is used to extract the region retained after recognition into the form of a paragraph document; The cleaning module is used to convert the extracted paragraph documents into a unified text format, delete invalid content, correct format and grammatical errors, screen content value, and unify the format; The fuzzy deduplication module is used to perform deduplication processing on the data processed by the cleaning module; A data quality inspection module is used to perform overall quality inspection on the data after deduplication processing and feedback defect information, so as to iteratively optimize the cleaning module according to the defect information; A feature extraction module is used to extract features of the data that passes through the data quality inspection module through a deep learning model to obtain corpus features; A semantic understanding module is used to extract semantic information based on the corpus features extracted by the feature extraction module through a trained semantic understanding model for use in the training and application of a large model; The result optimization module is used to optimize the data processing process according to the results and feedback information of the training and application of the large model based on the semantic information.

Images