Pre-training data set construction method, training method, equipment, medium and product

Abstract

The invention discloses a pre-training data set construction method, a training method, equipment, a medium and a product, and relates to the technical field of computer model training. The construction method comprises the following steps: acquiring original Chinese text data; cleaning the original Chinese text data to obtain a Chinese text corpus set; performing quality score labeling on part of texts in the Chinese text corpus set by using an open source large language model to obtain samples with labels; training a classifier by using the samples with the labels to obtain a high-quality classifier; and performing quality classification on the Chinese text corpus set by utilizing a high-quality classifier to obtain a high-quality Chinese text pre-training data set. According to the method, the overall quality of Chinese data is greatly improved, and then a data basis is provided for training of a high-performance Chinese large language model.

Description

Technical Field

[0001] The present invention relates to the field of computer model training technology, and in particular to a method for constructing a Chinese text pre-training data set, a large language model training method, equipment, storage medium and product. Background Art

[0002] The success of Large Language Models (LLMs) is mainly due to the availability of large, high-quality pre-trained corpora, which provide support for the basic knowledge and reasoning capabilities of large language models in a variety of tasks, from creative writing to complex problem solving. Among them, open source datasets such as The Pile and CommonCrawl have played an important role in promoting the development of LLMs, promoting collaboration, and establishing innovative benchmarks.

[0003] Existing researchers are more concerned about expanding high-quality data. Currently, the demand for pre-training data has exceeded 10 trillion tokens, highlighting two key directions in English pre-training: expanding the scale of data and improving data quality. Open source datasets have expanded rapidly, from collections such as The Pile (825GB) to larger datasets such as FineWeb (15TB), which draws heavily on the Common Crawl corpus. At the same time, the focus has shifted from rule-based filtering methods in early projects to model-driven methods, such as demonstrated by the FineWeb-Edu dataset.

[0004] Despite the rapid progress of English open source datasets, Chinese data is still severely underrepresented in the global web. Existing open source Chinese datasets (e.g., WuDao, SkyPile150B, and WanjuanV1) are limited in scale due to the scarcity of Chinese data sources on the Internet. In addition, there is limited research on improving the quality classification of Chinese web data, resulting in poor overall quality of Chinese data. These challenges have created huge obstacles to the development of high-performance Chinese large language models, highlighting the urgent need for more effective data filtering and quality classification methods. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a Chinese text pre-training dataset construction method, training method, device, medium and product to solve the problem of scarcity and poor overall quality of open source Chinese datasets.

[0006] In a first aspect, the present invention provides a method for constructing a Chinese text pre-training dataset, comprising: Get the original Chinese text data; Cleaning the original Chinese text data to obtain a Chinese text corpus; Using an open source large language model to perform quality scoring and annotation on part of the text in the Chinese text corpus to obtain labeled samples; Use labeled samples to train the classifier to obtain a high-quality classifier; The Chinese text corpus is quality classified using a high-quality classifier to obtain a high-quality Chinese text pre-training data set.

[0007] Furthermore, the original Chinese text data is cleaned to obtain a Chinese text corpus, including: Performing content security filtering on the original Chinese text data to filter out data containing unsafe content; Extract and clean the Chinese text data after content security filtering to delete invalid fields and obtain valid fields; De-duplicate the extracted and cleaned Chinese text data; Perform heuristic rule filtering on the deduplicated Chinese text data to eliminate outliers and excessively repeated content; The basic quality classifier is used to perform preliminary quality assessment on the Chinese text data after heuristic rule filtering to obtain the Chinese text corpus.

[0008] Furthermore, the DocParser parser is used to extract and clean the Chinese text data after the content security filtering process; The MinHash algorithm is used to deduplicate the extracted and cleaned Chinese text data.

[0009] Furthermore, the open source large language model selects the Qwen2-72B-Instruct model, and the Qwen2-72B-Instruct model performs quality scoring annotation on part of the text in the Chinese text corpus according to the quality scoring method in the FineWeb-edu dataset.

[0010] Furthermore, the classifier is obtained by adding a classification head on the basis of the BGE-M3 vector model.

[0011] Furthermore, when the classifier is trained, the embedding layer and the encoding layer of the BGE-M3 vector model are frozen, and only the newly added classification head is trained.

[0012] In a second aspect, the present invention further provides a large language model training method, comprising: Construct a high-quality Chinese text pre-training dataset using the Chinese text pre-training dataset construction method as described above; The large language model is trained and verified using the high-quality Chinese text pre-training dataset.

[0013] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program / instructions stored on the memory, wherein the processor executes the computer program / instructions to implement the steps in the Chinese text pre-training dataset construction method or the large language model training method as described above.

[0014] In a fourth aspect, the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps in the Chinese text pre-training dataset construction method or the large language model training method as described above.

[0015] In a fifth aspect, the present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the Chinese text pre-training dataset construction method or the large language model training method as described above.

[0016] The method for constructing a Chinese text pre-training data set provided by the present invention includes two stages: basic processing and high-quality processing. In the basic processing stage, the original Chinese text data is subjected to a preliminary quality assessment; in the high-quality processing stage, high-quality samples are first identified with the help of the Qwen2-72B-Instruct large-scale model, and then a high-quality classifier with 500 million parameters is obtained by training the high-quality samples. Finally, the high-quality classifier is used to efficiently filter the samples after the preliminary quality assessment of the billion-scale, and the final high-quality Chinese text pre-training data set is generated. The present invention greatly improves the overall quality of Chinese data, thereby providing a data basis for the training of high-performance Chinese large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a flow chart of a method for constructing a Chinese text pre-training dataset in an embodiment of the present invention; Figure 2 This is a structural diagram of the Roberta model in an embodiment of the present invention; Figure 3 Whether to freeze the F1 score of the backbone model during training in the embodiment of the present invention; Figure 4 is the F1 score of different learning rates during training in the embodiment of the present invention; wherein lr represents the learning rate; Figure 5is a comprehensive average score curve of different classifiers in the embodiment of the present invention, where B represents one billion; Figure 6 is the average score curve of the Chinese index of different classifiers in the embodiment of the present invention; wherein B represents one billion; Figure 7 This is a flow chart of a large language model training method in an embodiment of the present invention; Figure 8 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following is a clear and complete description of the technical solution in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Example 1 Figure 1 The flowchart of the method for constructing a Chinese text pre-training dataset provided by the present invention is shown. Figure 1 As shown, the method for constructing a Chinese text pre-training dataset includes the following steps: In step S1, original Chinese text data is obtained.

[0022] In this embodiment, the original Chinese text data covers a variety of Chinese corpus sources, including news, social media, and blogs, which enhances the coverage and representativeness of the dataset.

[0023] In step S2, the original Chinese text data is cleaned to obtain a Chinese text corpus (ie, the cci3.0 dataset).

[0024] In a specific embodiment of the present invention, the original Chinese text data is cleaned to obtain a Chinese text corpus, including: Step S2.1: Perform content security filtering on the original Chinese text data to filter out data containing unsafe content; Step S2.2: extracting and cleaning the Chinese text data after content security filtering to delete invalid fields and obtain valid fields; Step S2.3: performing deduplication processing on the extracted and cleaned Chinese text data; Step S2.4: Perform heuristic rule filtering on the deduplicated Chinese text data to eliminate outliers and excessively repeated content; Step S2.5: Use the basic quality classifier to perform preliminary quality assessment on the Chinese text data after heuristic rule filtering to obtain the Chinese text corpus, namely the CCI3.0 dataset.

[0025] In this embodiment, the original Chinese text data is subjected to content security filtering through keyword matching and classifier discrimination, and data from websites that may contain unsafe content is excluded, thereby ensuring that the data meets strict security standards.

[0026] Since the original Chinese text data comes from different sources of different institutions and has different data formats, the present invention uses DocParser to extract and clean the Chinese text data after content security filtering, delete invalid HTLM (Hyper Text Markup Language) tags, delete meaningless fields, and extract the content of specified fields.

[0027] DocParser is a document parsing component used for document content parsing. It supports 17 document formats including PDF, JPG, DOC, TXT, XLS, PPT, etc. It can parse document content, layout information, location coordinates, table structure and other contents.

[0028] In this embodiment, the MinHash algorithm is used to perform deduplication processing on the extracted and cleaned Chinese text data, ensuring the diversity of the data set and avoiding redundancy. The MinHash algorithm is an implementation method of LSH (local sensitive hashing), which is used to identify similar subsets in a data set. Therefore, it can identify near-duplicate documents in Chinese text data, and remove near-duplicate documents to achieve deduplication processing.

[0029] In the cleaning process (i.e., the basic process), the Chinese text data after deduplication is also filtered using heuristic rules to eliminate outliers and excessively repeated content. In this embodiment, the heuristic rules include filtering low-scoring documents, character string length filtering, filtering when the proportion of meaningless characters exceeds a threshold, filtering when the proportion of excessively repeated characters is high, and the like.

[0030] In this embodiment, the basic quality classifier is a quality classifier trained using the open source Chinese Web Text database.

[0031] The ChineseWeb Text database is a Chinese data set carefully selected from Common Crawl's massive web page data. It collects web page snapshots from 2021 to 2023, totaling 1.42TB of data. At the same time, each text in ChineseWeb Text is accompanied by a quantitative quality score, providing a reference standard for screening and use for preliminary quality assessment.

[0032] Steps S3 to S5 are the high-quality processing stage. In step S3, an open source large language model is used to perform quality scoring and annotation on part of the text in the Chinese text corpus obtained in step S2 to obtain labeled samples.

[0033] In a specific embodiment of the present invention, the open source large language model uses the Qwen2-72B-Instruct model, and the Qwen2-72B-Instruct model performs quality scoring annotation on part of the text in the Chinese text corpus obtained in step S2 according to the quality scoring method in FineWeb-edu, that is, the method in the FineWeb-edu data set is used to define high-quality samples. In this embodiment, the Chinese text corpus includes 1.5 billion samples, and the Qwen2-72B-Instruct model performs quality scoring annotation on 140,000 samples in the Chinese text corpus.

[0034] In step S4, the classifier is trained using labeled samples to obtain a high-quality classifier.

[0035] The classifier is trained using the 140,000 samples obtained in step S3 to obtain a high-quality classifier for high-quality Chinese educational content. The annotation capability of the large-scale Qwen2-72B-Instruct model (72B parameters) is distilled into a small-scale classifier (500 million parameters) suitable for large-scale data quality scoring, which can efficiently perform quality scoring or annotation on large-scale data at a lower cost.

[0036] In a specific embodiment of the present invention, the open source BGE-M3 vector model is extended and a classification head is added for regression output to obtain a classifier. When training the classifier, the original embedding layer and encoding layer are frozen, and only the newly added classification head is trained, and finally the classifier is converted into a binary classifier. The classifier is trained for multiple rounds and the learning rate is 3e -4 The classifier uses the Roberta model structure, such as Figure 2 As shown, C represents the category, T1, T2, T3, …, TN represent different input units, CLS (i.e. classification) represents the classification head, and Token1, Token2, Token3, …, TokenN represent different inputs.

[0037] When training the classifier, compare whether to freeze the backbone model part and the F1 score of the grid search of the learning rate on the same test set, such as Figure 3 and Figure 4 As shown. Figure 3 As shown in , the fewer the iterations, the higher the F1 score of the frozen backbone model part; the more the iterations, the higher the F1 score of the unfrozen backbone model part. Figure 4 As shown, the learning rate is 3e -4 The F1 score at is the best overall.

[0038] In step S5, a high-quality classifier is used to perform quality classification on the Chinese text corpus to obtain a high-quality Chinese text pre-training dataset (ie, the cci3.0-hq dataset).

[0039] The Chinese text corpus was quality classified using a high-quality classifier. The entire process took 410 A100 GPU hours, and the final dataset had a hard disk size of about 500G. It is currently the largest open source high-quality Chinese text pre-training dataset.

[0040] In order to evaluate the beneficial effect of the high-quality Chinese text pre-training dataset constructed by the present invention on training large language models, a dataset comparison experiment was designed. A 500 million parameter model trained from scratch was used to train 100 billion tokens, and a zero-sample setting evaluation was performed in 10 benchmark tests. This embodiment designs two types of dataset experiments: a mixed dataset experiment of Chinese and English codes and a pure Chinese dataset experiment. The pre-training model structure used in the experiment is shown in Table 1.

[0041] Table 1 Pre-training model parameter configuration Parameter English name Parameter Chinese name Specific value attention_dropout Attention deactivation ratio 0.0 bos_token_id Starting word ID 151849 eos_token_id End word ID 151850 hidden_act Hidden layer activation function SiLU——Sigmoid Linear Unit hidden_size Hidden layer dimensions 896 intermediate_size Middle layer dimension 2432 max_position_embeddings Maximum sequence length 4096 num_attention_heads Number of attention heads 14 num_hidden_layers Number of hidden layers 24 num_key_value_heads Number of key-value headers 2 pad_token_id Filler ID 151643 rms_norm_eps RMS normalized numerical stability parameter <![CDATA[1e -06 ]]> rope_theta Positional encoding scaling factor 10000 tie_word_embeddings Whether to bind the word embedding layers of the encoder and decoder True torch_dtype Pre-trained data types bfloat16——Brain Floating PointFormat (brain floating point 16 bits) vocab_size Vocabulary size 151851 The experimental results are shown in Tables 2 and 3. The experimental results show that the Chinese text pre-training dataset constructed by the present invention is significantly better than other Chinese datasets, such as SkyPile and WanjuanV1, in terms of comprehensive indicators and Chinese indicators, which demonstrates the effectiveness of the method of the present invention.

[0042] Table 2 Experimental results of Chinese-English code mixed dataset index skypile wanjuan-v1 cci3.0 cci3.0-hq ARC-C (Challenging Question Answering Benchmark) 0.270 0.277 0.265 0.269 ARC-E (Easy Answer Benchmark) 0.521 0.517 0.539 0.542 HellaSwag (common sense reasoning benchmark) 0.355 0.347 0.360 0.357 Winograd (common sense reasoning benchmark) 0.507 0.502 0.498 0.523 MMLU (Multi-Task Language Understanding Benchmark) 0.286 0.287 0.289 0.292 OBQA (Open Book Question Answering Benchmark) 0.334 0.312 0.326 0.318 PIQA (Physics Question Answering Benchmark) 0.651 0.651 0.652 0.648 SIQA (Social Knowledge Question Answering Benchmark) 0.380 0.387 0.375 0.394 CEval (Chinese Evaluation Benchmark) 0.279 0.275 0.278 0.296 CMMLU (Chinese Multi-Task Language Understanding) 0.294 0.286 0.292 0.309 <![CDATA[Average English (Average score across standard English indicators, including ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, and SIQA)]]> 0.413 0.410 0.413 0.418 <![CDATA[Average Chinese (Average score of Chinese indicators, including CEval and CMMU)]]> 0.287 0.280 0.285 0.303 Average (comprehensive average score) 0.388 0.384 0.388 0.395 Table 3 Experimental results of pure Chinese dataset index skypile wanjuan-v1 cci3.0 cci3.0-hq MMLU (Multi-Task Language Understanding Benchmark) 0.286 0.287 0.289 0.292 OBQA (Open Book Question Answering Benchmark) 0.334 0.312 0.326 0.318 PIQA (Physics Question Answering Benchmark) 0.651 0.651 0.652 0.648 SIQA (Social Knowledge Question Answering Benchmark) 0.380 0.387 0.375 0.394 CEval (Chinese Evaluation Benchmark) 0.279 0.275 0.278 0.296 CMMLU (Chinese Multi-Task Language Understanding Benchmark) 0.294 0.286 0.292 0.309 <![CDATA[Average English (Average score across standard English indicators, including ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, and SIQA)]]> 0.413 0.410 0.413 0.418 <![CDATA[Average Chinese (Average score of Chinese indicators, including CEval and CMMU)]]> 0.287 0.280 0.285 0.303 Average (comprehensive average score) 0.388 0.384 0.388 0.395 The Chinese and English names corresponding to the abbreviations in Tables 2 and 3 are shown in Table 4.

[0043] Table 4 Abbreviated Chinese and English names Abbreviation English name Chinese name ARC-C AI2 Reasoning Challenge - Challenge Set AI2 Reasoning Challenge - Challenge Set ARC-E AI2 Reasoning Challenge - Easy Set AI2 Reasoning Challenge - Simple Set HellaSwag HellaSwag Benchmark HellaSwag Benchmark Winograd Winograd Schema Challenge Winograd Schema Challenge MMLU Massive Multitask Language Understanding (MMLU) Large-scale multi-task language understanding OBQA OpenBook QA Open Book Q&A PIQA Physical Interaction Question Answering (PIQA) Physics Interactive Q&A SIQA Social IQa (SIQA) Social IQ Quiz CEval Chinese Evaluation (CEval) Chinese Assessment CMMLU Chinese MMLU (CMMU) Chinese Multi-task Language Understanding At the same time, the present invention also conducts a unified evaluation and comparison of the intermediate model results in the model training process. The evaluation and comparison results are as follows: Figure 5 and Figure 6 shown. Figure 5 and Figure 6 Among them, WanJuan is a high-quality English web text dataset obtained from CommonCrawl; Skypile is a large-scale Chinese language model pre-training dataset, which is taken from publicly available Chinese Internet web page data and has undergone strict filtering and extensive deduplication. It also uses models such as FastText and Bert to filter low-quality data; word unit token is the unit of text (word or character) processed by the large language model. According to Figure 5 and Figure 6 It can be seen that the high-quality dataset constructed by the present invention steadily exceeds other open source datasets as the amount of training increases.

[0044] One of the keys to the construction of a high-quality Chinese text pre-training dataset in the present invention is the high-quality classifier. In order to verify the effectiveness of the high-quality classifier of the present invention, the performance of the high-quality classifier of the present invention and other existing Chinese and English quality classifiers on the F1 index is compared, as shown in Table 5. As can be seen from Table 5, the high-quality classifier of the present invention performs well, surpassing the existing open source Chinese and English quality classifiers, such as the classifiers of FineWeb-edu, IndustryCorpus2 and ChineseWebText.

[0045] Table 5 Accuracy comparison of different classifiers

[0046] Example 2 like Figure 7 As shown, a large language model training method provided by an embodiment of the present invention includes the following steps: Step A1: construct a high-quality Chinese text pre-training dataset using the Chinese text pre-training dataset construction method in Example 1 of the present application; Step A2: Use the Chinese text pre-training dataset obtained in step A1 to train and verify the large language model.

[0047] Example 3 The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, the electronic device includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the Chinese text pre-training data set construction method in Example 1 of the present application or the large language model training method in Example 2 of the present application.

[0048] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) or the programs and / or data loaded from the storage part into the random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM, and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0049] The processor and memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0050] Although not shown, an embodiment of the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method for constructing a Chinese text pre-training dataset in Example 1 of the present application or the large language model training method in Example 2 of the present application is implemented.

[0051] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0052] Although not shown, an embodiment of the present invention also provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the Chinese text pre-training data set construction method in Example 1 of the present application or the large language model training method in Example 2 of the present application.

[0053] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for constructing a Chinese text pre-training dataset, characterized in that: The construction method comprises: Get the original Chinese text data; Cleaning the original Chinese text data to obtain a Chinese text corpus; Using an open source large language model to perform quality scoring and annotation on part of the text in the Chinese text corpus to obtain labeled samples; Use labeled samples to train the classifier to obtain a high-quality classifier; The Chinese text corpus is quality classified using a high-quality classifier to obtain a high-quality Chinese text pre-training data set.

2. The method for constructing a Chinese text pre-training dataset according to claim 1, characterized in that: The original Chinese text data is cleaned to obtain a Chinese text corpus, including: Performing content security filtering on the original Chinese text data to filter out data containing unsafe content; Extract and clean the Chinese text data after content security filtering to delete invalid fields and obtain valid fields; De-duplicate the extracted and cleaned Chinese text data; Perform heuristic rule filtering on the deduplicated Chinese text data to eliminate outliers and excessively repeated content; The basic quality classifier is used to perform preliminary quality assessment on the Chinese text data after heuristic rule filtering to obtain the Chinese text corpus.

3. The method for constructing a Chinese text pre-training dataset according to claim 2, characterized in that: Use DocParser to extract and clean up Chinese text data after content security filtering; The MinHash algorithm is used to deduplicate the extracted and cleaned Chinese text data.

4. The method for constructing a Chinese text pre-training dataset according to claim 1, characterized in that: The open source large language model uses the Qwen2-72B-Instruct model, and the Qwen2-72B-Instruct model performs quality scoring annotation on part of the text in the Chinese text corpus according to the quality scoring method in the FineWeb-edu dataset.

5. The method for constructing a Chinese text pre-training dataset according to any one of claims 1 to 4, characterized in that: The classifier is obtained by adding a classification head on the basis of the BGE-M3 vector model.

6. The method for constructing a Chinese text pre-training dataset according to claim 5, characterized in that: When the classifier is trained, the embedding layer and encoding layer of the BGE-M3 vector model are frozen, and only the newly added classification head is trained.

7. A large language model training method, characterized in that: The training method comprises: Constructing a high-quality Chinese text pre-training dataset using the Chinese text pre-training dataset construction method according to any one of claims 1 to 6; The large language model is trained and verified using the high-quality Chinese text pre-training dataset.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the steps in the method for constructing a Chinese text pre-training dataset as described in any one of claims 1 to 6 or the large language model training method as described in claim 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps in the method for constructing a Chinese text pre-training dataset as described in any one of claims 1 to 6 or the large language model training method as described in claim 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the method for constructing a Chinese text pre-training dataset as described in any one of claims 1 to 6 or the large language model training method as described in claim 7 are implemented.

Images