Large model pre-training corpus construction method and device

Abstract

The invention provides a large model pre-training corpus construction method and device, and relates to the technical field of data processing, and the method comprises the steps: inputting a plurality of corpus samples into a target large language model, and outputting a text scalar score corresponding to each corpus sample; taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, and obtaining a plurality of training samples; training a classifier based on the plurality of training samples to obtain a trained target classifier; wherein the computing resource required by the classifier is smaller than that of the target large language model; and screening each corpus sample in a preset corpus based on the target classifier to obtain a high-quality training corpus.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for constructing a large model pre-training corpus. Background Art

[0002] Pre-training data quality assessment is a critical and challenging task in the field of large language model development. The purpose is to filter out high-quality corpus from massive unstructured data to improve the pre-training effect of the language model. The quality and processing methods of pre-training data directly affect the performance of the language model in end-to-end tasks, including comprehension, reasoning, language, and subject capabilities. In the process of building a large language model, how to efficiently evaluate and filter out data that is beneficial to training is a key step in improving model performance.

[0003] Therefore, how to effectively construct large-scale model pre-training corpus has become a problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The present invention provides a method and device for constructing a large model pre-training corpus, which are used to solve the problem of how to effectively construct a large model pre-training corpus in the prior art.

[0005] The present invention provides a method for constructing a large model pre-training corpus, comprising: Inputting multiple corpus samples into a target large language model, and outputting text scalar scores corresponding to each of the corpus samples; Taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, obtaining a plurality of training samples; Training a classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; Based on the target classifier, each corpus sample in the preset corpus is screened to obtain high-quality training corpus.

[0006] According to a large model pre-training corpus construction method provided by the present invention, the steps of inputting multiple corpus samples into a target large language model and outputting text scalar scores corresponding to each of the corpus samples include: Obtaining a preset prompt template, wherein the preset prompt template is used to guide the target large language model to evaluate the text quality to output a text scalar score of each text; Based on the combination of the corpus sample and the preset prompt template, and inputting into the target large language model, the text scalar score corresponding to each of the corpus samples is output.

[0007] According to a large model pre-training corpus construction method provided by the present invention, a classifier is trained based on the multiple training samples to obtain a trained target classifier, including: For any training sample, input the training sample into the classifier, and output the text prediction score corresponding to the training sample; Calculating a loss value according to a text prediction score corresponding to the training sample and a text scalar score in the training sample using a preset loss function; If the loss value is less than a preset threshold, the classifier training is completed and a trained target classifier is obtained.

[0008] According to a large model pre-training corpus construction method provided by the present invention, each corpus sample in a preset corpus is screened based on the target classifier to obtain high-quality training corpus, including: Inputting each of the corpus samples into the target classifier to obtain a text scalar score of each of the corpus samples; Corpus samples whose text scalar scores are less than a preset score threshold are screened out to obtain high-quality training corpus whose text scalar scores are greater than or equal to the preset score threshold.

[0009] According to a method for constructing a large model pre-training corpus provided by the present invention, after screening each corpus sample in a preset corpus based on the target classifier to obtain high-quality training corpus, the method further includes: Pre-training a preset language model based on the high-quality training corpus, and stopping the training when a preset training condition is met to obtain a pre-trained preset language model; The performance of the preset language model is evaluated through a standardized evaluation platform.

[0010] The present application embodiment provides a large model pre-training corpus construction device, including: An input module, used to input multiple corpus samples into a target large language model, and output a text scalar score corresponding to each of the corpus samples; An acquisition module, used for taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, and acquiring multiple training samples; A training module, used for training a classifier based on the plurality of training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; The screening module is used to screen each corpus sample in the preset corpus based on the target classifier to obtain high-quality training corpus.

[0011] The present application embodiment provides a large model pre-training corpus construction device, the device is also used for: Obtaining a preset prompt template, wherein the preset prompt template is used to guide the target large language model to evaluate the text quality to output a text scalar score of each text; Based on the combination of the corpus sample and the preset prompt template, and inputting into the target large language model, the text scalar score corresponding to each of the corpus samples is output.

[0012] Optionally, the device is also used for: For any training sample, input the training sample into the classifier, and output the text prediction score corresponding to the training sample; Calculating a loss value according to a text prediction score corresponding to the training sample and a text scalar score in the training sample using a preset loss function; If the loss value is less than a preset threshold, the classifier training is completed and a trained target classifier is obtained.

[0013] Optionally, the device is also used for: Inputting each of the corpus samples into the target classifier to obtain a text scalar score of each of the corpus samples; Corpus samples whose text scalar scores are less than a preset score threshold are screened out to obtain high-quality training corpus whose text scalar scores are greater than or equal to the preset score threshold.

[0014] Optionally, the device is also used for: Pre-training a preset language model based on the high-quality training corpus, and stopping the training when a preset training condition is met to obtain a pre-trained preset language model; The performance of the preset language model is evaluated through a standardized evaluation platform.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for constructing a large model pre-training corpus as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for constructing a large model pre-training corpus.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned large model pre-training corpus construction methods.

[0018] The large-model pre-training corpus construction method and device provided by the present invention can significantly improve the performance of the language model in tasks such as understanding, generation and reasoning by screening high-quality training corpus. Using a small language model as a classifier can greatly reduce the required computing resources and energy consumption compared to directly using a large language model for screening. Through an automated screening process, large-scale corpora can be processed quickly and the efficiency of data processing can be improved. Giving priority to limited high-quality corpus for model training can more reasonably allocate data resources and improve training effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of a process for constructing a large model pre-training corpus provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a large model pre-training corpus construction device provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In the embodiment of the present application, the core of data screening is to sample the data set in some way to obtain a subset of the original data, and the quality assessment scheme is to screen the data by scoring the text quality. Here we need to define "data" first. First of all, the smallest data unit of the language model is Token, which can be composed of bytes, characters, subwords or larger components, such as vocabulary, etc., which is determined by the framework of the Tokenizer and its training method. Based on this, a single data sample can be defined here: Definition 1. A single data sample used to train or evaluate a model is represented as a data point in the data space , It is defined as an ordered set of Tokens, that is, ,in Indicates the length of this Token set.

[0023] For example, in the process of language modeling, It can be a token sequence from an Internet document, book, or blog text. In addition, since there is an upper limit on the length of the sequence input during language model training, long document data samples are usually split into multiple data points. This actually means that during the pre-training process, It may constitute part of a document, but not the entire document. A collection of data points is called a dataset. Here we define the concepts of dataset and dataset distribution: Definition 2. Dataset Represents a collection of some data points { ,in Represents the number of data points in the set.

[0024] Definition 3. Dataset The distribution of refers to the distribution of data points in the space of all possible data points.

[0025] With the concepts of data points and data sets, we can further elaborate on data processing. The essence of data processing is the process of creating a data set from a set of candidate data points. The created data set will be used to train or evaluate the language model. Before the data is processed, it generally goes through three stages: collection, labeling, and storage. Afterwards, once the training goal of the model is determined, the direction of data processing is to filter and sample data points that can maximize the desired goal.

[0026] Definition 4. The core of data processing is to select a data processing function The dataset And the objective function as input and from Filter, clean, and sample data points to create the final dataset , then based on the data set To train the model , so that Can maximize or minimize the objective function .

[0027] In practice, multiple data processing methods are often combined to improve the coverage and specificity of the final dataset. In this case, the different data processing steps are represented as components of the overall processing function , and the composite is represented as .

[0028] In the embodiment of the present application, the purpose is to find an efficient data processing function , which can map natural language text into a scalar space of quality distribution and filter out low-quality data, formally defined as

[0029] in, Represents the data points The function for quality assessment, Indicates a quality score threshold after evaluation. If the quality evaluation score is higher than this threshold, the data is high-quality data. , which can maximize the objective function , Representation Model Evaluation scores on the end-to-end side.

[0030] Figure 1 A flowchart of a method for constructing a large model pre-training corpus provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, including: Step 110, inputting a plurality of corpus samples into a target large language model, and outputting a text scalar score corresponding to each of the corpus samples; Collect and prepare multiple corpus samples, which can be text data from different sources and on different topics.

[0031] These corpus samples are fed into the target large language model as input data. This is usually done through API calls or direct interaction with the model's input interface. Using the model's evaluation capabilities, each input corpus sample is analyzed, and the model will evaluate the quality of the text based on its internal algorithms and learned knowledge.

[0032] The model outputs a text scalar score for each corpus sample, which reflects the performance of the text on predefined quality dimensions, such as correctness, completeness, coherence, etc. The score can be a continuous value, but in some cases, it may be necessary to map the continuous score to discrete levels or categories for subsequent processing.

[0033] Collect each corpus sample and its corresponding text scalar score to prepare for subsequent training sample creation or further analysis. If there are preset scoring criteria or thresholds, the output scores can be filtered according to these criteria to distinguish high-quality text samples.

[0034] Based on the output scores and possible feedback, the evaluation ability of the model can be iteratively optimized to improve the accuracy and reliability of the evaluation.

[0035] Optionally, the step of inputting a plurality of corpus samples into a target large language model and outputting a text scalar score corresponding to each of the corpus samples comprises: Obtaining a preset prompt template, wherein the preset prompt template is used to guide the target large language model to evaluate the text quality to output a text scalar score of each text; Based on the combination of the corpus sample and the preset prompt template, and inputting into the target large language model, the text scalar score corresponding to each of the corpus samples is output.

[0036] Prepare a preset prompt template (Prompt), which is designed to guide the language model to evaluate the quality of the text. In the prompt template, clearly specify the criteria for evaluating the quality of the text, such as correctness, completeness, coherence, etc.

[0037] Collect and prepare the corpus samples to be evaluated, which can be text data from different sources and topics. Combine each corpus sample with the preset prompt template to form the final input request. This may involve inserting the text sample into the specified position of the prompt template.

[0038] Optionally, the specific prompt design is as follows: You are a language model training corpus annotator, and your task is to evaluate the quality of each corpus. You need to score the quality of each text data on a scale of 0-5, where 0 represents the worst and 5 represents the best.

[0039] The assessment dimensions include the following: 1. Correctness: Check the text for errors in words or phrases, grammatical errors, or improper use of punctuation.

[0040] 2. Redundancy: Is there any redundant or repeated content in the text, is there any jump between paragraphs, and is there any logical relationship between them?

[0041] 3. Completeness: Does the text contain complete information? Are there any missing or unclear parts? Is there any half-sentence? Does the text have a beginning and an end?

[0042] 4. Coherence: Is the text content coherent? Is there any noise text such as advertisements inserted?

[0043] 5. Readability: Is the text readable, is the language clear, and is the paragraph well organized?

[0044] 6. Unity: Is the main theme of the text consistent as a whole? If the text is long, multiple themes are allowed, but the internal themes must be unified.

[0045] 7. Professionalism: If the text content involves a specific field or topic, check whether it uses correct and accurate terminology and whether there is in-depth analysis and understanding.

[0046] 8. Objectivity: Does the text describe the facts objectively and fairly? Is there any excessive subjective bias or erroneous interpretation?

[0047] 9. Value content: whether the text content contains valuable or useful information, such as encyclopedias, which have a higher value content

[0048] The scoring rules are as follows: 0 points: The text content performs very poorly in all evaluation dimensions, or has very low value content.

[0049] 1 point: The text content performs poorly in most evaluation dimensions, but performs averagely in a few dimensions, or has low value content.

[0050] 2 points: The text content performs averagely in most evaluation dimensions.

[0051] 3 points: The text content performs averagely in most evaluation dimensions, but performs well in a few dimensions.

[0052] 4 points: The text content performs well in most evaluation dimensions, but performs excellently in a few dimensions.

[0053] 5 points: The text content performs very well in all evaluation dimensions.

[0054] Require: Please rate the following text based on all the above evaluation dimensions. First, state your reasons for the evaluation and then give your evaluation score for the text quality.

[0055] constraint: 1. The evaluation score must be judged from the following six scores: [0, 1, 2, 3, 4, 5]. Decimal scores cannot appear, and scores cannot be output in Chinese characters.

[0056] 2. Output the evaluation score at the end of the output reason. The output template is as follows, where "X score" represents the score you judged for the given text: "...your reason.... The final quality evaluation score is X points."

[0057] The following text is a whole piece of content. You need to evaluate it and output a score as a whole: ...\{text to be rated\}...

[0058] Step 120, taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, and obtaining a plurality of training samples; In the embodiment of the present application, a certain number of text samples are collected from a preset corpus, and these samples should cover different topics and styles. Each corpus sample is evaluated using a target large language model and a preset prompt template to obtain a text scalar score for each sample.

[0059] Pair each corpus sample with its corresponding text scalar score to form a training sample. This training sample includes input (text sample) and output (scalar score). Make sure that each training sample is correctly labeled, that is, the text sample and its quality score are correctly matched.

[0060] Perform necessary preprocessing on the training samples, including text cleaning, standardization, word segmentation, etc., to meet the input requirements of the model. Collect all training samples to form a training set, which will be used to train the classifier.

[0061] Step 130, training a classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; In the embodiment of the present application, a suitable small language model architecture is selected, such as a simplified version of Transformer, recurrent neural network (RNN) or convolutional neural network (CNN), etc. The parameters of the model are set, including the number of layers, number of neurons, learning rate, optimizer, etc.

[0062] Load the prepared training samples into the training environment. Divide the training samples into training set, validation set, and test set. According to the model architecture and task type (such as regression or classification), select the appropriate loss function and evaluation metric and compile the model. Train the model using the training set data. During the training process, monitor the performance of the model on the validation set to avoid overfitting. According to the performance of the model on the validation set, adjust the hyperparameters to optimize the model performance.

[0063] Evaluate the final performance of the model on the test set to ensure that the model has good generalization ability. According to resource usage and model performance, further tune the model to achieve the best balance between resources and performance. Save the trained model as a file for subsequent deployment and use. Deploy the trained classifier to the production environment for automatic evaluation and screening of text data.

[0064] Step 140 , screening each corpus sample in the preset corpus based on the target classifier to obtain high-quality training corpus.

[0065] In the embodiment of the present application, it is ensured that the preset corpus has been collected, and preliminary cleaning and formatting have been performed to adapt to the input requirements of the classifier. The trained target classifier is loaded into the screening system to ensure that it can receive input and output scores.

[0066] The samples in the corpus are fed into the classifier in batches. This may require designing an automated script to handle large amounts of data. The classifier scores each corpus sample and then ranks all samples based on the score.

[0067] According to the quality requirements of the training corpus, set one or more screening criteria, such as score threshold, ranking percentage, etc. According to the set criteria, filter out high-quality samples that are above the threshold or within the required ranking percentage.

[0068] Evaluate the diversity and balance of the selected corpus in terms of topics, styles, fields, etc. to ensure the representativeness of the training data.

[0069] In the embodiments of the present application, by screening high-quality training corpus, the performance of the language model in tasks such as understanding, generation and reasoning can be significantly improved. Using a small language model as a classifier can greatly reduce the required computing resources and energy consumption compared to directly using a large language model for screening. Through an automated screening process, large-scale corpora can be processed quickly and the efficiency of data processing can be improved. Giving priority to limited high-quality corpora for model training can more reasonably allocate data resources and improve training results.

[0070] Figure 2 The large model pre-training corpus construction device provided in the embodiment of the present application is as follows: Figure 2 Nothing to do, including: The input module 210 is used to input multiple corpus samples into the target large language model and output the text scalar score corresponding to each of the corpus samples; The acquisition module 220 is used to take the corpus sample and the text scalar score corresponding to the corpus sample as a training sample to acquire multiple training samples; The training module 230 is used to train the classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than the target large language model; The screening module 240 is used to screen each corpus sample in the preset corpus based on the target classifier to obtain high-quality training corpus.

[0071] Optionally, the device is also used for: Obtaining a preset prompt template, wherein the preset prompt template is used to guide the target large language model to evaluate the text quality to output a text scalar score of each text; Based on the combination of the corpus sample and the preset prompt template, and inputting into the target large language model, the text scalar score corresponding to each of the corpus samples is output.

[0072] Optionally, the device is also used for: Inputting each of the corpus samples into the target classifier to obtain a text scalar score of each of the corpus samples; Corpus samples whose text scalar scores are less than a preset score threshold are screened out to obtain high-quality training corpus whose text scalar scores are greater than or equal to the preset score threshold.

[0073] Optionally, the device is also used for: Pre-training a preset language model based on the high-quality training corpus, and stopping the training when a preset training condition is met to obtain a pre-trained preset language model; The performance of the preset language model is evaluated through a standardized evaluation platform.

[0074] In the embodiments of the present application, by screening high-quality training corpus, the performance of the language model in tasks such as understanding, generation and reasoning can be significantly improved. Using a small language model as a classifier can greatly reduce the required computing resources and energy consumption compared to directly using a large language model for screening. Through an automated screening process, large-scale corpora can be processed quickly and the efficiency of data processing can be improved. Giving priority to limited high-quality corpora for model training can more reasonably allocate data resources and improve training results.

[0075] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the large model pre-training corpus construction method, the method comprising: inputting a plurality of corpus samples into the target large language model, and outputting the text scalar score corresponding to each of the corpus samples; Taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, obtaining a plurality of training samples; Training a classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; Based on the target classifier, each corpus sample in the preset corpus is screened to obtain high-quality training corpus.

[0076] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0077] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the large model pre-training corpus construction method provided by the above methods, the method comprising: inputting multiple corpus samples into a target large language model, and outputting a text scalar score corresponding to each of the corpus samples; Taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, obtaining a plurality of training samples; Training a classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; Based on the target classifier, each corpus sample in the preset corpus is screened to obtain high-quality training corpus.

[0078] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the large model pre-training corpus construction method provided by the above methods, the method comprising: inputting a plurality of corpus samples into a target large language model, and outputting a text scalar score corresponding to each of the corpus samples; Taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, obtaining a plurality of training samples; Training a classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; Based on the target classifier, each corpus sample in the preset corpus is screened to obtain high-quality training corpus.

[0079] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a large model pre-training corpus, characterized in that: include: Inputting multiple corpus samples into a target large language model, and outputting text scalar scores corresponding to each of the corpus samples; Taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, obtaining a plurality of training samples; Training a classifier based on the multiple training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; Based on the target classifier, each corpus sample in the preset corpus is screened to obtain high-quality training corpus.

2. The method for constructing a large model pre-training corpus according to claim 1, characterized in that: The step of inputting a plurality of corpus samples into a target large language model and outputting a text scalar score corresponding to each of the corpus samples comprises: Obtaining a preset prompt template, wherein the preset prompt template is used to guide the target large language model to evaluate the text quality to output a text scalar score of each text; Based on the combination of the corpus sample and the preset prompt template, and inputting into the target large language model, the text scalar score corresponding to each of the corpus samples is output.

3. The method for constructing a large model pre-training corpus according to claim 1, characterized in that: Training a classifier based on the multiple training samples to obtain a trained target classifier includes: For any training sample, input the training sample into the classifier, and output the text prediction score corresponding to the training sample; Calculating a loss value according to a text prediction score corresponding to the training sample and a text scalar score in the training sample using a preset loss function; If the loss value is less than a preset threshold, the classifier training is completed and a trained target classifier is obtained.

4. The method for constructing a large model pre-training corpus according to claim 3, characterized in that: Based on the target classifier, each corpus sample in the preset corpus is screened to obtain high-quality training corpus, including: Inputting each of the corpus samples into the target classifier to obtain a text scalar score of each of the corpus samples; Corpus samples whose text scalar scores are less than a preset score threshold are screened out to obtain high-quality training corpus whose text scalar scores are greater than or equal to the preset score threshold.

5. The method for constructing a large model pre-training corpus according to claim 1, characterized in that: After screening each corpus sample in the preset corpus based on the target classifier to obtain high-quality training corpus, the method further includes: Pre-training a preset language model based on the high-quality training corpus, and stopping the training when a preset training condition is met to obtain a pre-trained preset language model; The performance of the preset language model is evaluated through a standardized evaluation platform.

6. A large model pre-training corpus construction device, characterized in that: include: An input module, used to input multiple corpus samples into a target large language model, and output a text scalar score corresponding to each of the corpus samples; An acquisition module, used for taking the corpus sample and the text scalar score corresponding to the corpus sample as a training sample, and acquiring multiple training samples; A training module, used for training a classifier based on the plurality of training samples to obtain a trained target classifier; wherein the computing resources required by the classifier are less than those of the target large language model; The screening module is used to screen each corpus sample in the preset corpus based on the target classifier to obtain high-quality training corpus.

7. The large model pre-training corpus construction device according to claim 1, characterized in that: The device is also used for: Obtaining a preset prompt template, wherein the preset prompt template is used to guide the target large language model to evaluate the text quality to output a text scalar score of each text; Based on the combination of the corpus sample and the preset prompt template, and inputting into the target large language model, the text scalar score corresponding to each of the corpus samples is output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the large model pre-training corpus construction method as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for constructing a large model pre-training corpus as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for constructing a large model pre-training corpus as described in any one of claims 1 to 5.

Images