Mass high-quality data screening method

Abstract

The invention provides a mass high-quality data screening method, which relates to the technical field of large language models, and comprises the following steps: collecting a mass data set, and screening the data set to obtain a high-quality text; calculating the proportion of the token length of the high-quality text to the character length of the original text so as to divide the high-quality text into a plurality of uniform intervals; sampling the same number of text data from each interval to construct an initial training set, and distilling the initial training set to generate annotation data containing text quality scores; constructing a small parameter model, and carrying out joint training on the small parameter model through mathematical calculation of the thinking chain data and the annotation data; a small parameter model is used for screening mass data, and a large model is called again for distillation and iterative optimization until the screening accuracy of the small parameter model meets a preset threshold value; final data screening is completed through the small parameter models with the screening accuracy meeting a preset threshold value; the large model reasoning speed can be increased, and the computing power resource cost is remarkably reduced.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and more specifically, to a method for filtering massive amounts of high-quality data. Background Technology

[0002] With the rapid development of large language models, a large number of supervised and unsupervised datasets that can be used for training large models have been open-sourced in the Huggingface and Moda communities. High-quality datasets that solve practical problems and convey valuable information can promote the training of large models. Being able to select high-quality datasets from massive open-source datasets is an important foundation for constructing and training large industry models.

[0003] However, in the current training industry, large model training sets typically reach millions of data points (e.g., DeepSeek-R1-0528). These large models consume a lot of computing power and are inefficient. Furthermore, the massive text datasets they collect often come from diverse sources and vary in quality. Therefore, how to select high-quality data from massive text datasets has become a crucial step in training large language models. Summary of the Invention

[0004] The purpose of this invention is to provide a method for filtering massive amounts of high-quality data, which can improve the inference speed of large models and reduce the cost of computing resources.

[0005] The technical solution of this invention is as follows: Firstly, this application provides a method for filtering massive amounts of high-quality data, which includes the following steps: S1. Collect massive datasets and filter the datasets using a large model word segmenter to obtain high-quality text; S2. Calculate the ratio of token length to original text character length using a large model word segmenter, and divide the ratio into multiple uniform intervals. S3. Sample text data from each interval in equal quantities to construct an initial training set, and use a large model to distill the initial training set to generate labeled data containing text quality scores. S4. Construct a small parameter model based on labeled data, and jointly train the small parameter model using mathematical calculation thinking chain data and labeled data; S5. Use the trained small parameter model to filter the massive amount of data, and re-call the large model to distill and iteratively optimize the filtered abnormal data until the filtering accuracy of the small parameter model meets the preset threshold. S6. Obtain the enterprise's computing power cost, adjust the text quality scoring threshold according to the enterprise's computing power cost, and complete the final data filtering by filtering small parameter models whose accuracy meets the preset threshold.

[0006] Furthermore, in step S1, the aforementioned large-scale word segmenter is obtained by acquiring and training a massive amount of corpus.

[0007] Furthermore, in step S2, the above-mentioned division of the proportion into multiple uniform intervals includes dividing the proportion interval into five intervals: [0,0.2), [0.2,0.4), [0.4,0.6), [0.6,0.8), and [0.8,1], with the same number of samples in each interval.

[0008] Furthermore, the criteria for scoring the text quality mentioned above include accuracy, harmlessness, logic, and language quality.

[0009] Furthermore, in step S4, the aforementioned mathematical calculation thought chain data includes thought chain data of adding four positive integers, and each set of data includes input and output of the step-by-step calculation process.

[0010] Furthermore, in step S5, the abnormal data selected above includes data with JSON format errors, data whose sum of dimension scores does not match the total score, data whose scores exceed the specified range, and data with missing fields.

[0011] Secondly, this application provides an electronic device, comprising: Memory, used to store one or more programs; processor; When one or more of the above programs are executed by the above processor, a method for filtering massive amounts of high-quality data is implemented as described in any of the first aspects above.

[0012] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for filtering massive amounts of high-quality data as described in any of the first aspects above.

[0013] Compared with the prior art, the present invention has at least the following advantages or beneficial effects: (1) The present invention provides a method for screening massive high-quality data, which obtains a small parameter model by training the large model distillation data, so that the small parameter model achieves the same accuracy as the large model, reduces the consumption of inference resources, and significantly reduces the computing cost; (2) This invention uses a word segmenter to perform interval sampling, which ensures coverage of rare samples and enhances the generalization ability of the model; (3) This invention significantly improves the accuracy and output stability of small models through multi-task training (scoring + mathematical calculation) and iterative screening mechanism. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the steps of a method for filtering massive amounts of high-quality data according to the present invention. Figure 2 This is a schematic structural block diagram of an electronic device according to an embodiment of the present invention.

[0016] Icons: 101, memory; 102, processor; 103, communication interface. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0021] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0022] Example 1 Please see Figure 1 , Figure 1 The diagram shows the steps of a method for filtering massive amounts of high-quality data provided in an embodiment of this application.

[0023] This application provides a method for filtering massive amounts of high-quality data, which includes the following steps: S1. Collect massive datasets and filter the datasets using a large model word segmenter to obtain high-quality text; S2. Calculate the ratio of token length to original text character length using a large model word segmenter, and divide the ratio into multiple uniform intervals. S3. Sample text data from each interval in equal quantities to construct an initial training set, and use a large model to distill the initial training set to generate labeled data containing text quality scores. S4. Construct a small parameter model based on labeled data, and jointly train the small parameter model using mathematical calculation thinking chain data and labeled data; S5. Use the trained small parameter model to filter the massive amount of data, and re-call the large model to distill and iteratively optimize the filtered abnormal data until the filtering accuracy of the small parameter model meets the preset threshold. S6. Obtain the enterprise's computing power cost, adjust the text quality scoring threshold according to the enterprise's computing power cost, and complete the final data filtering by filtering small parameter models whose accuracy meets the preset threshold.

[0024] Among them, the small parameter model adopts the Qwen3-0.6B small model. The small parameter model is jointly trained by mathematical calculation thinking chain data and labeled data to improve the accuracy of the small model for the addition of four positive integers, so that the Qwen3-0.6B small model can reach the accuracy of the super large model.

[0025] In a preferred implementation, in step S1, the large model word segmenter is obtained by acquiring and training a massive amount of corpus.

[0026] In a preferred embodiment, step S2, dividing the ratio into multiple uniform intervals includes dividing the ratio interval into five intervals: [0,0.2), [0.2,0.4), [0.4,0.6), [0.6,0.8), and [0.8,1], with the same number of samples in each interval.

[0027] As a preferred implementation method, the criteria for text quality scoring include accuracy, harmlessness, logic, and language quality.

[0028] In a preferred embodiment, in step S4, the mathematical calculation thought chain data includes thought chain data of adding four positive integers, and each set of data includes input and output of the step-by-step calculation process.

[0029] In a preferred implementation, the abnormal data filtered in step S5 includes data with JSON format errors, data whose sum of dimension scores does not match the total score, data whose scores exceed the specified range, and data with missing fields.

[0030] Example 2 Please see Figure 2 , Figure 2 This is a schematic structural block diagram of an electronic device provided in an embodiment of this application.

[0031] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules. The processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used for signaling or data communication with other node devices.

[0032] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0033] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0034] It is understood that the structure shown in the figure is for illustrative purposes only. A method for filtering massive amounts of high-quality data may include more or fewer components than shown in the figure, or have a different configuration. The components shown in the figure can be implemented using hardware, software, or a combination thereof.

[0035] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0036] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0037] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0038] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0039] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for filtering massive amounts of high-quality data, characterized in that, Includes the following steps: S1. Collect massive datasets and filter the datasets using a large model word segmenter to obtain high-quality text; S2. Calculate the ratio of token length to original text character length using a large model word segmenter, and divide the ratio into multiple uniform intervals. S3. Sample text data from each interval in equal quantities to construct an initial training set, and use a large model to distill the initial training set to generate labeled data containing text quality scores. S4. Construct a small parameter model based on labeled data, and jointly train the small parameter model using mathematical calculation thinking chain data and labeled data; S5. Use the trained small parameter model to filter the massive amount of data, and re-call the large model to distill and iteratively optimize the filtered abnormal data until the filtering accuracy of the small parameter model meets the preset threshold. S6. Obtain the enterprise's computing power cost, adjust the text quality scoring threshold according to the enterprise's computing power cost, and complete the final data filtering by filtering small parameter models whose accuracy meets the preset threshold.

2. The method for filtering massive amounts of high-quality data as described in claim 1, characterized in that, In step S1, the large model word segmenter is obtained by acquiring massive amounts of corpus and training it.

3. The method for filtering massive amounts of high-quality data as described in claim 1, characterized in that, In step S2, dividing the ratio into multiple uniform intervals includes dividing the ratio interval into five intervals: [0,0.2), [0.2,0.4), [0.4,0.6), [0.6,0.8), and [0.8,1], with the same number of samples in each interval.

4. The method for filtering massive amounts of high-quality data as described in claim 1, characterized in that, The criteria for text quality scoring include accuracy, harmlessness, logic, and language quality.

5. The method for filtering massive amounts of high-quality data as described in claim 1, characterized in that, In step S4, the mathematical calculation thought chain data includes thought chain data of adding four positive integers, and each set of data includes input and output of step-by-step calculation process.

6. The method for filtering massive amounts of high-quality data as described in claim 1, characterized in that, In step S5, the filtered abnormal data includes data with JSON format errors, data whose sum of dimension scores does not match the total score, data whose scores exceed the specified range, and data with missing fields.

7. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the processor executes the one or more programs, it implements a method for filtering massive amounts of high-quality data as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a method for filtering massive amounts of high-quality data as described in any one of claims 1-6.

Images