Document data filtering method, big language model training method, equipment and medium

Through the document data filtering method, the problem of poor data quality in large language model training is solved. Low-quality data is filtered through classification and quality evaluation, which significantly improves the data set quality and model training effect.

CN118916336APending Publication Date: 2024-11-08江苏云从曦和人工智能有限公司
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410947083.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

During the training of large language models, the quality of massive Internet data is uneven, resulting in low-quality data affecting the model training efficiency and predicting output quality, such as inducing overfitting, affecting learning and prediction ability.

Method used

Provide a document data filtering method. By acquiring the document data set, performing classification and quality evaluation, determining whether the data meets the filtering conditions based on the document category and quality score, and using deduplication algorithm and preset filtering thresholds to ensure data quality.

Benefits of technology

It significantly improves the overall data quality of the document data set, improves the effectiveness of training large language models, reduces the impact of low-quality data, and improves the learning and prediction capabilities of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916336A_ABST
    Figure CN118916336A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and particularly provides a document data filtering method, a large language model training method, equipment and a medium. The technical problem that the training data quality of a large language model is poor is solved. The document data filtering method comprises the steps of obtaining a document data set, wherein the document data set comprises multiple pieces of document data; performing classification operation on the document data in the document data set, and determining a document category corresponding to each document data; performing quality evaluation on the document data in the document data set, and determining a quality score of the document data; based on the document category and the quality score corresponding to the document data, judging whether the document data meets a filtering condition or not; and if yes, filtering the document data. According to the method and the device, the low-quality document data meeting the filtering condition is filtered according to the document category and the quality score of the document data, so that the data quality of the document data set is effectively optimized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a document data filtering method, a large language model training method, an electronic device and a storage medium. Background Art

[0002] In the development of large language models, most models rely on massive amounts of Internet data for training, but the quality of this data varies. On the one hand, data such as high-quality academic articles and knowledge question-and-answer community conversations provide the model with rich knowledge points, which helps it to deeply learn and understand complex knowledge. On the other hand, low-quality data such as a large amount of repeated advertising texts and broken sentences fill the training set. They lack practical value and may not only reduce training efficiency, but also have a negative impact on the quality of the model's prediction output, for example, inducing the model to overfit, affecting its learning and prediction capabilities.

[0003] Accordingly, the art needs a new data filtering solution to solve the above problems. Summary of the invention

[0004] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of poor quality of training data in the training process of a large language model.

[0005] In a first aspect, a document data filtering method is provided, the method comprising: obtaining a document data set, the document data set comprising a plurality of document data; performing a classification operation on the document data in the document data set to determine a document category corresponding to each document data; performing a quality assessment on the document data in the document data set to determine a quality score of the document data; judging whether the document data meets a filtering condition based on the document category and the quality score corresponding to the document data; and if so, filtering the document data.

[0006] In a technical solution of the above-mentioned document data filtering method, before performing a classification operation on the document data in the document data set, the method also includes: based on a preset deduplication algorithm, performing a deduplication operation on the document data in the document data set to obtain a deduplicated document data set, and the preset deduplication algorithm includes a minimum hash algorithm.

[0007] In a technical solution of the above-mentioned document data filtering method, the classification operation is performed on the document data in the document data set to determine the document category corresponding to each document data, including: based on a trained document classification model, determining the matching results of the document data in the deduplicated document data set and the preset document category, the matching results including at least one similarity probability value; based on the matching results of the document data and the preset document category, determining the document category corresponding to the document data.

[0008] In a technical solution of the above-mentioned document data filtering method, determining the document category corresponding to the document data based on the matching result between the document data and the preset document category includes: determining the maximum similarity probability value among the similarity probability values ​​matching the document data with each preset document category; and determining the document category corresponding to the document data based on the document category corresponding to the maximum similarity probability value.

[0009] In a technical solution of the above-mentioned document data filtering method, the quality assessment of the document data in the document data set to determine the quality score of the document data includes: based on a trained document quality assessment model, determining the evaluation score corresponding to the document data in the deduplicated document data set in at least one preset evaluation dimension; based on the evaluation score corresponding to the document data in at least one preset evaluation dimension, determining the quality score of the document data.

[0010] In a technical solution of the above-mentioned document data filtering method, the preset evaluation dimensions include knowledge content, language coherence and text content uniformity; determining the quality score of the document data based on the evaluation score corresponding to the document data in at least one preset evaluation dimension includes: obtaining the evaluation score corresponding to at least one dimension of the document data among knowledge content, language coherence and text content uniformity; determining the quality score of the document data based on the evaluation score corresponding to the document data.

[0011] In a technical solution of the above-mentioned document data filtering method, based on the document category and quality score corresponding to the document data, it is judged whether the document data meets the filtering conditions; if so, the document data is filtered, including: based on the document category corresponding to the document data, obtaining a preset filtering threshold of the document category; judging whether the quality score of the document data is less than the preset filtering threshold of the document category; if so, performing a deletion operation on the document data.

[0012] In a second aspect, a large language model training method is provided, the method comprising: obtaining a document data set filtered by the document data filtering method described in any of the above technical solutions; and training a large language model based on the filtered document data set until the large language model converges.

[0013] In a third aspect, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned document data filtering method is implemented.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned document data filtering method.

[0015] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0016] The document data filtering method provided by the present application specifically includes: obtaining a document data set, the document data set includes multiple document data; performing classification operations on the document data in the document data set to determine the document category corresponding to each document data; performing quality assessment on the document data in the document data set to determine the quality score of the document data; judging whether the document data meets the filtering conditions based on the document category and quality score corresponding to the document data; if so, filtering the document data. The present application achieves the goal of effectively filtering low-quality document data based on the document category and quality score of the document data by accurately classifying the document data and assessing the quality score of the document data, thereby significantly improving the overall data quality of the document data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0018] Figure 1 This is a flowchart of the main steps of a document data filtering method according to an embodiment of the present application;

[0019] Figure 2 is a detailed step flow diagram of a document data filtering method according to an embodiment of the present application;

[0020] Figure 3 It is a flowchart of the main steps of a large language model training method according to an embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of the main structure of an electronic device according to an embodiment of the present application. Reference numerals:

[0022] 11: processor; 12: memory. DETAILED DESCRIPTION

[0023] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0024] In the description of the present application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, memory, and may also include software parts, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware or a combination of the two. Computer-readable storage media include any suitable medium that can store program code, such as a disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B or A and B. The singular terms "one" and "the" may also include plural forms.

[0025] In the development of large language models, most models rely on massive amounts of Internet data for training, but the quality of this data varies. On the one hand, data such as high-quality academic articles and knowledge question-and-answer community conversations provide the model with rich knowledge points, which helps it to deeply learn and understand complex knowledge. On the other hand, low-quality data such as a large amount of repeated advertising texts and broken sentences fill the training set. They lack practical value and may not only reduce training efficiency, but also have a negative impact on the quality of the model's prediction output, such as inducing the model to overfit, affecting its learning and prediction capabilities.

[0026] To this end, the present application provides a document data filtering method, which specifically includes: obtaining a document data set, which is used to train a large language model, and the document data set includes multiple document data; performing classification operations on the document data in the document data set to determine the document category corresponding to each document data; performing quality assessment on the document data in the document data set to determine the quality score of the document data; based on the document category and quality score corresponding to the document data, judging whether the document data meets the filtering conditions; if so, filtering the document data. The present application achieves the goal of effectively filtering low-quality document data in a document category based on the document category and quality score of the document data by accurately classifying the document data and evaluating the quality score of the document data, thereby significantly improving the overall data quality of the document data set and further improving the effect of large language model training.

[0027] See attached Figure 1 , Figure 1FIG. 1 is a flowchart of the main steps of a document data filtering method according to an embodiment of the present application. Figure 1 As shown, the document data filtering method in the embodiment of the present application mainly includes the following steps S101 to S104.

[0028] Step S101: Obtain a document dataset, where the document dataset is used to train a large language model and includes a plurality of document data.

[0029] In this embodiment, the document data set can be a collection of document data obtained by crawling data on relevant websites, or it can be a collection of pre-stored document data. The document data can be academic articles, conversations in knowledge question and answer communities, news, advertising texts, and other data.

[0030] Step S102: performing a classification operation on the document data in the document data set to determine the document category corresponding to each document data.

[0031] In this embodiment, the classification operation performed on the document data may be to classify the document data according to its content, and the document categories include politics, science, education, art, philosophy and the like.

[0032] Step S103: performing quality assessment on the document data in the document data set to determine a quality score of the document data.

[0033] Step S104: Based on the document category and quality score corresponding to the document data, determine whether the document data meets the filtering condition; if so, filter the document data.

[0034] Based on the method described in steps S101 to S104 above, the present application first obtains a document data set, classifies the document data in the document data set to determine the document category corresponding to each document data, and performs quality assessment on the document data in the document data set to obtain the quality score of the document data, and then determines whether the document data meets the filtering criteria based on the document category and quality score of the document data; if it is mixed, the document data is filtered. The present application achieves the goal of effectively filtering low-quality document data in a document category based on the document category and quality score of the document data by accurately classifying the document data and assessing the quality score of the document data, thereby significantly improving the overall data quality of the document data set.

[0035] The above steps S101 to S104 are further explained below.

[0036] With respect to step S101, a document dataset is obtained. The document dataset is applied to train a large language model, and the document dataset includes a plurality of document data.

[0037] Specifically, the document data set can be a collection of document data obtained by crawling data on relevant websites, or a collection of pre-stored document data. The document data can be academic articles, conversations in knowledge question-and-answer communities, news, advertising texts, and other data.

[0038] In one embodiment, before performing a classification operation on the document data in the document data set, the method further includes: performing a deduplication operation on the document data in the document data set based on a preset deduplication algorithm to obtain a deduplicated document data set, and the preset deduplication algorithm includes a minimum hash algorithm.

[0039] Specifically, due to the widespread dissemination and sharing of Internet information, different websites often publish similar or identical content. When crawling and storing this data, if deduplication is not performed, it will easily lead to redundancy of document data sets, thereby occupying a large amount of storage resources and may affect the efficiency of subsequent data processing and analysis.

[0040] Therefore, the minimum hash algorithm (MinHash) is used to calculate the text similarity of two document data, and similar document data is deleted based on the text similarity. First, the document data needs to be segmented using a preset segmentation tool to convert the document data into a set of words. Then, a hash value sequence is generated for each document data using the MinHash algorithm, and the minimum hash value in each sequence is recorded. By comparing the minimum hash value sequences of two document data, their Jaccard similarity is estimated. When the similarity of the two document data is greater than the preset similarity threshold, one of the document data is deleted. Deduplication of document data sets can help reduce data redundancy in document data sets and further improve the data quality of document data sets.

[0041] With respect to step S102, in one embodiment, the classification operation is performed on the document data in the document data set to determine the document category corresponding to each document data, including: based on a trained document classification model, determining the matching result between the document data in the deduplicated document data set and a preset document category, wherein the matching result includes at least one similarity probability value; based on the matching result between the document data and the preset document category, determining the document category corresponding to the document data.

[0042] In one embodiment, determining the document category corresponding to the document data based on the matching result between the document data and the preset document category includes: determining the maximum similarity probability value among the similarity probability values ​​of the document data matching each preset document category; and determining the document category corresponding to the document data based on the document category corresponding to the maximum similarity probability value.

[0043] Specifically, the document classification model is obtained by pre-training the BERT model. The BERT model (Bi di rectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture, which can capture the deep semantic information of words through bidirectional context understanding of text. In this embodiment, the BERT model is trained according to a pre-prepared data set for document classification to obtain a document classification model, the data set includes a training set and a test set, and the data set can be any document data, such as news articles, movie reviews, website posts, academic articles, conversations in knowledge question and answer communities, news, advertising texts, and other data.

[0044] Before inputting document data into the BERT model, some preprocessing of the document data is required. First, the document data is split into words using a preset word segmentation tool, and then these words are converted into an input format that the BERT model can understand. For example, the words are converted into an ID sequence that the BERT model can accept. The BERT model is trained using the training set. During the training process, the performance of the model is monitored and the training is stopped after the predetermined performance standard is reached.

[0045] Then use the test set to test the performance of the BERT model, convert the document data in the test set to the same format as the training set, and pass them to the BERT model for prediction. Calculate the accuracy of the model to evaluate the performance of the model. Use appropriate evaluation indicators (such as accuracy, recall, etc.) to evaluate the performance of the model. After the model meets the evaluation indicators, use the trained BERT model as a document classification model.

[0046] The preset document categories may include science, philosophy, education, art, politics and other categories. The document classification model is used to output the matching result between the document data and the preset document category based on the content of the document data. The matching result is the similarity probability value of the document data matching the preset document category. Specifically, the document data is segmented using a preset word segmentation tool to obtain multiple words, and then the multiple words are used as input and passed to the document classification model. The document classification model will output the similarity probability values ​​of the multiple words and each preset document category. It can be understood that the content of the document data may involve multiple fields and may have a certain degree of matching with multiple preset document categories. Therefore, the maximum similarity probability value among the similarity probability values ​​of the document data matching each preset document category is obtained, and the document category corresponding to the document data is determined based on the preset document category corresponding to the maximum similarity probability value.

[0047] With respect to step S103, in one embodiment, performing quality assessment on the document data in the document data set to determine the quality score of the document data includes: determining, based on a trained document quality assessment model, an assessment score corresponding to at least one preset assessment dimension of the document data in the deduplicated document data set; and determining the quality score of the document data based on the assessment score corresponding to at least one preset assessment dimension of the document data.

[0048] Specifically, the Bert model can be used as a basis, and the model can be trained in advance based on preset evaluation dimensions and training data to obtain a document quality assessment model. The document quality assessment model can comprehensively evaluate the document quality of the document data based on the preset evaluation dimensions, and obtain the evaluation score corresponding to the document data in at least one preset evaluation dimension, and then determine the quality score of the document data based on the evaluation score corresponding to the document data in at least one preset evaluation dimension. The quality score can predict the intrinsic value of the document.

[0049] The training process of the document quality assessment model includes the following steps:

[0050] Determine the evaluation dimensions: First, you need to determine the dimensions used to evaluate document quality. These dimensions may include but are not limited to the document's knowledge content, language coherence, and text content uniformity.

[0051] Collect training data: In order to train the document quality assessment model, it is necessary to collect training data containing annotated information. This data should include document samples and their scores on various evaluation dimensions.

[0052] Data preprocessing: Preprocess the collected document data, including word segmentation, stop word removal, stemming, etc., so as to convert it into a format that the model can process.

[0053] Model training: Use BERT as the base model and fine-tune the model through the preprocessed training data. In this process, the BERT model will be trained to predict the score of the document data in the preset evaluation dimension.

[0054] Model evaluation: Use a portion of document data that was not involved in training to evaluate the performance of the model. When the model performance meets the preset standards, a document quality evaluation model is obtained.

[0055] In one embodiment, the preset evaluation dimensions include knowledge content, language coherence and text content uniformity; determining the quality score of the document data based on the evaluation score corresponding to at least one preset evaluation dimension of the document data in the deduplicated document data set includes: obtaining the evaluation score corresponding to at least one dimension of the document data among knowledge content, language coherence and text content uniformity; determining the quality score of the document data based on the evaluation score corresponding to the document data.

[0056] Specifically, the document data is evaluated based on at least one dimension of knowledge content, language coherence, and text content uniformity to determine the evaluation scores of the document data corresponding to each preset evaluation dimension. Then, the evaluation scores of the document data corresponding to each preset evaluation dimension are combined into a comprehensive score through weighted average, summation, or other statistical methods, and the comprehensive score is used as the quality score of the document data.

[0057] With respect to step S104, in one embodiment, the determining whether the document data meets the filtering conditions is based on the document category and quality score corresponding to the document data; if so, filtering the document data, including: obtaining a preset filtering threshold of the document category based on the document category corresponding to the document data; determining whether the quality score of the document data is less than the preset filtering threshold of the document category; if so, deleting the document data.

[0058] Specifically, since the proportion of document data in each document category is different and the quality scores of document data in different document categories are unevenly distributed, a preset filtering threshold corresponding to each document category is set according to the document category. The corresponding preset filtering threshold is determined based on the document category corresponding to the document data, and it is determined whether the quality score of the document data is less than the preset filtering threshold. If so, it means that the document data belongs to low-quality data of the document category, and the document data is deleted.

[0059] See attached Figure 2 , Figure 2 FIG. 1 is a detailed flow chart of a method for filtering document data according to an embodiment of the present application. Figure 2 As shown, in this embodiment, the document data filtering method includes the following steps:

[0060] Step S201: Acquire a document data set, where the document data set includes a plurality of document data.

[0061] Step S202: Based on a preset deduplication algorithm, a deduplication operation is performed on the document data in the document data set to obtain a deduplication document data set.

[0062] Step S203: Based on the trained document classification model, determine the document category corresponding to a document data in the document data set after deduplication.

[0063] Step S204: Based on the trained document quality assessment model, determine the assessment score corresponding to the document data in at least one preset assessment dimension.

[0064] Step S205: Determine a quality score of the document data based on an evaluation score of the document data corresponding to at least one preset evaluation dimension.

[0065] Step S206: Based on the document category corresponding to the document data, a preset filtering threshold corresponding to the document category is obtained.

[0066] Step S207: determine whether the quality score of the document data is less than a preset filtering threshold corresponding to the document category; if so, execute step S208; if not, continue to execute step S203 for the next document data.

[0067] Step S208: performing a deletion operation on the document data, and continuing to perform a step of determining a document category corresponding to the document data on the next document data.

[0068] Another aspect of the present application also provides a large language model training method.

[0069] See attached Figure 3 , Figure 3 : is a flowchart of the main steps of a large language model training method according to an embodiment of the present application; in this embodiment, the large language model training method includes:

[0070] Step S301: obtaining a document data set filtered by the document data filtering method described in any of the above embodiments.

[0071] Step S302: training a large language model based on the filtered document data set until the large language model converges.

[0072] Specifically, the document dataset is mainly used to train the large language model and is the data support in the model learning and optimization process. Obtain a document dataset filtered by the above-mentioned document data filtering method, and use the filtered document dataset to train the large language model, set a preset deviation value, and complete the large language model training when the large language model converges to the preset deviation value. This application improves the problem of poor training data quality during the training of the large language model, ensures that high-quality training data is used for model training, and thus improves the training efficiency and model performance of the large language model.

[0073] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

[0074] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0075] Another aspect of the present application provides an electronic device.

[0076] See attached Figure 4 , Figure 4 exemplarily shows that the processor 11 and the memory 12 are connected in communication via a bus. In an embodiment of an electronic device according to the present application, the electronic device may include at least one processor 11; and a memory 12 connected in communication with the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented.

[0077] Furthermore, the present application also provides a computer-readable storage medium.

[0078] In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium may be configured to store a program for executing the document data filtering method of the above method embodiment, and the program may be loaded and run by a processor to implement the above document data filtering method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0079] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present application, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0080] It is understood by those skilled in the art that each module in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present application, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present application.

[0081] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0082] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0083] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0084] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A document data filtering method, characterized in that: The method comprises: Acquire a document data set, wherein the document data set includes a plurality of document data; Performing a classification operation on the document data in the document data set to determine the document category corresponding to each document data; Performing quality assessment on the document data in the document data set to determine a quality score of the document data; Based on the document category and quality score corresponding to the document data, it is determined whether the document data meets the filtering condition; if so, the document data is filtered.

2. The document data filtering method according to claim 1, characterized in that: Before performing a classification operation on the document data in the document data set, the method further includes: Based on a preset deduplication algorithm, a deduplication operation is performed on the document data in the document data set to obtain a deduplicated document data set, wherein the preset deduplication algorithm includes a minimum hash algorithm.

3. The document data filtering method according to claim 2, characterized in that: The performing a classification operation on the document data in the document data set to determine the document category corresponding to each document data includes: Based on the trained document classification model, determining the matching result between the document data in the deduplicated document data set and the preset document category, wherein the matching result includes at least a similarity probability value; Based on the matching result between the document data and the preset document category, the document category corresponding to the document data is determined.

4. The document data filtering method according to claim 3, characterized in that: The determining, based on the matching result between the document data and the preset document category, the document category corresponding to the document data includes: Determine the maximum similarity probability value among the similarity probability values ​​that match the document data with each preset document category; The document category corresponding to the document data is determined based on the document category corresponding to the maximum similarity probability value.

5. The document data filtering method according to claim 2, characterized in that: The step of performing quality assessment on the document data in the document data set to determine the quality score of the document data includes: Based on the trained document quality assessment model, determining the assessment scores corresponding to the document data in the deduplicated document data set in at least one preset assessment dimension; A quality score of the document data is determined based on an evaluation score corresponding to at least one preset evaluation dimension of the document data.

6. The document data filtering method according to claim 5, characterized in that: The preset evaluation dimensions include at least one of knowledge content, language coherence, and text content unity; The determining the quality score of the document data based on the evaluation score of the document data corresponding to at least one preset evaluation dimension includes: Obtaining an evaluation score corresponding to at least one dimension of the document data among knowledge content, language coherence, and text content uniformity; A quality score of the document data is determined based on the evaluation score corresponding to the document data.

7. The document data filtering method according to claim 1, characterized in that: determining whether the document data meets the filtering condition based on the document category and quality score corresponding to the document data; If yes, the document data is filtered, including: Based on the document category corresponding to the document data, obtaining a preset filtering threshold of the document category; Determining whether the quality score of the document data is less than a preset filtering threshold of the document category; If so, a deletion operation is performed on the document data.

8. A large language model training method, characterized in that: The method comprises: Acquire a document data set filtered by the document data filtering method according to any one of claims 1 to 7; The large language model is trained based on the filtered document data set until the large language model converges.

9. An electronic device comprising at least one processor and at least one memory, wherein the memory is suitable for storing a plurality of program codes, wherein: The program code is suitable for being loaded and run by the processor to execute the document data filtering method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the document data filtering method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Work scoring method and device, storage medium and terminal equipment

    CN109657144A

  • Text data filtering method and device and medium

    CN110347934A

  • Media file processing method and device

    CN110569377A

  • Text quality detection method and device, electronic equipment and storage medium

    CN117592456A

  • Document quality measurement

    US20140147048A1