Text corpus screening method and device, electronic equipment and storage medium

Abstract

The invention provides a text corpus screening method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a plurality of first text corpora; screening out a plurality of second text corpora of which the corpus quality meets a quality condition from the plurality of first text corpora; screening out a plurality of third text corpora of which the fact knowledge in the corpora belongs to a target subject from the plurality of second text corpora, the target subject being related to a model task of the target large language model; a plurality of target text corpora of which the knowledge hierarchy to which the fact knowledge belongs in the corpora meets the hierarchy condition are screened out from the plurality of third text corpora, the knowledge hierarchy is used for indicating the depth of the fact knowledge in the corpora, and the hierarchy condition comprises the standard that the depth of the fact knowledge in the third text corpora needs to reach; the multiple target text corpora are used for training the target large language model. According to the method, high-quality text corpora can be accurately screened out, the effect is better, and the performance of the target large language model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device, and storage medium for screening text corpus. Background Art

[0002] With the development of artificial intelligence technology, large language models have emerged. The performance of large language models depends largely on the text corpus used during training. Since the majority of text corpora used for training come from multiple sources, they are often contaminated with noise such as advertisements and web page tags, which affects model performance. Therefore, selecting high-quality text corpora is a key research focus in this field.

[0003] In related technologies, a common approach is to train a corpus screening model based on labeled sample corpora. Since each sample corpus's label indicates whether it contains noise, the corpus screening model can learn features from noise-free, high-quality sample corpora. The trained corpus screening model is then used to identify text corpora obtained from a multi-source network and select high-quality text corpora to train a large language model. The corpus screening model typically uses other advanced large language models that have already been trained.

[0004] However, in the above technical solution, the text corpus is only screened by identifying whether it contains noise, and the text corpus without noise may still have quality problems such as lack of substantive information and unclear logic. That is, the quality of the text corpus screened by the above method is still low, affecting the performance of the large language model. Summary of the Invention

[0005] The present disclosure provides a method, device, electronic device, and storage medium for screening text corpora, which can accurately screen out high-quality text corpora, i.e., achieve better screening results and help improve the performance of a target large language model. The technical solutions of the present disclosure are as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a method for screening text corpus is provided, comprising:

[0007] Acquire multiple first text corpora;

[0008] Selecting, from the plurality of first text corpora, a plurality of second text corpora whose corpora meet quality conditions, wherein the quality conditions include a standard to be met for the content of factual knowledge in the text corpora;

[0009] Filtering, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to a target discipline, the target discipline being relevant to a model task of a target large language model;

[0010] From the multiple third text corpora, multiple target text corpora are screened out, whose knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy conditions, wherein the knowledge hierarchy is used to indicate the depth of the factual knowledge in the corpus, and the hierarchy conditions include the standards to be achieved for the depth of the factual knowledge in the third text corpora, and the multiple target text corpora are used to train the target large language model.

[0011] According to another aspect of an embodiment of the present disclosure, a device for screening text corpus is provided, comprising:

[0012] An acquiring unit, configured to acquire a plurality of first text corpora;

[0013] A first screening unit is configured to screen out, from the plurality of first text corpora, a plurality of second text corpora whose corpora meet a quality condition, wherein the quality condition includes a standard to be met for the content of factual knowledge in the text corpora;

[0014] a second screening unit configured to screen out, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to a target subject, the target subject being related to a model task of a target large language model;

[0015] The third screening unit is configured to screen out, from the multiple third text corpora, multiple target text corpora whose knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy conditions, wherein the knowledge hierarchy is used to indicate the depth of the factual knowledge in the corpora, and the hierarchy conditions include the standards to be achieved for the depth of the factual knowledge in the third text corpora, and the multiple target text corpora are used to train the target large language model.

[0016] In some embodiments, the first screening unit is configured to perform a quality screening model to screen out a plurality of second text corpora whose corpus quality meets the quality condition from the plurality of first text corpora;

[0017] The second screening unit is configured to perform a subject screening model to screen out a plurality of third text corpora containing factual knowledge belonging to a target subject from the plurality of second text corpora;

[0018] The third screening unit is configured to execute a knowledge hierarchy screening model to screen out a plurality of target text corpora from the plurality of third text corpora, wherein the knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy condition, and the plurality of target text corpora are used to train the large language model;

[0019] Among them, the total number of model parameters of the quality screening model, the subject screening model and the knowledge level screening model is less than a preset value.

[0020] In some embodiments, the first screening unit comprises:

[0021] a first processing subunit configured to process any first text corpus among the plurality of first text corpora using a quality screening model to obtain a quality score of the first text corpus, where the quality score is used to indicate the quality of the first text corpus;

[0022] The first screening subunit is configured to screen out a plurality of second text corpora whose quality scores meet the quality condition from the plurality of first text corpora based on the quality scores of the plurality of first text corpora.

[0023] In some embodiments, the quality condition further includes at least one of a first indicator and a second indicator, wherein the first indicator is used to indicate the standard to be achieved by the expression style of the text corpus, and the second indicator is used to indicate the standard to be achieved by the educational background of the text corpus;

[0024] The first processing subunit is configured to process any first text corpus among the plurality of first text corpora using a quality screening model to obtain at least one of a first quality score and a second quality score of the first text corpus and a third quality score, wherein the first quality score is used to indicate an expression style of the first text corpus, the second quality score is used to indicate an educational background of the first text corpus, and the third quality score is used to indicate a content of factual knowledge in the first text corpus;

[0025] The screening subunit is configured to screen out the plurality of second text corpora from the plurality of first text corpora, wherein at least one of the first quality score and the second quality score and the third quality score of each second text corpus meet respective indicators in the quality conditions.

[0026] In some embodiments, the apparatus further comprises:

[0027] The first training unit is configured to obtain multiple sample corpus pairs, each sample corpus pair includes a first sample corpus, a second sample corpus and a reference confidence of each sample corpus, the reference confidence of each sample corpus is used to indicate the quality of the sample corpus compared with the quality of another sample corpus in the corpus pair; for each sample corpus pair in the multiple sample corpus pairs, the quality screening model is used to process the sample corpus pair to obtain the prediction confidence of each sample corpus in the sample corpus pair, the prediction confidence of each sample corpus is used to indicate the quality of the sample corpus predicted by the model compared with the quality of another sample corpus in the corpus pair; and the quality screening model is trained based on the reference confidence and prediction confidence of the multiple sample corpus pairs.

[0028] In some embodiments, the reference confidence of each sample corpus in each sample corpus pair is determined based on a first sample annotation model, where the first sample annotation model belongs to a large language model.

[0029] In some embodiments, the second screening unit is configured to perform classification of any second text corpus among the multiple second text corpora through the subject screening model, determine the subject classification result corresponding to the second text corpus, and the subject classification result is used to indicate the subject to which the factual knowledge in the second text corpus belongs; based on the subject classification results corresponding to the multiple second text corpora, screen out the multiple third text corpora whose factual knowledge in the corpus belongs to the target subject from the multiple second text corpora.

[0030] In some embodiments, the apparatus further comprises:

[0031] The second training unit is configured to obtain multiple third sample corpora, each third sample corpus includes a reference subject, and the reference subject of each third sample corpus is used to indicate the subject to which the factual knowledge in the third sample corpus belongs; for each third sample corpus in the multiple third sample corpora, the third sample corpus is processed by the subject screening model to obtain a predicted subject of the third sample corpus, and the predicted subject is used to indicate the subject to which the factual knowledge in the third sample corpus predicted by the model belongs; and the subject screening model is trained based on the reference subjects and predicted subjects of the multiple third sample corpora.

[0032] In some embodiments, the plurality of third sample corpora are screened based on the trained quality screening model, and the quality of the plurality of third sample corpora meets the quality condition.

[0033] In some embodiments, the third screening unit comprises:

[0034] a second processing subunit configured to process any third text corpus among the plurality of third text corpora using the knowledge hierarchy screening model to determine the knowledge hierarchy to which factual knowledge in the third text corpus belongs;

[0035] The second screening sub-unit is configured to perform, based on the knowledge level to which the factual knowledge in the multiple third text corpora belongs, screening out the multiple target text corpora from the multiple third text corpora, where the knowledge level to which the factual knowledge in the multiple target text corpora reaches the level threshold in the level condition.

[0036] In some embodiments, the second processing sub-unit is configured to execute matching of any third text corpus among the multiple third text corpora with multiple hierarchical rules through the knowledge hierarchy screening model; determine the hierarchy score of the third text corpus based on the number of target hierarchy rules in the multiple hierarchical rules, the target hierarchy rule is the hierarchy rule in the multiple hierarchical rules that matches the third text corpus, and the hierarchy score is positively correlated with the number of the target hierarchy rules; determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs based on the hierarchy score of the third text corpus, and the knowledge hierarchy is positively correlated with the hierarchy score.

[0037] In some embodiments, the apparatus further comprises:

[0038] The third training unit is configured to obtain multiple fourth sample corpora, each fourth sample corpus includes a reference level score, and the reference level score of each fourth sample corpus is used to indicate the knowledge level to which the factual knowledge in the fourth sample corpus belongs; for each fourth sample corpus in the multiple fourth sample corpora, the fourth sample corpus is processed by the knowledge level screening model to obtain a predicted level score of the fourth sample corpus, and the predicted level score is used to indicate the level score predicted by the model to which the factual knowledge in the fourth sample corpus belongs; based on the reference level scores and predicted level scores of the multiple fourth sample corpora, the knowledge level screening model is trained.

[0039] In some embodiments, the multiple fourth sample corpora are screened based on the trained quality screening model and the discipline screening model, the quality of the multiple fourth sample corpora meets the quality conditions, and the factual knowledge in the multiple fourth sample corpora belongs to the target discipline.

[0040] In some embodiments, the reference level scores of the plurality of fourth sample corpora include at least one of the following:

[0041] The reference level score of each fourth sample corpus is determined based on the second sample annotation model, where the second sample annotation model belongs to the large language model;

[0042] The reference level score of each fourth sample corpus is positively correlated with the number of level rules in the plurality of level rules that match the fourth sample corpus.

[0043] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device including:

[0044] one or more processors;

[0045] a memory for storing program codes executable by the processor;

[0046] The processor is configured to execute the program code to implement the above-mentioned text corpus screening method.

[0047] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned method for screening text corpora.

[0048] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the above-mentioned text corpus screening method when executed by a processor.

[0049] The solution provided by the embodiment of the present disclosure is that, for the multiple first text corpora initially acquired, first, based on the content of factual knowledge in the first text corpora, multiple second text corpora with factual knowledge content that meets the requirements are screened out from the multiple first text corpora, and then, based on the discipline to which the factual knowledge in the second text corpora belongs, multiple third text corpora whose factual knowledge belongs to the target discipline are screened out from the multiple second text corpora; finally, based on the level to which the factual knowledge in the third text corpora belongs, multiple target text corpora whose factual knowledge depth meets the level conditions are screened out from the multiple third text corpora, thereby ensuring that the multiple target text corpora finally screened out for model training not only contain factual knowledge, but also have discipline factual knowledge related to the model task, and the factual knowledge contained has sufficient depth, which is equivalent to having sufficient educational value or significance and is worthy of model learning. Therefore, model training through multiple target text corpora can significantly improve model performance. That is, from the perspective of the information required for model training, this solution breaks down the complex high-quality features in the text corpus into three levels: "whether it contains factual knowledge", "whether it contains the subject factual knowledge required for model training", and "whether the factual knowledge contained is deep enough to be worth learning". This can fully explore the characteristics of the text corpus and screen it layer by layer, so as to accurately screen out truly high-quality text corpora, that is, the screening effect is better, and then the model can be trained through high-quality text corpora, which is conducive to improving the performance of the model.

[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0052] Figure 1 The figure is a schematic diagram of an implementation environment of a method for screening text corpus according to an exemplary embodiment.

[0053] Figure 2 The figure is a flowchart of a method for screening text corpus according to an exemplary embodiment.

[0054] Figure 3 The figure is a flowchart of another method for screening text corpus according to an exemplary embodiment.

[0055] Figure 4 The figure is a schematic diagram showing a method of filtering text corpus according to an exemplary embodiment.

[0056] Figure 5The figure is a block diagram of a device for screening text corpus according to an exemplary embodiment.

[0057] Figure 6 It is a block diagram of a terminal according to an exemplary embodiment.

[0058] Figure 7 The figure is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0060] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0061] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the text corpus involved in this disclosure was obtained with full authorization.

[0062] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a method for screening text corpus according to an exemplary embodiment. Taking the electronic device as an example, see Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102.

[0063] The terminal 101 is at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player, an MP4 player, and a portable laptop computer. An application that supports obtaining text corpora is running on the terminal 101. The application can be a browser, a social application, a document management application, or an information application, etc., which is not limited in the embodiments of the present disclosure. A user can log in to the application through the terminal 101 to obtain the services provided by the application. The terminal 101 can be connected to the server 102 via a wireless network or a wired network, and can then send all collected text corpora to the server 102, which then filters out the applications required for large language model training.

[0064] Terminal 101 generally refers to one of multiple terminals. This embodiment uses terminal 101 as an example. Those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be a few terminals, or dozens, hundreds, or even more. This embodiment does not limit the number or device type of terminals.

[0065] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. The server 102 can receive multiple text corpora sent by the terminal 101, and filter out high-quality text corpora from the multiple text corpora so that the target large language model can be trained with high-quality text corpora in the future to improve the performance of the target large language model. In some embodiments, the number of the above-mentioned servers can be more or less, and the embodiments of the present disclosure are not limited to this. Of course, the server 102 also includes other functional servers to provide more comprehensive and diversified services.

[0066] Figure 2 is a flowchart of a method for screening text corpus according to an exemplary embodiment. Figure 2 As shown, the text corpus screening method is applied to the server and includes the following steps:

[0067] In step 201, the server obtains a plurality of first text corpora.

[0068] In the embodiments of the present disclosure, the sources of the multiple first text corpora may be the same or different, and the embodiments of the present application do not limit this. The first text corpora may be news corpora (i.e., texts from news media, such as newspapers, news websites, etc.), literary corpora (e.g., novels, plays, poems, etc.), social media corpora (i.e., colloquial texts on social media platforms), or professional field corpora (e.g., professional literature in the fields of medicine, law, science, etc.), and the embodiments of the present disclosure do not limit this.

[0069] In step 202, the server selects a plurality of second text corpora from the plurality of first text corpora, the corpora of which meet the quality conditions, where the quality conditions include the standards to be met for the content of factual knowledge in the text corpora.

[0070] In the embodiment of the present disclosure, for any first text corpus among the multiple first text corpora, the server obtains the quality of the first text corpus. The embodiment of the present disclosure does not limit the method for obtaining the quality of the text corpus. Then, based on the quality of the multiple first text corpora and the quality conditions, the server selects multiple second text corpora from the multiple first text corpora.

[0071] The quality of the first text corpus includes the content of factual knowledge in the first text corpus. Factual knowledge in a text corpus refers to specific information or data whose authenticity can be verified through scientific methods such as observation, experiment, or logical reasoning. The content of factual knowledge refers to the proportion of the number of characters used to describe factual knowledge in the first text corpus to the total content of the first text corpus. The content of factual knowledge in each second text corpus meets the knowledge content index in the quality condition. The disclosed embodiments do not limit the knowledge content index.

[0072] In step 203, the server selects from the plurality of second text corpora a plurality of third text corpora containing factual knowledge belonging to a target subject, where the target subject is related to the model task of the target large language model.

[0073] In this embodiment of the present disclosure, for any second text corpus among the plurality of second text corpora, the server determines the discipline to which the factual knowledge in the second text corpus belongs. This embodiment of the present disclosure does not limit the method for determining the discipline to which the factual knowledge belongs. Then, based on the disciplines to which the factual knowledge in the plurality of second text corpora belongs, the server obtains a plurality of third text corpora from the plurality of second text corpora. The factual knowledge in the plurality of third text corpora belongs to the target discipline.

[0074] Among them, the target discipline can be mathematics, physics, science or literature, etc., which is not limited in the embodiment of the present disclosure. The server can determine the target discipline based on the model task of the target large language model, thereby screening out multiple third text corpora from multiple second text corpora whose factual knowledge in the corpora belongs to the target discipline. For example, if the model task of the target large language model is to logically answer the user's scientific and technological questions, the target discipline can include mathematics, physics, science and other disciplines that focus on logic. Since the target discipline is related to the model task of the target large language model to be trained, the text corpora screened out subsequently for training the target large language model are guaranteed, which can improve the performance of the target large language model in processing model tasks.

[0075] In step 204, the server selects a plurality of target text corpora from the plurality of third text corpora, wherein the knowledge level to which the factual knowledge in the corpora belongs meets the hierarchical conditions. The knowledge level is used to indicate the depth of the factual knowledge in the corpora, and the hierarchical conditions include the standard to be achieved in the depth of the factual knowledge in the third text corpora. The plurality of target text corpora are used to train the target large language model.

[0076] In an embodiment of the present disclosure, for any third text corpus among multiple third text corpora, the server determines the knowledge level to which the factual knowledge in the third text corpus belongs. The embodiment of the present disclosure does not limit the method for determining the knowledge level to which the factual knowledge belongs. Then, based on the knowledge level and level conditions to which the factual knowledge in the multiple third text corpora belongs, the server obtains multiple target text corpora from the multiple third text corpora. The knowledge level to which the factual knowledge in the multiple target text corpora meets the knowledge level indicator in the level condition. The embodiment of the present disclosure does not limit the knowledge level indicator.

[0077] The knowledge level of the factual knowledge in the text corpus can include multiple levels, such as elementary school, junior high school, high school, and university. The knowledge level indicator can be high school level. Accordingly, the server selects multiple target text corpora from the multiple third text corpora in which the factual knowledge in the corpora reaches the high school level. In other words, the factual knowledge in the multiple target text corpora is at the high school level or above.

[0078] The disclosed embodiment provides a method for screening text corpora. For a plurality of first text corpora initially acquired, first, based on the content of factual knowledge in the first text corpora, a plurality of second text corpora with factual knowledge content meeting the standard are screened out from the plurality of first text corpora. Then, based on the discipline to which the factual knowledge in the second text corpora belongs, a plurality of third text corpora containing factual knowledge belonging to a target discipline are screened out from the plurality of second text corpora. Finally, based on the level to which the factual knowledge in the third text corpora belongs, a plurality of target text corpora with factual knowledge depth meeting the level conditions are screened out from the plurality of third text corpora. This ensures that the plurality of target text corpora ultimately screened out for model training not only contain factual knowledge, but also have disciplinary factual knowledge related to the model task, and the factual knowledge contained has sufficient depth, which is equivalent to having sufficient educational value or significance and is worthy of model learning. Therefore, model training through a plurality of target text corpora can significantly improve model performance. That is, from the perspective of the information required for model training, this solution breaks down the complex high-quality features in the text corpus into three levels: "whether it contains factual knowledge", "whether it contains the subject factual knowledge required for model training", and "whether the factual knowledge contained is deep enough to be worth learning". This can fully explore the characteristics of the text corpus and screen it layer by layer, so as to accurately screen out truly high-quality text corpora, that is, the screening effect is better, and then the model can be trained through high-quality text corpora, which is conducive to improving the performance of the model.

[0079] In some embodiments, selecting a plurality of second text corpora whose corpus quality meets a quality condition from a plurality of first text corpora includes:

[0080] Using a quality screening model, screening out a plurality of second text corpora whose corpus quality meets a quality condition from the plurality of first text corpora;

[0081] From the multiple second text corpora, multiple third text corpora containing factual knowledge belonging to the target discipline are selected, including:

[0082] Using the subject screening model, multiple third text corpora containing factual knowledge belonging to the target subject are screened from multiple second text corpora;

[0083] From the plurality of third text corpora, a plurality of target text corpora are selected, wherein the knowledge level of the factual knowledge in the corpora meets the level conditions, including:

[0084] Using the knowledge hierarchy screening model, multiple target text corpora are screened from multiple third text corpora, where the knowledge hierarchy of the factual knowledge in the corpora meets the hierarchy conditions. The multiple target text corpora are used to train the large language model.

[0085] Among them, the total number of model parameters of the quality screening model, the discipline screening model and the knowledge level screening model is less than the preset value.

[0086] The solution provided by the embodiment of the present disclosure adopts the form of a pipeline composed of a quality screening model, a subject screening model and a knowledge hierarchy screening model to perform screening in sequence. First, the second text corpus containing factual knowledge is screened out by the quality screening model, and then the third text corpus related to the subject to which the factual knowledge belongs and the model task is screened out from the screening results of the quality screening model by the subject screening model. Finally, the target text corpus with factual knowledge depth that meets the standard is screened out from the screening results of the subject screening model by the knowledge hierarchy screening model, thereby ensuring that truly high-quality text corpora can be accurately screened out, that is, the screening effect is better. On this basis, since the total number of model parameters of the quality screening model, the subject screening model and the knowledge hierarchy screening model is less than the preset value, that is, the scale of each of the above models is small, compared with directly adopting other trained advanced large language models, the inference time of each of the above small models will be shorter, thereby improving the screening efficiency while ensuring the quality screening effect.

[0087] In some embodiments, a quality screening model is used to screen out a plurality of second text corpora whose corpus quality meets the quality conditions from a plurality of first text corpora, including:

[0088] For any first text corpus among the multiple first text corpora, the first text corpus is processed using the quality screening model to obtain a quality score of the first text corpus, where the quality score is used to indicate the quality of the first text corpus;

[0089] Based on the quality scores of the plurality of first text corpora, a plurality of second text corpora whose quality scores meet the quality condition are screened out from the plurality of first text corpora.

[0090] The solution provided by the embodiment of the present disclosure determines the quality score of each first text corpus through a quality screening model, thereby screening out a second text corpus with a qualified quality score from multiple first text corpora. Since the quality score can reflect the quality of the first text corpus, that is, the quality score can accurately reflect the content of factual knowledge in the quality of the first text corpus, it is ensured that the screened second text corpora are texts with sufficient factual knowledge, thereby ensuring the quality of the text corpora subsequently screened out for model training, which is conducive to improving model performance.

[0091] In some embodiments, the quality condition further includes at least one of a first indicator and a second indicator, the first indicator being used to indicate the standard to be achieved by the expression style of the text corpus, and the second indicator being used to indicate the standard to be achieved by the educational background of the text corpus;

[0092] For any first text corpus among the multiple first text corpora, the first text corpus is processed using the quality screening model to obtain a quality score of the first text corpus, including:

[0093] For any first text corpus among the plurality of first text corpora, processing the first text corpus using the quality screening model to obtain at least one of a first quality score and a second quality score and a third quality score for the first text corpus, wherein the first quality score is used to indicate the expression style of the first text corpus, the second quality score is used to indicate the educational background of the first text corpus, and the third quality score is used to indicate the content of factual knowledge in the first text corpus;

[0094] Based on the quality scores of the plurality of first text corpora, a plurality of second text corpora having quality scores meeting a quality condition are screened out from the plurality of first text corpora, including:

[0095] A plurality of second text corpora are screened out from the plurality of first text corpora, wherein at least one of the first quality score and the second quality score and the third quality score of each second text corpus meet respective indicators in the quality condition.

[0096] The solution provided by the embodiment of the present disclosure can, in addition to considering the factual knowledge content in the text corpus, also consider at least one of the writing style and educational background of the text corpus during the quality screening process. The corpus screening is performed based on the quality scores of at least one of the writing style and educational background of the text corpus and each dimension of the factual knowledge content, so as to screen out text corpora that meet the standards in at least one of the writing style and educational background and each dimension of the factual knowledge content, thereby further improving the quality of the screened text corpus and achieving a better quality screening effect.

[0097] In some embodiments, the training process of the quality screening model includes:

[0098] Acquire multiple sample corpus pairs, each sample corpus pair including a first sample corpus, a second sample corpus, and reference confidence scores of each sample corpus, wherein the reference confidence score of each sample corpus is used to indicate the quality of the sample corpus compared to the quality of another sample corpus in the corpus pair;

[0099] For each of the multiple sample corpus pairs, the sample corpus pair is processed by the quality screening model to obtain a prediction confidence score for each sample corpus in the sample corpus pair. The prediction confidence score for each sample corpus is used to indicate the quality of the sample corpus predicted by the model compared to the quality of another sample corpus in the corpus pair.

[0100] The quality screening model is trained based on the reference confidence and prediction confidence of multiple sample corpora.

[0101] The solution provided by the embodiments of the present disclosure uses paired sample corpora to train the quality screening model, so that the quality screening model can determine the prediction confidence of each text corpus by comparing the gaps between the text corpora. The quality screening model is trained based on the prediction confidence and reference confidence of each text corpus in each sample corpus pair, so that the prediction confidence of the text corpus determined by the quality screening model is increasingly close to the reference confidence in the actual situation, that is, the performance of the quality screening model in determining the quality of the text corpus is improved, so that the quality screening model can more accurately screen out high-quality corpus texts.

[0102] In some embodiments, the reference confidence of each sample corpus in each sample corpus pair is determined based on a first sample annotation model, and the first sample annotation model belongs to a large language model.

[0103] The solution provided by the embodiments of the present disclosure uses a large language model to label paired sample corpora with reference confidence levels for the sample corpora used in the training process of the quality screening model. This not only saves labor costs, but also compares and labels paired sample corpora, which can make the reference confidence levels determined by the large language model more consistent, thereby obtaining more accurate and stable labeling results, greatly improving the labeling quality of the training data of the quality screening model, and further helping to improve the performance of the quality screening model so that the quality screening model can more accurately screen out high-quality corpus texts.

[0104] In some embodiments, a subject screening model is used to screen out a plurality of third text corpora containing factual knowledge belonging to a target subject from a plurality of second text corpora, including:

[0105] For any second text corpus among the plurality of second text corpora, classify the second text corpus using the subject screening model to determine a subject classification result corresponding to the second text corpus, where the subject classification result is used to indicate the subject to which the factual knowledge in the second text corpus belongs;

[0106] Based on the subject classification results corresponding to the plurality of second text corpora, a plurality of third text corpora in which the factual knowledge in the corpora belongs to the target subject are screened out from the plurality of second text corpora.

[0107] The solution provided by the embodiments of the present disclosure determines the discipline to which the factual knowledge in each second text corpus belongs through a discipline screening model, thereby screening out multiple third text corpora whose factual knowledge belongs to the target discipline from multiple second text corpora. Since the target discipline is related to the model task of the target large language model to be trained, the text corpora subsequently screened out for training the target large language model are guaranteed, which can improve the performance of the target large language model in processing model tasks, that is, it can improve the quality of samples used for model training.

[0108] In some embodiments, the training process of the subject screening model includes:

[0109] Acquire a plurality of third sample corpora, each of the third sample corpora including a reference subject, wherein the reference subject of each third sample corpus is used to indicate the subject to which the factual knowledge in the third sample corpus belongs;

[0110] For each of the plurality of third sample corpora, the subject screening model is used to process the third sample corpora to obtain a predicted subject for the third sample corpora, where the predicted subject indicates the subject to which the factual knowledge in the third sample corpora, as predicted by the model, belongs;

[0111] Based on the reference subjects and predicted subjects of multiple third-party sample corpora, a subject screening model is trained.

[0112] The solution provided by the embodiment of the present disclosure uses sample corpora containing reference disciplines to train the discipline screening model, so that the reference discipline of each sample corpus can be used as a label to guide the discipline screening model to gradually tend towards the reference label for the discipline to which the factual knowledge in the sample corpus belongs. That is, the predicted discipline of the text corpus determined by the discipline screening model is increasingly close to the reference discipline in the actual situation, thereby improving the accuracy of the discipline screening model in identifying the discipline to which the factual knowledge in the text corpus belongs, so that the text corpus whose factual knowledge belongs to the target discipline can be accurately screened out subsequently, thereby improving the screening effect.

[0113] In some embodiments, the plurality of third sample corpora are obtained by screening based on a trained quality screening model, and the quality of the plurality of third sample corpora meets the quality condition.

[0114] The solution provided by the embodiment of the present disclosure ensures that the quality of the third sample corpus used for training the subject screening model is high, because the third sample corpus used is obtained by screening the trained quality screening model. The subject screening model is trained on this basis, which facilitates the subject screening model to better learn the subject to which the factual knowledge in the text corpus belongs, and is beneficial to improving the performance of the subject screening model, so that the text corpus with factual knowledge belonging to the target subject can be accurately screened out in the future, thereby improving the screening effect.

[0115] In some embodiments, a knowledge hierarchy screening model is used to screen out a plurality of target text corpora from a plurality of third text corpora, wherein the knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy conditions, including:

[0116] For any third text corpus among the plurality of third text corpora, the third text corpus is processed by the knowledge hierarchy screening model to determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs;

[0117] Based on the knowledge level to which the factual knowledge in the plurality of third text corpora belongs, a plurality of target text corpora are screened out from the plurality of third text corpora, and the knowledge level to which the factual knowledge in the plurality of target text corpora reaches the level threshold in the level condition.

[0118] The solution provided by the embodiment of the present disclosure determines the knowledge level to which the factual knowledge in each third text corpus belongs through a knowledge level screening model, thereby screening out target text corpora that meet the knowledge level requirements from multiple third text corpora. Since the knowledge level reflects the depth of the factual knowledge in the third text corpora, the higher the depth of the factual knowledge, the greater the educational value or significance of the factual knowledge. By screening the target text corpora that meet the level requirements, it is ensured that the screened target text corpora all have sufficient educational value or significance and are worthy of model learning. Therefore, the subsequent training of the target large language model through the target text corpus can significantly improve the model performance.

[0119] In some embodiments, for any third text corpus among the plurality of third text corpora, processing the third text corpus using a knowledge hierarchy screening model to determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs includes:

[0120] For any third text corpus among the plurality of third text corpora, matching the third text corpus with the plurality of hierarchical rules respectively through the knowledge hierarchical screening model;

[0121] determining a hierarchical score of the third text corpus based on the number of target hierarchical rules among the plurality of hierarchical rules, wherein the target hierarchical rule is a hierarchical rule among the plurality of hierarchical rules that matches the third text corpus, and the hierarchical score is positively correlated with the number of target hierarchical rules;

[0122] Based on the hierarchy score of the third text corpus, the knowledge hierarchy to which the factual knowledge in the third text corpus belongs is determined, and the knowledge hierarchy is positively correlated with the hierarchy score.

[0123] The solution provided by the embodiment of the present disclosure is to determine, for each third text corpus, the hierarchical score of the third text corpus according to the number of hierarchical rules matched by the third text corpus. The more hierarchical rules matched by the third text corpus, the higher the corresponding hierarchical score, and thus the higher the determined knowledge hierarchy. This is equivalent to using a "bonus system" to determine the knowledge hierarchy of the third text corpus. Whenever the third text corpus hits a hierarchical rule, the knowledge hierarchy of the third text corpus may be improved. This method enables the knowledge hierarchy screening model to pay more attention to the features in the third text corpus to determine whether it matches the hierarchical rules, thereby being able to more accurately determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs, and then screen out target text corpora with better quality, that is, to improve the quality of screening.

[0124] In some embodiments, the training process of the knowledge hierarchy screening model includes:

[0125] Acquire a plurality of fourth sample corpora, each fourth sample corpus including a reference level score, wherein the reference level score of each fourth sample corpus is used to indicate the knowledge level to which the factual knowledge in the fourth sample corpus belongs;

[0126] For each of the plurality of fourth sample corpora, processing the fourth sample corpora using the knowledge hierarchy screening model to obtain a predicted hierarchy score for the fourth sample corpora, the predicted hierarchy score being used to indicate a hierarchy score to which factual knowledge in the fourth sample corpora, as predicted by the model, belongs;

[0127] A knowledge hierarchy screening model is trained based on the reference hierarchy scores and the predicted hierarchy scores of the plurality of fourth sample corpora.

[0128] The solution provided by the embodiment of the present disclosure uses sample corpora containing reference hierarchical scores to train the knowledge hierarchy screening model, so that the reference hierarchical score of each sample corpus can be used as a label to guide the knowledge hierarchy screening model to gradually tend to the reference hierarchical score for the hierarchy to which the factual knowledge in the sample corpus belongs. That is, the predicted hierarchical score of the text corpus determined by the knowledge hierarchy screening model is increasingly close to the reference hierarchical score in the actual situation, thereby improving the accuracy of the knowledge hierarchy screening model in identifying the hierarchy to which the factual knowledge in the text corpus belongs, so that the text corpus that meets the knowledge hierarchy standards can be accurately screened out in the future, thereby improving the screening effect.

[0129] In some embodiments, the plurality of fourth sample corpora are screened based on a trained quality screening model and a discipline screening model, the quality of the plurality of fourth sample corpora meet the quality conditions, and the factual knowledge in the plurality of fourth sample corpora belongs to the target discipline.

[0130] The solution provided by the embodiment of the present disclosure ensures that the quality of the fourth sample corpus used for training the knowledge hierarchy screening model is high, because the fourth sample corpus used is obtained by screening the trained quality screening model and subject screening model. The knowledge hierarchy screening model is trained on this basis, which facilitates the knowledge hierarchy screening model to better learn the level to which the factual knowledge in the text corpus belongs, and is beneficial to improving the performance of the knowledge hierarchy screening model, so that the text corpus that meets the level of factual knowledge can be accurately screened out in the future, thereby improving the screening effect.

[0131] In some embodiments, the reference level scores of the plurality of fourth sample corpora include at least one of the following:

[0132] The reference level score of each fourth sample corpus is determined based on the second sample annotation model, which belongs to the large language model;

[0133] The reference level score of each fourth sample corpus is positively correlated with the number of level rules among the multiple level rules that match the fourth sample corpus.

[0134] The solution provided by the embodiments of the present disclosure uses a large language model to label the sample corpus used in the training process of the knowledge hierarchy screening model with reference hierarchy scores. This not only saves labor costs, but also facilitates the large language model's powerful understanding ability for labeling, and can ensure the accuracy of the reference hierarchy scores to a certain extent.

[0135] For each fourth sample corpus, the hierarchical score of the fourth sample corpus is determined according to the number of hierarchical rules matched by the fourth sample corpus. The more hierarchical rules matched by the fourth sample corpus, the higher the corresponding hierarchical score, and thus the higher the determined knowledge level. This is equivalent to using a "bonus system" to determine the knowledge level of the fourth sample corpus. Every time the fourth sample corpus hits a hierarchical rule, the knowledge level of the fourth sample corpus may be improved. This method makes it possible to pay more attention to the features in the fourth sample corpus to determine whether they match the hierarchical rules, thereby more accurately determining the knowledge level to which the factual knowledge in the fourth sample corpus belongs, further ensuring the accuracy of the reference hierarchical score, and thus helping to improve the performance of the knowledge level screening model.

[0136] above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 3 FIG. 1 is a flowchart of another method for screening text corpus according to an exemplary embodiment. Taking the electronic device as an example, see Figure 3 , the method comprising:

[0137] In step 301, the server obtains a plurality of first text corpora.

[0138] In an embodiment of the present disclosure, before training the target large language model, the server can screen the acquired multiple first text corpora to select high-quality text corpora to train the target large language model.

[0139] In some embodiments, the multiple first text corpora come from different sources. Accordingly, the factual knowledge contained in the multiple first text corpora and the styles of the multiple first text corpora are different. On this basis, the factual knowledge and styles contained in the text corpora subsequently selected for training the target large language model are also different, ensuring the diversity of the training data of the target large language model and helping to improve the generalization ability and robustness of the target large language model.

[0140] In step 302, the server uses a quality screening model to screen out a plurality of second text corpora whose corpus quality meets the quality conditions from the plurality of first text corpora.

[0141] In the disclosed embodiments, the quality screening model can be constructed based on the Llama3-1.3B model or the BERT model. The disclosed embodiments do not limit the architecture of the quality screening model. Optionally, the number of model parameters in the quality screening model is less than a quantity threshold. In other words, the quality screening model can be a small-scale model. Compared to directly using other trained advanced large language models, using a small-scale model to determine the quality of the text corpus and perform screening will shorten the inference time used, thereby improving screening efficiency.

[0142] In the process of screening the second text corpus, for any first text corpus among the multiple first text corpora, the server processes the first text corpus through the quality screening model to obtain the quality score of the first text corpus. The quality score is used to indicate the quality of the first text corpus. Then, based on the quality scores of the multiple first text corpora, the server screens out multiple second text corpora whose quality scores meet the quality conditions from the multiple first text corpora. The solution provided by the embodiment of the present disclosure determines the quality score of each first text corpus through the quality screening model, thereby screening out the second text corpora with qualified quality scores from the multiple first text corpora. Since the quality score can reflect the quality of the first text corpus, that is, the quality score can accurately reflect the content of factual knowledge in the quality first text corpus, it is ensured that the screened second text corpora are texts with sufficient factual knowledge, thereby ensuring the quality of the text corpora subsequently screened out for model training, which is conducive to improving model performance.

[0143] Among them, the quality score can be used to indicate the content of factual knowledge in the first text corpus. Then, the server can screen out multiple second text corpora whose factual knowledge content reaches the content index in the quality condition from multiple first text corpora. Optionally, the quality score can be the proportion of the content describing factual knowledge in the first text corpus in the first text corpus. The content index refers to the minimum proportion of the content describing factual knowledge in the second text corpus in the second text corpus. For example, the content index is 50%, and accordingly, the server screens out multiple second text corpora whose factual knowledge content reaches 50% from multiple first text corpora. The embodiment of the present disclosure does not limit the content index.

[0144] In some embodiments, in addition to the standards required for the content of factual knowledge in the text corpus, the quality conditions may also include at least one of a first indicator and a second indicator. The first indicator indicates the standards required for the text corpus's expression style. The second indicator indicates the standards required for the text corpus's educational background. Accordingly, the quality score determined by the quality screening model includes at least one of the first and second quality scores and a third quality score. The first quality score indicates the expression style (writing style) of the first text corpus. A higher first quality score indicates a more standardized expression style (or more rigorous, more accurate wording, or clearer logic) of the first text corpus. The second quality score indicates the educational background of the first text corpus. A higher second quality score indicates a higher level of education in the first text corpus. Educational background levels can be divided, from low to high, into multiple levels, such as elementary school, junior high school, high school, technical secondary school, undergraduate, and graduate school, but this is not limited in the present embodiment. The third quality score indicates the content of factual knowledge in the first text corpus. A higher third quality score indicates a higher level of factual knowledge in the first text corpus.

[0145] Specifically, for any first text corpus among the multiple first text corpora, the server processes the first text corpus through the quality screening model to obtain the first quality score and at least one of the second quality score and the third quality score of the first text corpus. Then, the server screens out multiple second text corpora from the multiple first text corpora. The first quality score, at least one of the second quality score and the third quality score of each second text corpus meet the respective indicators in the quality conditions. The indicators for the expression style, educational background and factual knowledge content in the quality conditions can be the corresponding score thresholds, and the embodiments of the present disclosure do not limit the various score thresholds. For example, the score thresholds corresponding to the expression style, educational background and factual knowledge content are 0.5, then the server can screen out multiple second text corpora whose first quality score, second quality score and third quality score are all greater than 0.5 from the multiple first text corpora. The solution provided by the embodiment of the present disclosure can, in addition to considering the factual knowledge content in the text corpus, also consider at least one of the writing style and educational background of the text corpus during the quality screening process. The corpus screening is performed based on the quality scores of at least one of the expression style and educational background of the text corpus and each dimension of the factual knowledge content, so as to screen out text corpora that meet the standards in at least one of the writing style and educational background and each dimension of the factual knowledge content, thereby further improving the quality of the screened text corpus and achieving a better quality screening effect.

[0146] Among them, the educational background of the first text corpus can be determined based on information from multiple dimensions such as the educational background of the author of the first text corpus, the educational purpose of the first text corpus, the educational value of the first text corpus, the educational environment in which the first text corpus was produced, and the educational audience of the first text corpus, which is not limited in this embodiment of the present disclosure. The author's educational background refers to the author's level of education, professional training, and academic background. These factors may affect the author's writing style, content selection, and expression of opinions. The educational purpose refers to whether the first text corpus is intended to educate readers, convey factual knowledge or skills. Educational value refers to the educational significance and inspiration of the first text corpus to readers. The educational environment in which the first text corpus was produced refers to the educational system or cultural background in which the first text corpus was produced, such as school textbooks or online courses.

[0147] Alternatively, a quality screening model can be obtained by adding a three-category classification head to a language model. The language model can be the Llama3-1.3B model, which is not limited in this embodiment. Each classification corresponds to a dimension of "expression style," "factual knowledge content," and "educational background," so as to predict the quality score of the first text corpus in each of the three dimensions.

[0148] Before the above-mentioned quality screening model can be used to screen text corpora, it must be trained. The screening performance of the quality screening model depends on the training process. The following describes the training process of the quality screening model.

[0149] During the training process of the quality screening model, the server obtains multiple sample corpus pairs. Each sample corpus pair includes a first sample corpus, a second sample corpus, and a reference confidence of each sample corpus. The reference confidence of each sample corpus is used to indicate the quality of the sample corpus compared to the quality of another sample corpus in the corpus pair. Then, for each sample corpus pair in the multiple sample corpus pairs, the server processes the sample corpus pair through the quality screening model to obtain the prediction confidence of each sample corpus in the sample corpus pair. The prediction confidence of each sample corpus is used to indicate the quality of the sample corpus predicted by the model compared to the quality of another sample corpus in the corpus pair. Then, the server trains the quality screening model based on the reference confidence and prediction confidence of the multiple sample corpus pairs.

[0150] The solution provided by the embodiments of the present disclosure uses paired sample corpora to train the quality screening model, so that the quality screening model can determine the prediction confidence of each text corpus by comparing the gaps between the text corpora. The quality screening model is trained based on the prediction confidence and reference confidence of each text corpus in each sample corpus pair, so that the prediction confidence of the text corpus determined by the quality screening model is increasingly close to the reference confidence in the actual situation, that is, the performance of the quality screening model in determining the quality of the text corpus is improved, so that the quality screening model can more accurately screen out high-quality corpus texts.

[0151] The reference confidence of each sample corpus in each sample corpus pair may include at least one of a first reference confidence, a second reference confidence, and a third reference confidence. The first reference confidence of a sample corpus is used to indicate the expression style of the sample corpus compared to the expression style of another sample corpus in the corpus pair. For example, a larger first reference confidence indicates a more rigorous expression style of the sample corpus compared to the expression style of another sample corpus in the corpus pair. The second reference confidence of a sample corpus is used to indicate the educational background of the sample corpus compared to the educational background of another sample corpus in the corpus pair. For example, a larger second reference confidence indicates a higher level of educational background of the sample corpus compared to the educational background of another sample corpus in the corpus pair. The third reference confidence of a sample corpus is used to indicate the factual knowledge content of the sample corpus compared to the factual knowledge content of another sample corpus in the corpus pair. The larger the third reference confidence is, the more factual knowledge the sample corpus contains compared to the factual knowledge content of another sample corpus in the corpus pair.

[0152] Among them, the reference confidence of each sample corpus in each sample corpus pair is determined based on the first sample annotation model, and the first sample annotation model belongs to the large language model. The first sample annotation model can be Llama3-70B-Instruct, which is not limited in the embodiment of the present disclosure. The solution provided by the embodiment of the present disclosure uses a large language model to annotate the reference confidence of paired sample corpora with the sample corpora used in the training process of the quality screening model. This not only saves labor costs, but also compares and annotates paired sample corpora, which can make the reference confidence determined by the large language model more consistent, thereby obtaining more accurate and stable labeling results, greatly improving the label quality of the training data of the quality screening model, and thus helping to improve the performance of the quality screening model, so that the quality screening model can more accurately screen out high-quality corpus texts.

[0153] Among them, for each sample corpus pair, the server can process the first sample corpus and the second sample corpus in the sample corpus pair respectively through the first sample annotation model to respectively determine the quality parameters of the first sample corpus and the quality parameters of the second sample corpus, and then determine the reference quality parameters of the first sample corpus and the reference quality parameters of the second sample corpus based on the quality parameters of the first sample corpus and the quality parameters of the second sample corpus.

[0154] The server may determine the reference quality parameter of each sample corpus using the following formula 1. The second sample corpus is used as an example for illustration.

[0155] Formula 1:

[0156] P B>A =δ(s B -s A )

[0157] Among them, s A It is used to represent the quality score of the first sample corpus determined by the first sample annotation model; B It is used to represent the quality score of the second sample corpus determined by the first sample annotation model; δ is used to represent the sigmoid function; P B＞A The reference confidence level of the second sample corpus refers to the quality of the second sample corpus compared to the first sample corpus.

[0158] For each sample corpus in each sample corpus pair, the server can determine the prediction confidence of the sample corpus (including at least one of a first prediction confidence, a second prediction confidence, and a third prediction confidence) using the quality screening model. The first prediction confidence is used to indicate the expression style of the sample corpus. The second prediction confidence is used to indicate the educational background of the sample corpus. The third prediction confidence is used to indicate the content of factual knowledge in the sample corpus. Then, the server determines the model loss of the quality screening model based on the reference confidence and prediction confidence of the sample corpus pair, and adjusts the model parameters of the quality screening model with the goal of minimizing the model loss. The model loss of the quality screening model is used to represent the gap between the reference confidence and prediction confidence of each sample corpus in the sample corpus pair. If the quality screening model meets the training stop condition, the server stops training the quality screening model. The training stop condition can be that the model loss of the quality screening model is lower than the loss threshold, or it can be that the number of iterations of the quality screening model (i.e., the number of times the model parameters are adjusted) reaches a preset number, etc., which are not limited in the embodiments of the present disclosure.

[0159] In some embodiments, the server may determine the model loss of the quality screening model using the following formula 2.

[0160] Formula 2:

[0161]

[0162] Among them, s θ is used to represent the quality screening model; θ is used to represent the model parameters in the quality screening model; t A Used to represent the first sample corpus; s θ (t A ) is used to represent the quality score of the first sample corpus predicted by the quality screening model alone; t B Used to represent the second sample corpus; s θ (t B ) is used to indicate the quality score of the second sample corpus predicted by the quality screening model alone; δ is used to indicate the sigmoid function; δ(s θ (t B )-s θ (t A )) is used to represent the prediction confidence of the second sample corpus, that is, the quality of the second sample corpus compared with the quality of the first sample corpus; P B＞A Used to represent the reference confidence of the second sample corpus; L θ It is used to represent the model loss of the quality screening model; τ is used to represent the sample set of the quality screening model; E is used to represent expectation, which is used to measure the average loss of the model on the entire data distribution.

[0163] In step 303, the server uses the subject screening model to screen out, from the plurality of second text corpora, a plurality of third text corpora in which the factual knowledge in the corpora belongs to the target subject.

[0164] In the embodiment of the present disclosure, the subject screening model can be constructed based on the BERT model or the Llama3-1.3B model. The embodiment of the present disclosure does not limit the architecture of the subject screening model. Optionally, the number of model parameters in the subject screening model is less than the quantity threshold. That is, the subject screening model can be a small-scale model. Compared with directly using other trained advanced large language models, using a small-scale model to identify the discipline to which factual knowledge in the corpus belongs and to screen it will take less inference time, thereby improving screening efficiency.

[0165] In the process of screening the third text corpus, for any second text corpus among the multiple second text corpora, the server classifies the second text corpus through the subject screening model and determines the subject classification result corresponding to the second text corpus. The subject classification result is used to indicate the subject to which the factual knowledge in the second text corpus belongs. Then, based on the subject classification results corresponding to the multiple second text corpora, the server screens out multiple third text corpora whose factual knowledge in the corpus belongs to the target subject from the multiple second text corpora. The solution provided by the embodiment of the present disclosure determines the subject to which the factual knowledge in each second text corpus belongs through the subject screening model, thereby screening out multiple third text corpora whose factual knowledge belongs to the target subject from the multiple second text corpora. Since the target subject is related to the model task of the target large language model to be trained, the text corpus screened out for training the target large language model is guaranteed, which can improve the performance of the target large language model in processing the model task, that is, it can improve the quality of the samples used for model training.

[0166] Among them, the subject classification result may include the probability that the factual knowledge in the second text corpus belongs to each of the multiple candidate subjects. The subject with the highest probability in the subject classification result is the subject to which the factual knowledge in the second text corpus belongs. The embodiment of the present disclosure does not limit the number of multiple candidate subjects. For example, there are 65 candidate subjects. Accordingly, the subject screening model can be obtained by adding a 65 classification head on the basis of the BERT model. Each classification corresponds to a subject label, so that the second text corpus can be classified into any of the 65 subjects.

[0167] Before the aforementioned subject screening model can be used to screen text corpora, it must be trained. The screening performance of the subject screening model depends on the training process. The following describes the training process for the subject screening model.

[0168] During the training process of the discipline screening model, the server obtains multiple third sample corpora. Each third sample corpus includes a reference discipline, and the reference discipline of each third sample corpus indicates the discipline to which the factual knowledge in the third sample corpus belongs. For each of the multiple third sample corpora, the server processes the third sample corpora using the discipline screening model to obtain a predicted discipline for the third sample corpora. The predicted discipline indicates the discipline to which the factual knowledge in the third sample corpora is predicted by the model. The server then trains the discipline screening model based on the reference and predicted disciplines of the multiple third sample corpora. The solution provided by the disclosed embodiments uses sample corpora containing reference disciplines to train the discipline screening model. This allows the reference discipline of each sample corpus to be used as a label, guiding the discipline screening model to gradually lean towards the reference label for the discipline to which the factual knowledge in the sample corpus belongs. In other words, the predicted discipline of the text corpus determined by the discipline screening model is increasingly close to the actual reference discipline, thereby improving the accuracy of the discipline screening model in identifying the discipline to which the factual knowledge in the text corpus belongs. This allows the discipline screening model to subsequently accurately screen out text corpora containing factual knowledge belonging to the target discipline, thereby improving the screening effect.

[0169] Among them, the reference disciplines may include the probability that the factual knowledge in the third sample corpus belongs to each of the multiple candidate disciplines. The predicted disciplines include the probability that the factual knowledge in the third sample corpus predicted by the discipline screening model belongs to each of the multiple candidate disciplines. Then, the server can determine the model loss of the discipline screening model based on the reference disciplines and predicted disciplines of the multiple third sample corpora, and adjust the model parameters of the discipline screening model with the goal of minimizing the model loss. The model loss of the discipline screening model is used to represent the gap between the reference disciplines and predicted disciplines of the multiple third sample corpora. When the discipline screening model meets the training stop condition, the server stops training the discipline screening model. The training stop condition can be that the model loss of the discipline screening model is lower than the loss threshold, or it can also be that the number of iterations of the discipline screening model (that is, the number of times the model parameters are adjusted) reaches a preset number, etc., and the embodiments of the present disclosure do not limit this.

[0170] In some embodiments, the server may determine the model loss of the subject screening model using the following formula three.

[0171] Formula 3:

[0172]

[0173] Among them, p i,j It is used to represent the probability that the discipline of the factual knowledge in the third sample corpus of the i-th discipline predicted by the discipline screening model is the j-th discipline, that is, the predicted discipline; y i,jIt is used to express the probability that the discipline to which the factual knowledge in the i-th third sample corpus belongs is the j-th discipline, that is, the reference discipline in a real situation; N is used to express the number of third sample corpora; L is used to express the model loss of the discipline screening model; M is the total number of disciplines, such as M=65, which is not limited in the embodiment of the present disclosure. Formula three belongs to multi-category cross entropy loss. Multi-category cross entropy loss measures the difference between the model predicted probability distribution and the one-hot distribution of the true label. If the model prediction is similar to the true label distribution (that is, the predicted category probability is close to 1), the loss is small; conversely, if the model prediction is wrong, the loss is large. The discipline screening model predicts the probability that each corpus text belongs to each discipline, and takes the discipline with the largest probability as the discipline to which the factual knowledge in the corpus text belongs.

[0174] Among them, multiple third sample corpora are obtained by screening based on the trained quality screening model. The quality of multiple third sample corpora meets the quality conditions. That is, after the quality screening model is trained, it is beneficial for the trained quality screening model to screen out the sample data (i.e., the third sample corpora) required for the training of the subject screening model, and then train the subject screening model based on the screened sample data. The solution provided by the embodiment of the present disclosure ensures that the quality of the third sample corpora is high because the third sample corpora used for the training of the subject screening model are obtained by screening through the trained quality screening model. On this basis, the subject screening model is trained, which facilitates the subject screening model to better learn the subject to which the factual knowledge in the text corpus belongs, and is beneficial to improving the performance of the subject screening model, so that the text corpus whose factual knowledge belongs to the target subject can be accurately screened out later, thereby improving the screening effect.

[0175] In step 304, the server uses the knowledge hierarchy screening model to screen out multiple target text corpora from the multiple third text corpora, whose knowledge hierarchy of factual knowledge in the corpora meets the hierarchy conditions. The multiple target text corpora are used to train the large language model.

[0176] In the embodiment of the present disclosure, the knowledge level screening model is constructed based on Roberta, or based on the Bert model, or based on the Llama3-1.3B model. The embodiment of the present disclosure does not limit the architecture of the knowledge level screening model. Optionally, the number of model parameters in the knowledge level screening model is less than the quantity threshold. That is, the knowledge level screening model can be a small-scale model. Compared with directly using other trained advanced large language models, using a small-scale model to identify the knowledge level to which the factual knowledge in the corpus belongs and screening it will take less inference time, thereby improving screening efficiency.

[0177] During the process of screening target text corpora, for any third text corpora from the multiple third text corpora, the server processes the third text corpora using a knowledge hierarchy screening model to determine the knowledge hierarchy to which the factual knowledge in the third text corpora belongs. Then, based on the knowledge hierarchy to which the factual knowledge in the multiple third text corpora belongs, the server screens out multiple target text corpora from the multiple third text corpora. The knowledge hierarchy to which the factual knowledge in the multiple target text corpora meets the hierarchy threshold in the hierarchy condition. The solution provided by the disclosed embodiment determines the knowledge hierarchy to which the factual knowledge in each third text corpus belongs using a knowledge hierarchy screening model, thereby screening out target text corpora that meet the knowledge hierarchy criteria from the multiple third text corpora. Since the knowledge hierarchy reflects the depth of the factual knowledge in the third text corpora, the higher the depth of the factual knowledge, the greater the educational value or significance of the factual knowledge. By screening out target text corpora that meet the knowledge hierarchy criteria, it is ensured that the screened target text corpora all have sufficient educational value or significance and are worthy of model learning. Consequently, subsequent training of the target large language model with the target text corpora can significantly improve model performance.

[0178] Among them, the server can predict the probability that the factual knowledge in the third text corpus belongs to each level of multiple candidate levels through the knowledge hierarchy screening model, and use the level with the highest probability among the multiple candidate levels as the knowledge level to which the factual knowledge in the third text corpus belongs. The embodiment of the present disclosure does not limit the number of multiple candidate levels. For example, there are 6 levels, and accordingly, the knowledge hierarchy screening model can be obtained by adding a six-classification head on the basis of the roberta model. Each classification corresponds to a level, so that the third text corpus can be classified into any knowledge level of the 6 candidate levels.

[0179] In the process of determining the knowledge hierarchy to which each third text corpus belongs, for any third text corpus among the multiple third text corpora, the server matches the third text corpus with multiple hierarchical rules using a knowledge hierarchy screening model. The server then determines a hierarchy score for the third text corpus based on the number of target hierarchy rules among the multiple hierarchical rules. The target hierarchy rule is a hierarchy rule among the multiple hierarchical rules that matches the third text corpus. The hierarchy score is positively correlated with the number of target hierarchy rules. The server then determines the knowledge hierarchy to which the third text corpus belongs based on the hierarchy score of the third text corpus, where the knowledge hierarchy is positively correlated with the hierarchy score. The solution provided by the embodiment of the present disclosure is to determine, for each third text corpus, the hierarchical score of the third text corpus according to the number of hierarchical rules matched by the third text corpus. The more hierarchical rules matched by the third text corpus, the higher the corresponding hierarchical score, and thus the higher the determined knowledge hierarchy. This is equivalent to using a "bonus system" to determine the knowledge hierarchy of the third text corpus. Whenever the third text corpus hits a hierarchical rule, the knowledge hierarchy of the third text corpus may be improved. This method enables the knowledge hierarchy screening model to pay more attention to the features in the third text corpus to determine whether it matches the hierarchical rules, thereby being able to more accurately determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs, and then screen out target text corpora with better quality, that is, to improve the quality of screening.

[0180] In other embodiments, for any third text corpus among the plurality of third text corpora, the server can directly determine the knowledge hierarchy of the third text corpus using a knowledge hierarchy screening model. The hierarchy screening model does not need to determine the hierarchy score of the third text corpus. If the knowledge hierarchy screening model is to be able to directly determine the knowledge hierarchy of the text corpus, the knowledge hierarchy screening model can be trained in the following manner. The following describes the training process of the knowledge hierarchy screening model.

[0181] During the training process of the knowledge hierarchy screening model, the server obtains multiple fourth sample corpora. Each fourth sample corpus includes a reference hierarchy score. The reference hierarchy score of each fourth sample corpus is used to indicate the knowledge hierarchy to which the factual knowledge in the fourth sample corpus belongs. For each fourth sample corpus in the multiple fourth sample corpora, the server processes the fourth sample corpus through the knowledge hierarchy screening model to obtain a predicted hierarchy score of the fourth sample corpus, and the predicted hierarchy score is used to indicate the hierarchy score predicted by the model to which the factual knowledge in the fourth sample corpus belongs. The server trains the knowledge hierarchy screening model based on the reference hierarchy scores and predicted hierarchy scores of the multiple fourth sample corpora. The solution provided by the embodiment of the present disclosure uses sample corpora containing reference hierarchical scores to train the knowledge hierarchy screening model, so that the reference hierarchical score of each sample corpus can be used as a label to guide the knowledge hierarchy screening model to gradually tend to the reference hierarchical score for the hierarchy to which the factual knowledge in the sample corpus belongs. That is, the predicted hierarchical score of the text corpus determined by the knowledge hierarchy screening model is increasingly close to the reference hierarchical score in the actual situation, thereby improving the accuracy of the knowledge hierarchy screening model in identifying the hierarchy to which the factual knowledge in the text corpus belongs, so that the text corpus that meets the knowledge hierarchy standards can be accurately screened out in the future, thereby improving the screening effect.

[0182] Among them, the reference level score may include the probability that the factual knowledge in the fourth sample corpus belongs to each level in multiple candidate levels. The predicted level score includes the probability that the factual knowledge in the fourth sample corpus predicted by the knowledge level screening model belongs to each level in multiple candidate levels. Then, the server can determine the model loss of the knowledge level screening model based on the reference level scores and predicted level scores of multiple fourth sample corpora, and adjust the model parameters of the knowledge level screening model with the goal of minimizing the model loss. The model loss of the knowledge level screening model is used to represent the gap between the reference disciplines and the predicted disciplines of multiple fourth sample corpora. When the knowledge level screening model meets the training stop condition, the server stops training the knowledge level screening model. The training stop condition can be that the model loss of the knowledge level screening model is lower than the loss threshold, or it can also be that the number of iterations of the knowledge level screening model (i.e., the number of times the model parameters are adjusted) reaches a preset number, etc., and the embodiments of the present disclosure do not limit this.

[0183] In some embodiments, the server may determine the model loss of the knowledge level screening model using the following formula 4.

[0184] Formula 4:

[0185]

[0186] Among them, p i,jIt is used to represent the probability that the level of factual knowledge in the fourth sample corpus of the i-th layer is the j-th layer predicted by the knowledge level screening model, which is equivalent to the predicted level score; i,j represents the probability that the factual knowledge in the i-th fourth sample corpus belongs to the j-th level in real-world situations, equivalent to the reference level score; N represents the number of fourth sample corpora; L represents the model loss of the knowledge level screening model; and 6 represents the total number of levels. Formula 4 represents the multi-class cross-entropy loss and follows a similar principle to Formula 3, so I will not elaborate on it here.

[0187] Among them, multiple fourth sample corpora are obtained by screening based on the trained quality screening model and discipline screening model. The quality of the multiple fourth sample corpora meets the quality conditions. The factual knowledge in the multiple fourth sample corpora belongs to the target discipline. That is, the quality screening model is trained first; after the quality screening model is trained, the trained quality screening model is used to screen out the sample data (i.e., the third sample corpus) required for training the discipline screening model, and then the discipline screening model is trained based on the screened third sample corpus; after the discipline screening model is trained, the trained discipline screening model is used to screen out the sample data (i.e., the fourth sample corpus) required for the knowledge hierarchy screening model, and then the knowledge hierarchy screening model is trained based on the screened fourth sample corpus. The solution provided by the embodiment of the present disclosure ensures that the quality of the fourth sample corpus used for training the knowledge hierarchy screening model is high, because the fourth sample corpus used is obtained by screening the trained quality screening model and subject screening model. The knowledge hierarchy screening model is trained on this basis, which facilitates the knowledge hierarchy screening model to better learn the level to which the factual knowledge in the text corpus belongs, and is beneficial to improving the performance of the knowledge hierarchy screening model, so that the text corpus that meets the level of factual knowledge can be accurately screened out in the future, thereby improving the screening effect.

[0188] In some embodiments, the reference level scores of the plurality of fourth sample corpora may be determined based on at least one of the following items.

[0189] First, the reference level score of each fourth sample corpus is determined based on the second sample annotation model, and the second sample annotation model belongs to the large language model. The second sample annotation model can be Llama3-70B-Instruct, which is not limited in the embodiments of the present disclosure. The solution provided by the embodiments of the present disclosure uses a large language model to annotate the sample corpus with a reference level score for the sample corpus used in the training process of the knowledge level screening model, which not only saves labor costs, but also facilitates the powerful understanding ability of the large language model for annotation, and can guarantee the accuracy of the reference level score to a certain extent.

[0190] Second, the reference hierarchical score of each fourth sample corpus is positively correlated with the number of hierarchical rules that match the fourth sample corpus among the multiple hierarchical rules. The multiple hierarchical rules may include "containing a large number of professional terms" or "the text organization is strictly divided into paragraphs and sections", etc., which are not limited by the present embodiment. The solution provided by the present embodiment determines the hierarchical score of each fourth sample corpus based on the number of hierarchical rules matched by the fourth sample corpus. The more hierarchical rules matched by the fourth sample corpus, the higher the corresponding hierarchical score, and thus the higher the determined knowledge hierarchy. This is equivalent to using a "bonus system" to determine the knowledge hierarchy of the fourth sample corpus. Every time the fourth sample corpus hits a hierarchical rule, the knowledge hierarchy of the fourth sample corpus may be improved. This method enables more attention to be paid to the features in the fourth sample corpus to determine whether they match the hierarchical rules, thereby more accurately determining the knowledge hierarchy to which the factual knowledge in the fourth sample corpus belongs, further ensuring the accuracy of the reference hierarchical score, and thus helping to improve the performance of the knowledge hierarchy screening model.

[0191] The above-mentioned knowledge level screening model can screen out text corpora of corresponding knowledge levels that meet the training requirements according to the training requirements of the target large language model. Optionally, in the early stage of training of the target large language model, the knowledge level of the factual knowledge in the target text corpus screened out by the knowledge level screening model is lower than the preset level. This method enables the target large language model to converge quickly in the early stage of training by learning simple factual knowledge, which helps the model to establish basic language understanding and generation capabilities in the early stages, laying a solid foundation for subsequent learning of more complex content; and the learning of simple factual knowledge can keep the model stable in the early stages of training, avoiding instability in the training process due to premature exposure to complex data. This stability helps the model gradually adapt to the training process, reduces shocks during training, and prevents the model from focusing on specific complex patterns too early, thereby reducing the risk of overfitting and learning more general language features. In the annealing stage of the target large language model, the knowledge level of the factual knowledge in the target text corpus screened out by the knowledge level screening model is higher than the preset level. Learning complex factual knowledge during the annealing phase allows the model to further enhance its understanding of complex patterns and relationships, building on its existing foundational capabilities. This helps the model achieve higher performance levels on specific tasks. The lower learning rate during the annealing phase allows the model to fine-tune parameters to accommodate complex factual knowledge. This fine-tuning helps the model find optimal solutions within a complex solution space, thereby improving overall performance.

[0192] In some embodiments, the total number of model parameters for the quality screening model, the discipline screening model, and the knowledge hierarchy screening model is less than a preset value. The solution provided by the disclosed embodiments employs a pipeline consisting of the quality screening model, the discipline screening model, and the knowledge hierarchy screening model to perform screening in sequence. Because the total number of model parameters for the quality screening model, the discipline screening model, and the knowledge hierarchy screening model is less than a preset value, i.e., the scale of each of the aforementioned models is relatively small, the inference time of each of the aforementioned small models is shorter than directly employing other pre-trained advanced large language models, thereby improving screening efficiency while ensuring quality screening results.

[0193] In order to more clearly describe the method for screening text corpora provided by the embodiments of the present disclosure, the screening method is further described below with reference to the accompanying drawings. Figure 4 FIG. 1 is a schematic diagram showing a method for filtering text corpus according to an exemplary embodiment. Figure 4 The server uses paired sample corpora (first sample corpus + second sample corpus) to train a quality screening model based on three dimensions: writing style, factual knowledge, and educational background. A third sample corpus containing reference subjects is used to train a subject screening model. A fourth sample corpus containing reference hierarchical scores is used to train a knowledge hierarchy screening model. After model training, a pipeline consisting of the quality screening model, subject screening model, and knowledge hierarchy screening model is used to perform screening in sequence.

[0194] The disclosed embodiment provides a method for screening text corpora. For a plurality of first text corpora initially acquired, first, based on the content of factual knowledge in the first text corpora, a plurality of second text corpora with factual knowledge content meeting the standard are screened out from the plurality of first text corpora. Then, based on the discipline to which the factual knowledge in the second text corpora belongs, a plurality of third text corpora containing factual knowledge belonging to a target discipline are screened out from the plurality of second text corpora. Finally, based on the level to which the factual knowledge in the third text corpora belongs, a plurality of target text corpora with factual knowledge depth meeting the level conditions are screened out from the plurality of third text corpora. This ensures that the plurality of target text corpora ultimately screened out for model training not only contain factual knowledge, but also have disciplinary factual knowledge related to the model task, and the factual knowledge contained has sufficient depth, which is equivalent to having sufficient educational value or significance and is worthy of model learning. Therefore, model training through a plurality of target text corpora can significantly improve model performance. That is, from the perspective of the information required for model training, this solution breaks down the complex high-quality features in the text corpus into three levels: "whether it contains factual knowledge", "whether it contains the subject factual knowledge required for model training", and "whether the factual knowledge contained is deep enough to be worth learning". This can fully explore the characteristics of the text corpus and screen it layer by layer, so as to accurately screen out truly high-quality text corpora, that is, the screening effect is better, and then the model can be trained through high-quality text corpora, which is conducive to improving the performance of the model.

[0195] After quality screening of the text corpus using this solution, the resulting data is used throughout the pre-training phase, ranked from easy to difficult according to the knowledge hierarchy. In the experiment, the quality screening model, subject screening model, and knowledge hierarchy screening model used in this solution are small models with 355M parameters. They were trained from scratch on 100B of filtered data and compared with a baseline model with the same number of parameters trained on the currently open-source high-quality data fineweb. We selected nine common benchmark datasets: TriviaQA, Natural Questions, HellaSwag, WinoGrande, PIQA, SiQA, ARC-Challenge, BIG-Bench Hard, and AGIEval. For details, see Table 1 below.

[0196] Table 1

[0197]

[0198]

[0199] Table 1 shows the performance of the models pre-trained on FineWeb data and pre-trained on data selected by this solution on nine benchmark datasets. Table 1 shows the experimental results on nine benchmark datasets. It can be seen that the data selected by this solution effectively improves the factual knowledge capabilities of the base model, surpassing FineWeb on all nine benchmark datasets, with an average improvement of 1.4%.

[0200] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0201] Figure 5 This is a block diagram of a device for screening text corpus according to an exemplary embodiment. Figure 5 , the text corpus screening device includes:

[0202] An acquiring unit 501 is configured to acquire a plurality of first text corpora;

[0203] The first screening unit 502 is configured to screen out, from the plurality of first text corpora, a plurality of second text corpora whose corpora meet a quality condition, wherein the quality condition includes a standard to be met for the content of factual knowledge in the text corpora;

[0204] The second screening unit 503 is configured to screen out, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to a target subject, where the target subject is related to a model task of the target large language model;

[0205] The third screening unit 504 is configured to screen out multiple target text corpora from multiple third text corpora, whose knowledge hierarchy of factual knowledge in the corpora meets the hierarchy conditions. The knowledge hierarchy is used to indicate the depth of factual knowledge in the corpora, and the hierarchy conditions include the standards to be achieved for the depth of factual knowledge in the third text corpora. The multiple target text corpora are used to train the target large language model.

[0206] In some embodiments, the first screening unit 502 is configured to perform a quality screening model to screen out a plurality of second text corpora whose corpus quality meets the quality condition from the plurality of first text corpora;

[0207] The second screening unit 503 is configured to perform a subject screening model to screen out a plurality of third text corpora containing factual knowledge belonging to a target subject from the plurality of second text corpora;

[0208] The third screening unit 504 is configured to execute the knowledge hierarchy screening model to screen out a plurality of target text corpora from the plurality of third text corpora, wherein the knowledge hierarchy of the factual knowledge in the corpora meets the hierarchy conditions, and the plurality of target text corpora are used to train the large language model;

[0209] Among them, the total number of model parameters of the quality screening model, the discipline screening model and the knowledge level screening model is less than the preset value.

[0210] In some embodiments, the first screening unit 502 includes:

[0211] The first processing subunit is configured to process any first text corpus among the plurality of first text corpora using the quality screening model to obtain a quality score of the first text corpus, where the quality score is used to indicate the quality of the first text corpus;

[0212] The first screening subunit is configured to screen out a plurality of second text corpora whose quality scores meet a quality condition from the plurality of first text corpora based on the quality scores of the plurality of first text corpora.

[0213] In some embodiments, the quality condition further includes at least one of a first indicator and a second indicator, the first indicator being used to indicate the standard to be achieved by the expression style of the text corpus, and the second indicator being used to indicate the standard to be achieved by the educational background of the text corpus;

[0214] a first processing subunit configured to process any first text corpus from among the plurality of first text corpora using the quality screening model to obtain at least one of a first quality score and a second quality score and a third quality score for the first text corpus, wherein the first quality score is used to indicate an expression style of the first text corpus, the second quality score is used to indicate an educational background of the first text corpus, and the third quality score is used to indicate a content of factual knowledge in the first text corpus;

[0215] The screening subunit is configured to screen out a plurality of second text corpora from the plurality of first text corpora, wherein at least one of the first quality score and the second quality score and the third quality score of each second text corpus meet respective indicators in the quality condition.

[0216] In some embodiments, the apparatus further comprises:

[0217] The first training unit is configured to execute acquiring a plurality of sample corpus pairs, each sample corpus pair including a first sample corpus, a second sample corpus, and a reference confidence of each sample corpus, where the reference confidence of each sample corpus is used to indicate the quality of the sample corpus compared to the quality of another sample corpus in the corpus pair to which it belongs; for each sample corpus pair in the plurality of sample corpus pairs, processing the sample corpus pair through a quality screening model to obtain a prediction confidence of each sample corpus in the sample corpus pair, where the prediction confidence of each sample corpus is used to indicate the quality of the sample corpus predicted by the model compared to the quality of another sample corpus in the corpus pair; and training the quality screening model based on the reference confidence and prediction confidence of the plurality of sample corpus pairs.

[0218] In some embodiments, the reference confidence of each sample corpus in each sample corpus pair is determined based on a first sample annotation model, and the first sample annotation model belongs to a large language model.

[0219] In some embodiments, the second screening unit 503 is configured to perform classification of the second text corpus for any second text corpus among the multiple second text corpora through a subject screening model, and determine the subject classification result corresponding to the second text corpus, where the subject classification result is used to indicate the subject to which the factual knowledge in the second text corpus belongs; based on the subject classification results corresponding to the multiple second text corpora, multiple third text corpora are screened out from the multiple second text corpora, whose factual knowledge in the corpus belongs to the target subject.

[0220] In some embodiments, the apparatus further comprises:

[0221] The second training unit is configured to obtain multiple third sample corpora, each third sample corpus includes a reference subject, and the reference subject of each third sample corpus is used to indicate the subject to which the factual knowledge in the third sample corpus belongs; for each third sample corpus in the multiple third sample corpora, the third sample corpus is processed by a subject screening model to obtain a predicted subject of the third sample corpus, and the predicted subject is used to indicate the subject to which the factual knowledge in the third sample corpus predicted by the model belongs; and the subject screening model is trained based on the reference subjects and predicted subjects of the multiple third sample corpora.

[0222] In some embodiments, the plurality of third sample corpora are obtained by screening based on a trained quality screening model, and the quality of the plurality of third sample corpora meets the quality condition.

[0223] In some embodiments, the third screening unit 504 includes:

[0224] The second processing sub-unit is configured to process any third text corpus among the plurality of third text corpora using the knowledge hierarchy screening model to determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs;

[0225] The second screening sub-unit is configured to perform a screening of multiple target text corpora from the multiple third text corpora based on the knowledge level to which the factual knowledge in the multiple third text corpora belongs, and the knowledge level to which the factual knowledge in the multiple target text corpora belongs reaches the level threshold in the level condition.

[0226] In some embodiments, the second processing sub-unit is configured to execute matching of the third text corpus with multiple hierarchical rules respectively for any third text corpus among multiple third text corpora through a knowledge hierarchy screening model; determine the hierarchy score of the third text corpus based on the number of target hierarchy rules among the multiple hierarchical rules, the target hierarchy rule is the hierarchical rule among the multiple hierarchical rules that matches the third text corpus, and the hierarchy score is positively correlated with the number of target hierarchy rules; determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs based on the hierarchy score of the third text corpus, and the knowledge hierarchy is positively correlated with the hierarchy score.

[0227] In some embodiments, the apparatus further comprises:

[0228] The third training unit is configured to obtain multiple fourth sample corpora, each fourth sample corpus includes a reference level score, and the reference level score of each fourth sample corpus is used to indicate the knowledge level to which the factual knowledge in the fourth sample corpus belongs; for each fourth sample corpus in the multiple fourth sample corpora, the fourth sample corpus is processed by the factual knowledge level screening model to obtain a predicted level score of the fourth sample corpus, and the predicted level score is used to indicate the level score predicted by the model to which the factual knowledge in the fourth sample corpus belongs; based on the reference level scores and the predicted level scores of the multiple fourth sample corpora, the knowledge level screening model is trained.

[0229] In some embodiments, the plurality of fourth sample corpora are screened based on a trained quality screening model and a discipline screening model, the quality of the plurality of fourth sample corpora meet the quality conditions, and the factual knowledge in the plurality of fourth sample corpora belongs to the target discipline.

[0230] In some embodiments, the reference level scores of the plurality of fourth sample corpora include at least one of the following:

[0231] The reference level score of each fourth sample corpus is determined based on the second sample annotation model, which belongs to the large language model;

[0232] The reference level score of each fourth sample corpus is positively correlated with the number of level rules among the multiple level rules that match the fourth sample corpus.

[0233] The disclosed embodiment provides a text screening device, which first screens out a plurality of first text corpora with a content of factual knowledge that meets the requirements from the plurality of first text corpora based on the content of factual knowledge in the first text corpora, and then screens out a plurality of third text corpora containing factual knowledge belonging to a target discipline from the plurality of second text corpora based on the discipline to which the factual knowledge in the second text corpora belongs; and finally, screens out a plurality of target text corpora containing factual knowledge that meets the hierarchy requirements from the plurality of third text corpora based on the hierarchy to which the factual knowledge in the third text corpora belongs, thereby ensuring that the plurality of target text corpora finally screened out for model training not only contain factual knowledge, but also have disciplinary factual knowledge related to the model task, and the factual knowledge contained has sufficient depth, which is equivalent to having sufficient educational value or significance and is worthy of model learning. Therefore, model training through multiple target text corpora can significantly improve model performance. That is, from the perspective of the information required for model training, this solution breaks down the complex high-quality features in the text corpus into three levels: "whether it contains factual knowledge", "whether it contains the subject factual knowledge required for model training", and "whether the factual knowledge contained is deep enough to be worth learning". This can fully explore the characteristics of the text corpus and screen it layer by layer, so as to accurately screen out truly high-quality text corpora, that is, the screening effect is better, and then the model can be trained through high-quality text corpora, which is conducive to improving the performance of the model.

[0234] It should be noted that the text corpus screening device provided in the above embodiment only uses the division of the above functional units as an example to illustrate the screening of text corpora. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the text corpus screening device provided in the above embodiment and the text corpus screening method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0235] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0236] When an electronic device is provided as a terminal, Figure 6 FIG. 6 is a block diagram of a terminal 600 according to an exemplary embodiment. Figure 6The following is a block diagram of a terminal 600 according to an exemplary embodiment of the present disclosure. Terminal 600 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0237] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0238] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0239] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one computer program, which is used to be executed by the processor 601 to implement the text corpus screening method provided in the method embodiment of the present application.

[0240] In some embodiments, terminal 600 may optionally include a peripheral device interface 603 and at least one peripheral device. Processor 601, memory 602, and peripheral device interface 603 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 603 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0241] The peripheral device interface 603 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0242] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 604 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0243] Display screen 605 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. If display screen 605 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 605. These touch signals can be input as control signals to processor 601 for processing. Display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 605, located on the front panel of terminal 600. In other embodiments, there can be at least two display screens 605, located on different surfaces of terminal 600 or in a foldable design. In still other embodiments, display screen 605 can be a flexible display screen, located on a curved or foldable surface of terminal 600. Display screen 605 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 605 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0244] The camera assembly 606 is used to capture images or videos. In some embodiments, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0245] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 601 for processing, or input into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0246] Power supply 608 is used to power various components in terminal 600. Power supply 608 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 608 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is charged via a wired line, while a wireless rechargeable battery is charged via a wireless coil. The rechargeable battery can also support fast charging technology.

[0247] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0248] When an electronic device is provided as a server, Figure 7 This is a block diagram of a server according to an exemplary embodiment. The server 700 may vary greatly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 701 and one or more memories 702, wherein the memory 702 stores at least one program code, which is loaded and executed by the processor 701 to implement the text corpus screening method provided by each of the above method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server 700 may also include other components for implementing device functions, which will not be described in detail here.

[0249] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as memory 602 or memory 702 including instructions. The instructions can be executed by processor 601 of terminal 600 or processor 701 of server 700 to complete the above-mentioned text corpus screening method. Alternatively, the computer-readable storage medium can be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0250] A computer program product includes a computer program / instruction, which implements the above-mentioned text corpus screening method when executed by a processor.

[0251] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0252] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for screening text corpus, characterized in that: The method comprises: Acquire multiple first text corpora; Selecting, from the plurality of first text corpora, a plurality of second text corpora whose corpora meet quality conditions, wherein the quality conditions include a standard to be met for the content of factual knowledge in the text corpora; Filtering, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to a target discipline, the target discipline being relevant to a model task of a target large language model; From the multiple third text corpora, multiple target text corpora are screened out, whose knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy conditions, wherein the knowledge hierarchy is used to indicate the depth of the factual knowledge in the corpus, and the hierarchy conditions include the standards to be achieved for the depth of the factual knowledge in the third text corpora, and the multiple target text corpora are used to train the target large language model.

2. The method for screening text corpus according to claim 1, characterized in that: The step of selecting a plurality of second text corpora whose corpora meet the quality conditions from the plurality of first text corpora comprises: Using a quality screening model, screening out a plurality of second text corpora whose corpus quality meets a quality condition from the plurality of first text corpora; The step of screening out, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to a target subject, includes: Using the subject screening model, screening out from the plurality of second text corpora a plurality of third text corpora containing factual knowledge belonging to a target subject; The step of screening out, from the plurality of third text corpora, a plurality of target text corpora whose knowledge levels of factual knowledge in the corpora meet the level conditions comprises: Using the knowledge hierarchy screening model, a plurality of target text corpora are screened from the plurality of third text corpora, wherein the knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy conditions, and the plurality of target text corpora are used to train the large language model; Among them, the total number of model parameters of the quality screening model, the subject screening model and the knowledge level screening model is less than a preset value.

3. The method for screening text corpus according to claim 2, characterized in that: The method of screening out a plurality of second text corpora whose corpus quality meets the quality condition from the plurality of first text corpora by using the quality screening model includes: For any first text corpus among the plurality of first text corpora, processing the first text corpus through the quality screening model to obtain a quality score of the first text corpus, where the quality score is used to indicate the quality of the first text corpus; Based on the quality scores of the plurality of first text corpora, a plurality of second text corpora whose quality scores meet the quality condition are screened out from the plurality of first text corpora.

4. The method for screening text corpus according to claim 3, characterized in that: The quality condition further includes at least one of a first indicator and a second indicator, wherein the first indicator is used to indicate the standard to be achieved by the expression style of the text corpus, and the second indicator is used to indicate the standard to be achieved by the educational background of the text corpus; The step of processing any first text corpus among the plurality of first text corpora by using a quality screening model to obtain a quality score of the first text corpus includes: For any first text corpus among the plurality of first text corpora, processing the first text corpus using the quality screening model to obtain at least one of a first quality score and a second quality score of the first text corpus, and a third quality score, wherein the first quality score is used to indicate the expression style of the first text corpus, the second quality score is used to indicate the educational background of the first text corpus, and the third quality score is used to indicate the content of factual knowledge in the first text corpus; The step of screening out, from the plurality of first text corpora, a plurality of second text corpora whose quality scores meet the quality condition based on the quality scores of the plurality of first text corpora comprises: The plurality of second text corpora are selected from the plurality of first text corpora, and at least one of the first quality score and the second quality score and the third quality score of each second text corpus meet respective indicators in the quality conditions.

5. The method for screening text corpus according to claim 2, characterized in that: The training process of the quality screening model includes: Acquire multiple sample corpus pairs, each sample corpus pair including a first sample corpus, a second sample corpus, and reference confidence scores of each sample corpus, the reference confidence score of each sample corpus being used to indicate the quality of the sample corpus compared to the quality of another sample corpus in the corpus pair; For each sample corpus pair of the plurality of sample corpus pairs, processing the sample corpus pair using the quality screening model to obtain a prediction confidence score for each sample corpus in the sample corpus pair, wherein the prediction confidence score for each sample corpus indicates the quality of the sample corpus predicted by the model relative to the quality of another sample corpus in the corpus pair; The quality screening model is trained based on the reference confidence and the prediction confidence of the multiple sample corpus pairs.

6. The method for screening text corpus according to claim 5, characterized in that: The reference confidence of each sample corpus in each sample corpus pair is determined based on a first sample annotation model, where the first sample annotation model belongs to a large language model.

7. The method for screening text corpus according to claim 2, characterized in that: The subject screening model is used to screen out, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to the target subject, including: For any second text corpus among the plurality of second text corpora, classify the second text corpus using the subject screening model to determine a subject classification result corresponding to the second text corpus, where the subject classification result is used to indicate the subject to which the factual knowledge in the second text corpus belongs; Based on the subject classification results corresponding to the plurality of second text corpora, the plurality of third text corpora in which the factual knowledge in the corpora belongs to the target subject are screened out from the plurality of second text corpora.

8. The method for screening text corpus according to claim 2, characterized in that: The training process of the subject screening model includes: Acquire a plurality of third sample corpora, each of the third sample corpora including a reference discipline, wherein the reference discipline of each third sample corpus is used to indicate the discipline to which the factual knowledge in the third sample corpus belongs; For each of the plurality of third sample corpora, processing the third sample corpora using the discipline screening model to obtain a predicted discipline for the third sample corpora, where the predicted discipline indicates the discipline to which the factual knowledge in the third sample corpora, as predicted by the model, belongs; The discipline screening model is trained based on the reference disciplines and predicted disciplines of the plurality of third sample corpora.

9. The method for screening text corpus according to claim 8, characterized in that: The plurality of third sample corpora are obtained by screening based on the trained quality screening model, and the quality of the plurality of third sample corpora meets the quality condition.

10. The method for screening text corpus according to claim 2, characterized in that: The method of screening out, from the plurality of third text corpora, a plurality of target text corpora whose knowledge hierarchy of factual knowledge in the corpora meets the hierarchy conditions by using the knowledge hierarchy screening model includes: For any third text corpus among the plurality of third text corpora, the third text corpus is processed by the knowledge hierarchy screening model to determine the knowledge hierarchy to which factual knowledge in the third text corpus belongs; Based on the knowledge level to which the factual knowledge in the multiple third text corpora belongs, the multiple target text corpora are screened out from the multiple third text corpora, and the knowledge level to which the factual knowledge in the multiple target text corpora reaches the level threshold in the level condition.

11. The method for screening text corpus according to claim 10, characterized in that: The step of processing any third text corpus among the plurality of third text corpora by using the knowledge hierarchy screening model to determine the knowledge hierarchy to which the factual knowledge in the third text corpus belongs includes: For any third text corpus among the plurality of third text corpora, matching the third text corpus with a plurality of hierarchical rules respectively through the knowledge hierarchical screening model; determining a hierarchical score of the third text corpus based on the number of target hierarchical rules in the plurality of hierarchical rules, wherein the target hierarchical rule is a hierarchical rule in the plurality of hierarchical rules that matches the third text corpus, and the hierarchical score is positively correlated with the number of target hierarchical rules; Based on the hierarchy score of the third text corpus, a knowledge hierarchy to which the factual knowledge in the third text corpus belongs is determined, wherein the knowledge hierarchy is positively correlated with the hierarchy score.

12. The method for screening text corpus according to claim 2, characterized in that: The training process of the knowledge level screening model includes: Acquire a plurality of fourth sample corpora, each fourth sample corpus comprising a reference level score, the reference level score of each fourth sample corpus being used to indicate a knowledge level to which factual knowledge in the fourth sample corpus belongs; For each fourth sample corpus of the plurality of fourth sample corpora, processing the fourth sample corpus using the knowledge hierarchy screening model to obtain a predicted hierarchy score for the fourth sample corpus, the predicted hierarchy score being used to indicate a hierarchy score to which the factual knowledge in the fourth sample corpus belongs, as predicted by the model; The knowledge hierarchy screening model is trained based on the reference hierarchy scores and the predicted hierarchy scores of the plurality of fourth sample corpora.

13. The method for screening text corpus according to claim 12, characterized in that: The plurality of fourth sample corpora are screened based on the trained quality screening model and the discipline screening model, the quality of the plurality of fourth sample corpora meets the quality conditions, and the factual knowledge in the plurality of fourth sample corpora belongs to the target discipline.

14. The method for screening text corpus according to claim 12, characterized in that: The reference level scores of the plurality of fourth sample corpora include at least one of the following: The reference level score of each fourth sample corpus is determined based on the second sample annotation model, where the second sample annotation model belongs to the large language model; The reference level score of each fourth sample corpus is positively correlated with the number of level rules in the plurality of level rules that match the fourth sample corpus.

15. A device for screening text corpus, characterized in that: The device comprises: An acquiring unit, configured to acquire a plurality of first text corpora; A first screening unit is configured to screen out, from the plurality of first text corpora, a plurality of second text corpora whose corpora meet a quality condition, wherein the quality condition includes a standard to be met for the content of factual knowledge in the text corpora; a second screening unit configured to screen out, from the plurality of second text corpora, a plurality of third text corpora containing factual knowledge belonging to a target subject, the target subject being related to a model task of a target large language model; The third screening unit is configured to screen out, from the multiple third text corpora, multiple target text corpora whose knowledge hierarchy to which the factual knowledge in the corpora belongs meets the hierarchy conditions, wherein the knowledge hierarchy is used to indicate the depth of the factual knowledge in the corpora, and the hierarchy conditions include the standards to be achieved for the depth of the factual knowledge in the third text corpora, and the multiple target text corpora are used to train the target large language model.

16. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the text corpus screening method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for screening text corpora as described in any one of claims 1 to 14.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for screening text corpora according to any one of claims 1 to 14 is implemented.

Images