A text quality cleaning method, device and medium

Abstract

The application discloses a text quality cleaning method and device and medium, and mainly relates to the field of data classification. The method first samples internet corpus to obtain sampling data; and then extracts pre-set low-quality texts from the sampling data, wherein the low-quality texts include incoherent sentences, sentences with insufficient knowledge expression, and repeatedly appearing sentences, and the pre-set low-quality texts at least include two of the three types. As can be seen, compared with the traditional method which can only filter out one type of low-quality texts, the method provided by the application can filter out at least two types of low-quality texts, and the generality of the text quality classification model is improved by covering multiple types of low-quality texts.

Description

Technical Field

[0001] This application relates to the field of data classification, and in particular to a method, apparatus and medium for text quality cleaning. Background Technology

[0002] With the development of artificial intelligence, machines can now hear sounds through speech recognition technology and see images through image processing technology. However, the most challenging aspect is enabling machines to possess the language comprehension abilities of normal humans. To enable machines to understand human language, they first need to understand the meaning of each character or word. A simple method is to use a vector representation for each word and determine its meaning through a context window. However, this method produces static vectors. Due to the ambiguity of human language, the same word can have different meanings in different contexts, thus dynamic word representations have emerged. Pre-trained language models are currently the most popular method for dynamic feature extraction. Through unsupervised learning on large-scale corpora, more general language representations can be learned, and then fine-tuned to complete downstream tasks in different domains. Since language models learn through the contextual relationships of the corpus, the quality of the corpus used during pre-training largely determines the model's expressive power. With the advent of the big data era, various internet corpora can be obtained through web crawlers, but their quality varies greatly. Table 1 is a classification table of common low-quality texts in Internet corpora. As shown in Table 1, there are many types of low-quality texts in Internet corpora, and low-quality texts in Internet corpora must be removed during pre-training.

[0003] Table 1. Classification of common low-quality texts in Internet corpora

[0004]

[0005]

[0006] Since the low-quality texts used for training in existing text quality classification models often come from a certain category of low-quality texts in Table 1, they can only filter out one type of low-quality text. However, there are many types of low-quality texts in the Internet corpus, and existing text quality classification models cannot fully filter out all types of low-quality texts.

[0007] Therefore, how to filter out various low-quality texts from internet corpora is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] The purpose of this application is to provide a text quality cleaning method, apparatus, and medium for filtering low-quality text.

[0009] To address the aforementioned technical problems, this application provides a text quality cleaning method, comprising:

[0010] Sampling was performed on internet corpora to obtain sampled data;

[0011] Extract predefined low-quality text from the sampled data. Low-quality text includes incoherent sentences, sentences with insufficient knowledge expression, and sentences that appear repeatedly. The predefined low-quality text includes at least two of these categories.

[0012] Preferably, if the low-quality text is an incoherent sentence, then extracting the pre-defined low-quality text from the sampled data includes:

[0013] Calculate the perplexity of each statement in the sampled data;

[0014] Low-quality text is identified and extracted based on the perplexity of each statement, where the perplexity is proportional to the probability that a statement is low-quality text.

[0015] Preferably, if the low-quality text is a statement with insufficient knowledge expression, then extracting pre-defined low-quality text from the sampled data includes:

[0016] The TextRank algorithm based on jieba word segmentation is used to filter keywords in each sentence;

[0017] A metric for measuring keyword density is calculated based on the number of keywords in each statement and the statement length.

[0018] Low-quality text is identified and extracted based on indicators, where the indicators are inversely proportional to the probability that a sentence is low-quality text.

[0019] Preferably, if the low-quality text consists of recurring sentences, then extracting the pre-defined low-quality text from the sampled data includes:

[0020] Obtain the sentence vector;

[0021] Construct an Annoy index based on sentence vectors;

[0022] The Annoy index is used to count statements whose similarity exceeds a preset threshold.

[0023] The number of similar questions in each sentence is sorted to identify low-quality text for extraction. The number of similar questions is proportional to the probability that the sentence is low-quality text.

[0024] Preferably, if the low-quality text does not belong to any of the categories of incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences, then the pre-defined low-quality text extracted from the sampled data includes:

[0025] The sampled data is filtered based on keywords or regular expressions to obtain low-quality text, which is then extracted.

[0026] Preferably, it further includes:

[0027] The remaining sampled data after extracting low-quality text was determined to be high-quality text;

[0028] The text quality classification model was trained and optimized based on low-quality and high-quality text.

[0029] Calculate the sampling rate of high-quality text, and sample the high-quality text according to the sampling rate to obtain the pre-training dataset.

[0030] Preferably, training and optimizing the text quality classification model based on low-quality and high-quality text includes:

[0031] Construct a text quality classification training set based on low-quality and high-quality text;

[0032] The text quality classification model was trained and optimized using BERT based on the text quality classification training set.

[0033] Preferably, calculating the sampling rate of high-quality text includes:

[0034] The distribution of high-quality text at the article level, the total number of tokens for high-quality text, and the token distribution for each article are statistically analyzed.

[0035] The sampling rate is calculated based on the target token count and the total token count of the high-quality text.

[0036] To address the aforementioned technical problems, this application also provides a text quality cleaning apparatus, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to implement the steps of the above-described text quality cleaning method when executing a computer program.

[0039] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned text quality cleaning method.

[0040] This application provides a text quality cleaning method, which includes first sampling internet corpora to obtain sampled data; then extracting pre-defined low-quality text from the sampled data. Low-quality text includes incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences, with the pre-defined low-quality text including at least two of these categories. Therefore, compared to traditional methods that can only filter out one type of low-quality text, the method provided in this application covers multiple categories of low-quality text and can filter out at least two types, improving the versatility of the text quality classification model.

[0041] Furthermore, the text quality cleaning apparatus and medium provided in this application correspond to the text quality cleaning method, and the effects are as described above. Attached Figure Description

[0042] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart of a text quality cleaning method provided in this application;

[0044] Figure 2 A distribution chart of r index values ​​in the range of 0-0.35 in the sampling data provided in this application;

[0045] Figure 3 This application provides a probability distribution diagram of the number of articles.

[0046] Figure 4 This application provides a probability distribution diagram of the number of tokens in an article.

[0047] Figure 5 This application provides a structural diagram of a text quality cleaning device. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0049] The core of this application is to provide a text quality cleaning method to improve the versatility of text quality classification models.

[0050] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Figure 1 A flowchart illustrating a text quality cleaning method provided in this application. Figure 1 As shown, the method includes:

[0052] S1: Sample the internet corpus to obtain sampled data.

[0053] Spark, a fast and versatile computing engine designed specifically for large-scale data processing, can be used for various types of computations, including text processing and machine learning. In text processing, it primarily performs operations such as article deduplication and sensitive word filtering. The internet corpus in this embodiment is data that has undergone article deduplication and sensitive information filtering using Spark. It should be noted that text quality classification based on big data processing systems like Spark only involves simple keyword filtering and deduplication, without considering the semantics of the text. Therefore, using Spark alone for data cleaning is insufficient.

[0054] In this embodiment, due to the massive size of the crawled internet corpus—which remains large even after Spark cleaning, potentially exceeding TB levels—direct processing and analysis are difficult. Therefore, sampling of the internet corpus is necessary. In practice, the Spark-cleaned internet corpus can be stored on nodes, with each node storing several weeks' worth of data, stored in small files each week. Because the files are small but numerous, sampling rules can be designed to ensure the diversity and comprehensiveness of the sampled data. For example, X files can be randomly selected from all the acquired files, and 2*Y lines can be randomly sampled from each file. These 2*Y lines are then written into two separate sampling files for later use. X and Y can be set according to the data size. Furthermore, if the internet corpus is at the TB level, the sampling file size can be set to approximately 1GB to ensure a sampling rate of at least one ten-thousandth.

[0055] S2: Extract pre-defined low-quality text from the sampled data. Low-quality text includes incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences. The pre-defined low-quality text includes at least two of these categories.

[0056] In this embodiment, since most paragraphs of the corpus crawled from the Internet are relatively short and mostly consist of single sentences, low-quality text is uniformly represented in the form of sentences. This embodiment defines low-quality text into three types: incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences, and proposes corresponding extraction methods for each type. It should be noted that the types of low-quality text are not limited to the three mentioned above; specific classifications can be found in Table 1.

[0057] For incoherent sentences, perplexity (PPL) is used to rank them. A higher perplexity indicates lower quality and a higher probability of being low-quality text. For sentences with insufficient knowledge expression, such as navigation bars, keyword density is used for filtering. A metric 'r' is designed to measure keyword density and determine if a sentence is low-quality text. A smaller metric 'r' indicates fewer keywords and less valuable information, further increasing the probability of low-quality text. For recurring sentences, such as website descriptions and product advertisements, sentence vector clustering is used for filtering. Sentences with similar meanings are clustered, and the number of similar questions for each sentence is ranked to determine if it is low-quality text. A higher number of similar questions indicates a higher probability of low-quality text.

[0058] This embodiment proposes a text quality cleaning method. The method first samples internet corpora to obtain sampled data, and then extracts a pre-defined set of low-quality text from the sampled data. Low-quality text includes incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences. The pre-defined set of low-quality text includes at least two of these categories. This method covers multiple categories of low-quality text and can filter out at least two types of low-quality text, improving the versatility of the text quality classification model.

[0059] As mentioned in the above embodiments, low-quality text includes incoherent sentences. The extraction steps for incoherent sentences in low-quality text were not described in detail in the above embodiments; this embodiment provides a supplementary explanation. These steps include:

[0060] Calculate the perplexity of each statement in the sampled data;

[0061] Low-quality text is identified and extracted based on the perplexity of each statement, where the perplexity is proportional to the probability that a statement is low-quality text.

[0062] In this embodiment, for semantically incoherent statements, the Probability of Probability (PPL) is used to evaluate the statement's rationality. The higher the PPL of a statement, the higher the probability that the statement is low-quality text. After calculating the PPL of each statement, the statements are sorted according to their PPL and then evaluated to determine the statements that are considered low-quality text. The calculation first calculates the probability of the statement's occurrence, then raises it to the power of 1 / N to obtain the statement's PPL. The calculation formula is as follows: Among them, W1W2…W N Let P(W1W2…W2) represent the probability of N words or characters (tokens) appearing in a sentence, corresponding to those in the pre-trained language model dictionary. N )=P(W2|W1)P(W3|W1W2)…P(W N W1W2…W N-1 The probability of the entire sentence is calculated by multiplying the probabilities of each token. It should be noted that for a given set of k words, the probability of the (k+1)th word can be calculated using a model. In this embodiment, an autoregressive language model (GPT) is used. In practice, the appropriate model can be selected based on the specific circumstances; this embodiment does not limit the type of model used. Furthermore, since the calculated probability is relatively small, the logarithm can be applied to both sides of the equation when calculating the sentence's probability.

[0063] This embodiment describes the steps for extracting low-quality text when it is an incoherent sentence. The reasonableness of the sentence is evaluated by PPL. The higher the PPL of the sentence, the higher the probability that the sentence is low-quality text, which makes it easier to determine whether each sentence is an incoherent low-quality text.

[0064] As mentioned in the above embodiments, low-quality text also includes statements with insufficient knowledge representation. The extraction steps for statements with insufficient knowledge representation in the low-quality text were not described in detail in the above embodiments; this embodiment provides a supplementary explanation. These steps include:

[0065] The TextRank algorithm based on jieba word segmentation is used to filter keywords in each sentence;

[0066] A metric for measuring keyword density is calculated based on the number of keywords in each statement and the statement length.

[0067] Low-quality text is identified and extracted based on indicators, where the indicators are inversely proportional to the probability that a sentence is low-quality text.

[0068] In this embodiment, keyword density is used for filtering, aiming to identify sentences with fewer keywords, longer texts, and shorter texts by considering the number of keywords and text length. This embodiment uses the Jieba word segmentation algorithm (TextRank) based on co-occurrence frequency to filter keywords for each sentence. Keywords are extracted by statistically combining co-occurring words and frequencies, using frequency to represent co-occurrence weight.

[0069] In this embodiment, an index *r* is designed to measure keyword density and distinguish statements with insufficient knowledge expression. The smaller the *r* of a statement, the fewer keywords it contains, the less valuable information it contains, and the higher the probability that the statement is low-quality text. Let the number of keywords be N. k The text length is l, and N k If the maximum value of r is 30, then the formula for calculating r is r = (N k / m)[exp(N k / m) / min(l,50)][max(l-10,0) / l], where m is the importance enhancement threshold, and its value is related to N. k The maximum value is related to the value of r. min(l,50) is used to ensure that the value of r of the statement does not decrease due to the length of the statement. The 10 in l-10 represents the minimum word count threshold of the statement. When l < 10, r is 0. When N k When / m > 1, the growth rate of r increases, thus widening the gap in r values ​​between different statements. In other words, when N... k When / m>1, the larger the r of the statement, the lower the probability that the statement is low-quality text. Figure 2 This is a distribution chart of r-index values ​​ranging from 0 to 0.35 in 60,000 sampled data points, as shown below. Figure 2 As shown, when r is 0.05, more than 20,000 statements are considered to have insufficient knowledge expression, which is in line with actual expectations. Therefore, r = 0.05 can be used for keyword density filtering. It should be noted that the r value is related to the distinction criteria for statements with insufficient knowledge expression and the quality of the statements. The worse the quality of the statements and the higher the distinction criteria, the larger r is, and the more statements are considered to be low-quality text.

[0070] This embodiment describes the steps for extracting low-quality text when it is a statement with insufficient knowledge expression. An index r is designed based on the keyword density screening method. The smaller the r of a statement, the higher the probability that the statement is low-quality text, which makes it easier to determine whether each statement is low-quality text with insufficient knowledge expression.

[0071] As mentioned in the above embodiments, low-quality text also includes recurring statements. The extraction steps for low-quality text consisting of recurring statements were not described in detail in the above embodiments; this embodiment provides a supplementary explanation. These steps include:

[0072] Obtain the sentence vector;

[0073] Construct an Annoy index based on sentence vectors;

[0074] The Annoy index is used to count statements whose similarity exceeds a preset threshold.

[0075] The number of similar questions in each sentence is sorted to identify low-quality text for extraction. The number of similar questions is proportional to the probability that the sentence is low-quality text.

[0076] In this embodiment, sentence vectors are manually added to the beginning of sentences to represent their meaning; therefore, different sentences have different sentence vectors. This embodiment uses a high-dimensional vector approximate indexing algorithm (Approximate Nearest Neighbors Oh Yeah, Annoy) to count sentences with similarity exceeding a preset threshold based on their sentence vectors. Annoy builds a binary tree of high-dimensional vectors to find the nearest point to any query. After counting sentences with similarity exceeding the preset threshold, they are sorted according to the number of similar questions in each sentence. The more similar questions, the higher the repetition rate of the sentence, and the higher the probability that the sentence is low-quality text. For example, if the preset threshold is set to 80%, all sentences with similarity exceeding 80% are counted, and then low-quality text is filtered out based on the number of similar questions for each sentence. Alternatively, a brute-force search can be used to traverse all sentences and calculate similarity to count sentences with similarity exceeding the preset threshold. This embodiment does not limit the search method.

[0077] This embodiment describes the steps for extracting low-quality text when it consists of recurring sentences. Sentence vector clustering is used to count sentences with similarity exceeding a preset threshold. Low-quality text is determined based on the number of similar questions for each sentence. The more similar questions a sentence has, the higher the probability that the sentence is low-quality text, making it easier to determine whether each sentence is a recurring low-quality text.

[0078] The above embodiments detail the screening methods for three types of low-quality text: incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences. If the low-quality text does not fall into any of these three categories, a new screening step needs to be determined. This step includes:

[0079] The sampled data is filtered based on keywords or regular expressions to obtain low-quality text, which is then extracted.

[0080] In this embodiment, regular expressions can be used to filter timestamps, representing numbers through codes to filter timestamps in a certain preset format. Obviously, the filtering method is not limited to filtering based on keywords or regular expressions; appropriate filtering methods can be flexibly designed according to specific needs. After filtering the sampled data, a manual preliminary screening can be performed to confirm the existence of obviously low-quality text.

[0081] This embodiment provides supplementary explanations for situations where low-quality text does not fall into any of the categories of incoherent sentences, sentences with insufficient knowledge expression, or repetitive sentences, thus expanding the screening content and improving the screening method.

[0082] The above embodiments filtered four types of low-quality text. This embodiment trains the text quality classification model based on the filtered high-quality and low-quality text to adjust various parameters of the text quality classification model. Furthermore, since the high-quality text obtained after cleaning is still quite large, far exceeding the number of tokens required for pre-training the text quality classification model, it is necessary to sample it. This step includes:

[0083] The remaining sampled data after extracting low-quality text was determined to be high-quality text;

[0084] The text quality classification model was trained and optimized based on low-quality and high-quality text.

[0085] Calculate the sampling rate of high-quality text, and sample the high-quality text according to the sampling rate to obtain the pre-training dataset.

[0086] This embodiment determines that the remaining sampled data after extracting low-quality text is high-quality text. The text quality classification model is trained and optimized based on both low-quality and high-quality text to adjust its various parameters. Furthermore, this embodiment also samples cleaned internet corpora to provide a pre-training dataset for the text quality classification model, thereby improving its understanding capabilities.

[0087] The above embodiments did not provide a detailed explanation of the steps for training and optimizing the text quality classification model based on low-quality and high-quality text. This embodiment supplements this explanation. These steps include:

[0088] Construct a text quality classification training set based on low-quality and high-quality text;

[0089] The text quality classification model was trained and optimized using BERT based on the text quality classification training set.

[0090] In this embodiment, a text quality classification training set is constructed using both low-quality and high-quality text. To ensure the balance of the training set, a similar number of high-quality and low-quality texts are selected. Then, a denoising-based custom language model (BERT) is used to train and optimize the text quality classification model using binary classification. Furthermore, when finally confirming the training effect, i.e., evaluating the text quality classification model, manual evaluation or machine evaluation can be used, choosing the appropriate evaluation method based on the actual situation. It should be noted that the text quality classification model achieving its expected results means that most of the high-quality text is retained, while obviously low-quality text is removed.

[0091] This embodiment details the steps for training and optimizing a text quality classification model using low-quality and high-quality text. A text quality classification training set is constructed based on the low-quality and high-quality text. BERT is then used to train and optimize the text quality classification model based on the training set to adjust various parameters of the model.

[0092] The above embodiments did not provide a detailed explanation of the sampling rate calculation for high-quality text; this embodiment supplements this explanation. The step includes:

[0093] The distribution of high-quality text at the article level, the total number of tokens for high-quality text, and the token distribution for each article are statistically analyzed.

[0094] The sampling rate is calculated based on the target token count and the total token count of the high-quality text.

[0095] Figure 3 This application provides a probability distribution diagram of the number of articles. Figure 4 This application provides a probability distribution diagram of the number of tokens for an article. In this embodiment, since the text quality classification model needs to read the filtered high-quality text in the form of articles, it is necessary to statistically analyze the distribution of high-quality text at the article level. See [link to relevant documentation]. Figure 3 Furthermore, since the implementation tends to sample articles with more words—as longer articles often indicate more valuable information—it is also necessary to calculate the total number of tokens for high-quality texts and the token distribution for each article. (See [link to relevant documentation]). Figure 4 The formula for calculating the sampling rate is W = 1 - T. k / (T total *P k ), T k T represents the target number of tokens. total P represents the total number of tokens. kLet be the probability that the probability of high-quality text is greater than k, where k is the minimum probability of high-quality text to be retained. It should be noted that this sampling rate should ensure that as many high-quality texts as possible are sampled, and that at least the target number of tokens is obtained. Depending on the actual situation, the actual sampling rate can be set slightly higher than the calculated sampling rate. After calculating the sampling rate, all high-quality texts are sampled according to the sampling rate and the principle of random uniform sampling to obtain the pre-training dataset.

[0096] This embodiment provides a detailed explanation of the steps for calculating the sampling rate. The sampling rate is calculated based on the number of target tokens and the total number of tokens in the high-quality text, so that the high-quality text can be sampled according to the sampling rate to obtain the pre-training dataset.

[0097] The text quality cleaning method has been described in detail in the above embodiments. This application also provides embodiments of a text quality cleaning apparatus. It should be noted that this application describes the apparatus embodiments from a hardware perspective.

[0098] Figure 5 A structural diagram of a text quality cleaning apparatus provided in another embodiment of this application is shown below. Figure 5 As shown, the device includes:

[0099] Memory 20 is used to store computer programs;

[0100] The processor 21 is configured to implement the steps of the text quality cleaning method mentioned in the above embodiments when executing a computer program.

[0101] The text quality cleaning device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0102] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0103] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the text quality cleaning method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, high-quality text.

[0104] In some embodiments, the text quality cleaning device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0105] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the quality cleaning apparatus for this text and may include more or fewer components than illustrated.

[0106] The text quality cleaning apparatus provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the above-described text quality cleaning method, with the same effect.

[0107] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the text quality cleaning method described in the above method embodiments.

[0108] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] The computer-readable storage medium provided in this application includes the text quality cleaning method mentioned above, with the same effect.

[0110] The foregoing has provided a detailed description of the text quality cleaning method, apparatus, and medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0111] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A text quality cleaning method, characterized in that, include: Sampling was performed on internet corpora to obtain sampled data; Extract pre-defined low-quality text from the sampled data. The low-quality text includes incoherent sentences, sentences with insufficient knowledge expression, and sentences that appear repeatedly. The pre-defined low-quality text includes at least two of these categories. Wherein, if the low-quality text is a statement with insufficient knowledge representation, then the step of extracting the pre-defined low-quality text from the sampled data includes: The TextRank algorithm based on jieba word segmentation is used to filter keywords for each statement; an index for measuring keyword density is calculated based on the number of keywords and the statement length for each statement; low-quality text is identified and extracted based on the index, wherein the index is inversely proportional to the probability that the statement is low-quality text; Among them, the index of keyword density The number of keywords is The text length is , m is the importance enhancement threshold, and its value is related to It is related to the maximum value. This is used to ensure that the indicator value of a statement does not decrease due to the statement becoming too long.

2. The text quality cleaning method according to claim 1, characterized in that, If the low-quality text is an incoherent sentence, then extracting the pre-defined low-quality text from the sampled data includes: Calculate the perplexity of each statement in the sampled data; The low-quality text is determined and extracted based on the perplexity of each statement, wherein the perplexity is proportional to the probability that the statement is the low-quality text.

3. The text quality cleaning method according to claim 1, characterized in that, If the low-quality text is a recurring statement, then extracting the pre-defined low-quality text from the sampled data includes: Obtain the sentence vector; Construct the Annoy index based on the sentence vector; The Annoy index is used to count statements whose similarity exceeds a preset threshold. The number of similar questions in each of the statements is sorted to identify and extract the low-quality text, wherein the number of similar questions is proportional to the probability that the statement is the low-quality text.

4. The text quality cleaning method according to claim 1, characterized in that, If the low-quality text does not belong to any of the categories of incoherent sentences, sentences with insufficient knowledge expression, and repetitive sentences, then the step of extracting the pre-defined low-quality text from the sampled data includes: The sampled data is filtered based on keywords or regular expressions to obtain low-quality text, which is then extracted.

5. The text quality cleaning method according to claim 1, characterized in that, Also includes: The remaining sampled data after extracting the low-quality text is determined to be high-quality text; The text quality classification model is trained and optimized based on the low-quality text and the high-quality text. Calculate the sampling rate of the high-quality text, and sample the high-quality text according to the sampling rate to obtain a pre-training dataset.

6. The text quality cleaning method according to claim 5, characterized in that, The step of training and optimizing the text quality classification model based on the low-quality text and the high-quality text includes: Construct a text quality classification training set based on the low-quality text and the high-quality text; The text quality classification model is trained and optimized using BERT based on the text quality classification training set.

7. The text quality cleaning method according to claim 5, characterized in that, The calculation of the sampling rate of the high-quality text includes: The distribution of the high-quality text at the article level, the total number of tokens for the high-quality text, and the distribution of the number of tokens for each article are statistically analyzed. The sampling rate is calculated based on the target number of tokens and the total number of tokens in the high-quality text.

8. A text quality cleaning device, characterized in that, Includes memory used to store computer programs; A processor, configured to implement the steps of the text quality cleaning method as described in any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the text quality cleaning method as described in any one of claims 1 to 7.

Images