Text quality evaluation method and device, equipment, storage medium and product

Abstract

The invention discloses a text quality evaluation method and device, equipment, a storage medium and a product, the score of each original paragraph is determined by scoring the paragraph dimension of an original text, and the original paragraph with the lower score is regarded as a low-quality text; according to the method, sentence filtering is performed on an original paragraph, sentences with poor quality are removed, a reconstructed paragraph corresponding to the original paragraph is obtained, the paragraph text quality is determined by analyzing the scores of the original paragraph and the reconstructed paragraph, if the quality of the reconstructed paragraph is greatly improved compared with the quality of the original paragraph, the original paragraph is regarded as a low-quality text, and the text quality In the text quality evaluation process, manual text labeling is not needed, and the accuracy of text quality evaluation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a text quality evaluation method and device, equipment, storage medium and product. BACKGROUND

[0002] Large language model (LLM) is an artificial intelligence model based on deep learning technology, which has strong language understanding and generation ability. At present, large language models have received widespread attention in society. Large language models need large-scale text data for training, and high-quality text data can better reflect human language habits and expression methods, so that the large language model can more accurately understand and process natural language. Therefore, it is necessary to evaluate the quality of text data.

[0003] At present, the quality of text is mainly evaluated by building a classification model. This method needs to prepare various types of text in advance, manually label the quality of the text, train a classification model according to the labeled data, and then use the model to classify the quality of other text. This method requires people to manually label according to experience, and there are labeling errors in the labeling process, and it has a certain subjectivity, which cannot ensure the accuracy of the text label, so that the classification model obtained by training cannot accurately evaluate the quality of the text. SUMMARY

[0004] Therefore, the present application provides a text quality evaluation method, device, equipment, storage medium and product, which can divide the text into paragraphs and filter the sentences, calculate the scores of each paragraph before and after sentence filtering, and determine the quality of the paragraph text by analyzing the scores of each paragraph before and after sentence filtering. No manual text labeling is required, which improves the accuracy of text quality evaluation.

[0005] To achieve the above purpose, the embodiment of the present application provides a text quality evaluation method, comprising:

[0006] Obtain an original text and divide the original text into paragraphs to obtain at least one original paragraph;

[0007] Score the original paragraph to obtain the score of the original paragraph;

[0008] Perform sentence filtering processing on the original paragraph based on a preset sentence filtering rule to obtain a reconstructed paragraph;

[0009] Score the reconstructed paragraph to obtain the score of the reconstructed paragraph;

[0010] determining that the original paragraph is low-quality text when the score of the original paragraph is less than a preset qualified threshold, or when a difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold.

[0011] As an improvement of the above scheme, the scoring of the original paragraph to obtain the score of the original paragraph comprises:

[0012] scoring the paragraph elements of the original paragraph, and integrating the scores of all the paragraph elements in the original paragraph to obtain the score of the original paragraph; wherein the paragraph elements comprise at least one of lexical richness, text readability, information density, semantic consistency and emotional consistency.

[0013] As an improvement of the above scheme, the scoring of the paragraph elements of the original paragraph to obtain the score of the original paragraph comprises:

[0014] measuring the degree of lexical diversity of the original paragraph by using the average word symbol ratio of the language segment to obtain a lexical richness score;

[0015] calculating a text readability score according to the number of words, the number of phonetic radicals and the number of sentences in the original paragraph, wherein the text readability score is negatively correlated with the ratio of the number of words to the number of sentences and the ratio of the number of phonetic radicals to the number of words, respectively; or calculating the text readability according to the number of sentence average word class, the number of sentence average effective characters, the average effective character frequency ranking and the average effective character stroke number in the original paragraph, wherein the text readability score is positively correlated with the number of sentence average effective characters and the average effective character frequency ranking, respectively, and the text readability score is negatively correlated with the number of sentence average word class and the average effective character stroke number, respectively;

[0016] determining an information density score according to the information density of the original paragraph; wherein the information density score is positively correlated with the information density;

[0017] determining the similarity between all the sentences in the original paragraph, and calculating a semantic consistency score according to the similarity;

[0018] determining the emotional score of each of the sentences in the original paragraph according to the mapping relationship between the preset emotional type and the emotional score, and calculating an emotional consistency score according to the emotional scores of all the sentences in the original paragraph;

[0019] The score of the original paragraph is obtained by weighted summation of the lexical richness score, the text readability score, the information density score, the semantic consistency score, and the sentiment consistency score.

[0020] As an improvement to the above solution, the process of filtering the original paragraph based on preset sentence filtering rules to obtain a reconstructed paragraph includes:

[0021] Determine the sentence elements of each sentence in the original paragraph; wherein, the sentence elements include at least one of the following: number of lexical units, number of special symbols, number of stop words, word repetition rate, number of spelling errors, number of grammatical errors, and sentence structural integrity;

[0022] Sentences whose sentence elements do not conform to the preset sentence filtering rules are discarded to obtain reconstructed paragraphs.

[0023] As an improvement to the above solution, the step of discarding sentences whose sentence elements do not conform to the preset sentence filtering rules to obtain the reconstructed paragraph includes:

[0024] When the number of lexical units is less than a first set threshold, the number of special symbols is greater than a second set threshold, the number of stop words is less than a third set threshold, the word repetition rate is greater than a fourth set threshold, the number of spelling errors is greater than a fifth set threshold, the number of grammatical errors is greater than a sixth set threshold, or the sentence structure is incomplete, the sentence is discarded to obtain a reconstructed paragraph.

[0025] As an improvement to the above scheme, the sentence structure integrity must meet the following conditions to be complete: the sentence includes a subject and a predicate, and the number of nouns in the sentence is greater than or equal to a set word count threshold.

[0026] As an improvement to the above solution, the process of filtering the original paragraph based on preset sentence filtering rules to obtain a reconstructed paragraph includes:

[0027] Based on preset sentence filtering rules, the original paragraphs with scores greater than the preset qualified threshold are subjected to sentence filtering processing to obtain reconstructed paragraphs.

[0028] As an improvement to the above solution, the method further includes:

[0029] When the score of the original paragraph is greater than or equal to the preset qualified threshold, and the difference between the score of the reconstructed paragraph and the score of the original paragraph is less than or equal to the set difference threshold, the original paragraph is determined to be high-quality text; wherein, the high-quality text is used for training a large language model.

[0030] To achieve the above objectives, embodiments of the present invention also provide a text quality assessment device, comprising:

[0031] The text acquisition module is used to acquire raw text and divide the raw text into paragraphs to obtain at least one raw paragraph.

[0032] The first scoring module is used to score the original paragraph and obtain a score for the original paragraph;

[0033] The sentence filtering module is used to perform sentence filtering processing on the original paragraph based on preset sentence filtering rules to obtain the reconstructed paragraph;

[0034] The second scoring module is used to score the reconstructed paragraph and obtain a score for the reconstructed paragraph;

[0035] The quality assessment module is used to determine that the original paragraph is low-quality text when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold.

[0036] To achieve the above objectives, embodiments of the present invention also provide a text quality assessment device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the text quality assessment method as described in any of the above embodiments.

[0037] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the text quality assessment method as described in any of the above embodiments.

[0038] To achieve the above objectives, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the text quality assessment method as described in any of the above embodiments.

[0039] Compared with existing technologies, the text quality assessment method, apparatus, device, storage medium, and product disclosed in this invention first divide the acquired original text into paragraphs to obtain at least one original paragraph; then, the original paragraph is scored to obtain a score; the original paragraph is processed by sentence filtering based on preset sentence filtering rules to obtain a reconstructed paragraph, and the reconstructed paragraph is scored to obtain a score; when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold, the original paragraph is determined to be low-quality text. Therefore, this embodiment of the invention determines the score of each original paragraph by scoring the original text at the paragraph level, and regards the original paragraph with a lower score as low-quality text; by filtering sentences in the original paragraphs to remove sentences of poor quality, reconstructed paragraphs corresponding to the original paragraphs are obtained; the quality of the paragraph text is determined by analyzing the scores of the original paragraphs and the reconstructed paragraphs; if the quality of the reconstructed paragraphs is significantly improved compared to the original paragraphs, the original paragraphs are regarded as low-quality text. In the text quality assessment process, no manual text annotation is required, which improves the accuracy of text quality assessment. Attached Figure Description

[0040] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a text quality assessment method according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of a paragraph scoring structure provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the sentence filtering process provided in an embodiment of the present invention;

[0044] Figure 4 This is a flowchart illustrating a text quality assessment method according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of a text quality assessment device provided in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of a text quality assessment device provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] See Figure 1 This is a flowchart illustrating a text quality assessment method provided in an embodiment of the present invention.

[0049] Specifically, the text quality assessment method includes steps S1 to S5:

[0050] S1. Obtain the original text and divide the original text into paragraphs to obtain at least one original paragraph.

[0051] Specifically, the original text can be obtained from news websites, novel databases, Wikipedia, Baidu Encyclopedia, academic databases, or libraries. The source of the original text is not limited to the above specific sources. It can be obtained through web scraping technology or imported by users. Here, we will not limit the source and acquisition method of the original text.

[0052] For example, the paragraph division method is as follows:

[0053] I. Paragraph Division Based on Punctuation and Formatting: 1. Division by Specific Punctuation Marks: In most texts, punctuation marks such as periods, question marks, and exclamation marks usually indicate the end of a sentence. When these punctuation marks are followed by a certain number of spaces or line breaks, it is very likely that a paragraph is separated. 2. Judging by Line Breaks: Many texts use line breaks to indicate paragraph separation during typesetting. 3. Based on Indentation Formatting: In some documents, paragraphs are represented by first-line indentation, which is common in academic papers, formal reports, and other documents.

[0054] II. Paragraph division based on the logical structure of the text: 1. Division by topic: Analyzing the content of the text, when the topic or theme of the text changes significantly, new paragraphs can usually be created. 2. Division by chapter titles: For longer documents, such as books and long reports, chapter titles are usually used to divide different parts.

[0055] It is worth noting that the specific paragraph division method is not limited to the above methods, and the specific division method can be determined according to the actual situation.

[0056] S2. Score the original paragraph to obtain a score for the original paragraph.

[0057] Specifically, the method for evaluating paragraph scores is as follows: after the paragraphs are divided, each paragraph is scored according to its paragraph dimension to obtain a score for each paragraph.

[0058] S3. Based on preset sentence filtering rules, the original paragraph is subjected to sentence filtering processing to obtain the reconstructed paragraph.

[0059] Specifically, sentence filtering rules are pre-set, and these rules are used to filter sentences in the original paragraph, removing sentences of poor quality. Based on the sentence dimension, sentences of poor quality are removed, and the remaining high-quality sentences are reorganized to obtain the reconstructed paragraph.

[0060] S4. Score the reconstructed paragraph to obtain a score for the reconstructed paragraph.

[0061] Specifically, the reconstructed paragraphs are scored at the paragraph level, taking into account factors such as lexical richness, text readability, information density, semantic consistency, and sentiment consistency, and a score is calculated for the reconstructed paragraphs.

[0062] S5. When the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold, the original paragraph is determined to be low-quality text.

[0063] Specifically, the score of the original paragraph is compared with the score of the reconstructed paragraph. If the score of the original paragraph is less than the preset threshold, it indicates that the quality of the original paragraph is poor, and therefore the original paragraph is regarded as low-quality text. If the score of the original paragraph is greater than or equal to the preset threshold, but the score of the original paragraph is higher than the score of the reconstructed paragraph and the difference between the two is large, it indicates that there is a lot of low-quality data in the original paragraph, and therefore the original paragraph is regarded as low-quality text.

[0064] Compared with existing technologies, this invention determines the score of each original paragraph by scoring the original text at the paragraph level, and regards the original paragraph with a lower score as low-quality text. By filtering sentences in the original paragraphs and removing sentences of poor quality, reconstructed paragraphs corresponding to the original paragraphs are obtained. The quality of the paragraph text is determined by analyzing the scores of the original paragraphs and the reconstructed paragraphs. If the quality of the reconstructed paragraphs is significantly improved compared to the original paragraphs, the original paragraphs are regarded as low-quality text. In the text quality evaluation process, the training text (i.e., the original text) is evaluated by quantifying information at the essential level of text, such as sentences and paragraphs. It is applicable to general training texts, does not require manual text annotation, and improves the accuracy of text quality evaluation.

[0065] In a preferred embodiment, based on steps S1 to S5, step S2, which involves scoring the original paragraph to obtain its score, includes:

[0066] The paragraph elements of the original paragraph are scored, and the scores of all the paragraph elements in the original paragraph are combined to obtain the score of the original paragraph; wherein, the paragraph elements include at least one of lexical richness, text readability, information density, semantic consistency and sentiment consistency.

[0067] For example, see Figure 2 The paragraph scoring framework diagram shown illustrates the scoring of both original and reconstructed paragraphs at the paragraph level. Original paragraphs of the original text are scored to obtain their scores; reconstructed paragraphs of the reconstructed text are scored to obtain their scores. The paragraph-level scoring involves evaluating various paragraph elements, such as lexical richness, text readability, information density, semantic consistency, and sentiment consistency. This includes not only measuring the surface structure of sentences but also semantic levels such as semantic consistency and sentiment consistency, which can be used to measure deeper aspects of the text. It is understood that "reconstructed text" refers to the combination of all reconstructed paragraphs.

[0068] It is worth noting that paragraph elements are not limited to the specific elements mentioned above. Necessary paragraph elements can be selected according to the actual situation, and no specific restrictions are made here.

[0069] Further, the step of scoring the paragraph elements of the original paragraph and summing the scores of all the paragraph elements in the original paragraph to obtain the score of the original paragraph includes:

[0070] The lexical richness score is obtained by measuring the lexical diversity of the original paragraph using the average word-to-sign ratio of the language segments.

[0071] A text readability score is calculated based on the number of words, syllables, and sentences in the original paragraph, wherein the text readability score is negatively correlated with the ratio of the number of words to the number of sentences and the ratio of the number of syllables to the number of words, respectively; or, the text readability is calculated based on the average number of parts of speech per sentence, the average number of effective characters per sentence, the average effective character frequency ranking, and the average number of effective character strokes in the original paragraph, wherein the text readability score is positively correlated with the average number of effective characters per sentence and the average effective character frequency ranking, respectively, and negatively correlated with the average number of parts of speech per sentence and the average number of effective character strokes, respectively.

[0072] An information density score is determined based on the information density of the original paragraph; wherein the information density score is positively correlated with the information density.

[0073] Determine the pairwise similarity between all sentences in the original paragraph, and calculate the semantic consistency score based on the similarity.

[0074] Based on a preset mapping relationship between sentiment type and sentiment score, the sentiment score of each sentence in the original paragraph is determined according to the sentiment type of each sentence in the original paragraph, and the sentiment consistency score is calculated based on the sentiment scores of all sentences in the original paragraph.

[0075] The score of the original paragraph is obtained by weighted summation of the lexical richness score, the text readability score, the information density score, the semantic consistency score, and the sentiment consistency score.

[0076] For example, in the evaluation of paragraphs, paragraph scores are calculated by comprehensively considering various paragraph elements such as lexical richness, text readability, information density, semantic consistency, and sentiment consistency. The scoring methods for each paragraph element are described below:

[0077] (1) Lexical richness. The Mean Token to Type Ratio across Language Divisions (MTLD) is used to measure the diversity of a text's vocabulary. Lexical richness can be assessed by calculating the repetition rate of words in the text, resulting in a lexical richness value denoted as mtld_value. This method is applicable to both Chinese and English lexical richness measurement. In actual evaluation, mtld_value needs to be converted into a score. mtld_value and lexical richness score are positively correlated, for example: or, The specific formula for converting mtld_value to a lexical richness score is not limited to the formula mentioned above, as long as mtld_value and lexical richness score are positively correlated. Other existing techniques can also be used to determine the lexical richness score.

[0078] (2) Text Readability. For English text, the Flesch-Kincaid algorithm can be used to score text readability. This is a commonly used readability test metric to evaluate the readability of text. The Flesch-Kincaid algorithm returns a readability score, typically ranging from 0 to 100. A higher score indicates that the text is easier to read, denoted as FK_value. For Chinese text, an empirical calculation formula needs to be fitted based on the average number of part-of-speech categories per sentence (i.e., average number of part-of-speech categories per sentence), the average number of effective characters per sentence (i.e., average number of effective characters per sentence), the average frequency ranking of effective characters, and the average number of strokes of effective characters. The resulting readability score is denoted as zh_value.

[0079]

[0080] in,

[0081]

[0082] zh_value = -3.516 average number of parts of speech categories + 0.38 average number of valid characters + 0.016 average frequency of valid characters + 9.34 average number of strokes of valid characters -54.18;

[0083] The coefficients and intercepts of the text readability score formula, FK_value calculation formula, and zh_value calculation formula are not limited to the specific values ​​mentioned above, nor are the specific calculation formulas limited to the specific forms mentioned above. For example, text readability score = square of FK_value or zh_value, FK_value = set base value - square of (number of words / number of sentences) - (number of syllables - number of sentences), zh_value = -3 square root of the average number of part-of-speech categories per sentence + 0.3 square root of the average number of effective characters per sentence + 0.01 square root of the average frequency of effective characters + 9 square root of the average number of strokes of effective characters - 60. These calculation formulas can be set according to the actual situation and are not limited here.

[0084] It is worth noting that, taking a certain original paragraph as an example, the average effective character frequency ranking is determined in the following way: the frequency of each effective character in the original paragraph is counted, the effective characters in the original paragraph are ranked according to the calculated frequency of the characters, the frequency ranking of all effective characters in the original paragraph is obtained, and the average frequency ranking of all effective characters in the original paragraph is calculated to obtain the average effective character frequency ranking of the original paragraph.

[0085] (3) Information density. The information density score can be calculated using entropy, by counting the frequency p(w) of each word in the text.i Then entropy Information density N represents the total number of words.

[0086]

[0087] It is worth noting that the specific formula for calculating the information density score is not limited to the above specific form. For example, the information density score = arctanD can be set according to the actual situation, and is not limited here.

[0088] (4) Semantic consistency. First, sentence vectors for all sentences are calculated using an open-source pre-trained model. Then, a similarity matrix is ​​constructed between each pair of sentences in the paragraph. The similarity can be calculated using cosine vectors. The closer the sentences are semantically, the closer the cosine vector calculation result is to 1.

[0089]

[0090] Taking a simple sentence as an example, s ij This represents the cosine result of the i-th and j-th sentence vectors in the similarity matrix.

[0091] It is worth noting that, in addition to using cosine vectors to calculate similarity, Euclidean distance, Manhattan distance, and other similarity calculation methods can also be used to determine semantic consistency. Here, we will not limit the specific calculation method of semantic consistency.

[0092] (5) Sentiment Consistency. The sentiment model can adopt industry-standard models or be trained independently. First, it is necessary to calculate the sentiment of the sentence, which is generally divided into positive and negative. Positive sentiment is scored as 1 point and negative sentiment is scored as -1 point.

[0093]

[0094] Finally, by combining all the above indicators, the paragraph score is calculated:

[0095] Paragraph score = w1 * vocabulary richness score + w2 * text readability score + w3 *

[0096] Information density score + w4 * semantic consistency score + w5 * sentiment consistency score;

[0097] The weights w1 to w5 can be dynamically set according to the actual situation.

[0098] It is worth noting that positive emotions include pleasure, excitement, and surprise, while negative emotions include sadness, shock, and fear. In addition to the above-mentioned positive and negative categories, the emotional state of a sentence can also be further subdivided into pleasure, surprise, sadness, shock, and fear, etc. Pleasure and surprise are positively correlated with emotional consistency scores, as are sadness, shock, and fear. Moreover, the degree to which these emotions contribute to the emotional consistency score can be the same or different. That is, the mapping relationship between emotion type and emotion score can be set according to the actual situation. Here, we will not make specific restrictions on the emotional classification and calculation method of sentences.

[0099] In this implementation, when considering quantitative text metrics, the evaluation of paragraphs takes into account the applicability of the language as much as possible, and the methods involved are selected to meet the requirements of both Chinese and English. When special metrics, such as text readability, cannot meet the bilingual requirements, calculation rules that meet the score calculation requirements are customized for the text.

[0100] In a preferred embodiment, based on steps S1 to S5, step S3 involves performing sentence filtering on the original paragraph according to preset sentence filtering rules to obtain a reconstructed paragraph, including:

[0101] Determine the sentence elements of each sentence in the original paragraph; wherein, the sentence elements include at least one of the following: number of lexical units, number of special symbols, number of stop words, word repetition rate, number of spelling errors, number of grammatical errors, and sentence structural integrity;

[0102] Sentences whose sentence elements do not conform to the preset sentence filtering rules are discarded to obtain reconstructed paragraphs.

[0103] For example, see Figure 3 The flowchart shown illustrates the sentence filtering process. At the sentence level, it performs checks on the number of tokens (i.e., special characters), stop words, word repetition rate, spelling errors, grammatical errors, and sentence structure. It analyzes whether each sentence is of high quality, retains sentences that meet the filtering rules, and removes sentences that do not, i.e., sentences of poor quality. This achieves sentence filtering, for example, removing sentences with a small number of tokens or sentences with an excessive number of special characters.

[0104] Optionally, the sentence structure integrity check result includes complete and incomplete. A complete sentence structure requires the following conditions: the sentence includes a subject and a predicate, and the number of nouns in the sentence is greater than or equal to a set word count threshold. The specific value of the set word count threshold can be set according to actual circumstances and is not limited here.

[0105] It is worth noting that sentence elements are not limited to the specific examples mentioned above, and may also include other elements, such as changing "number of stop words" to "usage of modal particles" or "usage of non-substantive words," etc., which are not limited here.

[0106] Furthermore, the step of discarding sentences whose sentence elements do not conform to preset sentence filtering rules to obtain reconstructed paragraphs includes:

[0107] When the number of lexical units is less than a first set threshold, the number of special symbols is greater than a second set threshold, the number of stop words is less than a third set threshold, the word repetition rate is greater than a fourth set threshold, the number of spelling errors is greater than a fifth set threshold, the number of grammatical errors is greater than a sixth set threshold, or the sentence structure is incomplete, the sentence is discarded to obtain a reconstructed paragraph.

[0108] Specifically, a sentence filter is constructed by designing a series of evaluation rules (i.e., sentence filtering rules) to determine whether the current sentence meets the high-quality requirements. For example, the evaluation rules are detailed in the table below:

[0109]

[0110] The following is a detailed explanation of the meaning of the rules in the table above:

[0111] (1) Token count check (lexical count check). Token is a fundamental term in natural language processing and large language model training. A token can be considered the smallest comprehensible unit, such as a word, a character, a symbol, or a letter. The significance of token count check is to initially determine whether the current sentence contains sufficient information. Sentences with too few tokens are usually meaningless. For example, the sentence "My wife" encountered in the training text does not express any meaning and belongs to low-quality text, which should be filtered out.

[0112] (2) Special Character Check. Special characters are primarily targeted at low-quality webpage text or garbled text, excluding code and mathematical formulas. These special characters are usually caused by uncleaned or poorly cleaned text, affecting readability and introducing noise during subsequent training. For example, a web text dataset might contain text like the following: "...reliability, ventilation and heat dissipation, wiring pass-through rate.\na. Outer layer size. Size too large." When printing long lines, ... with the core element as the center, reading this text will be greatly disturbed by the special symbols. Such low-quality sentences should be filtered out.

[0113] (3) Stop-word checking. Stop words are usually words that appear very frequently in a language but carry little meaningful information during actual content analysis. For example, "the", "is", "at", "which" in English, and "在", "从", "的", "得" in Chinese. Although these stop words do not have much meaning, they are essential connecting words for people to write and communicate. It is normal for a sentence to contain more than one stop word; otherwise, the overall sentence would be too rigid and lack the flexibility of a normal sentence. Especially for English, in the sentence "The grass has grown up", if the stop words are removed and it becomes "Grass has grown up", it is neither in line with the habit nor grammatically correct.

[0114] (4) Word repetition rate checking. This type of checking is to determine whether the current sentence is abnormal. The repeated appearance of certain words affects normal reading. For example: "Four-column hydraulic press\nSingle-column hydraulic press\nGantry hydraulic press\n?\nFrame hydraulic press\n?\n......\nSingle-column hydraulic press\nGantry hydraulic press\nFrame hydraulic press\nFour-column hydraulic press\nHorizontal hydraulic press", the repetition rate of the word "hydraulic press" in the current sentence is too high. The reason is that there are repeated and circular sentences, and such sentences should be considered low-quality sentences.

[0115] (5) Spelling error checking. The main purpose of spelling error checking is to ensure the accuracy of the text used as training data. Spelling errors are regarded as a form of data noise. Eliminating this noise helps the model capture the essential features of the language more accurately, rather than learning incorrect word patterns due to spelling mistakes in the words.

[0116] (6) Grammar error checking. The purpose of grammar error checking is to ensure the accuracy and readability of the text. Grammar errors may lead to inconsistent text structures, which in turn affect the generalization ability of the model. For example, in a wrong text "The grasshas to grown up", while other similar texts are "The tree has grown up". After the model learns these two texts, it may learn incorrect patterns, resulting in grammar errors in subsequent text generation.

[0117] (7) Sentence structure checking. A high-quality and meaningful sentence generally contains at least a "subject" and a "predicate", and at least some nouns. For example, "我走" (I walk), "你跑" (You run), etc. These contain a subject and a predicate and are sentences. However, considering the above other rules, such sentences actually have little meaning and low quality. A better way of expression would be "我走回家了" (I walked home), "你跑回家吧" (You run home), etc.

[0118] It is understandable that, in considering quantitative text metrics, this implementation method takes into account the applicability of the language as much as possible when evaluating sentences, and strives to meet the scoring requirements of bilingual Chinese and English texts.

[0119] It is worth noting that the evaluation rules for sentence filters are not limited to the table above and can be set according to the actual situation. There are no restrictions here. Optionally, sentence filters are single-sentence filters, and the evaluation rules for single-sentence filters are set for single sentences.

[0120] In a preferred embodiment, based on steps S1 to S5, the sentence filtering process performed on the original paragraph based on preset sentence filtering rules in step S3 to obtain the reconstructed paragraph includes: performing sentence filtering process on the original paragraph with a score greater than the preset qualified threshold based on preset sentence filtering rules to obtain the reconstructed paragraph.

[0121] Specifically, in order to reduce the computational load of the text quality assessment method and improve the assessment efficiency, sentence filtering and re-scoring are not performed on original paragraphs with scores below the preset passing threshold. Sentence filtering and re-scoring are only performed on original paragraphs with scores above the preset passing threshold.

[0122] In a preferred embodiment, based on steps S1 to S5, the method further includes:

[0123] When the score of the original paragraph is greater than or equal to the preset qualified threshold, and the difference between the score of the reconstructed paragraph and the score of the original paragraph is less than or equal to the set difference threshold, the original paragraph is determined to be high-quality text; wherein, the high-quality text is used for training a large language model.

[0124] Specifically, after text quality assessment, high-quality texts are selected for training the large language model, enabling the trained large language model to perform natural language processing better.

[0125] Further, see Figure 4 To make the text quality assessment method clearer, the entire assessment process is briefly described below:

[0126] The original text is first divided into multiple paragraphs (e.g., ...). Figure 4 The system consists of two paragraphs (Paragraph 1 and Paragraph 2), each composed of multiple sentences. Each paragraph is first evaluated at the paragraph level, scoring based on lexical richness, text readability, information density, semantic consistency, and sentiment consistency. These scores are then weighted and summed. If a paragraph's score falls below a certain threshold, the text quality of that paragraph is deemed substandard.

[0127] If the score of the original paragraph is higher than the preset passing threshold, each sentence of the original paragraph will be processed through a sentence filter, which includes rules for checking the number of tokens, special symbols, stop words, word repetition rate, spelling errors, grammar errors, and sentence structure. Sentences that meet all the rules will be retained; otherwise, they will be labeled as low-quality sentences and will be discarded in subsequent paragraph scoring.

[0128] Sentences filtered by the sentence filter are reassembled into new paragraphs (i.e., reconstructed paragraphs). These reconstructed paragraphs are then scored again to calculate a new score. If the new score is higher than the previous score, and the increase exceeds a set difference threshold, the original paragraph will still be considered low-quality text.

[0129] Compared with existing technologies, the text quality assessment method provided in this invention first divides the acquired original text into paragraphs to obtain at least one original paragraph; then, it scores the original paragraphs to obtain a score; based on preset sentence filtering rules, it performs sentence filtering on the original paragraphs to obtain reconstructed paragraphs, and scores the reconstructed paragraphs to obtain a score; when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold, the original paragraph is determined to be low-quality text. Therefore, this invention determines the score of each original paragraph by scoring the original text at the paragraph level, and considers original paragraphs with lower scores as low-quality text; by filtering sentences in the original paragraphs to remove sentences of poor quality, it obtains reconstructed paragraphs corresponding to the original paragraphs; by analyzing the scores of the original paragraphs and the reconstructed paragraphs, it determines the text quality of the paragraphs; if the quality of the reconstructed paragraph is significantly improved compared to the original paragraph, the original paragraph is considered low-quality text. In the text quality assessment process, no manual text annotation is required, thus improving the accuracy of text quality assessment.

[0130] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a text quality assessment device provided in an embodiment of the present invention. The text quality assessment device 20 includes:

[0131] The text acquisition module 21 is used to acquire the original text and divide the original text into paragraphs to obtain at least one original paragraph.

[0132] The first scoring module 22 is used to score the original paragraph and obtain a score for the original paragraph;

[0133] The sentence filtering module 23 is used to perform sentence filtering processing on the original paragraph based on preset sentence filtering rules to obtain the reconstructed paragraph;

[0134] The second scoring module 24 is used to score the reconstructed paragraph and obtain a score for the reconstructed paragraph;

[0135] The quality assessment module 25 is used to determine that the original paragraph is low-quality text when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold.

[0136] In one implementation, the first scoring module 22 is specifically used for:

[0137] The process of scoring the original paragraph to obtain a score for the original paragraph includes:

[0138] The paragraph elements of the original paragraph are scored, and the scores of all the paragraph elements in the original paragraph are combined to obtain the score of the original paragraph; wherein, the paragraph elements include at least one of lexical richness, text readability, information density, semantic consistency and sentiment consistency.

[0139] In one implementation, the first scoring module 22 is specifically used for:

[0140] The lexical richness score is obtained by measuring the lexical diversity of the original paragraph using the average word-to-sign ratio of the language segments.

[0141] A text readability score is calculated based on the number of words, syllables, and sentences in the original paragraph, wherein the text readability score is negatively correlated with the ratio of the number of words to the number of sentences and the ratio of the number of syllables to the number of words, respectively; or, the text readability is calculated based on the average number of parts of speech per sentence, the average number of effective characters per sentence, the average effective character frequency ranking, and the average number of effective character strokes in the original paragraph, wherein the text readability score is positively correlated with the average number of effective characters per sentence and the average effective character frequency ranking, respectively, and negatively correlated with the average number of parts of speech per sentence and the average number of effective character strokes, respectively.

[0142] An information density score is determined based on the information density of the original paragraph; wherein the information density score is positively correlated with the information density.

[0143] Determine the pairwise similarity between all sentences in the original paragraph, and calculate the semantic consistency score based on the similarity.

[0144] Based on a preset mapping relationship between sentiment type and sentiment score, the sentiment score of each sentence in the original paragraph is determined according to the sentiment type of each sentence in the original paragraph, and the sentiment consistency score is calculated based on the sentiment scores of all sentences in the original paragraph.

[0145] The score of the original paragraph is obtained by weighted summation of the lexical richness score, the text readability score, the information density score, the semantic consistency score, and the sentiment consistency score.

[0146] In one embodiment, the sentence filtering module 23 is specifically used for:

[0147] Determine the sentence elements of each sentence in the original paragraph; wherein, the sentence elements include at least one of the following: number of lexical units, number of special symbols, number of stop words, word repetition rate, number of spelling errors, number of grammatical errors, and sentence structural integrity;

[0148] Sentences whose sentence elements do not conform to the preset sentence filtering rules are discarded to obtain reconstructed paragraphs.

[0149] In one implementation, discarding sentences whose sentence elements do not conform to preset sentence filtering rules to obtain a reconstructed paragraph includes:

[0150] When the number of lexical units is less than a first set threshold, the number of special symbols is greater than a second set threshold, the number of stop words is less than a third set threshold, the word repetition rate is greater than a fourth set threshold, the number of spelling errors is greater than a fifth set threshold, the number of grammatical errors is greater than a sixth set threshold, or the sentence structure is incomplete, the sentence is discarded to obtain a reconstructed paragraph.

[0151] In one implementation, the sentence structure integrity is complete if the following conditions are met: the sentence includes a subject and a predicate, and the number of nouns in the sentence is greater than or equal to a set word count threshold.

[0152] In one embodiment, the sentence filtering module 23 is specifically used for:

[0153] Based on preset sentence filtering rules, the original paragraphs with scores greater than the preset qualified threshold are subjected to sentence filtering processing to obtain reconstructed paragraphs.

[0154] In one embodiment, the quality assessment module 25 is further configured to:

[0155] When the score of the original paragraph is greater than or equal to the preset qualified threshold, and the difference between the score of the reconstructed paragraph and the score of the original paragraph is less than or equal to the set difference threshold, the original paragraph is determined to be high-quality text; wherein, the high-quality text is used for training a large language model.

[0156] It is worth noting that the specific working process of the text quality assessment device can be referred to the working process of the text quality assessment method described in the above embodiments, and will not be repeated here.

[0157] Compared with existing technologies, the text quality assessment device disclosed in this invention first divides the acquired original text into paragraphs to obtain at least one original paragraph; then, it scores the original paragraphs to obtain a score; based on preset sentence filtering rules, it performs sentence filtering on the original paragraphs to obtain reconstructed paragraphs, and scores the reconstructed paragraphs to obtain a score; when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold, the original paragraph is determined to be low-quality text. Therefore, this invention determines the score of each original paragraph by scoring the original text at the paragraph level, and considers original paragraphs with lower scores as low-quality text; by filtering sentences in the original paragraphs to remove sentences of poor quality, it obtains reconstructed paragraphs corresponding to the original paragraphs; by analyzing the scores of the original paragraphs and the reconstructed paragraphs, it determines the text quality of the paragraphs; if the quality of the reconstructed paragraph is significantly improved compared to the original paragraph, the original paragraph is considered low-quality text. In the text quality assessment process, no manual text annotation is required, thus improving the accuracy of text quality assessment.

[0158] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a text quality assessment device provided in an embodiment of the present invention. The text quality assessment device 30 includes a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, it implements the steps as described in the above-described text quality assessment method embodiment, for example... Figure 1 The steps S1 to S5 described above; or, when the processor 31 executes the computer program, it implements the functions of each module in the above-described device embodiments.

[0159] For example, the computer program can be divided into one or more modules, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the text quality assessment device. For example, the computer program can be divided into multiple modules, each with the following specific functions:

[0160] The text acquisition module 21 is used to acquire the original text and divide the original text into paragraphs to obtain at least one original paragraph.

[0161] The first scoring module 22 is used to score the original paragraph and obtain a score for the original paragraph;

[0162] The sentence filtering module 23 is used to perform sentence filtering processing on the original paragraph based on preset sentence filtering rules to obtain the reconstructed paragraph;

[0163] The second scoring module 24 is used to score the reconstructed paragraph and obtain a score for the reconstructed paragraph;

[0164] The quality assessment module 25 is used to determine that the original paragraph is low-quality text when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold.

[0165] The specific working process of each module can be referred to the working process of the text quality assessment device described in the above embodiments, and will not be repeated here.

[0166] The text quality assessment device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The text quality assessment device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the text quality assessment device may also include input / output devices, network access devices, buses, etc.

[0167] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the text quality assessment device, connecting all parts of the device via various interfaces and lines.

[0168] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the text quality assessment device by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0169] If the modules integrated into the text quality assessment device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0170] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the text quality assessment method as described in any of the above embodiments.

[0171] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A text quality assessment method, characterized in that, include: Obtain the original text and divide the original text into paragraphs to obtain at least one original paragraph; The original paragraph is scored to obtain a score for the original paragraph; The original paragraph is processed by sentence filtering based on preset sentence filtering rules to obtain a reconstructed paragraph. The reconstructed paragraph is scored to obtain a score for the reconstructed paragraph; When the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold, the original paragraph is determined to be low-quality text.

2. The text quality assessment method as described in claim 1, characterized in that, The process of scoring the original paragraph to obtain a score for the original paragraph includes: The paragraph elements of the original paragraph are scored, and the scores of all the paragraph elements in the original paragraph are combined to obtain the score of the original paragraph; wherein, the paragraph elements include at least one of lexical richness, text readability, information density, semantic consistency and sentiment consistency.

3. The text quality assessment method as described in claim 2, characterized in that, The process of scoring the paragraph elements of the original paragraph and summing the scores of all the paragraph elements in the original paragraph to obtain the score of the original paragraph includes: The lexical richness score is obtained by measuring the lexical diversity of the original paragraph using the average word-to-sign ratio of the language segments. A text readability score is calculated based on the number of words, syllables, and sentences in the original paragraph, wherein the text readability score is negatively correlated with the ratio of the number of words to the number of sentences and the ratio of the number of syllables to the number of words, respectively; or, the text readability is calculated based on the average number of parts of speech per sentence, the average number of effective characters per sentence, the average effective character frequency ranking, and the average number of effective character strokes in the original paragraph, wherein the text readability score is positively correlated with the average number of effective characters per sentence and the average effective character frequency ranking, respectively, and negatively correlated with the average number of parts of speech per sentence and the average number of effective character strokes, respectively. An information density score is determined based on the information density of the original paragraph; wherein the information density score is positively correlated with the information density. Determine the pairwise similarity between all sentences in the original paragraph, and calculate the semantic consistency score based on the similarity. Based on a preset mapping relationship between sentiment type and sentiment score, the sentiment score of each sentence in the original paragraph is determined according to the sentiment type of each sentence in the original paragraph, and the sentiment consistency score is calculated based on the sentiment scores of all sentences in the original paragraph. The score of the original paragraph is obtained by weighted summation of the lexical richness score, the text readability score, the information density score, the semantic consistency score, and the sentiment consistency score.

4. The text quality assessment method as described in claim 1, characterized in that, The original paragraph is processed by sentence filtering based on preset sentence filtering rules to obtain a reconstructed paragraph, including: Determine the sentence elements of each sentence in the original paragraph; wherein, the sentence elements include at least one of the following: number of lexical units, number of special symbols, number of stop words, word repetition rate, number of spelling errors, number of grammatical errors, and sentence structural integrity; Sentences whose sentence elements do not conform to the preset sentence filtering rules are discarded to obtain reconstructed paragraphs.

5. The text quality assessment method as described in claim 4, characterized in that, The step of discarding sentences whose sentence elements do not conform to the preset sentence filtering rules to obtain a reconstructed paragraph includes: When the number of lexical units is less than a first set threshold, the number of special symbols is greater than a second set threshold, the number of stop words is less than a third set threshold, the word repetition rate is greater than a fourth set threshold, the number of spelling errors is greater than a fifth set threshold, the number of grammatical errors is greater than a sixth set threshold, or the sentence structure is incomplete, the sentence is discarded to obtain a reconstructed paragraph.

6. The text quality assessment method as described in claim 4, characterized in that, The sentence structure must be complete if it meets the following conditions: the sentence includes a subject and a predicate, and the number of nouns in the sentence is greater than or equal to a set word count threshold.

7. The text quality assessment method according to any one of claims 1 to 6, characterized in that, The original paragraph is processed by sentence filtering based on preset sentence filtering rules to obtain a reconstructed paragraph, including: Based on preset sentence filtering rules, the original paragraphs with scores greater than the preset qualified threshold are subjected to sentence filtering processing to obtain reconstructed paragraphs.

8. The text quality assessment method according to any one of claims 1 to 6, characterized in that, Also includes: When the score of the original paragraph is greater than or equal to the preset qualified threshold, and the difference between the score of the reconstructed paragraph and the score of the original paragraph is less than or equal to the set difference threshold, the original paragraph is determined to be high-quality text; wherein, the high-quality text is used for training a large language model.

9. A text quality assessment device, characterized in that, include: The text acquisition module is used to acquire raw text and divide the raw text into paragraphs to obtain at least one raw paragraph. The first scoring module is used to score the original paragraph and obtain a score for the original paragraph; The sentence filtering module is used to perform sentence filtering processing on the original paragraph based on preset sentence filtering rules to obtain the reconstructed paragraph; The second scoring module is used to score the reconstructed paragraph and obtain a score for the reconstructed paragraph; The quality assessment module is used to determine that the original paragraph is low-quality text when the score of the original paragraph is less than a preset qualified threshold, or when the difference between the score of the reconstructed paragraph and the score of the original paragraph is greater than a set difference threshold.

10. A text quality assessment device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the text quality assessment method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the text quality assessment method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the text quality assessment method as described in any one of claims 1 to 8.

Images