Chinese corpus cleaning method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention discloses a Chinese corpus cleaning method and device, electronic equipment and a readable storage medium, relates to the technical field of natural language processing, and aims to improve the accuracy of an output result of a trained model. The method comprises the following steps: determining whether semantics of adjacent paragraphs in a Chinese corpus to be cleaned are coherent or not; dividing each paragraph into at least one corpus sample according to whether the semantics of the adjacent paragraphs are coherent or not, so that the semantics of the paragraphs in each corpus sample in the at least one corpus sample are coherent, and the semantics of the paragraphs of different corpus samples are not coherent; and cleaning the at least one corpus sample according to a preset cleaning rule.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a Chinese corpus cleaning method, device, electronic device and readable storage medium. Background Art

[0002] In today's information age, the processing and analysis of large-scale Chinese corpora are of great significance to fields such as natural language processing (NLP) and machine learning. However, Chinese corpora in practical applications often contain problems such as noise, which has a significant negative impact on the models generated by subsequent training. In the existing technology, for the Chinese corpus to be cleaned, noise and other factors are directly cleaned, and the model is trained using the cleaned samples. Since Chinese corpora may contain many randomly distributed paragraphs, the model trained using the samples will have a low accuracy rate in the output results during use. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a Chinese corpus cleaning method, device, electronic device and readable storage medium to facilitate improving the accuracy of the output results of the trained model.

[0004] In a first aspect, an embodiment of the present application provides a Chinese corpus cleaning method, comprising: determining whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent; dividing each paragraph into at least one corpus sample based on whether the semantics of the adjacent paragraphs are coherent, so that the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other; and cleaning the at least one corpus sample according to preset cleaning rules.

[0005] According to a specific implementation method of an embodiment of the present application, the adjacent paragraphs include a first paragraph and a second paragraph adjacent to the first paragraph, and the first paragraph is located before the second paragraph in the Chinese corpus to be cleaned; determining whether the semantics of the adjacent paragraphs in the Chinese corpus to be cleaned are coherent includes: determining whether the semantics of the first paragraph and the second paragraph are coherent according to a preset artificial intelligence model.

[0006] According to a specific implementation method of the embodiment of the present application, determining whether the semantics of the first paragraph and the second paragraph are coherent according to a preset artificial intelligence model includes: in the first paragraph, obtaining the first content between the end position of the first paragraph and the first position determined according to a first preset rule; in the second paragraph, obtaining the second content between the starting position of the second paragraph and the second position determined according to a second preset rule; determining whether the semantics of the first content and the second content are coherent according to the preset first artificial intelligence model; and determining whether the semantics of the first paragraph and the second paragraph are coherent according to whether the semantics of the first content and the second content are coherent.

[0007] According to a specific implementation method of the embodiment of the present application, in the first paragraph, obtaining the first content between the end position of the first paragraph and the first position determined according to the first preset rule includes: in the first paragraph, starting from the text at the end position of the first paragraph, in the direction of the start position of the first paragraph, sequentially determining a first preset number of characters; and determining the determined first preset number of characters as the first content.

[0008] According to a specific implementation method of an embodiment of the present application, determining whether the semantics of the first content and the second content are consistent based on a preset first artificial intelligence model includes: inputting the first content and the second content into a preset first artificial intelligence model, determining a first probability that the second content is semantically consistent with the first content and a second probability that the second content is semantically inconsistent with the first content; and determining whether the semantics of the first content and the second content are semantically consistent based on the first probability and the second probability.

[0009] According to a specific implementation method of an embodiment of the present application, determining whether the semantics of the first paragraph and the second paragraph are coherent based on a preset artificial intelligence model includes: inputting the first paragraph into a preset second artificial intelligence model to determine a first vector corresponding to the first paragraph; inputting the second paragraph into the preset second artificial intelligence model to determine a second vector corresponding to the second paragraph; and determining whether the semantics of the first paragraph and the second paragraph are coherent based on the similarity between the first vector and the second vector.

[0010] According to a specific implementation method of an embodiment of the present application, the cleaning of the at least one corpus sample according to a preset cleaning rule includes at least one of the following: normalizing the punctuation marks in each corpus sample in the at least one corpus sample; identifying whether there are sentences of a preset type for each corpus sample of the at least one corpus sample; if so, clearing the sentences of the preset type; identifying whether there are preset invisible characters for each corpus sample in the at least one corpus sample; if so, clearing the preset invisible characters.

[0011] According to a specific implementation method of an embodiment of the present application, the preset type of sentence includes at least one of the following: a sentence that is repeated with other sentences in the same corpus sample, a sentence that is irrelevant to the meaning expressed by each paragraph in the same corpus sample, and a sentence without punctuation at the end.

[0012] According to a specific implementation of an embodiment of the present application, after cleaning the at least one corpus sample according to a preset cleaning rule, the method further includes: for each corpus sample in the at least one corpus sample, determining whether the semantics of the sentences adjacent to each other in each paragraph in each corpus sample are sequentially coherent; if not, removing the corpus sample containing the incoherent sentences; or, if the first sentence in a corpus sample is semantically incoherent with the second sentence and the third sentence respectively, determining whether the semantics of the second sentence and the third sentence are semantically coherent; wherein the second sentence, the first sentence and the third sentence are arranged adjacent to each other in sequence; if the semantics of the second sentence and the third sentence are semantically coherent, removing the first sentence.

[0013] According to a specific implementation method of an embodiment of the present application, after cleaning the at least one corpus sample according to a preset cleaning rule, the method further includes: determining, for each corpus sample in the at least one corpus sample, whether the semantics of a single sentence in each paragraph in each corpus sample conforms to the preset rule; if the semantics of the first sentence in the first paragraph of a corpus sample does not conform to the preset rule, correcting the first sentence.

[0014] In a second aspect, an embodiment of the present application provides a Chinese corpus cleaning device, comprising: a first determination module, used to determine whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent; a division module, used to divide each paragraph into at least one corpus sample according to whether the semantics of adjacent paragraphs are coherent, so that the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other; a cleaning module, used to clean the at least one corpus sample according to preset cleaning rules.

[0015] According to a specific implementation method of an embodiment of the present application, the adjacent paragraphs include a first paragraph and a second paragraph adjacent to the first paragraph, and the first paragraph is located before the second paragraph in the Chinese corpus to be cleaned; the first determination module includes: a determination sub-module, which is used to determine whether the semantics of the first paragraph and the second paragraph are coherent according to a preset artificial intelligence model.

[0016] According to a specific implementation method of an embodiment of the present application, the determination submodule includes: a first acquisition unit, used to acquire, in the first paragraph, the first content between the end position of the first paragraph and the first position determined according to a first preset rule; a second acquisition unit, used to acquire, in the second paragraph, the second content between the start position of the second paragraph and the second position determined according to a second preset rule; a first determination unit, used to determine, based on a preset first artificial intelligence model, whether the semantics of the first content and the second content are coherent; and a second determination unit, used to determine, based on whether the semantics of the first content and the second content are coherent, whether the semantics of the first paragraph and the second paragraph are coherent.

[0017] According to a specific implementation method of an embodiment of the present application, the first acquisition unit is specifically used to: in the first paragraph, starting from the text at the end position of the first paragraph, determine a first preset number of characters in sequence toward the start position of the first paragraph; and determine the determined first preset number of characters as the first content.

[0018] According to a specific implementation method of an embodiment of the present application, the first determination unit is specifically used to: input the first content and the second content into a preset first artificial intelligence model, determine a first probability that the second content is semantically coherent with the first content and a second probability that the second content is semantically incoherent with the first content; and determine whether the first content and the second content are semantically coherent based on the first probability and the second probability.

[0019] According to a specific implementation method of an embodiment of the present application, the determination submodule is specifically used to: input the first paragraph into a preset second artificial intelligence model to determine a first vector corresponding to the first paragraph; input the second paragraph into the preset second artificial intelligence model to determine a second vector corresponding to the second paragraph; and determine whether the semantics of the first paragraph and the second paragraph are coherent based on the similarity between the first vector and the second vector.

[0020] According to a specific implementation method of an embodiment of the present application, the cleaning module includes at least one of the following sub-modules: a normalization sub-module, used to perform normalization processing on punctuation marks in each corpus sample in the at least one corpus sample; a first clearing sub-module, used to identify whether there are sentences of a preset type for each corpus sample of the at least one corpus sample; if so, clear the sentences of the preset type; a second clearing sub-module, used to identify whether there are preset invisible characters for each corpus sample in the at least one corpus sample; if so, clear the preset invisible characters.

[0021] According to a specific implementation method of an embodiment of the present application, the preset type of sentence includes at least one of the following: a sentence that is repeated with other sentences in the same corpus sample, a sentence that is irrelevant to the meaning expressed by each paragraph in the same corpus sample, and a sentence without punctuation at the end.

[0022] According to a specific implementation method of the embodiment of the present application, the device also includes: a first clearing module, which is used to determine, for each corpus sample in the at least one corpus sample, whether the semantics of the adjacent sentences in each paragraph in each corpus sample are coherent in sequence after the cleaning module cleans the at least one corpus sample according to a preset cleaning rule; if not, the corpus sample containing the incoherent sentences is cleared; or, a second clearing module, which is used to determine, after the cleaning module cleans the at least one corpus sample according to a preset cleaning rule, whether the semantics of the second sentence and the third sentence are coherent if the first sentence in a corpus sample is semantically incoherent with the second sentence and the third sentence respectively; wherein the second sentence, the first sentence and the third sentence are arranged adjacent to each other in sequence; if the semantics of the second sentence and the third sentence are coherent, the first sentence is cleared.

[0023] According to a specific implementation method of an embodiment of the present application, the device also includes: a second determination module, which is used to determine, for each corpus sample in the at least one corpus sample, whether the semantics of a single sentence in each paragraph in each corpus sample conforms to the preset rules after the cleaning module cleans the at least one corpus sample according to the preset cleaning rules; and an error correction module, which is used to correct the first sentence in the first paragraph of a corpus sample if the semantics of the first sentence do not conform to the preset rules.

[0024] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; a power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, and is used to execute the Chinese corpus cleaning method described in any of the aforementioned implementation methods.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the Chinese corpus cleaning method described in any of the aforementioned implementation methods.

[0026] The Chinese corpus cleaning method, device, electronic device, and readable storage medium of this embodiment divide each paragraph into at least one corpus sample based on whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent, so that the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other. Then, the at least one corpus sample is cleaned according to a preset cleaning rule. Because before cleaning, each paragraph is divided into at least one corpus sample based on whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent, and the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other, the semantics of adjacent paragraphs in each corpus sample obtained by the division are coherent. Based on this, each training sample obtained is semantically coherent at least at the paragraph level. Such training samples are used to train a model, and the trained model outputs the accuracy of the results during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A flowchart of a Chinese corpus cleaning method provided in one embodiment of the present application;

[0029] Figure 2 A schematic diagram of the structure of a Chinese corpus cleaning device provided in one embodiment of the present application;

[0030] Figure 3 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described are only a portion of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0032] In order to enable those skilled in the art to better understand the technical concepts, implementation plans and beneficial effects of the embodiments of the present application, specific examples are described in detail below.

[0033] Figure 1 A flow chart of a Chinese corpus cleaning method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the Chinese corpus cleaning method of this embodiment may include:

[0034] S101: Determine whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent.

[0035] The Chinese corpus to be cleaned may be of a large scale and have multiple paragraphs, such as a large scale of web page data.

[0036] In this embodiment, paragraphs are used as units to determine whether the semantics between adjacent paragraphs are coherent.

[0037] Semantic coherence of adjacent paragraphs can mean that, semantically, the intention expressed by the latter paragraph in the adjacent paragraphs can be coherently connected with the intention expressed by the previous paragraph.

[0038] Semantic incoherence between adjacent paragraphs may mean that semantically, the intention expressed by the latter paragraph in the adjacent paragraphs cannot be coherently connected with the intention expressed by the previous paragraph.

[0039] S102. Divide each paragraph into at least one corpus sample according to whether adjacent paragraphs are semantically coherent, so that each paragraph in each corpus sample in the at least one corpus sample is semantically coherent and paragraphs in different corpus samples are semantically incoherent with each other.

[0040] In this embodiment, at least one corpus sample is divided based on whether adjacent paragraphs are semantically coherent. If one sample A includes multiple paragraphs, these multiple paragraphs are semantically coherent; the paragraphs in another sample B are semantically incoherent with the paragraphs in the other samples. It is understood that the obtained corpus sample may include a sample containing only one paragraph.

[0041] Specifically, if the sample to be cleaned includes paragraphs 1, 2, 3, 4, and 5, and they are adjacent in sequence, if paragraphs 1 and 2 are semantically coherent, paragraph 3 is incoherent with both paragraphs 2 and 4, and paragraph 4 and paragraph 5 are coherent, then paragraphs 1 and 2 can be aggregated into sample A, paragraph 3 into sample B, and paragraphs 4 and paragraph 5 into sample C; if paragraphs 1, 2, and 3 are semantically coherent, paragraphs 4 and 5 are semantically coherent, and paragraph 4 is semantically incoherent with paragraph 3, then paragraphs 1, 2, and 3 can be aggregated into sample D, and paragraphs 4 and 5 into sample E.

[0042] S103: Clean at least one corpus sample according to a preset cleaning rule.

[0043] In this embodiment, the corpus sample obtained in S102 is cleaned. It can be understood that each corpus sample in the at least one corpus sample obtained is cleaned separately, and each cleaned corpus sample can be used as training data (sample) to train the model.

[0044] In this embodiment, each corpus sample may be cleaned of noise.

[0045] In this embodiment, each paragraph is divided into at least one corpus sample according to whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent, so that the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other. Then, the at least one corpus sample is cleaned according to a preset cleaning rule. Because before cleaning, each paragraph is divided into at least one corpus sample according to whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent, and the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other, the semantics between adjacent paragraphs in each corpus sample in the at least one corpus sample obtained by the division are coherent, and each training sample obtained based on this is semantically coherent at least at the paragraph level. Such training samples are used to train the model, and the trained model outputs the accuracy of the result during use.

[0046] In some examples, each adjacent paragraph includes a first paragraph and a second paragraph adjacent to the first paragraph, and the first paragraph is located before the second paragraph in the Chinese corpus to be cleaned.

[0047] Exemplarily, each adjacent paragraph includes paragraph A and paragraph B, paragraph A is adjacent to paragraph B, and paragraph A is before paragraph B in the Chinese corpus to be cleaned.

[0048] In some examples, determining whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent (S101) may include:

[0049] S101a: Determine whether the semantics of the first paragraph and the second paragraph are coherent based on a preset artificial intelligence model.

[0050] In this embodiment, an artificial intelligence model is used to determine whether the semantics of two adjacent paragraphs, namely the first paragraph and the second paragraph, are coherent, so as to improve the accuracy of determining whether the semantics of the two adjacent paragraphs are coherent.

[0051] As an optional embodiment, determining whether the semantics of the first paragraph and the second paragraph are coherent according to a preset artificial intelligence model (S101a) may include:

[0052] A1. In a first paragraph, obtain first content between the end position of the first paragraph and a first position determined according to a first preset rule.

[0053] In this embodiment, part of the content can be intercepted from the first paragraph, and the part of the content is the content from the first position to the end position of the first paragraph. The first position can be determined according to a first preset rule, and the first preset rule can be a rule based on the number of words or characters, or a rule based on the number of sentences.

[0054] The first content may include only text, or may include punctuation marks in addition to text.

[0055] In some examples, in a first paragraph, obtaining first content (A1) between an end position of the first paragraph and a first position determined according to a first preset rule may include:

[0056] A11. In the first paragraph, starting from the text at the end of the first paragraph and moving toward the beginning of the first paragraph, determine a first preset number of texts in sequence.

[0057] A12. Determine the first preset number of characters as the first content.

[0058] In some examples, the first preset number of characters may be 256 characters, and the 256 characters are searched for sequentially from the characters at the end of the first paragraph to the front of the first paragraph.

[0059] In some examples, the content of the last sentence in the first paragraph may be determined, and the content of the last sentence may be determined as the first content.

[0060] A2. In the second paragraph, obtain the second content between the start position of the second paragraph and the second position determined according to the second preset rule.

[0061] The second preset rule of this embodiment may be the same as or different from the first preset rule.

[0062] In the second paragraph, the second content is obtained in a manner similar to the manner of obtaining the first content in A1.

[0063] Specifically, in a specific example, in the second paragraph, starting from the text at the start position of the second paragraph, a second preset number of characters are determined in sequence toward the end position of the second paragraph; the determined second preset number of characters are determined as the second content.

[0064] The second preset number may be the same as or different from the first preset number.

[0065] In another specific example, in the second paragraph, the content of a sentence at the beginning of the paragraph may be determined, and the content of the sentence may be determined as the first content.

[0066] A3. Determine whether the semantics of the first content and the second content are coherent based on a preset first artificial intelligence model.

[0067] The first content and the second content can be input into an artificial intelligence model, i.e., a preset first artificial intelligence model. The preset first artificial intelligence model determines whether the semantics of the first content and the second content are consistent based on the input first content and the second content.

[0068] In some examples, determining whether the first content and the second content are semantically coherent based on a preset first artificial intelligence model (A3) may include:

[0069] A31. Input the first content and the second content into a preset first artificial intelligence model to determine a first probability that the second content is semantically coherent with the first content and a second probability that the second content is semantically incoherent with the first content.

[0070] In some examples, the preset first artificial intelligence model may be a model capable of predicting whether two parts of content are semantically coherent, specifically a model capable of next sentence prediction.

[0071] Through the preset first artificial intelligence model, the probability of semantic coherence and incoherence between the second content and the first content can be determined, and then, based on the probability of coherence and incoherence, it can be determined whether the semantics of the second content and the first content are coherent.

[0072] A32. Determine whether the first content and the second content are semantically coherent based on the first probability and the second probability.

[0073] If the first probability is greater than the second probability, it is determined that the first content and the second content are semantically coherent; if the first probability is less than or equal to the second probability, it is determined that the first content and the second content are semantically incoherent.

[0074] A4. Determine whether the semantics of the first and second contents are coherent based on whether the semantics of the first and second contents are coherent.

[0075] If the first content and the second content are semantically coherent, it is determined that the first paragraph and the second paragraph are semantically coherent; if the first content and the second content are semantically incoherent, it is determined that the first paragraph and the second paragraph are semantically incoherent.

[0076] In this embodiment, whether the semantics of the first content of the first paragraph and the second content of the second paragraph are semantically coherent is determined, which helps to reduce the amount of calculation required to determine whether the semantics of the first paragraph and the second paragraph are semantically coherent, thereby improving calculation efficiency.

[0077] As an alternative embodiment, determining whether the semantics of the first paragraph and the second paragraph are coherent according to a preset artificial intelligence model (S101a) may include:

[0078] B1. Input the first paragraph into a preset second artificial intelligence model to determine a first vector corresponding to the first paragraph.

[0079] In some examples, the first vector may be a vector representing the intention expressed in the first paragraph, and the preset second artificial intelligence model has the function of summarizing the intention of a paragraph of text and expressing it in the form of a vector.

[0080] B2. Input the second paragraph into a preset second artificial intelligence model to determine a second vector corresponding to the second paragraph.

[0081] In some examples, the second vector may be a vector representing the intention expressed by the second paragraph.

[0082] B3. Determine whether the semantics of the first paragraph and the second paragraph are coherent based on the similarity between the first vector and the second vector.

[0083] This step can be implemented through the preset second artificial intelligence model, or through a functional model other than the preset second artificial intelligence model, which is not limited in this embodiment.

[0084] In some examples, the cleaning of the at least one corpus sample according to a preset cleaning rule (S103) may include at least one of the following:

[0085] S103a: Perform normalization processing on punctuation marks in each corpus sample in at least one corpus sample.

[0086] You can check the punctuation marks in each corpus sample and normalize each punctuation mark, for example, unify English punctuation marks into Chinese punctuation marks.

[0087] S103b: For each corpus sample of the at least one corpus sample, identify whether there is a sentence of a preset type; if so, remove the sentence of the preset type.

[0088] In some examples, the preset type of statement includes at least one of the following:

[0089] Sentences that are repeated with other sentences in the same corpus sample, sentences that are irrelevant to the meaning expressed by the paragraphs in the same corpus sample, and sentences without punctuation at the end.

[0090] In the corpus sample, if there are two identical sentences, one of them will be removed.

[0091] Other contents such as advertisements are often inserted into the Chinese corpus to be cleaned. These contents are sentences that are irrelevant to the meaning expressed by each paragraph in the same corpus sample.

[0092] In each paragraph of the corpus sample, there are sentences without punctuation marks at the end. Such sentences are non-standard sentences and should be removed.

[0093] S103c: for each corpus sample in the at least one corpus sample, identify whether there is a preset invisible character; if so, remove the preset invisible character.

[0094] Invisible characters may be spaces in a specific format, and the invisible characters need to be removed from the corpus sample.

[0095] Normalizing the punctuation marks in each corpus sample, removing preset types of sentences, and removing preset invisible characters can make the corpus sample more "clean". When using such corpus samples to train the model, the output results of the obtained model will be more accurate during use.

[0096] In some examples, after cleaning at least one corpus sample according to a preset cleaning rule (S103), the method may further include:

[0097] S104 . For each corpus sample in the at least one corpus sample, determine whether the semantics of consecutively adjacent sentences in each paragraph in each corpus sample are consecutively coherent; if not, remove the corpus sample containing the incoherent sentences.

[0098] If a sample contains three consecutive paragraphs A, B, and C, and if B is semantically coherent with A and C is semantically coherent with B, then A, B, and C are three semantically coherent paragraphs. If B is semantically coherent with A and C is semantically incoherent with B, then the corpus sample is semantically incoherent at the sentence level, and the corpus sample is removed, that is, it is not used to train the model.

[0099] In some examples, after cleaning at least one corpus sample according to a preset cleaning rule (S103), the method may further include:

[0100] S105: If the semantics of the first sentence in a corpus sample are not coherent with the second sentence and the third sentence respectively, determine whether the semantics of the second sentence and the third sentence are coherent.

[0101] The second statement, the first statement and the third statement are arranged adjacent to each other in sequence; if the second statement and the third statement are semantically coherent, the first statement is cleared.

[0102] A corpus sample includes sentences A, B, and C. When sentence B is semantically incoherent with sentence A and sentence B is semantically incoherent with sentence C, determine whether sentences A and C are semantically coherent; if sentences A and C are semantically coherent, remove sentence B; if sentences A and C are semantically incoherent, determine whether sentence D adjacent to sentence C is coherent with sentence A; if sentence D is coherent with sentence A, remove sentence C.

[0103] In some examples, determining whether the semantics between two adjacent sentences are coherent can be achieved using an artificial intelligence model with next sentence prediction function.

[0104] In some examples, after cleaning at least one corpus sample according to a preset cleaning rule (S103), the method may further include:

[0105] S106 : For each corpus sample in the at least one corpus sample, determine whether the semantics of a single sentence in each paragraph in each corpus sample conforms to a preset rule.

[0106] The preset rule in this embodiment can be based on whether a certain meaning can be accurately expressed. For example, the sentence "going to Macau for dinner" expresses the destination of "going" (Macau) and the purpose of "going to Macau" (eating). Therefore, the sentence reasonably expresses the relevant meaning and therefore meets the preset rule. However, for example, "going to dinner tomorrow," "going" is usually not followed by a word indicating time, as this does not clearly express the true intention of the sentence. Therefore, this sentence does not meet the preset rule.

[0107] In some examples, a single sentence may be segmented and the perplexity of each word may be calculated. If the perplexity value is greater than a preset value, the sentence does not comply with the preset rules.

[0108] S107: If the semantics of the first sentence in the first paragraph of a corpus sample does not conform to a preset rule, correct the first sentence.

[0109] The first sentence may be corrected so that the sentence can accurately express a certain meaning.

[0110] In some examples, a first statement that does not conform to preset rules can be input into a preset model, and the preset model outputs a correct statement corresponding to the first statement. The correct statement is used to replace the first statement, thereby correcting the first statement.

[0111] One usage scenario of this embodiment may be: before training a model, it is necessary to clean the training data, so that the training data is of higher quality. The higher the quality of the data, the better the effect of the model trained.

[0112] Another use scenario of this embodiment is that it can be widely used in the scenario of automated essay correction, and can automatically check for typos and semantic incoherence in texts.

[0113] Figure 2 A schematic diagram of the structure of a Chinese corpus cleaning device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the Chinese corpus cleaning device of this embodiment includes: a first determination module 11, used to determine whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent; a division module 12, used to divide each paragraph into at least one corpus sample according to whether the semantics of adjacent paragraphs are coherent, so that the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other; a cleaning module 13, used to clean the at least one corpus sample according to preset cleaning rules.

[0114] The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0115] The device of this embodiment divides each paragraph into at least one corpus sample according to whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent, so that the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other. Then, the at least one corpus sample is cleaned according to a preset cleaning rule. Because before cleaning, each paragraph is divided into at least one corpus sample according to whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent, and the paragraphs in each corpus sample in the at least one corpus sample are semantically coherent, and the paragraphs in different corpus samples are semantically incoherent with each other, the semantics between adjacent paragraphs in each corpus sample in the at least one corpus sample obtained by the division are coherent, and each training sample obtained based on this is semantically coherent at least at the paragraph level. Such training samples are used to train the model, and the trained model outputs the accuracy of the result during use.

[0116] As an optional implementation, the adjacent paragraphs include a first paragraph and a second paragraph adjacent to the first paragraph, and the first paragraph is located before the second paragraph in the Chinese corpus to be cleaned; the first determination module includes: a determination submodule, which is used to determine whether the semantics of the first paragraph and the second paragraph are coherent according to a preset artificial intelligence model.

[0117] As an optional embodiment, the determination submodule includes: a first acquisition unit, used to acquire, in the first paragraph, the first content between the end position of the first paragraph and the first position determined according to a first preset rule; a second acquisition unit, used to acquire, in the second paragraph, the second content between the start position of the second paragraph and the second position determined according to a second preset rule; a first determination unit, used to determine, based on a preset first artificial intelligence model, whether the semantics of the first content and the second content are coherent; and a second determination unit, used to determine, based on whether the semantics of the first content and the second content are coherent, whether the semantics of the first paragraph and the second paragraph are coherent.

[0118] As an optional implementation, the first acquisition unit is specifically used to: in the first paragraph, starting from the text at the end position of the first paragraph, determine a first preset number of characters in sequence toward the start position of the first paragraph; and determine the determined first preset number of characters as the first content.

[0119] As an optional embodiment, the first determination unit is specifically used to: input the first content and the second content into a preset first artificial intelligence model, determine a first probability that the second content is semantically coherent with the first content and a second probability that the second content is semantically incoherent with the first content; and determine whether the first content and the second content are semantically coherent based on the first probability and the second probability.

[0120] As an optional implementation, the determination submodule is specifically used to: input the first paragraph into a preset second artificial intelligence model to determine a first vector corresponding to the first paragraph; input the second paragraph into the preset second artificial intelligence model to determine a second vector corresponding to the second paragraph; and determine whether the semantics of the first paragraph and the second paragraph are coherent based on the similarity between the first vector and the second vector.

[0121] As an optional embodiment, the cleaning module includes at least one of the following: a normalization submodule, used to normalize the punctuation marks in each corpus sample of the at least one corpus sample; a first clearing submodule, used to identify whether there are sentences of a preset type for each corpus sample of the at least one corpus sample; if so, clear the sentences of the preset type; a second clearing submodule, used to identify whether there are preset invisible characters for each corpus sample of the at least one corpus sample; if so, clear the preset invisible characters.

[0122] As an optional implementation, the preset type of sentence includes at least one of the following: a sentence that is repeated with other sentences in the same corpus sample, a sentence that is irrelevant to the meaning expressed by each paragraph in the same corpus sample, and a sentence without punctuation at the end.

[0123] As an optional embodiment, the device further includes: a first clearing module, which is used to determine, for each corpus sample in the at least one corpus sample, whether the semantics of the adjacent sentences in each paragraph in each corpus sample are sequentially coherent after the cleaning module cleans the at least one corpus sample according to a preset cleaning rule; if not, the corpus sample containing the incoherent sentences is cleared; or, a second clearing module, which is used to determine, if the first sentence in a corpus sample is semantically incoherent with the second sentence and the third sentence respectively, whether the semantics of the second sentence and the third sentence are coherent after the cleaning module cleans the at least one corpus sample according to a preset cleaning rule; wherein the second sentence, the first sentence and the third sentence are arranged adjacent to each other in sequence; if the semantics of the second sentence and the third sentence are semantically coherent, the first sentence is cleared.

[0124] As an optional embodiment, the device also includes: a second determination module, which is used to determine, for each corpus sample in the at least one corpus sample, whether the semantics of individual sentences in each paragraph in each corpus sample conform to the preset rules after the cleaning module cleans the at least one corpus sample according to the preset cleaning rules; and an error correction module, which is used to correct the first sentence in the first paragraph of a corpus sample if the semantics of the first sentence do not conform to the preset rules.

[0125] The device of the above embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0126] Figure 3 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application is shown in FIG. Figure 3 As shown, it may include: a shell 61, a processor 62, a memory 63, a circuit board 64 and a power supply circuit 65, wherein the circuit board 64 is placed inside the space enclosed by the shell 61, and the processor 62 and the memory 63 are arranged on the circuit board 64; the power supply circuit 65 is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory 63 is used to store executable program code; the processor 62 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 63, and is used to execute any of the Chinese corpus cleaning methods provided in the aforementioned embodiments, so that corresponding beneficial technical effects can also be achieved, which have been described in detail above and will not be repeated here.

[0127] The aforementioned electronic devices exist in various forms, including but not limited to:

[0128] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0129] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0130] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0131] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0132] (5) Other electronic devices with data interaction functions.

[0133] Correspondingly, an embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement any of the Chinese corpus cleaning methods provided in the aforementioned embodiments, thereby also achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0134] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0135] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0136] In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] For the convenience of description, the above device is described as being divided into various units / modules based on their functions. Of course, when implementing this application, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.

[0138] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0139] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A Chinese corpus cleaning method, characterized in that: include: Determine whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent; Dividing each paragraph into at least one corpus sample according to whether adjacent paragraphs are semantically coherent, so that each paragraph in each corpus sample in the at least one corpus sample is semantically coherent and paragraphs in different corpus samples are semantically incoherent with each other; The at least one corpus sample is cleaned according to a preset cleaning rule.

2. The method according to claim 1, characterized in that Each adjacent paragraph includes a first paragraph and a second paragraph adjacent to the first paragraph, and the first paragraph is located before the second paragraph in the Chinese corpus to be cleaned; The step of determining whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent includes: According to a preset artificial intelligence model, determine whether the semantics of the first paragraph and the second paragraph are coherent.

3. The method according to claim 2, characterized in that The determining, based on a preset artificial intelligence model, whether the semantics of the first paragraph and the second paragraph are coherent includes: In the first paragraph, obtaining first content between the end position of the first paragraph and a first position determined according to a first preset rule; In the second paragraph, obtaining second content between the start position of the second paragraph and a second position determined according to a second preset rule; determining, based on a preset first artificial intelligence model, whether the first content and the second content are semantically coherent; Whether the semantics of the first content and the second content are coherent is determined based on whether the semantics of the first paragraph and the second paragraph are coherent.

4. The method according to claim 3, characterized in that The step of obtaining, in the first paragraph, first content between the end position of the first paragraph and a first position determined according to a first preset rule includes: In the first paragraph, starting from the text at the end of the first paragraph and moving toward the start of the first paragraph, a first preset number of texts are sequentially determined; The first preset number of characters determined is determined as the first content.

5. The method according to claim 3, characterized in that The determining, based on a preset first artificial intelligence model, whether the semantics of the first content and the second content are coherent includes: Inputting the first content and the second content into a preset first artificial intelligence model to determine a first probability that the second content is semantically coherent with the first content and a second probability that the second content is semantically incoherent with the first content; Whether the first content and the second content are semantically consistent is determined based on the first probability and the second probability.

6. The method according to claim 2, characterized in that The determining, based on a preset artificial intelligence model, whether the semantics of the first paragraph and the second paragraph are coherent includes: Inputting the first paragraph into a preset second artificial intelligence model to determine a first vector corresponding to the first paragraph; Inputting the second paragraph into the preset second artificial intelligence model to determine a second vector corresponding to the second paragraph; Whether the first paragraph and the second paragraph are semantically coherent is determined based on the similarity between the first vector and the second vector.

7. The method according to claim 1, characterized in that The cleaning of the at least one corpus sample according to a preset cleaning rule includes at least one of the following: performing normalization processing on punctuation marks in each corpus sample in the at least one corpus sample; For each corpus sample of the at least one corpus sample, identifying whether there is a sentence of a preset type; if so, removing the sentence of the preset type; For each corpus sample of the at least one corpus sample, identifying whether there is a preset invisible character; If it exists, the preset invisible characters are cleared.

8. A Chinese corpus cleaning device, characterized in that: include: The first determination module is used to determine whether the semantics of adjacent paragraphs in the Chinese corpus to be cleaned are coherent; a division module, configured to divide each paragraph into at least one corpus sample according to whether adjacent paragraphs are semantically coherent, so that each paragraph in each corpus sample of the at least one corpus sample is semantically coherent and paragraphs in different corpus samples are semantically incoherent with each other; The cleaning module is used to clean the at least one corpus sample according to a preset cleaning rule.

9. An electronic device, characterized in that: The electronic device includes: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the method described in any one of claims 1 to 7 above.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 7.

Images