A text screening method for massive high-noise colloquial short texts

Abstract

The invention proposes a text screening method for massive high-noise colloquial short texts, which belongs to the field of natural language processing. The method comprises the following steps: preprocessing training corpus and target text to be screened; extracting sentence information from annotated positive class corpus in the preprocessed training corpus to distinguish between strong business-related sentences and weak business-related sentences; performing sentence matching on the preprocessed target text using the extracted sentence information, classifying the matching results of the strong business-related sentences as positive class texts, and performing the following steps on the matching results of the weak business-related sentences; performing text processing on both the target text and the training corpus, and converting the processed text into a word vector representation; training a text classification model using the word vector representation of the training corpus, and inputting the word vector representation of the target text into the trained text classification model to classify the text, thereby realizing text screening of the target text.

Description

Technical Field

[0001] The present invention relates to a screening method for massive high-noise colloquial short texts, which can reduce the noise of texts with high error rates and screen texts according to semantic similarity and sentence information, and belongs to the field of natural language processing. Background Art

[0002] With the diversification and convenience of communication technology and the rapid development of computer networks, remote communication between people has become more cost-effective and high-quality. The development of the Internet has also made it easier for people to express themselves online. These communications and expressions are different from written texts. Spoken short texts have some characteristics that are clearly different from common written materials:

[0003] 1. Complex and non-standard grammar: Since people are often not as strict and standardized in speaking as in writing, they are more likely to use convenience and conformity to habits. It is very common in colloquial short texts to have many clauses, reversed word order, and unclear reference.

[0004] 2. Colloquial vocabulary: In speech communication, especially in informal daily communication, people may habitually use many colloquial words, such as dialect-specific words, common English words, abbreviations, and popular words;

[0005] 3. Accents and dialects: People in different places have different language expression habits, and regional differences are huge. For example, the words and sentence patterns of Cantonese are quite different from those of Mandarin, which is based on northern dialects. These differences will cause high differences in the expression of colloquial short texts;

[0006] 4. High noise: Since some spoken short texts may come from voice translation, voice signal transmission depends on the communication environment. The instability of network signals and environmental noise will greatly affect the quality of voice signals, resulting in errors and fragmentary omissions in some spoken short texts.

[0007] 5. Shorter sentences: In most cases, people do not often use complex sentences with elaborate modifications in their spoken expressions. Instead, they tend to use concise and clear expressions, and they contain many meaningless words that only express responses, such as "um" and "hello". Effective sentences are generally shorter.

[0008] The characteristics of spoken short texts, such as high noise, high error rate, poor standardization, and short sentence length, make text classification extremely difficult.

[0009] Text classification is an important part of natural language processing tasks. The steps of Chinese text classification are often divided into three parts: data preprocessing, text representation, and classification using classification models. Data preprocessing includes data cleaning, Chinese word segmentation, part-of-speech tagging, etc. Text representation is to digitize Chinese words to facilitate the calculation of classification models. The mainstream text classification models are roughly classic classification models based on statistical learning, such as naive Bayes classifier, support vector machine, Rocchio algorithm, KNN, etc. The statistical learning-based methods have small computational complexity, low complexity, and require less training expectations. However, the accuracy of these algorithms is highly dependent on the quality of the text. It is difficult to deal with highly complex and highly diverse texts, and they are greatly affected by text noise. If the text length is too short, the classification accuracy may be greatly reduced, which is not enough to deal with the classification of massive short spoken texts.

[0010] The machine learning technology that has emerged in recent years, especially the deep learning algorithm implemented by highly complex neural networks, has achieved excellent results in multiple branches of natural language processing tasks and has almost become the only choice for handling complex text classification tasks. However, these deep learning algorithms have great costs and prices while achieving good results. (1) Lack of explainability: The input of a highly complex neural network is generally a vector (word vector or sentence vector) obtained by preprocessing the text, which is used to digitize natural language and output as a prediction result or classification result. The intermediate process is vague and difficult to understand and control. (2) Huge amount of computation: The training of deep neural networks often requires a huge amount of computation. The training time can easily take several days. The parameters are often relatively complex, and it takes a long time to trial and error to select the appropriate parameters. (3) Limited by the training corpus: The final effect of the deep learning algorithm is closely related to the quality and quantity of the training corpus. The more complex the network, the more training data it requires. However, the cost of manual annotation of colloquial short texts is very high, and it cannot cope with corpora of different genres and fields. If you want to solve the text classification problem in a certain field, you can only spend a lot of time building a corpus in that field, such as medicine, news media, etc., and then train a model for the corpus in that field. Summary of the invention

[0011] The purpose of the present invention is to realize a text screening method for massive high-noise colloquial short texts, to solve the problems of large noise interference, large training corpus demand, low classification accuracy, etc. in the classification of colloquial short texts, and the model can be improved and corrected through business annotation as needed to further improve the classification accuracy.

[0012] To achieve the above object, the present invention adopts the following technical solutions:

[0013] A text screening method for massive high-noise colloquial short texts includes the following steps:

[0014] Preprocess the training corpus and the target text to be screened;

[0015] Extract sentence information from the annotated positive class corpus in the preprocessed training corpus, and take sentences containing the annotated keywords (that is, related to the business) as sentences that are strongly related to the business, and sentences that do not contain the annotated keywords as sentences that are weakly related to the business;

[0016] The extracted sentence information is used to perform sentence matching on the preprocessed target text, and the matching results of the business-strongly related sentences are classified as positive texts, and the matching results of the business-weakly related sentences are processed in the following steps;

[0017] Perform text processing on both the target text and the training corpus, and convert the processed text into word vector representation;

[0018] Use the word vector representation of the training corpus to train the text classification model, input the word vector representation of the target text into the trained text classification model to classify the text, and realize text screening of the target text.

[0019] Furthermore, text classification models include TextCNN, FastText, TextRNN, HierarchicalAttention Network, seq2seq with attention and other models, and the TextCNN model is preferred.

[0020] Furthermore, after the target text is screened, it is detected whether the screening result contains annotation information. If so, the training corpus is updated according to the annotation information, and the text classification model (such as the TextCNN model) is retrained.

[0021] Furthermore, preprocessing includes word segmentation, noise reduction and pinyin substitution.

[0022] Furthermore, the pre-processing step includes:

[0023] First, segment the words using the Jieba Chinese word segmenter;

[0024] Secondly, the text is error-checked using a sliding window, and the language model is constructed by calculating the maximum likelihood estimate using the n-gram model. If the calculated probability is lower than the legal threshold, the text at the sliding window is judged to be erroneous.

[0025] Again, for errors at the character level, a dictionary of near-phonetic words is used to obtain a candidate set, and the legitimacy of the sentence in the sliding window is calculated using the n-gram model. All candidate set results are compared and sorted to obtain the optimal corrected word; for errors at the word level, the pinyin is directly used to replace the word.

[0026] Furthermore, the sentence pattern matching step includes: extracting sentence patterns from sentences in the text; comparing similarities between sentence patterns, and selecting the sentence pattern with the greatest similarity for matching.

[0027] Furthermore, the sentence pattern extraction step includes: segmenting the sentence and marking the parts of speech; for the subject, object, and complement components in the sentence that represent the object of the action, remove the noun components including names of people, places, and organizations, and replace them with their part-of-speech tags, and retain the pronouns; replace the time and place in the adverbial with part-of-speech tags, and replace the modifying components in the attributive with a unified character representation; form a list of alternative words for the verb and conjunction parts, and the sentence pattern is represented by a list of vocabulary and part-of-speech tags.

[0028] Furthermore, the step of comparing the similarity between sentences includes: converting semantically meaningful words into word vectors, and calculating the Euclidean distance of their word vectors in the vector space based on the edit distance; treating verbs in the same candidate verb list as the same words, and for verbs not in the candidate verb list, selecting the words with the minimum distance to the words in the word list as the words with the maximum similarity, and performing sentence matching based on the above-mentioned same words or words with the maximum similarity.

[0029] Furthermore, text processing is the operation of limiting the length of the text, removing colloquial words, and merging repeated words.

[0030] Furthermore, the steps of text processing include:

[0031] Use a sliding window to scan the text, and only keep the first word that appears twice or more in the sliding window;

[0032] Establish a colloquial word database and remove meaningless colloquial words;

[0033] Texts with less than 5 valid words were eliminated, and the average length of the training texts was calculated. Texts with a length of more than 1.5 times the average length were truncated at the end of the sentence closest to the average length to retain complete sentences.

[0034] Furthermore, the steps of converting text into word vector representation are as follows: using the word-to-vector model to train word vectors for the processed text, and representing the text as word vectors; for words replaced by pinyin, calculating the pinyin similarity based on the edit distance, and finding the word with the highest pronunciation similarity to the pinyin based on the similarity, and using the word vector of the word as the word vector of the pinyin replacement word to obtain the final word vector representation.

[0035] Compared with deep learning methods and classic natural language processing algorithms based on statistical learning, the method of the present invention first uses a text error correction algorithm that takes pinyin similarity into account to perform noise reduction on colloquial short texts, and integrates techniques such as sentence extraction and matching, rule filtering, and semantic similarity analysis. It has a relatively small amount of calculation, high accuracy, and requires less training corpus. The accuracy of the model can be improved based on feedback from business annotation information.

[0036] The main innovations of this invention are: (1) a pre-processing and denoising method for massive high-noise colloquial short text data, correcting word granularity errors, and using pinyin to replace word granularity errors that are difficult to correct; (2) a lightweight semantic relevance screening method. Traditional text classification has high requirements for the corpus and high computational complexity. This method proposes a lightweight colloquial short text screening method that integrates idiomatic sentence extraction and matching, word vector representation combining vocabulary and pinyin similarity, and semantic relevance analysis based on text classification models (such as TextCNN models). (3) a feedback model correction method based on business annotation information to sustainably improve the model classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart for the classification of colloquial short texts.

[0038] Figure 2 This is an example diagram of Chinese sentence extraction and matching results.

[0039] Figure 3 This is an example diagram of text simplification.

[0040] Figure 4 This is the training flow chart of the TextCNN text classification model. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the method and steps for extracting idiomatic sentence patterns from natural language text according to the present invention are further described in detail below in conjunction with the accompanying drawings.

[0042] The main purpose of the present invention is to provide a text screening method for massive high-noise colloquial short texts, perform high-accuracy classification based on a small amount of training corpus, and perform feedback correction on the classification model based on the text annotated by business personnel.

[0043] According to the first aspect of the present invention, a Chinese text error correction algorithm based on improved pinyin similarity fuzzy matching is used: according to the characteristics of spoken short texts, most vocabulary errors are similar-sounding words and homophones (due to problems with pinyin input or phonetic translation). Chinese text error correction is divided into error detection, error correction and pinyin replacement. The error detection part detects errors from both character granularity and word granularity to obtain a candidate set of suspected error positions; for the error correction part, for errors at the character granularity, the n-gram language model can be used for correction (using the candidate dictionary to obtain alternative words to replace the error position, and then calculating the sentence legality through the n-gram model, comparing and sorting all candidate set results, and obtaining the optimal corrected word); for errors at the word granularity, due to the high noise and high error rate of spoken short texts and the short sentences, it is difficult to use the algorithm to infer the corrected word through other parts of the sentence. At this time, the pinyin is used to replace the word.

[0044] According to the second aspect of the present invention, text pattern correlation matching is performed based on idiomatic sentence patterns extracted from training corpus. For certain specific types of text, there are idiomatic expression sentence patterns. Commonly used sentence patterns in the positive class corpus in the training corpus are extracted, and sentence pattern matching is performed on the corpus to be classified, so that most of the negative class texts can be eliminated. Sentence pattern extraction part: first, the sentence is segmented and dependency syntactic analyzed, and the words representing specific action objects are eliminated based on the rules, and the sentence trunk is retained. The sentence pattern matching part converts the words in the sentence trunk into word vectors using the word-to-vector algorithm, calculates the word similarity based on the word vector, and calculates the similarity of the overall sentence pattern based on the edit distance.

[0045] According to the third aspect of the present invention, the corpus is processed and represented by word vectors according to the characteristics of the colloquial short text, and a text classification model (the TextCNN model is taken as an example below) is used to perform semantic relevance analysis on the target classified corpus. In the sentence matching stage, the matching results obtained from some sentences with weak business relevance cannot be determined to be positive texts of business concern, and only the sentence patterns are constrained. There is no restriction on the specific objects discussed in the text, so it is necessary to further classify the text at the semantic level. According to the characteristics of the colloquial short text, this method removes colloquial words, merges repeated words, and imposes length constraints on the colloquial short text:

[0046] (1) Eliminate colloquial words: In colloquial short texts, modal particles, meaningless words expressing responses, and common conjunctions in spoken language appear very frequently. These words lengthen the text, while the length of the TextCNN input vector is fixed and limited. Eliminating these words can make the input vector contain more effective information, thereby improving the accuracy.

[0047] (2) Merging repeated words: When people are communicating informally, they tend to repeat the same words or phrases several times in a row to ensure that the other party can hear clearly or out of habit. In this case, the repeated part should be retained only once, which not only preserves its semantics but also avoids it occupying the effective length meaninglessly.

[0048] (3) Length limitation: Due to the differences in the length of spoken texts, there may be large differences in the length of positive and negative texts when training. However, length cannot be used as a basis for classification. The TextCNN neural network can easily learn the length of the non-zero part of the input vector as a feature. Therefore, it is necessary to truncate overly long texts so that their length is roughly similar to the average length of the corpus. For texts that are too short, they need to be eliminated because they have too little effective information.

[0049] These operations are designed for the characteristics of short spoken texts and the learning characteristics of TextCNN neural networks, which can improve the classification accuracy of TextCNN, but cannot be combined in the preprocessing step, which may affect sentence matching. After the training corpus is processed as above, the word-to-vector algorithm is used to train Chinese text into word vectors and input into the shallow TextCNN neural network. Since the network has low complexity and can better capture the local relevance of the text, accurate text classification can be achieved with less training corpus.

[0050] According to the fourth aspect of the present invention, the model is corrected in a feedback manner according to the business annotation data. In a large amount of colloquial short texts, various possibilities may occur, and because the training corpus is extremely limited, it is still possible to misclassify the text, so it is necessary to use the annotation information of the classification result data by the business personnel to correct the model and continuously improve the accuracy of the model. The positive and negative corpora in the training corpus can be updated according to the standard information of the business personnel, the common positive sentence patterns can be updated, and the seed vocabulary can be updated, thereby improving the accuracy of the classification results.

[0051] Figure 1The flow chart of colloquial short text classification is given. As shown in the figure, first, both the training corpus and the target text to be screened are preprocessed; second, the training corpus contains texts and their labels. The labels include two types: positive (indicating that the text is related to the business) and negative (indicating that the text is not related to the business). The sentence information of the annotated positive corpus (i.e., business-related text) in the training corpus is extracted respectively, and the strong business-related sentences and weak business-related sentences are distinguished; third, the text is sentence-matched. The matching results of the sentence patterns that are strongly related to the business can be directly classified as positive texts, and the matching results of the sentence patterns that are weakly related to the business need to be analyzed in the next step; fourth, the target text and the training corpus are subjected to length restrictions, colloquial words are removed, and repeated words are merged. Fourth, the word vector is trained using the word-to-vector algorithm to convert the text into a representation of the word vector. The training corpus is used to train the TextCNN neural network. The trained model is used to classify the target text and the desired results are screened out. Fifth, if the business personnel provide the annotation information for the screening results, the training corpus is updated and the model is retrained.

[0052] The specific method of the preprocessing part is: first use the jieba Chinese word segmenter to segment words. Since there are typos in the sentence, the segmentation results often have segmentation errors. The n-gram model is based on the Markov model hypothesis. The probability of a word only depends on the previous word or the previous few words of the word. Typos in Chinese text are localized. In the error detection part, you only need to select a sliding window of reasonable length to check whether there are typos. Due to the characteristics of colloquial short texts, the sliding window length should be short. The n-gram model constructs a language model by calculating the maximum likelihood estimate. If its probability is lower than the legal threshold, it is judged that an error occurs here. In the error correction part, for word-level errors, a dictionary of near-phonetic words is used to obtain a candidate set, and then the sentence legality in the sliding window is calculated through the n-gram language model. All candidate set results are compared and sorted to obtain the optimal corrected word. For word-level errors, due to the high noise of the text to be classified, the candidate set may be too large, and pinyin is directly used as a substitute.

[0053] Sentence matching part: Figure 2Examples of Chinese sentence pattern extraction and matching results are given. First, sentence patterns are extracted from complete sentences. The sentences are tokenized, dependency syntactic analyzed, and part-of-speech tagged. For parts representing the action objects such as the subject, object, and object complement in the sentence, noun components such as personal names, place names, and organization names are removed, replaced by their part-of-speech tags, pronouns are retained, time and place in adverbials are also replaced by part-of-speech tags, and the modifying components in attributives are uniformly replaced by 'de' to represent the modifying components. The verb and conjunction parts form a list of alternative words, such as ['ask', 'consult', 'inquire']. The sentence pattern is represented as a list of words and part-of-speech tags. Second, the similarity between sentence patterns is compared, and the sentence pattern with the greatest similarity is selected for matching. The sentence pattern matching method is based on the edit distance (i.e., the Levenshtein distance). According to the characteristics of natural language, some words such as auxiliary words do not have very important semantic functions and are ignored when calculating the distance. Any verbs in the same list of alternative verbs are regarded as the same. If not in the list of alternative words, the minimum distance from the words in the word list is taken. When calculating the distance of words, they are converted into word vectors, and the Euclidean distance of their word vectors in the vector space is calculated, and the sigmoid function is used to map it to the interval [0, 1].

[0054] Model classification part: Text processing is performed on both the text to be classified and the training corpus, and the steps are as follows:

[0055] (1) Use a sliding window to scan the text, and only retain the first occurrence of words that appear two or more times within the sliding window;

[0056] (2) Establish a colloquial word library and remove meaningless colloquial words that appear;

[0057] (3) Length limit. After the processing of the first two steps, texts with less than 5 valid words are removed. Calculate the average length of the training text, and for texts whose length is more than 1.5 times the average length, intercept them. The interception position is the end of the sentence closest to the average length to retain the complete sentence.

[0058] Figure 3 It is a diagram showing the example of text simplification, which shows the difference before and after text processing.

[0059] Use the word-to-vector model to train word vectors for the processed text and represent the text as word vectors; for the words replaced by pinyin due to the noise of colloquial short texts, calculate the pinyin similarity based on the edit distance, and find the word with the highest similarity in pronunciation to this pinyin according to the similarity (if there are multiple words with the same pronunciation, take the word with the highest frequency of occurrence in the corpus), and use the word vector of this word as the word vector of the pinyin replacement word. Then use the training text to train the TextCNN network, and after obtaining the classification model, classify the text to be classified. Figure 4Flowchart for TextCNN model training.

[0060] It should be noted and understood that various modifications and improvements can be made to the present invention described in detail above without departing from the spirit and scope of the present invention required by the appended claims. Therefore, the scope of the technical solution claimed is not limited by any specific exemplary teaching given.

Claims

1. A text screening method for massive high-noise colloquial short texts, characterized in that: The following steps are involved: Preprocess the training corpus and the target text to be screened; Preprocessing includes word segmentation, noise reduction and pinyin replacement. The steps of preprocessing include: first, segment words through the Jieba Chinese word segmenter; second, use the sliding window to detect errors in the text, and use the n-gram model to calculate the maximum likelihood estimate to construct a language model. If the calculated probability is lower than the legal threshold, the text at the sliding window is judged to be wrong; third, for errors at the word granularity, use the dictionary of near-phonetic words to obtain the candidate set, calculate the legality of the sentence in the sliding window through the n-gram model, compare and sort all the candidate set results, and get the optimal corrected word; for errors at the word granularity, directly use the pinyin to replace the word; Extract sentence information from the annotated positive class corpus in the preprocessed training corpus, and take sentences containing the annotated keywords as business-strongly relevant sentences, and sentences not containing the annotated keywords as business-weakly relevant sentences; The extracted sentence information is used to perform sentence matching on the preprocessed target text, and the matching results of the business-strongly related sentences are classified as positive texts, and the matching results of the business-weakly related sentences are processed in the following steps; Both the target text and the training corpus are processed, and the processed text is converted into a word vector representation; the steps of converting the text into a word vector representation are as follows: the word vector is trained on the processed text using the word-to-vector model, and the text is represented as a word vector; for words replaced by pinyin, the pinyin similarity is calculated based on the edit distance, and the word with the highest pronunciation similarity to the pinyin is found according to the similarity, and the word vector of the word is used as the word vector of the pinyin replacement word to obtain the final word vector representation; Use the word vector representation of the training corpus to train the text classification model, input the word vector representation of the target text into the trained text classification model to classify the text, and realize text screening of the target text.

2. The method according to claim 1, characterized in that The steps of sentence matching include: extracting sentence patterns from sentences in the text; comparing the similarities between sentence patterns, and selecting the sentence pattern with the greatest similarity for matching.

3. The method according to claim 2, characterized in that The steps of sentence extraction include: segmenting the sentence and marking the parts of speech; for the subject, object, and complement components in the sentence that represent the object of the action, remove the noun components including names of people, places, and organizations, and replace them with their part-of-speech tags, and retain the pronouns; replace the time and place in the adverbial with part-of-speech tags, and replace the modifying components in the attributive with a unified character representation; form a list of alternative words for the verb and conjunction parts, and the sentence pattern is represented by a list of vocabulary and part-of-speech tags.

4. The method according to claim 2, characterized in that The steps of comparing the similarity between sentences include: converting semantically meaningful words into word vectors, and calculating the Euclidean distance of the word vectors in the vector space based on the edit distance; treating verbs in the same candidate verb list as identical words, and selecting the words with the smallest distance from the verbs in the word list for verbs not in the candidate word list as words with the greatest similarity, and performing sentence matching based on the identical words or words with the greatest similarity.

5. The method according to claim 1, characterized in that Text processing is the operation of limiting the length of text, removing colloquial words, and merging repeated words.

6. The method according to claim 5, characterized in that The steps of text processing include: Use a sliding window to scan the text, and only keep the first word that appears twice or more in the sliding window; Establish a colloquial word database and remove meaningless colloquial words; Texts with less than 5 valid words were eliminated, and the average length of the training texts was calculated. Texts with a length of more than 1.5 times the average length were truncated at the end of the sentence closest to the average length to retain complete sentences.

7. The method according to claim 1, characterized in that After text screening of the target text, check whether the screening result contains annotation information. If so, update the training corpus according to the annotation information and retrain the text classification model.

Images