A method for content recognition and correction of text-specific sentences

Abstract

The application discloses a content recognition and error correction method for specific sentences in a text, and belongs to the field of natural language processing. The method combines multiple technologies, including regular matching, hfl / rbt3 model upper and lower sentence prediction, location-sensitive hash technology, and Jaccard similarity calculation method, and is aimed at specific contexts such as important speech recognition and error correction of specific sentences. Various methods are combined to accurately identify the important speech and other gold sentence parts in news articles. By improving the minhash algorithm, it is more effective in processing specific texts, and Jaccard similarity calculation is used for matching of short texts, so that it also has high accuracy in error recognition. The method can effectively extract and verify specific sentences such as important speech content, and plays an important role in improving efficiency and accuracy for news release.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for content recognition and error correction of specific sentences in text. Background Technology

[0002] The scenario of identifying text content in an article is similar to that of citation identification, specifically targeting the task of hidden citations. Citation identification is the process of identifying and extracting citations from academic text. Currently, a common approach is to use deep neural networks and attention mechanisms to extract reference text from the citation context. This task can be viewed as a classification or sequence labeling task. For example, in sequence labeling, the model is trained to predict the attributes of words in the text, thereby determining which sentences are citation sentences. Early implementations also used the sliding window approach, selecting an appropriate window size to choose citation segments and then using text similarity matching to identify hidden citations.

[0003] In the existing technology, hfl / rbt3 is a RoBERTa-based pre-trained Chinese language model. It is a small model with 3 layers and 12 attention heads. It was released by Harbin Institute of Technology (HIT) as part of its Chinese-BERT-wwm project. hfl / rbt3 can be used for various natural language processing tasks, such as text classification, named entity recognition, sentiment analysis, etc. It can be easily loaded and used with Transformers or PaddleHub libraries.

[0004] Location-Sensitive Hashing (LSH) is a fast near-neighbor search technique for massive amounts of high-dimensional data, offering high efficiency. Its basic idea is to hash the input high-dimensional feature vector into a low-dimensional feature space using a hash function. This hashing method ensures that points that are close in the original feature space remain close in the low-dimensional space after hashing; that is, the probability of collision between close points is higher than the probability of collision between distant points. LSH effectively reduces the complexity of data storage and retrieval and is suitable for various data types, including text, images, and audio.

[0005] For the specific operation of position-sensitive hashing, firstly, a series of hash functions need to be defined to map high-dimensional input vectors to low-dimensional output values. The hash functions should satisfy the property that similar input vectors have a high probability of being mapped to the same output value, while different input vectors have a low probability of being mapped to the same output value. Secondly, the resulting hash value is divided into several parts, denoted as keys, and stored in a hash table along with the original input vector, where the key is the original input vector and the value is the original input vector. Next, to find the nearest neighbor of the query vector, the same hash function is applied, and the corresponding output value is searched in the hash table. All input vectors that share at least one key with the query vector are retrieved, and their distances to the query vector are calculated. The closest vector is returned as the approximate nearest neighbor.

[0006] Minhash is a technique for quickly estimating the similarity between two sets. Invented by Andrei Broder, it was initially used in the AltaVista search engine to detect and remove duplicate web pages. It has also been applied to large-scale clustering problems, such as clustering documents based on the similarity of their word sets. The minhash value is actually related to Jaccard similarity: the Jaccard similarity between two sets is equal to the probability that their random permutations yield the same minhash value. Therefore, minhash is often used to estimate Jaccard similarity, and the minhash algorithm provides a fast approximation of the Jaccard similarity between two sets.

[0007] The aforementioned technologies have different applications in different scenarios; however, there is no comprehensive method for specific sentence recognition (such as important speeches, poems in articles, famous quotes from important figures, etc.). For specific application scenarios, the algorithm needs to be appropriately modified according to the specific scenario. In certain scenarios, deep learning is not necessarily the best choice; simple rule-based recognition is sometimes more accurate and efficient than deep learning. This invention utilizes a rule-based method to achieve recognition of specific sentences in articles. Summary of the Invention

[0008] This invention addresses the identification of specific sentences in articles by comprehensively utilizing existing methods to construct a text-oriented identification and error correction method. The method is roughly divided into two parts: an identification module and a matching module. The identification module is mainly based on regular expression matching and the Roberta model, while the matching module is mainly based on position-sensitive hashing and Jaccard similarity. The identification results are accurate and reliable.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] This invention provides a method for content recognition and error correction of specific sentences in text, comprising the following steps:

[0011] S1. Apply regular expression matching rules to initially identify the key phrases. The identification situation is mainly divided into two categories: one is that they are enclosed in double quotes. This type can be correctly matched after regular expression and proceeds to step S3; the other type is not enclosed in double quotes and proceeds to step S2 for further processing.

[0012] S2. Use the trained hfl / rbt3 model to predict the next and next sentences, determine the key sentence, and proceed to step S3.

[0013] S3. Collect the database of memorable quotes and segment the identified quotes into sentences. For each sentence, determine the matching method based on sentence length. When the sentence length is greater than or equal to a set threshold, use the minhash matching method to calculate the sentence's minhash value. Store the hash values ​​corresponding to the sentences in the database, along with the sentences themselves, in a hash table. Use an inverted index to find the most similar sentence and proceed to step S4. When the sentence length is less than the set threshold, use the Jaccard similarity matching method to calculate the Jaccard value between the identified sentence and other sentences in the database. Find the sentence with the highest Jaccard value and proceed to step S4.

[0014] S4. Starting from the first sentence, use it as the target sentence to match the database. If a match is found, compare the subsequent parts of the sentence in the database with the subsequent parts of the identified key phrase to see if there are any errors. Output the recognition result and the erroneous parts.

[0015] Furthermore, in step S2, the training of the hfl / rbt3 model uses positive and negative samples from the upper and lower sentences. The positive samples are from the upper and lower sentences in the Golden Sentence Database, while the negative samples are collected manually. After data augmentation, the negative samples are input into the pre-trained model hfl / rbt3 for training. Data augmentation methods include, but are not limited to, random deletion, random replacement, and random addition.

[0016] Furthermore, the specific steps for further judgment using the trained model in step S2 are as follows:

[0017] S201. First, obtain a paragraph without quotation marks. Divide the paragraph into sentences, using periods, exclamation marks, question marks, and ellipses as the basis for sentence division.

[0018] S202. Then, the first sentence is taken as the golden sentence, and the next sentence is taken as the next sentence. After being processed by the tokenizer of the hfl / rbt3 model, it is input into the hfl / rbt3 model to judge the return value.

[0019] S203. Initialize the results. The first sentence is the golden sentence. Then input it into the model. If the return value is 1, it means that the next sentence is also a golden sentence. Add the sentence to the result sentence. Repeat step S202. Use the sentence as the previous sentence. Then judge whether the following sentence is the next sentence. If the return value is 0, it means that the next sentence is not a golden sentence. Then exit and stop judging. When all sentences have been traversed, the process ends, and the result sentence, which is the golden sentence part, is obtained.

[0020] Furthermore, the specific steps for performing the minimum hash operation on the sentence in step S3 are as follows:

[0021] S301. First, perform 3-shingle processing on the sentence to obtain a list of shingles;

[0022] S302. Define the number of random permutations and transformations as perm_num, and then randomly generate perm_num coefficients of affine transformation functions for a and b, i.e.:

[0023]

[0024] Where m is a Mersenne prime, taken as 2. 61 -1, where a and b are random integers less than the maximum value, and the maximum value is 2. 32 -1;

[0025] S303. For each hash function, perform a hash operation on each shingle, and find the minimum value as the minimum hash value of this hash function. The perm_num minimum hash functions form the minimum hash value of the final original text.

[0026] Furthermore, the specific steps for performing the storage operation on the database sentence in step S3 are as follows:

[0027] (1) Define the number of random permutations as perm_num, divide all random permutations into b segments, each segment has r hashes. Assume the true Jaccard similarity between two texts is js, then the probability that a corresponding part of the perm_num minimum hashes between the two texts is equal is equal to js. The probability that the minimum hash signatures of the two texts are all equal in a certain segment, that is, the probability of becoming a candidate pair, is:

[0028]

[0029] The relationship between document Jaccard similarity and the probability of becoming a candidate pair is an S-shaped curve. The similarity at the half-probability point (0.5 Jaccard similarity) is the threshold. At this point, the approximate relationship between b, r, and the threshold is as follows:

[0030]

[0031] Based on the provided threshold, perm_num, false positive weight, and false negative weight, the optimal values ​​of b and r are obtained. False positives are dissimilar text pairs that are hashed into the same bucket, and false negatives are similar text pairs that are not hashed into the same bucket. The weights of false positives and false negatives are set to 0.5 each.

[0032] (2) Calculate the minhash value for each sentence in the database, and then divide the obtained perm_num hashes into b blocks. Each block contains r minimum hashes. The content of the b blocks is used as the b-segment hash value (key) of the sentence.

[0033] (3) Record the hash values ​​(keys) of segment b into a hash table (HashTables), where each hash value (key) is an index, i.e. a key, and the index value is the corresponding sentence representation. At the same time, create a dictionary whose key is the sentence representation, denoted as mi, where i is an integer starting from 0, and the value is the sentence itself.

[0034] Furthermore, the specific steps for inverted index matching in step S3 are as follows:

[0035] For the target sentence, first determine the sentence length. If it is greater than a certain value, perform a minhash and obtain its corresponding hash value (key). In the hash table, iterate through all hash values ​​to find a candidate set with some identical hash values. Then, find the instance in the candidate set that has the closest Hamming distance to the target sentence. If it is within the tolerance range, output it; otherwise, mark it as unrecognized.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention identifies specific sentences, such as important speeches, based on regular expression matching, without directly using deep learning models for citation identification. This results in more accurate identification and does not require the collection of large amounts of data, thus reducing the cost of manual annotation.

[0038] 2. This invention further identifies the key quotes based on the Roberta model, considering whether all directly quoted parts without quotation marks are key quotes. This method makes the identification of key quotes more accurate.

[0039] 3. This invention uses position-sensitive hashing (minhash) and Jaccard similarity calculation to match key phrases and identify errors. Minhash maps high-dimensional feature vectors to low-dimensional feature vectors, storing long sentences in hash form. It can be computed and stored offline and uses an inverted index for similarity matching, making it highly efficient for massive amounts of long text. Jaccard similarity calculation is more accurate for short sentences. The combination of these two methods balances efficiency and accuracy.

[0040] 4. This invention uses 3-shingle to process the target sentence instead of jieba segmentation, resulting in higher accuracy and processing speed. Due to the special nature of important speeches and other key sentences, even a single word error is generally unacceptable. Jieba segmentation, which calls third-party libraries, takes longer, and minhash calculation after removing stop words and punctuation widens the gap between the erroneous sentence and the original sentence. Therefore, the simple 3-shingle is more effective.

[0041] In summary, this invention proposes a content recognition and error correction method for specific sentences in text. This method combines multiple technologies, including regular expression matching, HFL / RBT3 model for sentence prediction, position-sensitive hashing (minhash), and the Jaccard similarity calculation method. It can accurately identify specific sentences, such as important speeches in news articles. Furthermore, by improving the minhash algorithm, it becomes more effective in processing specific texts, achieving higher accuracy in error recognition. This plays a crucial role in important national fields such as news releases. This method can also be extended to identify poetic lines or famous quotes from other important figures in articles. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments recorded in this invention, and those skilled in the art can obtain other drawings based on these drawings.

[0043] Figure 1 A flowchart is provided for an embodiment of the present invention to illustrate a method for content recognition and error correction of specific sentences in text.

[0044] Figure 2 The flowchart illustrates the training and inference process of the sentence prediction model provided in this embodiment of the invention.

[0045] Figure 3 A flowchart of the minhash operation provided in an embodiment of the present invention.

[0046] Figure 4This is a schematic diagram illustrating the storage principle of the minhash hash table provided in an embodiment of the present invention.

[0047] Figure 5 This is a diagram showing the complete input and output results provided for an embodiment of the present invention. Detailed Implementation

[0048] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] The following example, using an important speech, illustrates a content recognition and error correction method for specific sentences in text, as described in this invention. Figure 1 As shown, it includes the following steps:

[0050] Step 1: Regular expression matching and recognition

[0051] Using regular expression matching rules, we first identify important speeches, which are also referred to as key quotes. The identification results are mainly divided into two categories: those enclosed in double quotes, which can be correctly matched after regular expression processing and require no further processing; and those not enclosed in double quotes, which require further processing.

[0052] Step Two: Training and Inference of the Sentence Prediction Model

[0053] like Figure 2 This demonstrates the steps for fine-tuning and inference of the pre-trained model hfl / rbt3. First, positive and negative samples of the preceding and following sentences of important speeches are collected. Positive samples are obtained from a database of key quotes, while negative samples are collected manually. Therefore, there is an imbalance in the positive and negative sample data. Data augmentation is then performed on the negative samples. Next, lexical analysis is performed on the data, and then it is input into the pre-trained model hfl / rbt3 for model training.

[0054] The specific steps for inference using the trained model are as follows:

[0055] S1. First, obtain a paragraph without quotation marks. Divide the paragraph into sentences, using periods, exclamation marks, question marks, and ellipses as the basis for sentence division.

[0056] S2. Then, the first sentence is taken as the golden sentence, and the next sentence is taken as the next sentence. After being processed by the tokenizer of the hfl / rbt3 model, it is input into the hfl / rbt3 model to judge the return value.

[0057] S3. If the return value is 1, it means that the next sentence is also a golden sentence. Add it to the result sentence and repeat step S2. Treat this sentence as the previous sentence and then judge whether the following sentence is the next sentence. If the return value is 0, it means that the next sentence is not a golden sentence. Exit and stop judging. When all sentences have been traversed, the process ends, and the result sentence, which is the golden sentence part, is obtained.

[0058] Step 3: minhash operation

[0059] like Figure 3 The general steps of minhash are shown. The specific steps for performing minhash on a sentence are described below:

[0060] S1. First, the text 3-shingle is processed;

[0061] S2. Define the number of random permutations and transformations as perm_num, and then randomly generate perm_num coefficients of affine transformation functions for a and b, i.e.:

[0062]

[0063] Where m is a Mersenne prime, here we take 2. 61 -1, where a and b are random integers less than the maximum value, and the maximum value is 2. 32 -1.

[0064] S3. For each hash function, perform a hash operation on each shingle, and find the minimum value as the minimum hash value of this hash function. The perm_num minimum hash functions form the minimum hash value of the final original text.

[0065] Step 4: Minhash Index Storage

[0066] The sentences in the collected important speech database are stored in a hash table to facilitate quick matching of similar sentences later, such as... Figure 4 As shown, the specific steps for performing storage operations on database sentences are as follows:

[0067] S1. Consider dividing perm_num into b segments, each with r hashes. Assuming the true Jaccard similarity between two texts is js, the probability that corresponding parts of the perm_num minimum hashes of the two texts are equal is equal to js. Therefore, the probability that all hashes of the minimum hash signatures of the two texts are equal in a certain segment is:

[0068]

[0069] In other words, it represents the probability of becoming a candidate pair. The relationship between document Jaccard similarity and the probability of becoming a candidate pair is an S-shaped curve, largely unaffected by b and r. The similarity at the 1 / 2 point of the candidate probability is the threshold, which is 0.5 Jaccard similarity. At this point, the approximate relationship between b, r, and the threshold is as follows:

[0070]

[0071] Based on the provided threshold, perm_num, false positive weight, and false negative weight, the optimal values ​​of b and r are obtained. False positives are dissimilar text pairs that are hashed into the same bucket, and false negatives are similar text pairs that are not hashed into the same bucket. Generally, the weights of false positives and false negatives are set to 0.5 each, which means that we want both cases to occur as little as possible.

[0072] S2. Calculate the minhash value for each sentence in the database, and then divide the obtained perm_num hashes into b blocks. Each block contains r minimum hashes, and the content of the b blocks is used as the b-segment hash value (key) of the sentence.

[0073] S3. Record the hash values ​​(keys) of segment b into a hash table, where each hash value is an index, that is, a key, and the index value is the corresponding sentence representation, such as "m1". At the same time, create a dictionary with the sentence representation "mi" as the key, i as an integer starting from 0, and the value as the sentence itself.

[0074] Step 5: Identifying errors in the identified key phrases

[0075] For the identified key phrases in the input, the key phrases are segmented into sentences. Starting from the first sentence, they are used as the target sentences for matching in the database. If a match is found, the subsequent parts of that sentence in the database are compared with the subsequent parts of the identified key phrase to check for errors.

[0076] For matching the target sentence, first determine the sentence length. If it is greater than a certain value, perform a minhash and obtain its corresponding hash value (key). In the hash table, iterate through all hash values ​​to find a candidate set with some identical hash values. Then, find the instance in the candidate set that has the closest Hamming distance to the target sentence. If it is within the tolerance range, output it; otherwise, mark it as unrecognized. If the sentence length is less than a certain value, calculate the Jaccard similarity between the target sentence and other sentences in the database. Find the one with the highest similarity. If the similarity value is within a certain range, output it; otherwise, mark it as unrecognized.

[0077] Figure 5The complete input and output results are shown. After inputting an article, the above-mentioned key sentence recognition and similarity matching are performed to obtain the original sentence that is similar to the target sentence, and the original sentence and the recognized sentence are output.

[0078] In summary, the content recognition and error correction method for specific sentences in text proposed in this invention combines multiple technologies, including regular expression matching, HFL / RBT3 model for sentence prediction, position-sensitive hashing (minhash), and Jaccard similarity calculation. This method can accurately identify important speeches in news articles. Furthermore, by improving the minhash algorithm, it becomes more effective in processing specific texts, resulting in higher accuracy in error recognition. This method can also be extended to identify poetic lines or famous quotes from other important figures in articles.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for content recognition and error correction of specific sentences in text, characterized in that, Includes the following steps: S1. Apply regular expression matching rules to initially identify the key phrases. The identification situation is mainly divided into two categories: one is that they are enclosed in double quotes. This type can be correctly matched after regular expression and proceeds to step S3; the other type is not enclosed in double quotes and proceeds to step S2 for further processing. S2. Use the trained hfl / rbt3 model to predict the next and next sentences, determine the key sentence, and proceed to step S3. The specific steps for predicting the preceding and following sentences using the trained model in step S2 are as follows: S201. First, obtain a paragraph without quotation marks. Divide the paragraph into sentences, using periods, exclamation marks, question marks, and ellipses as the basis for sentence division. S202. Then, if the first sentence is the key sentence, then that sentence is the previous sentence and the next sentence is the next sentence. After being processed by the word analyzer of the hfl / rbt3 model, it is input into the hfl / rbt3 model to judge the return value. S203. Initialize the results. The first sentence is the key sentence. Then input it into the model. If the return value is 1, it means that the next sentence is also a key sentence. Add the sentence to the result sentence. Repeat step S202. Use this sentence as the previous sentence. Then determine whether the following sentence is the next sentence. If the return value is 0, it means that the next sentence is not a golden sentence, so the process will stop and no longer continue. The process ends when all sentences have been traversed, thus obtaining the result sentence, which is the golden sentence part. S3. Collect the database of memorable quotes and segment the identified quotes into sentences. For each sentence, determine the matching method based on sentence length. When the sentence length is greater than or equal to a set threshold, use the minhash matching method to calculate the minimum hash value of the sentence. Store the hash values ​​corresponding to the sentences in the database as segments and sentences in a hash table. Use an inverted index to find the most similar sentence and proceed to step S4. When the sentence length is less than the set threshold, use the Jaccard similarity matching method to calculate the Jaccard value of the identified sentence and other sentences in the database. Find the sentence with the highest Jaccard value and proceed to step S4. The specific steps for storing the database sentence in step S3 are as follows: (1) Define the number of random permutations as perm_num, divide all random permutations into b segments, each segment has r hashes. Assume the true Jaccard similarity between two texts is js, then the probability that a corresponding part of the perm_num minimum hashes between the two texts is equal is equal to js. The probability that the minimum hash signatures of the two texts are all equal in a certain segment, that is, the probability of becoming a candidate pair, is: ， The relationship between document Jaccard similarity and the probability of becoming a candidate pair is an S-shaped curve. The similarity at the half-probability point (i.e., a Jaccard similarity of 0.5) is the threshold. At this point, the relationship between b, r, and the threshold is estimated as follows: ， Based on the provided threshold, perm_num, false positive weight, and false negative weight, the optimal values ​​of b and r are obtained. False positives are dissimilar text pairs that are hashed into the same bucket, and false negatives are similar text pairs that are not hashed into the same bucket. The weights of false positives and false negatives are set to 0.5 each. (2) Calculate the minhash value for each sentence in the database, and then divide the obtained perm_num hashes into b blocks, with r minimum hashes in each block. The content of the b blocks is used as the b-segment hash value of the sentence. (3) Record the hash values ​​of segment b into a hash table, where each hash value is an index, i.e. a key, and the index value is the corresponding sentence representation. At the same time, create a dictionary whose key is the sentence representation, denoted as mi, where i is an integer starting from 0, and the value is the sentence itself. S4. Starting from the first sentence, use it as the target sentence to match the database. If a match is found, compare the subsequent parts of the sentence in the database with the subsequent parts of the identified key phrase to see if there are any errors. Output the recognition result and the erroneous parts.

2. The content recognition and error correction method for specific sentences in text as described in claim 1, characterized in that, In step S2, the training of the hfl / rbt3 model uses positive and negative samples from the upper and lower sentences. The positive samples are from the upper and lower sentences in the Golden Sentence Database, while the negative samples are collected manually. After data augmentation, the negative samples are input into the pre-trained model hfl / rbt3 for training. Data augmentation methods include, but are not limited to, random deletion, random replacement, and random addition.

3. The content recognition and error correction method for specific sentences in text according to claim 1, characterized in that, The specific steps for performing the minimum hash operation on the sentence in step S3 are as follows: S301. First, perform 3-shingle processing on the sentence to obtain a list of shingles; S302. Define the number of random permutations and transformations as perm_num, and then randomly generate perm_num coefficients of affine transformation functions for a and b, i.e.: ， Where m is a Mersenne prime, taken as 2. 61 -1, where a and b are random integers less than the maximum value, and the maximum value is 2. 32 -1; S303. For each hash function, perform a hash operation on each shingle, and find the minimum value as the minimum hash value of this hash function. The perm_num minimum hash functions form the minimum hash value of the final original text.

4. The content recognition and error correction method for specific sentences in text according to claim 1, characterized in that, The specific steps for inverted index matching in step S3 are as follows: For the target sentence, first determine the sentence length. If it is greater than a certain value, perform minhash and obtain its corresponding hash value. In the hash table, traverse all hash values ​​by index and find a candidate set with some identical hash values. Then, find the instance in the candidate set that has the closest Hamming distance to the target sentence. If it is within the tolerance range, output it; otherwise, mark it as unrecognized.

Images