Semantic-level matching text duplicate checking method for high-concurrency scene

Abstract

The invention belongs to the technical field of natural language processing, and provides a semantic-level matching text duplicate checking method for a high-concurrency scene in order to solve the technical problem that in the prior art, accuracy and duplicate checking efficiency are low in the aspect of text duplicate checking, and the semantic-level matching text duplicate checking method for the high-concurrency scene comprises the steps that 1, candidate texts are stored in a database; 2, dividing the query text into query statements, and combining the query statements into batch query data; step 3, performing multi-field weighted query of parallel processing on batch query data; 4, calculating the number of repeated characters of the query statement and the matching statement; 5, duplicate checking results are saved, query texts are combined into batch query data, and the duplicate checking efficiency in a high-concurrency scene is improved by combining parallel processing multi-field weighted query and a distributed lock.

Description

Technical Field

[0001] The invention belongs to the technical field of natural language processing. Background Art

[0002] Text duplication detection aims to detect the similarity between two or more texts and identify repeated or similar parts. Existing text duplication detection methods include keyword matching and full-text indexing. The keyword matching method divides the text into continuous phrases or character sequences by counting the word frequency in the text or using the n-gram model, and then compares the overlap degree of the fragments to calculate the similarity; full-text indexing methods such as Elasticsearch are a distributed, real-time search and analysis engine built on Lucene, which can quickly store, search and analyze large amounts of data, and supports multiple data types and complex query syntax. By establishing an inverted index, each word is mapped to a list of documents containing the word, thereby achieving fast search and comparison. Therefore, Elasticsearch is widely used in text duplication detection in various scenarios.

[0003] However, most methods for calculating the repetition rate of duplicate checking results only consider complete duplication or duplicate content, which makes it difficult to accurately identify subtle differences and similarities between texts. Moreover, in the case of large amounts of text content or high concurrency scenarios, the efficiency of existing text duplication checking methods is also low. Summary of the invention

[0004] In order to solve the technical problems of low accuracy and efficiency in text duplication checking in the prior art, the present invention proposes a "semantic level matching text duplication checking method for high concurrency scenarios".

[0005] Semantic matching text duplication checking method for high concurrency scenarios, such as Figure 1 As shown, the following steps are included:

[0006] Step 1: Candidate text storage: split the candidate text into sentences and store them in the database, and record the sentences and the corresponding original candidate texts;

[0007] Step 2: The query text is divided into query statements, and the query statements are combined into batch query data;

[0008] Step 3: Implementing multi-field weighted query for batch query data in parallel in the database, wherein the multi-field weighted query includes word segmentation processing and word segmentation result matching processing, wherein the word segmentation processing can divide the batch query data into individual words, and the word segmentation result matching processing can achieve semantic level matching, matching a single identical word and its synonyms, thereby obtaining a matching statement for the query statement;

[0009] Step 4: Calculate the number of repeated characters in the query statement and the matching statement in step 3: First, filter the matching statements for stop words; merge the matching statements without stop words, and count the number of characters in the merged text as the intermediate result. If the character length exceeds the set threshold, the number of characters in the merged text is used as the valid count; Calculate the number of repeated characters:

[0010] 1. Exact match: If the query statement is the same as the matching statement, and the character length of the matching statement exceeds the set threshold, the number of characters contained in the matching statement is regarded as the number of repeated characters;

[0011] 2. Partial matching: If the matching statement contains a query statement and the character length of the query statement exceeds the set threshold, the number of characters contained in the query statement is counted as the number of repeated characters;

[0012] 3. Fragment matching: For situations other than full matching and partial matching, the query statement is compared with the matching statement of the merged identifier to traverse the character content between all identifiers. If the character length exceeds the set threshold, the character length is accumulated and the accumulated result is used as the number of repeated characters. Among them, if the character content contains synonym matching results, the character length is calculated according to the original vocabulary length corresponding to the synonym;

[0013] Step 5: Save the duplicate checking results and use the distributed lock in the Redisson distributed toolkit to implement distributed duplication checking result saving.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] By combining query texts into batch query data, combined with parallel processing of multi-field weighted queries and distributed locks, the efficiency of duplicate checking in high-concurrency scenarios is improved; by post-processing the matching statements, stop words are removed and valid matching statements are merged. On this basis, the number of repeated characters is calculated through a variety of methods such as full matching, partial matching, and fragment matching and accumulation, to achieve semantic-level matching, improve the accuracy of repeated character statistics, and accurately identify subtle differences and similarities between texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in the field without creative work by adopting the embodiments of the present invention are within the scope of protection of the present invention.

[0018] In step 1, the candidate text is split into candidate statements through regular expressions.

[0019] In step 2, the query text is divided into query statements through regular expressions, and the constructed regular expressions include Chinese punctuation marks.

[0020] The multi-field weighted query in step 3 is implemented through the MultiMatchQuery query type in Elasticsearch. MultiMatchQuery allows queries of the same text to be executed on multiple different fields. Optionally, the minimum match degree is set to 75%. At the same time, parallel processing is implemented through the CompletableFuture class in Java, which can simplify the asynchronous programming model and make it easier to write non-blocking code.

[0021] Furthermore, the synonym matching is implemented by querying the organized synonym library in the target field. The synonym library includes synonyms and abbreviations of the vocabulary in the target field. Taking the policy field as an example, for example, "housing provident fund" and "provident fund", "social security" and "social insurance", "Ministry of Human Resources and Social Security" and "MOHRSS", etc. At the same time, for the case of synonym matching, the corresponding original words are recorded.

[0022] In step 4, first, stop word filtering is performed on the matching statements. The stop words are used to perform regular matching on the matching statements obtained in step 3 to obtain the matching statements after removing stop words. Among them, stop words are characters that appear frequently in the language but have low information content, such as "de", "shi", "you", "dan", "ge", "zhe", "wo", "ni", etc. After using stop words for regular matching, the stop word characters in the statement are marked as set identifiers.

[0023] The matching statements after removing stop words are merged. Since there are cases where two or more stop words are adjacent in the matching statements, there are two or more identifiers adjacent in the matching statements after removing stop words, so the redundant identifiers are removed. First, the identifier symbols are matched through regular expressions to obtain the positions of all matching identifiers in the matching statements after removing stop words, and check whether there is character content in the adjacent positions. If there is character content, keep the character content therein to obtain the matching statements with merged identifiers.

[0024] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

[0025] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments, and other arrangements may be designed, and that a variety of simple variations may be made to the technical solution of the present invention, provided that they do not deviate from the scope of the present invention as defined by the appended claims, and that these simple variations all fall within the scope of protection of the present invention.

Claims

1. A semantic matching text duplication checking method for high-concurrency scenarios, characterized in that: The following steps are involved: Step 1: Candidate text storage: split the candidate text into sentences and store them in the database, and record the sentences and the corresponding original candidate texts; Step 2: The query text is divided into query statements, and the query statements are combined into batch query data; Step 3: Implementing multi-field weighted query for batch query data in parallel in the database, wherein the multi-field weighted query includes word segmentation processing and word segmentation result matching processing, wherein the word segmentation processing can divide the batch query data into individual words, and the word segmentation result matching processing can achieve semantic level matching, matching a single identical word and its synonyms, thereby obtaining a matching statement for the query statement; Step 4: Calculate the number of repeated characters in the query statement and the matching statement in step 3: First, filter the matching statement for stop words; Merge the matching statements without stop words, and count the number of characters in the merged text as the intermediate result. If the character length exceeds the set threshold, the number of characters in the merged text is used as the valid count; calculate the number of repeated characters:

1. Exact match: If the query statement is the same as the matching statement, and the character length of the matching statement exceeds the set threshold, the number of characters contained in the matching statement is regarded as the number of repeated characters; 2. Partial matching: If the matching statement contains a query statement and the character length of the query statement exceeds the set threshold, the number of characters contained in the query statement is regarded as the number of repeated characters; 3. Fragment matching: For situations other than full matching and partial matching, the query statement is compared with the matching statement of the merged identifier to traverse the character content between all identifiers. If the character length exceeds the set threshold, the character length is accumulated and the accumulated result is used as the number of repeated characters. Among them, if the character content contains synonym matching results, the character length is calculated according to the original vocabulary length corresponding to the synonym; Step 5. Save the duplicate checking results: Use the distributed lock in the Redisson distributed toolkit to save the distributed duplicate checking results.

2. The semantic level matching text duplication checking method for high concurrency scenarios according to claim 1 is characterized in that: In the step 1, the candidate text is split into candidate sentences by using regular expressions.

3. The semantic level matching text duplication checking method for high concurrency scenarios according to claim 1 is characterized in that: In step 2, the query text is divided into query statements by using a regular expression, and the constructed regular expression contains Chinese punctuation marks.

4. The semantic level matching text duplication checking method for high concurrency scenarios according to claim 1 is characterized in that: In step 3, multi-field weighted query is implemented through the MultiMatchQuery query type in Elasticsearch, and the minimum matching degree is set to 75%; and parallel processing is implemented through the CompletableFuture class in Java.

5. The semantic level matching text duplication checking method for high concurrency scenarios according to claim 1 is characterized in that: In step 4, the matching statements that remove stop words are merged. First, the regular matching identifier symbol is used to obtain the positions of all matching identifiers in the matching statement that removes stop words, and it is checked whether there is character content in adjacent positions. If there is character content, the character content is retained to obtain the matching statement of the merged identifier.

Images