Mass text duplicate checking method, system and equipment in real scene and storage medium

Abstract

The invention discloses a mass text duplicate checking method, system and device in a real scene and a storage medium, belongs to the technical field of text duplicate checking, and solves the technical problems that an existing text duplicate checking algorithm is low in duplicate checking speed and lacks semantic duplicate checking when facing mass text data in the real scene. The method comprises the following steps: establishing a document library comprising a vector library and a character library according to a real scene; selecting a vectorization model and performing fine tuning training optimization to obtain a vectorization processing model; all documents used for checking duplicate checking are obtained, and a duplicate checking database is generated; obtaining a document needing to be subjected to duplicate checking and generating data to be subjected to duplicate checking; and sequentially performing vector similarity comparison, same word number screening and continuous same word number screening on the data to be subjected to duplicate checking and the duplicate checking database to obtain final text content hit by duplicate checking. The method can give consideration to multiple checking of semantic level and character level, and effectively improves the calculation efficiency of checking in massive texts.

Description

Technical Field

[0001] The present invention relates to the technical field of text duplication checking, and more specifically, to a method, system, device and storage medium for checking for duplicate content in massive texts in real scenarios. Background Art

[0002] In the current academic and office fields, there has not yet been an efficient duplicate checking method that can simultaneously meet semantic matching and character-level matching to completely curb plagiarism and plagiarism in the process of academic creation and office document writing. This undoubtedly brings huge challenges to the duplicate checking and plagiarism verification of academic and office documents. Plagiarists often steal the original author's academic and office achievements by modifying text and replacing sentences.

[0003] Since it is difficult to conduct convenient comparison and verification of the massive amount of existing documents, this may not only mislead readers and reduce the credibility of information, but may also lead to the theft of others' achievements and academic fraud. These problems may have extremely serious and adverse negative impacts on individuals, companies, academia and even the entire society.

[0004] However, existing text duplication checking algorithms have problems such as slow duplication checking speed and lack of semantic duplication checking when faced with massive text data in real scenarios. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art. The first purpose of the present invention is to provide a method for checking for duplicate content in massive texts in a real scenario, which can effectively solve the challenges in the current practical application of text checking for duplicate content. While integrating a variety of checking for duplicate content related algorithms to work together, it takes into account multiple checks at the semantic level and the character level, and effectively improves the computational efficiency of checking for duplicate content in massive texts.

[0006] The second objective of the present invention is to provide a massive text duplication checking system in a real scenario.

[0007] A third object of the present invention is to provide a computer device.

[0008] A fourth objective of the present invention is to provide a computer storage medium.

[0009] In order to achieve the above-mentioned first objective, the present invention provides a method for checking for duplicate content in a massive amount of text in a real scenario, comprising the following steps:

[0010] Step 1: Establish a document library including a vector library and a text library according to a real scenario, wherein the vector library is used to store vector data after the text is vectorized, and the text library is used to store text data corresponding to the vector data, and establish an association relationship between the vector library and the text library according to an index;

[0011] Step 2: Select a vectorization model, and fine-tune the vectorization model according to the document library to obtain a vectorization processing model;

[0012] Step 3: Obtain all documents for checking duplicate content, pre-process them using the vectorization processing model, and then import them into the document library to obtain a duplicate content database;

[0013] Step 4: Obtain the document to be checked for duplicates, and use the vectorization processing model to perform preprocessing to obtain the data to be checked for duplicates;

[0014] Step 5: perform vector similarity comparison between the data to be checked for duplicate content and the data stored in the duplicate checking database to obtain high-similarity data in the duplicate checking database;

[0015] Step 6: Screen the data to be checked for duplicates and the data with high similarity for the same number of characters to obtain text data with the same number of characters threshold;

[0016] Step 7: Filter the data to be checked for duplicates and the text data with the same word count threshold for the same number of words to obtain the final text content that is checked for duplicates.

[0017] As a further improvement, the vector library is the open source vector library Milvus, and the text library is a MySQL database.

[0018] Furthermore, the vectorized model is an open source model BCEmbedding model;

[0019] The fine-tuning training specifically includes selecting no less than 5,000 sentences from the document library, manually annotating at least one similar sentence for each sentence, and using these sentences to fine-tune the open source model BCEmbedding model to obtain a vectorized processing model that is more suitable for the current usage scenario.

[0020] Furthermore, in step 3, all the documents are split into several sentences according to the document splitting logic, and the split sentences are vectorized using the vectorization processing model, and the vectors and corresponding sentences obtained after the vectorization processing are respectively stored in the vector library and the text library in the document library to obtain a duplicate checking database;

[0021] The document splitting logic is as follows:

[0022] According to the input length limit of the BCEmbedding model, 500 words are selected as the upper limit;

[0023] The document is segmented into sentences using the coarse granularity of period, exclamation mark, question mark and double quote;

[0024] If the word count is still greater than 500 after the coarse-grained splitting, the document is split into sentences using the fine-grained splitting of period, exclamation mark, question mark, comma, colon, semicolon, comma, double quote, and single quote.

[0025] If the word count is still around 500 after fine-grained splitting, split the document by directly cutting it off at 500 characters.

[0026] Further, in step 5, the sentences of the duplicate checking data are vectorized using the vectorization processing model to obtain a vector array, and the Cosine distance calculation method is used to calculate the Cosine distances between the vectors in the vector array and all the vectors in the vector library in the duplicate checking database one by one, and the vectors in the vector library whose Cosine distances are greater than or equal to the vector similarity threshold are screened out, and the text data corresponding to the vector is obtained as the high-similarity data;

[0027] The vector similarity threshold is:

[0028] The Cosine distance normalized to between 0 and 1 is used as the screening threshold value, and 0.75 is specifically selected as the threshold using the scene experience method.

[0029] Furthermore, in step 6, the sentences of the data to be checked for duplicates are compared with the text data one by one for the number of identical words, and text data with a number of words greater than or equal to a threshold value are further selected from the text data as text data with a number of words greater than or equal to a threshold value;

[0030] The thresholds for the number of identical words are:

[0031] The ratio of the number of identical words normalized between 0 and 1 was used as the screening threshold value, and 0.80 was specifically selected as the threshold using the scenario experience method.

[0032] Furthermore, in step 7, the text data of the sentence of the data to be checked for duplicates with the same word count threshold are compared one by one for the number of consecutive identical words, and finally the text data with the number of consecutive identical words greater than or equal to the threshold is selected as the text content hit by the final duplicate check;

[0033] The threshold for consecutive identical words is:

[0034] The specific number of consecutive identical characters is used as the screening threshold value, and the scenario experience method is used to specifically select 12 characters as the threshold.

[0035] In order to achieve the above-mentioned second objective, the present invention provides a massive text duplication checking system in a real scenario, comprising:

[0036] A document library establishment module is used to establish a document library including a vector library and a text library according to a real scenario, wherein the vector library is used to store vector data after the text is vectorized, and the text library is used to store text data corresponding to the vector data, and an association relationship between the vector library and the text library is established according to an index;

[0037] A vectorization model training module, used for selecting a vectorization model, and fine-tuning, training and optimizing the vectorization model according to the document library to obtain a vectorization processing model;

[0038] A first acquisition module is used to acquire all documents used for checking duplicates, and after pre-processing using the vectorization processing model, import them into the document library to obtain a duplicate checking database;

[0039] The second acquisition module is used to obtain the document to be checked for duplicate content, and use the vectorization processing model to perform preprocessing to obtain the data to be checked for duplicate content;

[0040] A vector similarity comparison module is used to compare the vector similarity of the data to be checked for duplicate content with the data stored in the duplicate checking database to obtain high-similarity data in the duplicate checking database;

[0041] The same word count comparison module is used to screen the same word counts of the to-be-checked duplicate data and the high-similarity data to obtain text data with the same word count threshold;

[0042] The module for comparing the number of consecutive identical characters is used to screen the data to be checked for duplicates and the text data with the same number of characters threshold for the number of consecutive identical characters to obtain the text content that is finally checked for duplicates.

[0043] In order to achieve the third objective above, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned method for checking for duplicate content in a massive amount of text in a real scenario.

[0044] In order to achieve the fourth objective mentioned above, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for checking for duplicate content of massive texts in a real scenario mentioned above is implemented.

[0045] Beneficial Effects

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] The present invention can effectively solve the challenges in the current practical application of text duplication checking. While integrating multiple duplication checking related algorithms to work together, it takes into account multiple checks at the semantic level and the character level, and effectively improves the computational efficiency of duplication checking in massive texts. It is very suitable for duplication checking of massive text data in real scenarios, and has a very broad market application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the specific embodiments in the accompanying drawings.

[0050] See also Figure 1 , a method for checking duplicate content of a large amount of text in a real scenario, including the following steps 1 to 7:

[0051] Step 1: Establish a document library including a vector library and a text library according to the real scenario. The vector library is used to store vector data after the text is vectorized, and the text library is used to store the text data corresponding to the vector data. Establish an association relationship between the vector library and the text library according to the index.

[0052] In this embodiment, the vector library is the open source vector library Milvus, and the text library is a MySQL database.

[0053] Step 2: Select a vectorization model and fine-tune the vectorization model based on the document library to obtain a vectorization processing model.

[0054] The vectorized model is the open source model BCEmbedding model.

[0055] The specific fine-tuning training is to select no less than 5,000 sentences from the document library, and manually annotate at least one similar sentence for each sentence. These sentences are used to fine-tune the open source model BCEmbedding model to obtain a vectorized processing model that is more suitable for the current usage scenario.

[0056] For example, select 5,000 sentences from the document library, and manually annotate 5,000 similar sentences for each sentence, and use these 5,000 pairs of sentences to fine-tune the open source model BCEmbedding model.

[0057] Step 3: Obtain all documents used for duplicate checking, pre-process them using a vectorized processing model, and then import them into the document library to obtain a duplicate checking database.

[0058] Specifically, in step 3, all documents are split into several sentences according to the document splitting logic, and the split sentences are vectorized using a vectorization processing model. The vectors and corresponding sentences obtained after the vectorization processing are respectively stored in the vector library and the text library in the document library to obtain a duplicate checking database.

[0059] The document splitting logic is as follows:

[0060] According to the input length limit of the BCEmbedding model, 500 words are selected as the upper limit;

[0061] The document is segmented into sentences using the coarse granularity of period, exclamation mark, question and double quote (i.e., !? "");

[0062] If the word count is still greater than 500 after the coarse-grained splitting, the document is split into sentences using the fine-grained splitting of period, exclamation mark, question, comma, colon, semicolon, comma, double quotes, and single quotes (i.e., !?, :;, ""'');

[0063] If the word count is still around 500 after fine-grained splitting, split the document by directly cutting it off at 500 characters.

[0064] Step 4: Obtain the document that needs to be checked for duplicate content, and use the vectorized processing model to pre-process the document to obtain the data to be checked for duplicate content.

[0065] That is, the document to be checked for duplicate content is split into several sentences according to the document splitting logic in step 3.

[0066] Step 5: Perform a vector similarity comparison between the data to be checked for duplicate content and the data stored in the duplicate checking database to obtain high-similarity data in the duplicate checking database.

[0067] In step 5, the sentences of the data to be checked for duplicates are vectorized using a vectorization processing model to obtain a vector array, and the Cosine distance calculation method is used to calculate the Cosine distance between the vectors in the vector array and all the vectors in the vector library in the duplicate checking database one by one, and the vectors in the vector library whose Cosine distance is greater than or equal to the vector similarity threshold are screened out, and the text data corresponding to the vector is obtained as high-similarity data.

[0068] The vector similarity threshold is:

[0069] The Cosine distance normalized to between 0 and 1 is used as the screening threshold value, and 0.75 is selected as the threshold using the scene experience method. Of course, the threshold selection can be flexibly adjusted according to the scene manual experience.

[0070] Step 6: Filter the data to be checked for duplicates and the data with high similarity by the same number of characters to obtain text data with the same threshold number of characters.

[0071] In step 6, the sentences of the data to be checked for duplicates are compared with the text data one by one for the number of identical characters, and text data with a number of characters greater than or equal to a threshold value for the number of identical characters is further screened out from the text data as text data with a threshold value for the number of identical characters.

[0072] The thresholds for the number of identical words are:

[0073] The ratio of the number of identical words normalized between 0 and 1 is used as the screening threshold value, and 0.80 is selected as the threshold using the scenario experience method. Of course, the threshold selection can be flexibly adjusted according to the scenario manual experience.

[0074] Step 7: Filter the text data to be checked for duplicates and the text data with the same word count threshold for the same number of words to obtain the final text content that is checked for duplicates.

[0075] The text data with the same word count threshold of the sentences in the data to be checked for duplicates are compared one by one for the number of consecutive identical words, and finally the text data with the number of consecutive identical words greater than or equal to the threshold is selected as the text content hit in the final duplicate check.

[0076] The threshold for consecutive identical words is:

[0077] The specific number of consecutive identical characters is used as the screening threshold value, and the scene experience method is used to specifically select 12 characters as the threshold. Of course, the threshold selection can be flexibly adjusted according to the scene manual experience.

[0078] A massive text duplication checking system in a real scenario, comprising:

[0079] The document library establishment module is used to establish a document library including a vector library and a text library according to real scenarios. The vector library is used to store vector data after the text is vectorized, and the text library is used to store the text data corresponding to the vector data. The association relationship between the vector library and the text library is established according to the index;

[0080] The vectorization model training module is used to select a vectorization model and fine-tune the vectorization model according to the document library to obtain a vectorization processing model;

[0081] The first acquisition module is used to acquire all documents used for checking duplicates, and after pre-processing using a vectorized processing model, import them into a document library to obtain a duplicate checking database;

[0082] The second acquisition module is used to obtain the documents to be checked for duplicate content and use the vectorized processing model to perform preprocessing to obtain the data to be checked for duplicate content;

[0083] The vector similarity comparison module is used to compare the vector similarity of the data to be checked for duplicate content with the data stored in the duplicate checking database to obtain the high-similarity data in the duplicate checking database;

[0084] The same word count comparison module is used to screen the same word count between the data to be checked for duplicates and the data with high similarity, and obtain text data with the same word count threshold;

[0085] The module for comparing the number of consecutive identical characters is used to screen the text data to be checked for duplicates and the text data with the same number of characters as the threshold, and obtain the final text content that is checked for duplicates.

[0086] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the above-mentioned method for checking for duplicate content in a massive amount of text in a real scenario.

[0087] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for checking for duplicate content in a massive amount of text in a real scenario.

[0088] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for checking for duplicate content in a large amount of text in a real scenario, characterized in that: The following steps are involved: Step 1: Establish a document library including a vector library and a text library according to a real scenario, wherein the vector library is used to store vector data after the text is vectorized, and the text library is used to store text data corresponding to the vector data, and establish an association relationship between the vector library and the text library according to an index; Step 2: Select a vectorization model, and fine-tune the vectorization model according to the document library to obtain a vectorization processing model; Step 3: Obtain all documents for checking duplicate content, pre-process them using the vectorization processing model, and then import them into the document library to obtain a duplicate content database; Step 4: Obtain the document to be checked for duplicates, and use the vectorization processing model to perform preprocessing to obtain the data to be checked for duplicates; Step 5: perform vector similarity comparison between the data to be checked for duplicate content and the data stored in the duplicate checking database to obtain high-similarity data in the duplicate checking database; Step 6: Screen the data to be checked for duplicates and the data with high similarity for the same number of characters to obtain text data with the same number of characters threshold; Step 7: Filter the data to be checked for duplicates and the text data with the same word count threshold for the same number of words to obtain the final text content that is checked for duplicates.

2. The method for checking duplicate content of a large amount of text in a real scene according to claim 1 is characterized in that: The vector library is the open source vector library Milvus, and the text library is a MySQL database.

3. The method for checking duplicate content of a large amount of text in a real scene according to claim 1 is characterized in that: The vectorized model is an open source model BCEmbedding model; The fine-tuning training specifically includes selecting no less than 5,000 sentences from the document library, manually annotating at least one similar sentence for each sentence, and using these sentences to fine-tune the open source model BCEmbedding model to obtain a vectorized processing model that is more suitable for the current usage scenario.

4. The method for checking duplicate content of a large amount of text in a real scene according to claim 1, characterized in that: In step 3, for all the documents, each document is split into a plurality of sentences according to the document splitting logic, and the split sentences are vectorized using the vectorization processing model, and the vectors and corresponding sentences obtained after the vectorization processing are respectively stored in the vector library and the text library in the document library to obtain a duplicate checking database; The document splitting logic is as follows: According to the input length limit of the BCEmbedding model, 500 words are selected as the upper limit; The document is segmented into sentences using the coarse granularity of period, exclamation mark, question mark and double quote; If the word count is still greater than 500 after the coarse-grained splitting, the document is split into sentences using the fine-grained splitting of period, exclamation mark, question mark, comma, colon, semicolon, comma, double quote, and single quote. If the word count is still around 500 after fine-grained splitting, split the document by directly cutting it off at 500 characters.

5. The method for checking duplicate content of a large amount of text in a real scene according to claim 1, characterized in that: In step 5, the sentences of the duplicate checking data are vectorized using the vectorization processing model to obtain a vector array, and the Cosine distance calculation method is used to calculate the Cosine distances between the vectors in the vector array and all the vectors in the vector library in the duplicate checking database one by one, and the vectors in the vector library whose Cosine distances are greater than or equal to the vector similarity threshold are screened out, and the text data corresponding to the vector is obtained as the high-similarity data; The vector similarity threshold is: The Cosine distance normalized to between 0 and 1 is used as the screening threshold value, and 0.75 is specifically selected as the threshold using the scene experience method.

6. The method for checking duplicate content of a large amount of text in a real scene according to claim 5, characterized in that: In step 6, the sentences of the data to be checked for duplicates are compared with the text data one by one for the same number of words, and text data with a number of words greater than or equal to a threshold value are further selected from the text data as text data with a same number of words threshold value; The thresholds for the number of identical words are: The ratio of the number of identical words normalized between 0 and 1 was used as the screening threshold value, and 0.80 was specifically selected as the threshold using the scenario experience method.

7. The method for checking duplicate content of a large amount of text in a real scene according to claim 6, characterized in that: In step 7, the text data of the sentence with the same word count threshold of the data to be checked for duplicates are compared one by one for the number of consecutive identical words, and finally the text data with the number of consecutive identical words greater than or equal to the threshold is selected as the text content hit by the final duplicate check; The threshold for consecutive identical words is: The specific number of consecutive identical characters is used as the screening threshold value, and the scenario experience method is used to specifically select 12 characters as the threshold.

8. A massive text duplication checking system in a real scenario, characterized in that: include: A document library establishment module is used to establish a document library including a vector library and a text library according to a real scenario, wherein the vector library is used to store vector data after the text is vectorized, and the text library is used to store text data corresponding to the vector data, and an association relationship between the vector library and the text library is established according to an index; A vectorization model training module, used for selecting a vectorization model, and fine-tuning, training and optimizing the vectorization model according to the document library to obtain a vectorization processing model; A first acquisition module is used to acquire all documents used for checking duplicates, and after pre-processing using the vectorization processing model, import them into the document library to obtain a duplicate checking database; The second acquisition module is used to obtain the document to be checked for duplicate content, and use the vectorization processing model to perform preprocessing to obtain the data to be checked for duplicate content; A vector similarity comparison module is used to compare the vector similarity of the data to be checked for duplicate content with the data stored in the duplicate checking database to obtain high-similarity data in the duplicate checking database; The same word count comparison module is used to screen the same word counts of the to-be-checked duplicate data and the high-similarity data to obtain text data with the same word count threshold; The module for comparing the number of consecutive identical characters is used to screen the data to be checked for duplicates and the text data with the same number of characters threshold for the number of consecutive identical characters to obtain the text content that is finally checked for duplicates.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements a method for checking for duplicate content in a massive amount of text in a real scenario as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a method for checking for duplicate content in a massive amount of text in a real scenario as described in any one of claims 1 to 7.

Images