Cross-database text duplicate checking method, device and equipment and readable storage medium

Abstract

The invention provides a cross-database text duplicate checking method, device and equipment and a readable storage medium, and relates to the technical field of data security, and the method comprises the following steps: segmenting a document sample into paragraph samples and sentence samples, and respectively storing the paragraph samples and the sentence samples into corresponding paragraph databases and sentence databases; calculating a sentence sample vector of each sentence sample and a paragraph sample vector of each paragraph sample, and respectively storing the sentence sample vectors and the paragraph sample vectors into a corresponding sentence database and a paragraph database; sequentially comparing the paragraphs to be subjected to duplicate checking with paragraph sample vectors in each paragraph library to obtain similar paragraphs; segmenting the paragraph to be subjected to duplicate checking into sentences to be subjected to duplicate checking; and comparing the sentence sample vectors contained in the similar paragraphs with the to-be-duplicated sentences to obtain similar sentences, and obtaining a duplicate checking result of the to-be-duplicated text according to the similar sentences in all the sentence databases. According to the method, data security is protected, and meanwhile, the requirement of cross-database duplicate checking is met.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to a method, device, equipment and readable storage medium for checking for duplicate text across databases. Background Art

[0002] In the field of academic research and education, text duplication detection technology is a key tool to ensure academic integrity and originality. With the development of digitization and networking, the application of text duplication detection technology is becoming more and more extensive, and its technical field covers many aspects, including but not limited to duplication detection algorithms or methods, duplication detection systems or platforms, duplication detection technology applications, duplication detection data sets or corpora, duplication detection result evaluation methods, and duplication detection related equipment or tools. Although duplication detection technology has made progress in many aspects, existing duplication detection systems are usually limited to a single database or data source and lack the ability to perform text duplication detection between different databases, which poses a challenge in terms of data security and privacy protection. At present, there is no literature or patent involving the technology of realizing cross-database text duplication detection under the premise of ensuring data security. Summary of the invention

[0003] The purpose of the present invention is to provide a cross-database text duplication checking method, device, equipment and readable storage medium, which can realize the need of cross-database duplication checking while protecting data security, so as to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0004] In a first aspect, the present application provides a cross-database text duplication checking method, comprising:

[0005] Acquire multiple text databases, and establish corresponding paragraph libraries and sentence databases for each text database;

[0006] Obtain document samples contained in the text database, and after dividing the document samples into paragraph samples and sentence samples, store them in corresponding paragraph libraries and sentence databases respectively;

[0007] Calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector in the corresponding sentence database and paragraph database respectively;

[0008] The text to be checked for duplicates is divided into paragraphs to be checked for duplicates, and the paragraphs to be checked for duplicates are sequentially compared with the paragraph sample vectors in each paragraph library to obtain similar paragraphs;

[0009] The paragraph to be checked for duplicates is divided into sentences to be checked for duplicates, and sentence sample vectors contained in similar paragraphs are extracted from a sentence database; the sentence sample vectors contained in the similar paragraphs are compared with the sentences to be checked for duplicates to obtain similar sentences;

[0010] The duplicate checking result of the text to be checked is obtained based on similar sentences in all sentence databases.

[0011] In a second aspect, the present application also provides a cross-database text duplication checking device, comprising:

[0012] Text database acquisition module: used to acquire multiple text databases and establish corresponding paragraph databases and sentence databases for each text database;

[0013] Document sample processing module: used to obtain document samples contained in the text database, and divide the document samples into paragraph samples and sentence samples, and store them in the corresponding paragraph library and sentence database respectively;

[0014] Sample vector calculation module: used to calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector into the corresponding sentence database and paragraph database respectively;

[0015] Paragraph comparison module: used to divide the text to be checked for duplicates into paragraphs to be checked for duplicates, and compare the paragraphs to be checked for duplicates with the paragraph sample vectors in each paragraph library in turn to obtain similar paragraphs;

[0016] Sentence comparison module: used for dividing the paragraph to be checked for duplicate content into sentences to be checked for duplicate content, extracting sentence sample vectors contained in similar paragraphs from the sentence database; comparing the sentence sample vectors contained in the similar paragraphs with the sentences to be checked for duplicate content to obtain similar sentences;

[0017] Duplicate check result summary module: used to obtain duplicate check results of the text to be checked for duplicate based on similar sentences in all sentence databases.

[0018] In a third aspect, the present application also provides a cross-database text duplication checking device, comprising:

[0019] Memory for storing computer programs;

[0020] A processor is used to implement the steps of the cross-database text duplication checking method when executing the computer program.

[0021] In a fourth aspect, the present application further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned cross-database based text duplication checking method are implemented.

[0022] The beneficial effects of the present invention are:

[0023] The cross-database text duplicate checking method provided by the present invention divides the text to be checked for duplicate into a number of paragraphs to be checked for duplicate and sentences to be checked for duplicate, and sends the paragraphs to be checked for duplicate and the sentences to be checked for duplicate to different text databases for comparison, thereby effectively ensuring data security. In the duplicate checking process, similar paragraphs are first determined by paragraph comparison, and then sentence samples in the similar paragraphs are compared with the sentences to be checked for duplicate, further determining sentences with high similarity, and obtaining duplicate checking results. While ensuring data security, the efficiency and accuracy of the duplicate checking work are improved, and the burden of manual audit is reduced.

[0024] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 It is a schematic diagram of the flow of the cross-database text duplication checking method described in an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of the structure of the cross-database text duplication checking device described in an embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of the structure of the cross-database text duplication checking device described in an embodiment of the present invention.

[0029] Markings in the figure:

[0030] 800. Cross-database text duplication checking device; 801. Processor; 802. Memory; 803. Multimedia component; 804. I / O interface; 805. Communication component. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Embodiment 1:

[0034] This embodiment provides a cross-database text duplication checking method.

[0035] See also Figure 1 , the figure shows that the method includes:

[0036] S1. Obtain multiple text databases [A1, A2, ..., A i ,…,A n ], establish corresponding paragraph library and sentence database for each text database;

[0037] In this embodiment, the multiple text databases are isolated databases, each of which contains a number of document samples, and the same document samples may exist between different text databases;

[0038] Specifically, the step S1 includes:

[0039] S11. Get the text database ID;

[0040] S12. Establishing a paragraph library and a sentence database for each text database, respectively, wherein the paragraph library and sentence database have the same label as the text database;

[0041] In this embodiment, the text database A i For example, its paragraph library is P i , the sentence database is S i ;

[0042] Based on the above embodiments, the method further includes:

[0043] S2. Obtain document samples contained in the text database, and divide the document samples into paragraph samples and sentence samples, and store them in the corresponding paragraph library and sentence database respectively;

[0044] Specifically, step S2 includes:

[0045] S21. Read the document sample, the document sample ID and the text database ID to which it belongs, and clean the document sample to remove headers, footers, comments, blank characters, and line breaks;

[0046] S22. Segment the cleaned document sample based on blank lines and / or paragraph marks to obtain several paragraph samples;

[0047] S23. Get the position and order of the paragraph sample in the document sample, generate the paragraph ID and paragraph number [P i1 ,P i2 ,…,P in ], where P in Represents the text database A i The nth paragraph sample of the data is obtained to facilitate the rapid location and citation of the paragraph in subsequent processing;

[0048] S24. The paragraph sample attributes are formed by the paragraph number and the document sample ID to which it belongs, and the paragraph ID, paragraph attributes and paragraph content of the paragraph sample are stored in the paragraph library corresponding to the text database ID.

[0049] Specifically, the step S2 further includes:

[0050] S21. Get punctuation marks in the paragraph text;

[0051] S22. Write a regular expression to match punctuation marks and divide the paragraph sample into several sentence samples;

[0052] S23. Get the position and order of the sentence sample in the paragraph sample, generate the sentence ID and sentence number [S ij1 ,S ij2 ,…,S ijm ], where S ijm Represents the text database A i The mth sentence sample of the jth paragraph sample;

[0053] S24. The sentence sample attributes are formed by the sentence number and the paragraph sample attributes to which it belongs, and the sentence ID, sentence sample attributes and sentence content are stored in the sentence database corresponding to the text database ID.

[0054] Based on the above embodiments, the method further includes:

[0055] S3. Calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector in the corresponding sentence database and paragraph database respectively;

[0056] Specifically, step S3 includes:

[0057] S31. Input the sentence sample into the word segmenter, perform word segmentation on the sentence sample to obtain a number of words, and convert the words into a word sequence, specifically, arranging the words in the original order in the sentence to form a subsequence;

[0058] S32. Input the word sequence into the BERT model to obtain the word vector, and calculate the average value of several word vectors to obtain the sentence sample vector of the sentence sample:

[0059]

[0060] Among them, svec ijk Represents the text database A i The kth sentence vector of the i-th paragraph sample, T represents the number of words contained in the sentence sample, vec ijkt Represents the word vector of the tth word in the sentence;

[0061] S33. According to the connection relationship and semantic relationship between sentence samples, a sentence sample relationship graph is constructed, and the sentence sample weight is calculated using the sentence sample relationship graph;

[0062] Specifically, the step S33 includes:

[0063] S331. Calculate the similarity between any two sentence sample vectors in the paragraph sample:

[0064]

[0065] Among them, w gk Represents the cosine similarity between the kth sentence vector and the gth sentence vector.

[0066] S332. Construct a similarity matrix, and use the similarity between the sentence sample vectors as the elements of the similarity matrix;

[0067] Take sentence sample S ij1 ,S ij2 , S ij3 For example, the corresponding sentence sample vector is svec ij1 、svec ij2 、svec ij3 , construct a similarity matrix based on the sentence samples:

[0068]

[0069] Among them, the elements on the diagonal are all 1, indicating that the similarity between each sentence sample and itself is 1.

[0070] S333. Establish a sentence sample relationship graph with sentence samples as nodes, connection relationships between sentence samples as edges, and elements in the similarity matrix as edge weights;

[0071] S334. Construct a Rank function model and define the initial Rank value of each sentence sample;

[0072] S33. Input the sentence sample relationship graph into the Rank function model to iteratively calculate the Rank value of each sentence sample;

[0073]

[0074] In the formula, Rank(S ijk ) represents the Rank value of the k-th sentence sample vector, Rank(S′ ijk ) represents the updated Rank value, P ij Represents the text database A i The jth paragraph sample of , d represents the damping coefficient, W gl It represents the cosine similarity between the lth sentence vector and the gth sentence vector, the gth sentence sample is connected to the kth sentence sample, and the lth sentence sample is connected to the gth sentence sample.

[0075] S335. When the number of iterative calculations is greater than the maximum preset number, the calculated Rank value of the sentence sample is used as the sentence sample weight.

[0076] S34. Obtain all sentence samples that constitute the paragraph sample;

[0077] S35. The paragraph vector of the paragraph sample is calculated by the sentence sample weights and sentence sample vectors of all sentence samples:

[0078]

[0079] Where pvec ij represents the vector of the jth paragraph in the i-th text database, K represents the number of sentences contained in the paragraph, svec ijk Represents the kth sentence vector of the paragraph;

[0080] Based on the above embodiments, the method further includes:

[0081] S4. Divide the text to be checked for duplicates into paragraphs to be checked for duplicates, and compare the paragraphs to be checked for duplicates with the paragraph sample vectors in each paragraph library in turn to obtain similar paragraphs;

[0082] Specifically, step S4 includes:

[0083] S41. Split the text to be checked for duplicates into several paragraphs to be checked for duplicates [DP1, DP2, ..., DP u ], calculate the vector dpvec of the paragraph to be checked for duplicates u ;

[0084] S42. Sending several of the paragraph vectors to be checked for duplicates to each paragraph library, and calculating the paragraph similarity between the paragraph vector to be checked for duplicates and each paragraph sample vector in the paragraph library in turn:

[0085]

[0086] Where pcos(θ) represents pvec ij with dpvec u The cosine similarity of

[0087] By using the formula, all the paragraphs to be checked for duplicate content are sent to the paragraph library corresponding to each text database in turn, each paragraph sample in the paragraph library is traversed, and the similarity between each paragraph sample and the current paragraph to be checked for duplicate content is calculated in turn, so that the paragraph similarity between each paragraph to be checked for duplicate content and each paragraph sample can be obtained;

[0088] S43. When the paragraph similarity is greater than a first preset threshold, the paragraph sample corresponding to the paragraph sample vector is taken as a similar paragraph;

[0089] The calculated paragraph similarity is compared with the first preset threshold in sequence. In this embodiment, the first preset threshold is set according to the requirements of the duplicate checking. The higher the duplicate checking rate requirement, the smaller the first preset threshold. When the paragraph similarity is greater than the first preset threshold, the compared paragraph template is regarded as a similar paragraph. It should be noted that a paragraph to be checked for duplicates may have multiple similar paragraphs.

[0090] Based on the above embodiments, the method further includes:

[0091] S5. The paragraph to be checked for duplicates is divided into sentences to be checked for duplicates, and sentence sample vectors of similar paragraphs are extracted from the sentence database; the sentence sample vectors of similar paragraphs are compared with the sentences to be checked for duplicates to obtain similar sentences;

[0092] Specifically, step S5 includes:

[0093] S51. Segment the paragraph to be checked for duplicates into sentences to be checked for duplicates [DS u1 ,DSu2 ,…,DS uv ] and calculate the vector dsvec of the sentence to be checked uv ;

[0094] S52. Get the paragraph ID of the similar paragraph, and search the corresponding sentence sample vector svec in the sentence database according to the paragraph ID ijk ;

[0095] S53. Calculate the sentence similarity between the sentence vector to be checked for duplicates and the found sentence sample vector:

[0096]

[0097] Where scos(θ) represents dsvec uv With svec ijk The cosine similarity of .

[0098] S54. When the sentence similarity is greater than a second preset threshold, the sentence sample corresponding to the sentence sample vector is taken as a similar sentence.

[0099] Based on the above embodiments, the method further includes:

[0100] S6. Obtain the duplicate checking result of the text to be checked based on similar sentences in all sentence databases;

[0101] Specifically, step S6 includes:

[0102] Obtain all similar sentences and their sample attributes found in a sentence database to obtain duplicate checking data of the current sentence database;

[0103] The duplicate check data in all sentence databases are obtained, and the duplicate check results of the text to be checked for duplicates are obtained after deduplication operations are performed on all the duplicate check data. In this embodiment, since there are the same document samples in different databases, it is necessary to dedupe the repeated similar sentences to obtain the final set of similar sentences, that is, the duplicate check results.

[0104] Embodiment 2:

[0105] like Figure 2 As shown, this embodiment provides a cross-database text duplication checking device, the device comprising:

[0106] Text database acquisition module: used to acquire multiple text databases and establish corresponding paragraph databases and sentence databases for each text database;

[0107] Document sample processing module: used to obtain document samples contained in the text database, and divide the document samples into paragraph samples and sentence samples, and store them in the corresponding paragraph library and sentence database respectively;

[0108] Sample vector calculation module: used to calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector into the corresponding sentence database and paragraph database respectively;

[0109] Paragraph comparison module: used to divide the text to be checked for duplicates into paragraphs to be checked for duplicates, and compare the paragraphs to be checked for duplicates with the paragraph sample vectors in each paragraph library in turn to obtain similar paragraphs;

[0110] Sentence comparison module: used for dividing the paragraph to be checked for duplicate content into sentences to be checked for duplicate content, extracting sentence sample vectors contained in similar paragraphs from the sentence database; comparing the sentence sample vectors contained in the similar paragraphs with the sentences to be checked for duplicate content to obtain similar sentences;

[0111] Duplicate check result summary module: used to obtain duplicate check results of the text to be checked for duplicate based on similar sentences in all sentence databases.

[0112] Based on the above embodiment, the document sample processing module includes:

[0113] Document sample reading unit: used to read document samples, document sample IDs and the text database IDs to which they belong, and clean the document samples to remove headers, footers, comments and blank characters;

[0114] Paragraph sample segmentation unit: used to segment the cleaned document sample based on blank lines and / or paragraph marks to obtain several paragraph samples;

[0115] Paragraph sample attribute generation unit: used to obtain the position and order of the paragraph sample in the document sample and generate the paragraph ID and paragraph number;

[0116] Paragraph sample storage unit: used to form paragraph sample attributes from the paragraph number and the document sample ID to which it belongs, and store the paragraph ID, paragraph attributes and paragraph content of the paragraph sample into the paragraph library corresponding to the text database ID.

[0117] Based on the above embodiment, the sample vector calculation module includes:

[0118] A word segmentation processing unit is used to input a sentence sample into a word segmenter, perform word segmentation on the sentence sample to obtain a number of words, and convert the words into a word sequence;

[0119] BERT model processing unit: used to input word sequences into the BERT model to obtain word vectors, and calculate the average value of several word vectors to obtain sentence sample vectors of sentence samples;

[0120] Sentence sample relationship graph construction unit: used to construct a sentence sample relationship graph according to the connection relationship and semantic relationship between sentence samples;

[0121] A sentence sample weight calculation unit: used to calculate the sentence sample weight using the sentence sample relationship graph;

[0122] Paragraph vector calculation unit: used to obtain all sentence samples constituting a paragraph sample, and calculate the paragraph vector of the paragraph sample from the sentence sample weights and sentence sample vectors of all sentence samples.

[0123] Based on the above embodiment, the sentence sample weight calculation unit includes:

[0124] Similarity calculation unit: used to calculate the similarity between any two sentence sample vectors in the paragraph sample;

[0125] Similarity matrix construction unit: used to construct a similarity matrix, taking the similarity between sentence sample vectors as the elements of the similarity matrix;

[0126] Sentence sample relationship graph establishment unit: used to establish a sentence sample relationship graph with sentence samples as nodes, connection relationships between sentence samples as edges, and elements in a similarity matrix as edge weights;

[0127] Rank function model unit: used to build the Rank function model and define the initial Rank value of each sentence sample;

[0128] Rank value iterative calculation unit: used to input the sentence sample relationship graph into the Rank function model to iteratively calculate the Rank value of each sentence sample;

[0129] Weight determination unit: used to use the calculated Rank value of the sentence sample as the sentence sample weight when the number of iterative calculations is greater than the maximum preset number.

[0130] Based on the above embodiment, the paragraph comparison module includes:

[0131] The to-be-checked duplicate paragraph segmentation unit is used to segment the to-be-checked duplicate text into several to-be-checked duplicate paragraphs and calculate the to-be-checked duplicate paragraph vectors;

[0132] A paragraph similarity calculation unit is used to send a plurality of the paragraph vectors to be checked for duplicates to each paragraph library, and sequentially calculate the paragraph similarity between the paragraph vector to be checked for duplicates and each paragraph sample vector in the paragraph library;

[0133] A similar paragraph determining unit is used to take the paragraph sample corresponding to the paragraph sample vector as a similar paragraph when the paragraph similarity is greater than a first preset threshold.

[0134] Based on the above embodiment, the sentence comparison module includes:

[0135] The to-be-checked-for-duplicate-sentences segmentation unit is used to segment the to-be-checked-for-duplicate-sentences paragraph into to-be-checked-for-duplicate-sentences and calculate the to-be-checked-for-duplicate-sentences vectors;

[0136] Sentence sample vector query unit: used to obtain the paragraph ID of similar paragraphs, and search the corresponding sentence sample vector in the sentence database according to the paragraph ID;

[0137] Sentence similarity calculation unit: used to calculate the sentence similarity between the sentence vector to be checked for duplicates and the found sentence sample vector;

[0138] A similar sentence determination unit is used to take the sentence sample corresponding to the sentence sample vector as a similar sentence when the sentence similarity is greater than a second preset threshold.

[0139] It should be noted that, regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0140] Embodiment 3:

[0141] Corresponding to the above method embodiment, a cross-database text duplication checking device is also provided in this embodiment. The cross-database text duplication checking device described below and the cross-database text duplication checking method described above can refer to each other.

[0142] Figure 3 FIG. 8 is a block diagram of a cross-database text duplication checking device 800 according to an exemplary embodiment. Figure 3 As shown, the cross-database text duplicate checking device 800 may include: a processor 801 and a memory 802. The cross-database text duplicate checking device 800 may also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0143] The processor 801 is used to control the overall operation of the cross-database text duplication checking device 800 to complete all or part of the steps in the above-mentioned cross-database text duplication checking method. The memory 802 is used to store various types of data to support the operation of the cross-database text duplication checking device 800, and these data may include, for example, instructions for any application or method operating on the cross-database text duplication checking device 800, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 802 or sent via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules can be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the cross-database text duplication checking device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include: Wi-Fi module, Bluetooth module, NFC module.

[0144] In an exemplary embodiment, the cross-database text duplication checking device 800 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned cross-database text duplication checking method.

[0145] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned cross-database text duplication checking method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions can be executed by the processor 801 of the cross-database text duplication checking device 800 to complete the above-mentioned cross-database text duplication checking method.

[0146] Embodiment 4:

[0147] Corresponding to the above method embodiment, a readable storage medium is also provided in this embodiment. The readable storage medium described below and the cross-database text duplication checking method described above can refer to each other.

[0148] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the cross-database text duplication checking method of the above method embodiment.

[0149] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0151] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A cross-database text duplication checking method, characterized in that: include: Acquire multiple text databases, and establish corresponding paragraph libraries and sentence databases for each text database; Obtain document samples contained in the text database, and after dividing the document samples into paragraph samples and sentence samples, store them in corresponding paragraph libraries and sentence databases respectively; Calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector in the corresponding sentence database and paragraph database respectively; The text to be checked for duplicates is divided into paragraphs to be checked for duplicates, and the paragraphs to be checked for duplicates are sequentially compared with the paragraph sample vectors in each paragraph library to obtain similar paragraphs; The paragraph to be checked for duplicates is divided into sentences to be checked for duplicates, and sentence sample vectors contained in similar paragraphs are extracted from a sentence database; the sentence sample vectors contained in the similar paragraphs are compared with the sentences to be checked for duplicates to obtain similar sentences; The duplicate checking result of the text to be checked is obtained based on similar sentences in all sentence databases.

2. The cross-database text duplication checking method according to claim 1, characterized in that ,Get the document samples contained in the text database, and divide the document samples into paragraph samples and sentence samples, and store them in the corresponding paragraph library and sentence database respectively, including: Read the document sample, the document sample ID and the text database ID to which it belongs, and clean the document sample to remove headers, footers, comments and blank characters; Segment the cleaned document sample based on blank lines and / or paragraph marks to obtain several paragraph samples; Get the position and order of the paragraph sample in the document sample, and generate the paragraph ID and paragraph number; The paragraph sample attribute is composed of the paragraph number and the document sample ID to which it belongs. The paragraph ID, paragraph attribute and paragraph content of the paragraph sample are stored in the paragraph library corresponding to the text database ID.

3. The cross-database text duplication checking method according to claim 1, characterized in that , calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector in the corresponding sentence database and paragraph database respectively, including: Inputting a sentence sample into a word segmenter, performing word segmentation on the sentence sample to obtain a plurality of words, and converting the plurality of words into a word sequence; Input the word sequence into the BERT model to obtain the word vector, and calculate the average of several word vectors to obtain the sentence sample vector of the sentence sample; According to the connection relationship and semantic relationship between the sentence samples, a sentence sample relationship graph is constructed, and the sentence sample weights are calculated using the sentence sample relationship graph; Obtain all sentence samples that constitute the paragraph sample; The paragraph vector of the paragraph sample is calculated by the sentence sample weights and sentence sample vectors of all sentence samples.

4. The cross-database text duplication checking method according to claim 1, characterized in that , divide the text to be checked for duplicates into paragraphs to be checked for duplicates, and compare the paragraphs to be checked for duplicates with the paragraph sample vectors in each paragraph library in turn to obtain similar paragraphs, including: Divide the text to be checked for duplicate content into several paragraphs to be checked for duplicate content, and calculate the paragraph vectors to be checked for duplicate content; Sending a number of the paragraph vectors to be checked for duplicates to each paragraph library, and calculating the paragraph similarity between the paragraph vector to be checked for duplicates and each paragraph sample vector in the paragraph library in turn; When the paragraph similarity is greater than a first preset threshold, the paragraph sample corresponding to the paragraph sample vector is taken as a similar paragraph.

5. A cross-database text duplication checking device, characterized in that: include: Text database acquisition module: used to acquire multiple text databases and establish corresponding paragraph databases and sentence databases for each text database; Document sample processing module: used to obtain document samples contained in the text database, and divide the document samples into paragraph samples and sentence samples, and store them in the corresponding paragraph library and sentence database respectively; Sample vector calculation module: used to calculate the sentence sample vector of each sentence sample and the paragraph sample vector of each paragraph sample, and store the sentence sample vector and the paragraph sample vector into the corresponding sentence database and paragraph database respectively; Paragraph comparison module: used to divide the text to be checked for duplicates into paragraphs to be checked for duplicates, and compare the paragraphs to be checked for duplicates with the paragraph sample vectors in each paragraph library in turn to obtain similar paragraphs; Sentence comparison module: used for dividing the paragraph to be checked for duplicate content into sentences to be checked for duplicate content, extracting sentence sample vectors contained in similar paragraphs from the sentence database; comparing the sentence sample vectors contained in the similar paragraphs with the sentences to be checked for duplicate content to obtain similar sentences; Duplicate check result summary module: used to obtain duplicate check results of the text to be checked for duplicate based on similar sentences in all sentence databases.

6. The cross-database text duplication checking device according to claim 5, characterized in that: The document sample processing module includes: Document sample reading unit: used to read document samples, document sample IDs and the text database IDs to which they belong, and clean the document samples to remove headers, footers, comments and blank characters; Paragraph sample segmentation unit: used to segment the cleaned document sample based on blank lines and / or paragraph marks to obtain several paragraph samples; Paragraph sample attribute generation unit: used to obtain the position and order of the paragraph sample in the document sample and generate the paragraph ID and paragraph number; Paragraph sample storage unit: used to form paragraph sample attributes from the paragraph number and the document sample ID to which it belongs, and store the paragraph ID, paragraph attributes and paragraph content of the paragraph sample into the paragraph library corresponding to the text database ID.

7. The cross-database text duplication checking device according to claim 5, characterized in that: The sample vector calculation module includes: A word segmentation processing unit is used to input a sentence sample into a word segmenter, perform word segmentation on the sentence sample to obtain a number of words, and convert the words into a word sequence; BERT model processing unit: used to input word sequences into the BERT model to obtain word vectors, and calculate the average value of several word vectors to obtain sentence sample vectors of sentence samples; Sentence sample relationship graph construction unit: used to construct a sentence sample relationship graph according to the connection relationship and semantic relationship between sentence samples; A sentence sample weight calculation unit: used to calculate the sentence sample weight using the sentence sample relationship graph; Paragraph vector calculation unit: used to obtain all sentence samples constituting a paragraph sample, and calculate the paragraph vector of the paragraph sample from the sentence sample weights and sentence sample vectors of all sentence samples.

8. The cross-database text duplication checking device according to claim 5, characterized in that: The paragraph comparison module includes: The to-be-checked duplicate paragraph segmentation unit is used to segment the to-be-checked duplicate text into several to-be-checked duplicate paragraphs and calculate the to-be-checked duplicate paragraph vectors; A paragraph similarity calculation unit is used to send a plurality of the paragraph vectors to be checked for duplicates to each paragraph library, and sequentially calculate the paragraph similarity between the paragraph vector to be checked for duplicates and each paragraph sample vector in the paragraph library; A similar paragraph determining unit is used to take the paragraph sample corresponding to the paragraph sample vector as a similar paragraph when the paragraph similarity is greater than a first preset threshold.

9. A cross-database text duplication checking device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the cross-database text duplication checking method as described in any one of claims 1 to 4 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the cross-database text duplication checking method as described in any one of claims 1 to 4.

Images