Paper plagiarism detection method and device based on similarity

Abstract

The invention belongs to the technical field of computers, and discloses a paper plagiarism detection method and device based on similarity, and the method comprises the steps: obtaining a to-be-detected paper, and carrying out the cleaning of the to-be-detected paper, and obtaining a target paper; obtaining a historical paper database, and screening to obtain a plurality of to-be-compared papers; calculating digital fingerprints and word frequency vectors of the target paper and the papers to be compared; calculating fingerprint similarity between the target paper and each to-be-compared paper based on the digital fingerprints; calculating the word frequency similarity between the target paper and each to-be-compared paper based on the word frequency vector; and obtaining a plagiarism detection result according to the fingerprint similarity and the word frequency similarity. According to the method, the data volume needing to be processed in similarity calculation can be reduced, and the calculation precision is ensured while the detection efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for detecting paper plagiarism based on similarity. Background Art

[0002] With the development of the Internet and the convenient access to information, plagiarism and plagiarism have become increasingly prominent. In order to ensure the maintenance of academic atmosphere and research quality, an efficient and accurate paper duplication detection algorithm has become an indispensable tool. The paper duplication detection algorithm analyzes and compares multiple text files to determine the similarity and repetition rate between them. Conventional paper duplication detection algorithms mainly include word frequency algorithm, cosine similarity algorithm, SimHash algorithm, etc.

[0003] However, since the paper text is long, the database resources that need to be compared are very large, which will increase the computational cost of the word frequency algorithm and reduce its efficiency. The calculation process of the cosine similarity algorithm is relatively complicated and will also affect its efficiency due to the increase in data volume. Although the SimHash algorithm is more efficient when processing large-scale data sets, it has lower accuracy and cannot accurately locate the plagiarized parts. Therefore, the existing paper duplication detection algorithms cannot meet both detection accuracy and efficiency at the same time because they have to process a large amount of data text. Summary of the invention

[0004] The present application provides a method and device for detecting plagiarism of a paper based on similarity, which can reduce the amount of data that needs to be processed for similarity calculation, while improving detection efficiency and ensuring calculation accuracy.

[0005] In a first aspect, the present application provides a similarity-based paper plagiarism detection method, comprising:

[0006] Obtain the papers to be tested and clean them to obtain the target papers;

[0007] Obtain a historical paper database and screen multiple papers to be compared;

[0008] Calculate the digital fingerprints and word frequency vectors of the target paper and each paper to be compared;

[0009] Calculate the fingerprint similarity between the target paper and each paper to be compared based on the digital fingerprint;

[0010] Calculate the word frequency similarity between the target paper and each paper to be compared based on the word frequency vector;

[0011] The plagiarism detection results are obtained based on fingerprint similarity and word frequency similarity.

[0012] Furthermore, the above-mentioned paper to be detected is obtained and cleaned to obtain the target paper, including:

[0013] Remove the header, footer, references and preset stop words of the paper to be tested;

[0014] Convert all capital letters of the paper to be tested to lowercase letters;

[0015] Perform lemma restoration on the English text of the paper to be tested to obtain the target paper.

[0016] Furthermore, the above-mentioned historical paper database is obtained, and multiple papers to be compared are screened, including:

[0017] Extract the keywords of the target paper; use the inverted index method to screen the papers to be compared in the historical paper database; the title or abstract of the paper to be compared contains at least one keyword.

[0018] Furthermore, the above calculation of the digital fingerprints and word frequency vectors of the target paper and each paper to be compared includes:

[0019] Segment the target paper or the paper to be compared to obtain multiple tokens;

[0020] One or more hash functions are used to process each word to obtain a digital fingerprint;

[0021] Classify and count each word to obtain multiple words of the same type and their corresponding frequencies;

[0022] Construct a word frequency vector based on each similar word and its corresponding frequency.

[0023] Furthermore, the above-mentioned use of one or more hash functions to process each word to obtain a digital fingerprint includes:

[0024] Put each word into different hash functions and map them to get the corresponding hash value;

[0025] The hash values ​​corresponding to different hash functions are combined into a hash vector;

[0026] The hash vector is normalized to obtain a digital fingerprint.

[0027] Furthermore, the above-mentioned use of one or more hash functions to process each word to obtain a digital fingerprint includes:

[0028] Random step: randomly arrange each word into a hash function and map it to get a hash value;

[0029] Repeat the random steps multiple times to get multiple different hash values;

[0030] The hash vector composed of each hash value is standardized to obtain a digital fingerprint.

[0031] Furthermore, the method also includes:

[0032] Use a syntactic analyzer to obtain the syntactic trees of the target paper and each paper to be compared;

[0033] Convert each syntax tree into a format to obtain each syntax tree in a treebank format;

[0034] Extract the corresponding syntactic pattern from each syntactic tree in the treebank format;

[0035] Calculate the syntactic similarity between the target paper and each paper to be compared based on the syntactic pattern;

[0036] The plagiarism detection results are obtained based on syntactic similarity, fingerprint similarity and word frequency similarity.

[0037] Furthermore, syntactic patterns are substructures of the treebank, including noun phrases and verb phrases.

[0038] Furthermore, the above-mentioned calculation of the syntactic similarity between the target paper and each paper to be compared based on the syntactic pattern includes:

[0039] Detect the number of overlapping phrases between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared;

[0040] Calculate the edit distance between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared;

[0041] The syntactic similarity is obtained by multiplying the inverse of the edit distance by the number of overlapping phrases.

[0042] Furthermore, the plagiarism detection results obtained according to the fingerprint similarity and word frequency similarity include:

[0043] Calculate the first similarity average of the fingerprint similarity and word frequency similarity with the paper to be compared;

[0044] If the first similarity average value is greater than the first preset threshold, it is determined that the target paper has plagiarism problems.

[0045] Furthermore, the plagiarism detection results obtained according to syntactic similarity, fingerprint similarity and word frequency similarity include:

[0046] Calculate the second similarity average of syntactic similarity, fingerprint similarity and word frequency similarity;

[0047] If the syntactic similarity is greater than the second preset threshold and the first similarity average is greater than the third preset threshold, or the second similarity average is greater than the first preset threshold, it is determined that the target paper has plagiarism problems;

[0048] Among them, the first preset threshold is smaller than the third preset threshold.

[0049] Furthermore, the method also includes:

[0050] Input the target paper and the paper to be compared into the trained first model respectively to obtain the corresponding fine features;

[0051] The two obtained fine features are input into the trained second model to obtain feature similarity;

[0052] Determine whether the feature similarity is greater than the fingerprint similarity between the target paper and the paper to be compared;

[0053] If so, the feature similarity is taken as the fingerprint similarity.

[0054] Furthermore, the method also includes:

[0055] Obtain multiple training papers and input them into the word embedding model respectively to obtain the text vector corresponding to each training paper;

[0056] Use grid search technology to obtain the optimal convolution kernel parameters and build a convolutional deep learning model;

[0057] The text vectors of each training paper are input into the convolutional deep learning model in turn, and the regularization technology is used to inactivate random neurons of the convolutional deep learning model to obtain the trained first model.

[0058] In a second aspect, the present application embodiment provides a similarity-based paper plagiarism detection device, comprising:

[0059] The cleaning module is used to obtain the papers to be tested and clean them to obtain the target papers;

[0060] The screening module is used to obtain the historical paper database and screen multiple papers to be compared;

[0061] The calculation module is used to calculate the digital fingerprint and word frequency vector of the target paper and each paper to be compared;

[0062] Fingerprint module, used to calculate the fingerprint similarity between the target paper and each paper to be compared based on the digital fingerprint;

[0063] The word frequency module is used to calculate the word frequency similarity between the target paper and each paper to be compared based on the word frequency vector;

[0064] The judgment module is used to obtain the plagiarism detection result according to the fingerprint similarity and the word frequency similarity.

[0065] Furthermore, the cleaning module comprises:

[0066] A removal unit is used to remove the header, footer, references and preset stop words of the paper to be tested;

[0067] A letter conversion unit, used to convert all the capital letters of the paper to be tested into lowercase letters;

[0068] The word form restoration unit is used to restore the English text of the paper to be tested to obtain the target paper.

[0069] Furthermore, the screening module is used to extract various keywords of the target paper, and use the inverted index method to screen the papers to be compared in the historical paper database; there is at least one keyword in the title or abstract of the paper to be compared.

[0070] Furthermore, the above-mentioned calculation module includes:

[0071] The segmentation unit is used to segment the target paper or the paper to be compared to obtain multiple word units;

[0072] A hash unit, used to process each word using one or more hash functions to obtain a digital fingerprint;

[0073] A statistical unit is used to classify and count each word unit to obtain multiple similar words and corresponding frequencies;

[0074] The vector unit is used to construct a word frequency vector based on each similar word and its corresponding frequency.

[0075] Furthermore, the hash unit is used to put each word into a different hash function, map it to obtain a corresponding hash value; form a hash vector with the hash values ​​corresponding to each different hash function; and standardize the hash vector to obtain a digital fingerprint.

[0076] Furthermore, the hash unit is used to randomly arrange each word into a hash function to map a hash value; repeat multiple times to obtain multiple different hash values; and standardize the hash vector composed of each hash value to obtain a digital fingerprint.

[0077] Furthermore, the device also includes:

[0078] The syntax tree module is used to obtain the syntax trees of the target paper and the papers to be compared using a syntax analyzer;

[0079] The tree bank module is used to convert the format of each syntax tree to obtain each syntax tree in the tree bank format;

[0080] A syntax extraction module, used to extract corresponding syntax patterns from each syntax tree in the treebank format;

[0081] The syntactic similarity module is used to calculate the syntactic similarity between the target paper and each paper to be compared based on the syntactic pattern;

[0082] The judgment module is also used to obtain plagiarism detection results based on syntactic similarity, fingerprint similarity and word frequency similarity.

[0083] Furthermore, the above syntactic similarity module includes:

[0084] Overlapping units are used to detect the number of overlapping phrases between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared;

[0085] The edit distance unit is used to calculate the edit distance between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared;

[0086] The syntactic similarity calculation unit is used to multiply the inverse of the edit distance by the number of overlapping phrases to obtain the syntactic similarity.

[0087] Furthermore, the judgment module is used to calculate a first similarity average value of fingerprint similarity and word frequency similarity with the paper to be compared; if the first similarity average value is greater than a first preset threshold, it is judged that the target paper has a plagiarism problem.

[0088] Furthermore, the judgment module is also used to calculate the second similarity average of syntactic similarity, fingerprint similarity and word frequency similarity; if the syntactic similarity is greater than the second preset threshold and the first similarity average is greater than the third preset threshold, or the second similarity average is greater than the first preset threshold, it is judged that the target paper has plagiarism problems; wherein the first preset threshold is less than the third preset threshold.

[0089] Furthermore, the device also includes:

[0090] A fine module is used to input the target paper and the paper to be compared into the trained first model respectively to obtain the corresponding fine features;

[0091] A feature similarity module is used to input the obtained two refined features into the trained second model to obtain feature similarity;

[0092] The feature updating module is used to determine whether the feature similarity is greater than the fingerprint similarity between the target paper and the paper to be compared; if so, the feature similarity is used as the fingerprint similarity.

[0093] Furthermore, the device also includes:

[0094] The word embedding module is used to obtain multiple training papers and input them into the word embedding model respectively to obtain the text vector corresponding to each training paper;

[0095] Parameter module, which is used to obtain the optimal convolution kernel parameters using grid search technology and build a convolutional deep learning model;

[0096] The training module is used to input the text vectors of each training paper into the convolutional deep learning model in sequence, and at the same time use regularization technology to inactivate random neurons of the convolutional deep learning model to obtain a trained first model.

[0097] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a similarity-based paper plagiarism detection method as described in any of the above embodiments are performed.

[0098] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a similarity-based paper plagiarism detection method as in any of the above embodiments are implemented.

[0099] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0100] The embodiment of the present application provides a method for detecting plagiarism of papers based on similarity. Firstly, the papers to be compared are determined by screening the historical paper database, thereby reducing the amount of data required for subsequent processing and improving the overall calculation efficiency of the similarity. Secondly, fingerprint similarity is calculated based on digital fingerprints and word frequency similarity is calculated based on word frequency vectors, and the two similarities are combined to determine whether the paper is plagiarized. The generation of digital fingerprints is not affected by changes in text details and has high robustness. The results of the word frequency algorithm can accurately locate the plagiarized parts in the target paper, thereby ensuring the calculation accuracy while improving the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 A flowchart of a similarity-based paper plagiarism detection method provided as an exemplary embodiment of the present application.

[0102] Figure 2 A flow chart of cleaning steps is provided for an exemplary embodiment of the present application.

[0103] Figure 3 A flowchart of the steps of calculating word frequency vectors and digital fingerprints is provided for an exemplary embodiment of the present application.

[0104] Figure 4 A flowchart of the steps of calculating a digital fingerprint is provided for an exemplary embodiment of the present application.

[0105] Figure 5 A flowchart of digital fingerprint calculation steps is provided for another exemplary embodiment of the present application.

[0106] Figure 6 A flowchart of the syntax tree analysis steps provided for an exemplary embodiment of the present application.

[0107] Figure 7 A flowchart of the syntactic similarity calculation steps provided for an exemplary embodiment of the present application.

[0108] Figure 8 A flowchart of the fingerprint similarity updating steps provided for an exemplary embodiment of the present application.

[0109] Fig. 9 A flowchart of the first model training step provided for an exemplary embodiment of the present application.

[0110] Fig.10 A structural diagram of a similarity-based paper plagiarism detection device provided as an exemplary embodiment of the present application.

[0111] Fig.11 A structural diagram of a cleaning module provided for an exemplary embodiment of the present application.

[0112] Fig.12 A structural diagram of a computing module provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0113] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0114] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0115] The terms "first", "second", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0116] Terminology explanation:

[0117] 1) Token: refers to the smallest unit of meaning in a text after word segmentation. In different languages ​​and contexts, tokens may have different forms:

[0118] Word: In languages ​​like English that use spaces to separate words, a token is usually an individual word.

[0119] Character: In some cases, especially in some Asian languages, a token may refer to a single character.

[0120] Phrase or n-gram: In some text processing tasks, a word unit can be a combination of multiple consecutive words or characters, such as bigram (two-tuple), trigram (three-tuple), etc.

[0121] Root or Stem: In text preprocessing using stemming or lemmatization, a lemma may be the base or standard form of a word.

[0122] 2) Term Frequency: refers to the number of times a specific word appears in text data. It is a basic concept in information retrieval and text mining, used to evaluate the importance of a word to a text file.

[0123] First, count each word in the text and calculate the total number of times each word appears in the text. The frequency of a specific word can be expressed as follows:

[0124]

[0125] Among them, t is the word whose frequency is to be calculated (updated to word-unit in this application), d is the text containing the word (the target paper or the paper to be compared in this application), and the total number of entries refers to the total number of all non-repeated words in the document.

[0126] In some cases, the word frequencies are normalized to eliminate the effect of document length on word frequencies:

[0127]

[0128] The normalized word frequency value is usually between 0 and 1.

[0129] It should be noted that word frequency does not take into account the importance or semantics of a word in a document, it only reflects the frequency of its occurrence. Therefore, it may overestimate the importance of common words, so the TF-IDF weight is usually calculated in combination with the Inverse Document Frequency (IDF) to more accurately assess the significance of a word.

[0130] 3) Hash function: A mathematical function that takes an input (or "key") and calculates a fixed-size numerical output, often called a "hash value", "hash code" or "digest". A hash function produces the same output for the same input whenever it is calculated. Hash functions typically map larger input data to a smaller output space, such as mapping an input of arbitrary length to a hash value of fixed length. Ideally, a hash function will evenly distribute the input in the output space and avoid aggregation in the output. It is almost impossible to reverse the original input data from the hash value. Hash functions used in cryptography also have some additional security features, such as resistance to pre-image attacks and birthday attacks.

[0131] Common hash functions include MD5, SHA-1, SHA-256, etc. They have different security performance and efficiency in different application scenarios. However, no hash function is completely collision-resistant, so choosing a suitable hash function needs to be determined based on specific application needs and security requirements.

[0132] 4) Syntactic Tree (or Parse Tree): It is a graphical representation used to show the grammatical structure of a sentence. In natural language processing (NLP) and computational linguistics, a syntactic tree uses a tree structure to represent the hierarchical and dependency relationships between words in a sentence. Each node in the syntactic tree represents a grammatical unit, which can be a single word, a phrase, or an entire sentence. The top of the syntactic tree is the root node, which represents the entire sentence. Internal nodes usually represent phrases or clauses, which are the components of a sentence. Leaf nodes are individual words in a sentence and are located at the bottom of the tree. The lines branching out from each internal node represent grammatical relationships, such as subject-predicate structure, verb-object structure, etc.

[0133] The edges in the syntax tree represent the dependency relationships between words, such as modification relationships, domination relationships, etc. The syntax tree reflects the hierarchical structure of the sentence, showing how phrases are combined into larger structures. Nodes can also contain grammatical role information, such as noun phrases (NP), verb phrases (VP), etc. In some syntax trees, leaf nodes also contain part-of-speech information of the vocabulary, such as nouns, verbs, adjectives, etc. Syntax trees are usually automatically generated by syntactic analyzers (Parsers), which build tree structures based on certain grammatical rules or statistical models.

[0134] 5) CNN model (Convolutional Neural Network): It is a deep learning model that is particularly suitable for processing data with obvious grid-like topological structures. Its internal structure mainly includes:

[0135] Convolutional Layer: It performs convolution operation with the input data through the convolution kernel to extract local features and introduces nonlinearity using activation functions (such as ReLU).

[0136] Activation function: such as ReLU, used to increase the nonlinear expression ability of the network.

[0137] Pooling layer: Use maximum pooling or average pooling to reduce the spatial dimension of the data and extract the main features.

[0138] Fully Connected Layer: At the end of the network, the feature maps extracted by the convolutional layer and the pooling layer are converted into the final output, such as classification results, feature extraction results, etc.

[0139] CNN training is performed through the back-propagation algorithm and the gradient descent algorithm, and the network weights are updated by calculating the gradient of the loss function. Compared with the method of manually designing features, CNN can automatically learn the features in the data. The advantages of the CNN model are parameter sharing, automatic feature extraction, and good adaptability to the spatial structure of the input data.

[0140] 6) DNN model (deep neural network): It is a machine learning model composed of multiple neuron layers. It imitates the structure and working principle of the human brain neural network, and achieves high-performance solutions for complex tasks through hierarchical feature learning and weight adjustment. The basic unit of DNN is a neuron. Each neuron receives input from other neurons and changes the impact of the input on the neuron by adjusting the weight. The neural network can approximate complex functions through multiple layers of nonlinear hidden layers, achieving a universal approximation effect. The internal structure of DNN includes:

[0141] Input Layer: Receives raw data input.

[0142] Hidden Layers: It is composed of multiple neurons. The output of each layer of neurons is used as the input of the next layer, and nonlinear transformation is achieved through activation functions.

[0143] Output Layer: Generates the final prediction result according to task requirements, such as classification or regression.

[0144] In the wave of artificial intelligence and machine learning, DNN has become a powerful tool for solving complex problems with its powerful feature learning ability and nonlinear processing ability. The advantage of DNN lies in its powerful feature learning ability. Compared with the traditional manual feature design method, DNN can automatically extract useful features from raw data, greatly improving the generalization ability of the model. In addition, the highly nonlinear characteristics of DNN enable it to handle complex nonlinear relationships.

[0145] See also Figure 1 , the present application embodiment provides a similarity-based paper plagiarism detection method, including:

[0146] Step S1, obtain the paper to be detected and clean it to obtain the target paper.

[0147] Step S2, obtaining a historical paper database, and screening to obtain multiple papers to be compared.

[0148] Among them, the screening method can use a trained clustering analysis model, input the target paper into it, and the clustering analysis model will perform cluster analysis on the historical paper database, classify similar papers into the same cluster, and then compare the target paper with the center or representative papers of these clusters to obtain multiple papers to be compared.

[0149] Specifically, conventional paper duplication detection algorithms usually perform detection on the entire historical paper database, and improve detection accuracy through comprehensive comparison; but for algorithms with more complex calculations such as word frequency and cosine similarity, this requires very large computing power or calculation time, resulting in low duplication detection efficiency. Therefore, this application first screens the historical paper database, and then uses a comprehensive judgment of the two algorithms to ensure calculation accuracy, so that accurate detection results can be obtained while reducing the amount of data.

[0150] Step S3, calculating the digital fingerprint and word frequency vector of the target paper and each paper to be compared.

[0151] Before calculating the digital fingerprint and word frequency vector of the paper to be compared, each paper to be compared may be subjected to the same cleaning process as the target paper, so as to further improve the efficiency of step S3.

[0152] Step S4, calculating the fingerprint similarity between the target paper and each paper to be compared based on the digital fingerprint.

[0153] Specifically, the Hamming distance or Jaccard coefficient between the digital fingerprint of the target paper and the digital fingerprint of the paper to be compared can be calculated as the fingerprint similarity of the two digital fingerprints.

[0154] However, the calculation of Hamming distance requires the digital fingerprints of the target paper and the paper to be compared to have the same length, which poses requirements for the fingerprint algorithm. Although Jaccard coefficient can handle digital fingerprints of unequal lengths, since the Jaccard coefficient is calculated based on the union of two digital fingerprints, it is easy to cause a relatively high fingerprint similarity even for a small intersection. Therefore, the Jaccard coefficient performs weaker than Hamming distance in terms of accuracy.

[0155] Step S5: Calculate the word frequency similarity between the target paper and each paper to be compared based on the word frequency vectors.

[0156] Specifically, the word frequency similarity can be obtained by calculating the Euclidean distance or Manhattan distance between two word frequency vectors.

[0157] In practical applications, before calculating the word frequency similarity, the word frequency vectors of the target paper and the papers to be compared can be weighted by TF-IDF to reduce the influence of common words, increase the weight of professional terms, and further improve the accuracy of the word frequency similarity.

[0158] Step S6: Obtain the plagiarism detection result based on the fingerprint similarity and the word frequency similarity.

[0159] A method for detecting paper plagiarism based on similarity provided by the above embodiments. First, determine the papers to be compared by screening the historical paper database, thereby reducing the amount of data required for subsequent processing and improving the overall calculation efficiency of similarity. Second, calculate the fingerprint similarity based on digital fingerprints and the word frequency similarity based on word frequency vectors respectively, and combine the two similarities to determine whether there is plagiarism in the paper. The generation of digital fingerprints is not affected by text detail changes and has high robustness. The result of the word frequency algorithm can accurately locate the part of the target paper where plagiarism exists, so as to ensure the calculation accuracy while improving the detection efficiency.

[0160] Please refer to Figure 2 , in some embodiments, the above process of obtaining the paper to be detected and cleaning it to obtain the target paper includes:

[0161] Step S11: Remove the header, footer, references, and preset stop words of the paper to be detected.

[0162] Among them, the preset stop words are words such as "de", "le", "he", etc., which are irrelevant to vocabulary and do not contain semantic information. Removing these contents can reduce data sparsity and improve the accuracy and efficiency of word segmentation.

[0163] Step S12: Convert all uppercase letters in the paper to be detected into lowercase letters.

[0164] The case conversion operation here is to avoid judging the same word as different words due to different cases (such as "Apple" and "apple"), thereby affecting the classification of word units and the calculation accuracy of word frequency.

[0165] Step S13, performing morphological restoration on the English text of the paper to be detected to obtain the target paper.

[0166] Among them, word form restoration refers to converting vocabulary into its dictionary form, such as converting "better" to "good", which is equivalent to unifying synonyms and improving the accuracy of word classification.

[0167] In addition to the above, you can also perform stemming on the target paper, for example, restoring vocabulary to its basic form and removing tense descriptions, such as converting "running" to "run".

[0168] The various cleaning processes provided in the above embodiments can reduce the noise in the text, improve the quality of the digital fingerprint and word frequency vector, make them better reflect the semantic content of the text, and thus improve the calculation accuracy of the similarity.

[0169] In some embodiments, the above-mentioned acquisition of the historical paper database and screening to obtain multiple papers to be compared include:

[0170] Extract the keywords of the target paper; use the inverted index method to screen the papers to be compared in the historical paper database; the title or abstract of the paper to be compared contains at least one keyword.

[0171] Specifically, the method of performing inverted indexing in titles and abstracts based on keywords to determine the papers to be compared given in the above embodiment is faster than the aforementioned clustering analysis model in screening, because here only the titles and abstracts of each paper need to be traversed, but the screening accuracy is weaker than the clustering analysis model; in actual application, it can be selected according to the specific technical field of the target paper. For technical fields with more research directions and a wider range, the clustering analysis model can be used for screening first. For some less popular technical fields with fewer related research, the inverted index method can be used first.

[0172] See also Figure 3 In some embodiments, the above calculation of the digital fingerprints and word frequency vectors of the target paper and each paper to be compared may specifically include the following steps:

[0173] Step S31, segment the target paper or the paper to be compared to obtain multiple word-grams.

[0174] Step S32: use one or more hash functions to process each word to obtain a digital fingerprint.

[0175] Step S33, classifying and counting each word-gram to obtain a plurality of similar word-grams and corresponding frequencies.

[0176] Step S34, constructing a word frequency vector according to each similar word unit and the corresponding frequency.

[0177] Specifically, we build a vocabulary containing all unique tokens, which will be used to map the tokens in the paper into the vector space, and then count the number of occurrences of each token in the vocabulary.

[0178] The vocabulary is converted into a vector form, where each dimension of the vector corresponds to a word in the vocabulary. If a word appears in the paper, its corresponding dimension value is the frequency of occurrence. The same type of word in this application refers to the same word, that is, after splitting to obtain multiple words, the same words are counted, and the frequency of occurrence, i.e., word frequency, is calculated based on the number of the same words.

[0179] Furthermore, in order to eliminate the influence of the length of the paper text on the word frequency, it is possible to choose to standardize the word frequency vectors of both the target paper and the paper to be compared (for example, each dimension is divided by the total number of words in the paper text).

[0180] Furthermore, if the dimensional space of the obtained word frequency vector is very large, PCA (principal component analysis) can be used to reduce the dimensionality of the word frequency vector. This is only applicable to the case where the dimensional space is extremely large. In the actual practice, in order to ensure the accuracy of the similarity, the dimensions of most word frequency vectors do not meet the standard of extremely large dimensional space, and no dimensionality reduction is required.

[0181] See also Figure 4 In some embodiments, the above-mentioned use of one or more hash functions to process each word to obtain a digital fingerprint may specifically include the following steps:

[0182] Step S3211, put each word into different hash functions, and map to obtain the corresponding hash value.

[0183] Specifically, suppose a set of hash functions is H = {h1,h2,h3,...h k,}, where h i is a hash function, each hash function is different, and k is the number of hash functions. Each word is input into the hash function in turn to obtain the corresponding function value, and the minimum value of the function value of all words is taken as the hash value mapped by this hash function.

[0184] For example, assuming that the words segmented from the target paper or the paper to be compared are apple, banana, and cherry, they are respectively input into the first hash function: h1(apple)=17, h1(banan)=11, h1(cherry)=5, where the function value 5 is the minimum value of all function values, and 5 is used as the hash value of h1.

[0185] Step S3212: The hash values ​​corresponding to the different hash functions are combined into a hash vector.

[0186] Step S3213, normalize the hash vector to obtain a digital fingerprint.

[0187] Specifically, in order to improve the computational efficiency of fingerprint similarity, the obtained hash vector is standardized so that each value in the digital fingerprint is between 0 and 1, which has better generalization ability.

[0188] See also Figure 5 In some embodiments, the above-mentioned use of one or more hash functions to process each word to obtain a digital fingerprint may specifically include the following steps:

[0189] Step S3221, random step: randomly arrange each word unit and put it into the hash function to map it to obtain a hash value.

[0190] Step S3222, repeat the random steps multiple times to obtain multiple different hash values.

[0191] Specifically, each word is first randomly arranged to disrupt the order of the words to reduce the impact of the original data structure on the results; then the disrupted arrangement is used to circularly shift the words in the arrangement to the right to generate n different arrangement orders, where n is the number of words obtained by segmenting the target paper or the paper to be compared.

[0192] Assume that the hash function used in this embodiment is h c , input each word in the i-th (1≤i≤n) order into the hash function h c , obtain the corresponding function value, sort the function values ​​according to the position of their corresponding words in the i-th arrangement sequence, and select the first non-zero function value in the function value arrangement as the hash value corresponding to the arrangement sequence.

[0193] Execute the above steps for the first k permutations, and use the obtained k hash values ​​as hash vectors; this is equivalent to "cyclically shifting the word elements in the permutation to the right" to generate the permutation, and this step only needs to be repeated k times.

[0194] Step S3223, normalize the hash vector composed of each hash value to obtain a digital fingerprint.

[0195] The above embodiment provides a process of constructing a digital fingerprint using a hash function. In the specific implementation process, the calculation method using multiple hash functions is relatively simple and can provide accurate fingerprint similarity calculation, but the calculation cost is high when processing large-scale data sets, especially when the number of papers to be compared is large; while the method of constructing a digital fingerprint using a single hash function has high calculation efficiency and is very suitable for use when the number of papers to be compared is large, but because the estimated variance of the digital fingerprint obtained by a single hash function is smaller than that of multiple hash functions, the label in terms of fingerprint similarity calculation accuracy is weaker than that of multiple hash functions.

[0196] See also Figure 6 In some embodiments, the method further comprises:

[0197] Step S01, using a syntactic analyzer to obtain the syntactic trees of the target paper and each paper to be compared.

[0198] Among them, the syntactic analyzer can be selected from Stanford Parser, SpaCy, NLTK and other analyzers.

[0199] Step S02, converting the format of each syntax tree to obtain each syntax tree in a tree library format.

[0200] Specifically, the syntactic tree in the Treebank format retains the structured information of the sentences in the paper, including the hierarchy of phrases, dependencies, and syntactic roles, which helps to gain a deeper understanding of the semantic and grammatical characteristics of the sentences. Moreover, the Treebank format supports the application of more complex syntactic similarity comparison algorithms, such as the edit distance algorithm, which can more carefully evaluate the differences between sentences. In addition, the Treebank format is very versatile. Not only do many existing NLP tools and libraries support this format, but it can also be adjusted according to multiple languages ​​to achieve cross-language sentence comparison.

[0201] Step S03: extract corresponding syntactic patterns from each syntactic tree in the treebank format.

[0202] Among them, the syntactic pattern is the substructure of the tree bank, including noun phrases and verb phrases.

[0203] Step S04, calculating the syntactic similarity between the target paper and each paper to be compared based on the syntactic pattern.

[0204] Step S05, obtaining a plagiarism detection result according to syntactic similarity, fingerprint similarity and word frequency similarity.

[0205] Although the fingerprint similarity mentioned in the aforementioned embodiments of the present application can quickly identify the similarity of texts, it does not have a deep enough understanding of semantics, and word frequency similarity focuses on the comparison of word-unit dimensions. Therefore, although the combination of the two is more accurate than conventional duplicate checking algorithms, it does not have a sufficient understanding of the structural information in the paper, including the components of phrases and sentences.

[0206] Therefore, this application further adds syntactic similarity comparison on top of fingerprint and word frequency similarity comparison. Syntactic analysis can reveal the dependencies and semantic roles between words in a text, reduce misjudgments caused by text editing or format changes, and has good robustness to noise and variations in the text, thereby greatly improving the accuracy of paper duplication detection.

[0207] See also Figure 7 In some embodiments, the above-mentioned calculation of the syntactic similarity between the target paper and each paper to be compared based on the syntactic pattern may specifically include the following steps:

[0208] Step S041, detecting the number of overlapping phrases between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared.

[0209] Among them, the number of overlapping phrases refers to how many identical phrases there are in the syntactic patterns of the two papers.

[0210] Specifically, detecting whether two syntactic patterns match can be processed using a tree-recursive neural network (Tree-RNN).

[0211] Step S042, calculating the edit distance between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared.

[0212] Among them, the edit distance (Levenshtein Distance) is calculated by calculating the minimum number of single-character editing (insertion, deletion or replacement) operations required to convert one string into another. In this application, it is the minimum number of editing operations required to convert the syntactic pattern of the target paper into the excitation pattern of the text to be compared. When calculating the edit distance, dynamic programming can be introduced, that is, by establishing a two-dimensional array (DP table) to record the intermediate results, and gradually filling the array to find the minimum edit distance. In dynamic programming, the solution of each subproblem depends on the solution of its adjacent subproblems, and the state transition equation is usually based on three operations: inserting nodes, deleting nodes, and replacing nodes.

[0213] Step S043: multiply the inverse of the edit distance by the number of overlapping phrases to obtain the syntactic similarity.

[0214] Specifically, the edit distance can reflect the similarity or difference between the syntactic structures of two texts. A smaller edit distance means that the two texts are more syntactically similar. Therefore, this application multiplies the inverse of the edit distance by the number of overlapping phrases to obtain the syntactic similarity, so that the size of the syntactic similarity is proportional to the possibility of plagiarism.

[0215] The above embodiment calculates the syntactic similarity based on the edit distance and the number of overlapping phrases, fully quantifies the differences between the syntactic trees, indirectly reflects the semantic differences between the two paper texts, and accurately identifies the slight changes in the sentences.

[0216] In some embodiments, the plagiarism detection result obtained according to the fingerprint similarity and the word frequency similarity includes:

[0217] Calculate the first similarity average value of the fingerprint similarity and word frequency similarity with the paper to be compared; if the first similarity average value is greater than the first preset threshold, it is determined that the target paper has plagiarism problems.

[0218] In the prior art, conventional operations usually involve determining whether a single similarity exceeds a corresponding threshold value, but this determination method is overly dependent on the accuracy of the corresponding similarity calculation process and has a poor fault tolerance rate. For example, in the two methods of digital fingerprint calculation mentioned in the above embodiments, using different calculation methods has a slight impact on the final similarity result. The present application calculates the average of the two similarities, which is equivalent to combining the similarity of the two papers in text content and word frequency, thereby improving the fault tolerance rate, accuracy, and stability of plagiarism judgment (if a text is rewritten by changing the word order or using synonyms, the word frequency similarity alone may not be able to detect plagiarism, but the fingerprint similarity may still be able to identify the similarity).

[0219] In one embodiment, the plagiarism detection result obtained according to the syntactic similarity, fingerprint similarity and word frequency similarity includes:

[0220] Calculate the second similarity average of syntactic similarity, fingerprint similarity and word frequency similarity; if the syntactic similarity is greater than the second preset threshold and the first similarity average is greater than the third preset threshold, or the second similarity average is greater than the first preset threshold, then it is determined that the target paper has plagiarism problems; wherein the first preset threshold is less than the third preset threshold.

[0221] This application provides two judgment methods here. One is to calculate the average value of three similarities. This method is more accurate than the judgment of a single similarity or two similarity average values ​​(i.e., the first similarity average value); the other is to judge the syntactic similarity and the first similarity average value separately. At this time, the judgment threshold of the first similarity average value, i.e., the third preset threshold value is greater than the first preset threshold value. It can be considered that the plagiarism judgment of fingerprint similarity and word frequency similarity is more relaxed. This method focuses more on the judgment of semantics and sentence structure. Which one to use can be selected according to the needs of researchers.

[0222] See also Figure 8 In some embodiments, the method further comprises:

[0223] Step S71, input the target paper and the paper to be compared into the trained first model respectively to obtain corresponding fine features.

[0224] Step S72: input the obtained two fine features into the trained second model to obtain feature similarity.

[0225] Among them, the first model is a convolutional neural network (CNN model), and the second model is a deep neural network (DNN model).

[0226] Step S73, determining whether the feature similarity is greater than the fingerprint similarity between the target paper and the paper to be compared.

[0227] Step S74: If yes, the feature similarity is used as the fingerprint similarity.

[0228] As described in the above embodiments, the digital fingerprint calculation based on the hash function is affected by the number of hash functions. The digital fingerprints calculated by a single or multiple hash functions are different, resulting in different calculation results of the fingerprint similarity, especially when the calculation methods of the digital fingerprints used in the target paper and the paper to be compared are different; therefore, in order to avoid such unexpected situations, the present application further adds the fingerprint similarity calculation of the model, calculates the fingerprint similarity again through the first model and the second model, and compares it with the fingerprint similarity obtained by using the hash function, and selects the larger value as the fingerprint similarity used for subsequent judgment.

[0229] The above embodiment updates the fingerprint similarity by adding two new models, thereby achieving a "double insurance" for the fingerprint similarity and avoiding inaccurate calculation of the fingerprint similarity due to an inappropriate selection of the number of hash functions.

[0230] See also Fig. 9 In some embodiments, the method further comprises:

[0231] Step S81, obtain multiple training papers and input them into the word embedding model respectively to obtain the text vector corresponding to each training paper.

[0232] Among them, word embedding models can be selected from Word2Vec, GloVe, FastText, etc.

[0233] Specifically, the present application uses text vectors for training in order to ensure that the dimensions of each training data input into the CNN model are the same, thereby speeding up the training efficiency of the CNN model.

[0234] Step S82, use grid search technology to obtain the optimal convolution kernel parameters and build a convolutional deep learning model.

[0235] Specifically, determining the optimal convolution kernel size and stride is one of the key steps in building a CNN model. The choice of these parameters will affect the model's ability to capture local features. The grid search in this application systematically searches for the optimal convolution kernel size and stride, and can quickly determine the optimal convolution kernel parameters corresponding to the task requirements of this application. The stride determines the interval at which the convolution kernel slides on the input data. A larger stride can reduce the spatial dimension of the output feature map, which helps reduce the amount of calculation, but may lose some information. The stride is usually set to 1 or 2.

[0236] Step S83, input the text vectors of each training paper into the convolutional deep learning model in sequence, and use regularization technology to inactivate random neurons of the convolutional deep learning model to obtain a trained first model.

[0237] Specifically, during the training process, Dropout randomly inactivates neurons (or features) in the convolutional deep learning model according to a predetermined probability (usually a hyperparameter, such as 0.5), that is, sets them to zero. This means that in each iteration, the architecture of the network will be slightly different. Since only a part of the neurons are activated in each iteration, this reduces the complex co-adaptation relationship between neurons, that is, prevents the network from over-relying on a specific combination of features.

[0238] In a sense, Dropout is equivalent to training multiple different models (different at each iteration due to random dropout) and averaging the predictions of these models at test time. This ensemble approach can improve the generalization ability of the model.

[0239] See also Fig.10 Another embodiment of the present application provides a similarity-based paper plagiarism detection device, comprising:

[0240] The cleaning module 101 is used to obtain the paper to be detected and clean it to obtain the target paper.

[0241] The screening module 102 is used to obtain a historical paper database and screen a plurality of papers to be compared.

[0242] The calculation module 103 is used to calculate the digital fingerprint and word frequency vector of the target paper and each paper to be compared.

[0243] The fingerprint module 104 is used to calculate the fingerprint similarity between the target paper and each paper to be compared based on the digital fingerprint.

[0244] The word frequency module 105 is used to calculate the word frequency similarity between the target paper and each paper to be compared based on the word frequency vector.

[0245] The judgment module 106 is used to obtain a plagiarism detection result according to the fingerprint similarity and the word frequency similarity.

[0246] See also Fig.11 Furthermore, the cleaning module 101 includes:

[0247] The removal unit 11 is used to remove the header, footer, references and preset stop words of the paper to be detected.

[0248] The letter conversion unit 12 is used to convert all the capital letters of the paper to be detected into lowercase letters.

[0249] The word form restoration unit 13 is used to perform word form restoration on the English text of the paper to be detected to obtain the target paper.

[0250] Furthermore, the screening module 102 is used to extract various keywords of the target paper, and use the inverted index method to screen the papers to be compared in the historical paper database; there is at least one keyword in the title or abstract of the paper to be compared.

[0251] See also Fig.12 , further, the calculation module 103 includes:

[0252] The segmentation unit 31 is used to segment the target paper or the paper to be compared to obtain a plurality of word-grams.

[0253] The hash unit 32 is used to process each word using one or more hash functions to obtain a digital fingerprint.

[0254] The statistical unit 33 is used to classify and count each word-gram to obtain a plurality of similar word-grams and corresponding frequencies.

[0255] The vector unit 34 is used to construct a word frequency vector according to each similar word element and the corresponding frequency.

[0256] Furthermore, the hash unit 32 is used to put each word into different hash functions to map and obtain corresponding hash values; to form a hash vector from the hash values ​​corresponding to different hash functions; and to standardize the hash vector to obtain a digital fingerprint.

[0257] Furthermore, the hash unit 32 is used to randomly arrange each word into a hash function to map a hash value; repeat the process multiple times to obtain multiple different hash values; and standardize the hash vector composed of each hash value to obtain a digital fingerprint.

[0258] Furthermore, the device also includes:

[0259] The syntax tree module is used to obtain the syntax trees of the target paper and each paper to be compared using a syntax analyzer.

[0260] The tree bank module is used to convert the format of each syntax tree to obtain each syntax tree in the tree bank format.

[0261] The syntax extraction module is used to extract the corresponding syntax patterns from each syntax tree in the tree bank format.

[0262] The syntactic similarity module is used to calculate the syntactic similarity between the target paper and each paper to be compared based on the syntactic pattern.

[0263] The judgment module is also used to obtain plagiarism detection results based on syntactic similarity, fingerprint similarity and word frequency similarity.

[0264] Furthermore, the above syntactic similarity module includes:

[0265] Overlapping units are used to detect the number of overlapping phrases between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared.

[0266] The edit distance unit is used to calculate the edit distance between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared.

[0267] The syntactic similarity calculation unit is used to multiply the inverse of the edit distance by the number of overlapping phrases to obtain the syntactic similarity.

[0268] Furthermore, the judgment module 106 is used to calculate a first similarity average value of fingerprint similarity and word frequency similarity with the paper to be compared; if the first similarity average value is greater than a first preset threshold, it is judged that the target paper has a plagiarism problem.

[0269] Furthermore, the judgment module 106 is also used to calculate the second similarity average of syntactic similarity, fingerprint similarity and word frequency similarity; if the syntactic similarity is greater than the second preset threshold and the first similarity average is greater than the third preset threshold, or the second similarity average is greater than the first preset threshold, it is judged that the target paper has plagiarism problems.

[0270] Among them, the first preset threshold is smaller than the third preset threshold.

[0271] Furthermore, the device also includes:

[0272] The fine module is used to input the target paper and the paper to be compared into the trained first model respectively to obtain the corresponding fine features.

[0273] The feature similarity module is used to input the obtained two fine features into the trained second model to obtain feature similarity.

[0274] The feature updating module is used to determine whether the feature similarity is greater than the fingerprint similarity between the target paper and the paper to be compared; if so, the feature similarity is used as the fingerprint similarity.

[0275] Furthermore, the device also includes:

[0276] The word embedding module is used to obtain multiple training papers and input them into the word embedding model respectively to obtain the text vector corresponding to each training paper.

[0277] The parameter module is used to obtain the optimal convolution kernel parameters using grid search technology and build a convolutional deep learning model.

[0278] The training module is used to input the text vectors of each training paper into the convolutional deep learning model in sequence, and at the same time use regularization technology to inactivate random neurons of the convolutional deep learning model to obtain a trained first model.

[0279] For the specific limitations of a similarity-based paper plagiarism detection device provided in this embodiment, please refer to the above embodiment of a similarity-based paper plagiarism detection method, which will not be repeated here. Each module in the above-mentioned similarity-based paper plagiarism detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0280] In the embodiments disclosed in the present application, it should be understood that the disclosed products can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional module in each embodiment of the present invention can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0281] The embodiment of the present application provides a computer device, which may include a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of a similarity-based paper plagiarism detection method as in any of the above embodiments.

[0282] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the above embodiment of a paper plagiarism detection method based on similarity, which will not be described in detail here.

[0283] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a similarity-based paper plagiarism detection method as in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0284] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of a paper plagiarism detection method based on similarity, which will not be described in detail here.

[0285] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0286] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0287] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A similarity-based paper plagiarism detection method, characterized in that: include: Obtain the papers to be tested and clean them to obtain the target papers; Obtain a historical paper database and screen multiple papers to be compared; Calculate the digital fingerprint and word frequency vector of the target paper and each of the papers to be compared; Calculate the fingerprint similarity between the target paper and each of the papers to be compared based on the digital fingerprint; Calculate the word frequency similarity between the target paper and each of the papers to be compared based on the word frequency vector; A plagiarism detection result is obtained according to the fingerprint similarity and the word frequency similarity.

2. The similarity-based paper plagiarism detection method according to claim 1, characterized in that: The step of calculating the digital fingerprint and word frequency vector of the target paper and each of the papers to be compared includes: Segmenting the target paper or the paper to be compared to obtain multiple word-grams; Processing each of the word elements using one or more hash functions to obtain the digital fingerprint; Classifying and counting the word-grams to obtain a plurality of similar word-grams and corresponding frequencies; The word frequency vector is constructed according to each of the similar word elements and the corresponding frequencies.

3. The similarity-based paper plagiarism detection method according to claim 2, characterized in that: The step of using one or more hash functions to process each word to obtain the digital fingerprint includes: Put each of the word elements into different hash functions and map them to obtain corresponding hash values; The hash values ​​corresponding to the different hash functions are combined into a hash vector; The hash vector is normalized to obtain the digital fingerprint.

4. The similarity-based paper plagiarism detection method according to claim 2, characterized in that: The step of using one or more hash functions to process each word to obtain the digital fingerprint includes: Random step: randomly arranging each word element and putting it into the hash function to map and obtain a hash value; Repeat the random step multiple times to obtain multiple different hash values; The hash vector composed of the hash values ​​is standardized to obtain the digital fingerprint.

5. The similarity-based paper plagiarism detection method according to claim 1, characterized in that: Also includes: Using a syntactic analyzer to obtain the syntactic trees of the target paper and each of the papers to be compared; Converting the syntax trees into different formats to obtain the syntax trees in a treebank format; extracting corresponding syntactic patterns from each of the syntactic trees in the treebank format; Calculating the syntactic similarity between the target paper and each of the papers to be compared based on the syntactic pattern; A plagiarism detection result is obtained according to the syntactic similarity, the fingerprint similarity and the word frequency similarity.

6. The similarity-based paper plagiarism detection method according to claim 5, characterized in that: The calculating the syntactic similarity between the target paper and each of the papers to be compared based on the syntactic pattern includes: Detecting the number of overlapping phrases between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared; Calculating the edit distance between the syntactic pattern of the target paper and the syntactic pattern of the paper to be compared; The syntactic similarity is obtained by multiplying the inverse of the edit distance by the number of overlapping phrases.

7. The similarity-based paper plagiarism detection method according to claim 1, characterized in that: Also includes: Input the target paper and the paper to be compared into the trained first model respectively to obtain corresponding fine features; Inputting the obtained two refined features into the trained second model to obtain feature similarity; Determine whether the feature similarity is greater than the fingerprint similarity between the target paper and the paper to be compared; If so, the feature similarity is used as the fingerprint similarity.

8. The similarity-based paper plagiarism detection method according to claim 7, characterized in that: Also includes: Obtain multiple training papers and input them into the word embedding model respectively to obtain text vectors corresponding to the training papers; Use grid search technology to obtain the optimal convolution kernel parameters and build a convolutional deep learning model; The text vectors of the training papers are sequentially input into the convolutional deep learning model, and the regularization technology is used to inactivate random neurons of the convolutional deep learning model to obtain the trained first model.

9. A similarity-based paper plagiarism detection device, characterized in that: include: The cleaning module is used to obtain the papers to be tested and clean them to obtain the target papers; The screening module is used to obtain the historical paper database and screen multiple papers to be compared; A calculation module, used to calculate the digital fingerprint and word frequency vector of the target paper and each of the papers to be compared; A fingerprint module, used for calculating the fingerprint similarity between the target paper and each of the papers to be compared based on the digital fingerprint; A word frequency module, used for calculating the word frequency similarity between the target paper and each of the papers to be compared based on the word frequency vector; A judgment module is used to obtain a plagiarism detection result according to the fingerprint similarity and the word frequency similarity.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the similarity-based paper plagiarism detection method as described in any one of claims 1 to 8 are implemented.

Images