Terminal-based paper duplicate checking method, terminal and storage medium

[0052] In order to facilitate a better understanding of the present invention, the present invention will be further explained below in conjunction with the accompanying drawings of related embodiments. The embodiments of the present invention are shown in the drawings, but the present invention is not limited to the above-mentioned preferred embodiments. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more fully.

[0053] The logic and/or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium, For use by instruction execution systems, devices, or equipment (such as computer-based systems, systems including processors, or other systems that can fetch and execute instructions from instruction execution systems, devices, or equipment), or combine these instruction execution systems, devices Or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, device, or device or in combination with these instruction execution systems, devices, or devices.

[0054] More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) with one or more wiring, portable computer disk cases (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable and editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program can be printed, because it can be used, for example, by optically scanning the paper or other media, and then editing, interpreting, or other suitable media if necessary. The program is processed in a manner to obtain the program electronically and then stored in the computer memory.

[0055] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the foregoing embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a logic gate circuit for implementing logic functions on data signals Discrete logic circuits, application-specific integrated circuits with suitable combinational logic gates, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0056] In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean specific features described in conjunction with the embodiment or example , Structure, materials or features are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner.

[0057] See figure 1 , Is a flowchart of a method for checking duplicate papers based on a terminal according to an embodiment of the present invention, including steps S10 to S30.

[0058] Step S10, respond to the user's paper duplicate check request;

[0059] When the user needs to check the paper, it will initiate a paper check request, and the terminal will respond to the user's paper check request. For example, a duplicate check button may be provided on the display interface of the terminal. When the paper document is uploaded successfully, the duplicate check button can be triggered to initiate a paper check request, and the processor of the terminal receives the duplicate check button in the terminal display module After the submitted paper duplicate check instruction, you can respond to the user's paper duplicate check request.

[0060] Step S20, fingerprint matching the paper with the text content in a preset database to determine the similarity between the paper and the text content;

[0061] It should be noted that the preset database can be composed of a local database and a network database. The local database can collect published papers, which can be in .doc format and/or .pdf format; understandably, when the collected papers are in .pdf format, they can be downloaded from .pdf format according to the open source tool pdfbox provided by Apache Extract specific file content from the thesis file. The file content in the network database can be used to crawl document resources from the Internet, and the document resources can be denoised document resources.

[0062] In the process of fingerprint matching, if the two files are exactly the same, the fingerprints of the two files are exactly the same. The fingerprint calculation generally uses a hash algorithm, but the traditional hash algorithm can only ensure that the fingerprint calculated from the original content is as uniform and random as possible. For two identical documents, their original content must be the same, but for two With different fingerprints, traditional hashing algorithms will not provide additional information except that their original content is different, which makes it difficult to calculate the similarity of papers. In actual use, even if the original content of the two files differ by only one byte, the corresponding fingerprints are likely to differ greatly. Therefore, in this embodiment, the similarity hash algorithm can be used to replace the traditional hash The algorithm respectively calculates the fingerprint of the paper and the fingerprint corresponding to the text content in the preset database. The main idea of ​​similarity hashing algorithm is to reduce dimensionality, map high-dimensional feature vectors to low-dimensional feature vectors, and perform fingerprint matching through the Hamming distance of two feature vectors to determine whether the paper is similar to the text content.

[0063] It should be noted that when the similarity hash algorithm is used to evaluate the similarity between the content of the paper and the text, the criterion for judging whether the text is similar through the Hamming distance is related to the number of fingerprints. Taking the 64-bit similarity hash fingerprint as an example, we can think that papers with a Hamming distance within 3 are highly similar to the text content based on experience.

[0064] In step S30, at least three pages are displayed on the terminal display interface at the same time; the at least three pages include: a first paper document page used to receive an edit and modify instruction from the user, and is dedicated to display that the similarity exceeds a preset The threshold is the second paper document page corresponding to the marked sentence in the paper, and the page used to index and display the text content corresponding to the sentence.

[0065] After determining the similarity between the paper and the text content, the similarity can be compared with a preset threshold, where the threshold can be set to one or more. The similarity is a value greater than or equal to 0 and less than or equal to 1, and the threshold is the same, but it is generally displayed in the form of percentage, for example, it can be set to 30% or 50%.

[0066] When displaying on the terminal display interface, at least three pages will be displayed, one of which is the first paper document page, which can receive editing and modification instructions sent by the user through the keyboard, microphone, mouse, and/or touch screen, and then edit according to the Modification instructions modify the content of the paper and/or add content. The second paper document page and the first paper document page are both the same paper submitted by the user for duplication check. The user can adjust different preset thresholds through the terminal display interface, so that the marked sentences displayed on the second paper document page are based on actual conditions. Adjust the preset threshold settings. For example, a marked sentence with a similarity of more than 50% in the paper is displayed on the second paper document page, and the marked sentence may be highlighted by means of color or bolding. Further, when multiple preset thresholds are set to display the marked sentences, they can also be distinguished by different colors. For example, the marked sentences with a similarity of more than 50% to 80% can be used for the sentence with a yellow background color. The highlighting of the sentence; marked sentences with a similarity of more than 80% can be highlighted by the red background color. In addition, regarding the index and the page displaying the text content corresponding to the sentence, the corresponding text content can be indexed and displayed according to the sentence marked in the second paper document page selected by the user, in addition to displaying similar text content , It can also display at least one of the author of the similar text content, the name of the journal where the paper is published, the name of the article, the publication time, the similarity corresponding to the similar text, and the modification suggestions for users.

[0067] In addition, it should be noted that the display of the page in the terminal display interface utilizes the FreeMarker technology to load the web page template, thereby generating HTML web page data for displaying the page.

[0068] This embodiment responds to a user’s paper duplicate check request; fingerprints the paper with the text content in a preset database to determine the similarity between the paper and the text content; simultaneously displays on the terminal display interface At least three pages; the at least three pages include: a first paper document page used to receive an edit and modify instruction from the user, dedicated to displaying the correspondingly marked sentences in the paper when the similarity exceeds a preset threshold The second paper document page of, and a page used to index and display the text content corresponding to the sentence. In order to enable the user to refer to the sentence marked in the second paper document page on the terminal display interface, and the page displaying the corresponding text content of the marked sentence, make corresponding edits and modifications on the first paper document page, without the need to follow the test report Content Find the sentence or paragraph to be modified from the paper, which is simple and intuitive; and the terminal display interface also displays the page where the user needs to modify the paper, the paper document page that has been marked with the sentence to be modified, and the page of the text content corresponding to the index of the sentence to be modified In turn, it is convenient for users to refer to and modify operations, saving time for modifying the paper.

[0069] In other embodiments, the terminal display interface is also provided with a duplicate check button, and after the step S30, the following steps may be further included:

[0070] Detect a user's click operation through the check button, and trigger a re-determination of the similarity between the paper and the text content when the user's click operation is detected;

[0071] According to the re-determined similarity, the second paper document page is updated.

[0072] It should be noted that in the traditional paper duplication check scheme, when the user's paper revision is completed, the paper must be re-uploaded to the paper duplication software. It is not immediately known whether the repetition rate meets the requirements, so the paper needs to be revised repeatedly. By setting the duplicate check button, based on the three interfaces displayed on the terminal display interface, the similarity of the revised paper is directly calculated again according to the network database and the local database to find the difference between the revised paper and the text content. Similarity and similar content make it convenient for users to receive updated feedback on the paper's duplicate check results and modify the paper, which is convenient and intuitive.

[0073] See figure 2 , Is a detailed flowchart of step S20 in a method for checking paper duplicates based on a terminal provided by another embodiment of the present invention, including steps S21 to S25.

[0074] Step S21, obtaining fingerprints corresponding to all text contents in the preset database and the total number of words of the paper;

[0075] The similarity is determined by dividing all the similar words by the total words of the paper, and the total words of the paper can be directly obtained through the software supported by the terminal display interface. The main requirement is the number of similar words. This involves fingerprint comparison with the text content in the local database and the network database, so it is necessary to obtain the fingerprint corresponding to each text content.

[0076] Specifically, the method for acquiring the fingerprint corresponding to each text content may be to segment the text content according to a preset standard, and then calculate the fingerprint of each unit after the segmentation through a similarity hash algorithm. Optionally, the text content can be segmented in units of sentences.

[0077] Further, when the fingerprint corresponding to the text content is an N-digit fingerprint, after the step S21, the step may be further included:

[0078] Divide the fingerprints corresponding to all text content into M blocks to form M fingerprint blocks, where each fingerprint block has an N/M fingerprint;

[0079] Using fingerprints with N/M bits as keywords, establish inverted indexes for M fingerprint blocks;

[0080] In order to improve the real-time response speed of the paper duplicate check, a multi-level index can be established for the massive fingerprints corresponding to all text content in the preset database to achieve the purpose of improving the response speed. The fingerprints corresponding to the text content are all N-bit fingerprints. For example, N is equal to 64. When building a multi-level index, the 64-bit fingerprints can be divided into 4 blocks, that is, M is equal to 4, and each fingerprint block has 16-bit fingerprints. . Using the 16-bit fingerprint of each fingerprint block as a keyword, an inverted index associated with the keyword and the text content can be established. The inverted index is equivalent to the mapping relationship table between the keyword and the text content corresponding to the keyword. The number of row indexes is the same as the number of fingerprint blocks.

[0081] Step S22, segmenting the paper to form a paper unit, and using the paper unit as input data, and calculating the paper unit fingerprint corresponding to each paper unit by a similarity hash algorithm;

[0082] Calculating the fingerprint of the paper can also be similar to the fingerprint of the text content. When performing specific fingerprint matching, if the fingerprint corresponding to the text content in the preset database is an N-digit fingerprint, the fingerprint corresponding to the paper should also be an N-digit fingerprint. Is an integer greater than 0.

[0083] Step S23, searching for all fingerprints similar to the fingerprint of the paper unit from the fingerprints corresponding to all the text content;

[0084] When performing fingerprint matching processing, it is also necessary to divide the fingerprints corresponding to the paper equally. Still taking N-digit fingerprints as an example, the fingerprints corresponding to the paper can be divided into M blocks with the same time-sharing fingerprints corresponding to the text content to form M blocks of the paper fingerprints, and each block of the paper fingerprints has an N/M-digit fingerprint . Since the fingerprints corresponding to the text content are also divided into M blocks, similar fingerprints can be found by comparing each fingerprint block with each paper fingerprint block.

[0085] Based on the above analysis, when the fingerprints corresponding to the text content are N-digit fingerprints, the step S23 includes the steps:

[0086] Divide all the paper unit fingerprints into M blocks to form M paper fingerprint blocks, where each paper fingerprint block has an N/M fingerprint;

[0087] Compare each paper fingerprint block with each fingerprint block in turn to find all similar fingerprints.

[0088] Step S24, load corresponding similar text content according to all the similar fingerprints found;

[0089] It is understandable that there are similar fingerprints between the fingerprints corresponding to the text content and the fingerprints corresponding to the paper. Then there must be similar text content in the original text data between the text content and the paper. The correspondence between the fingerprint and the text content can be Relations, search with similar fingerprints, so as to load similar text content.

[0090] Still taking the inverted index established in this embodiment as an example, the step S24 may include the steps:

[0091] Determine the fingerprint block to which each similar fingerprint belongs;

[0092] Using the similar fingerprint as a keyword, search for similar text content corresponding to the similar fingerprint from the inverted index where the fingerprint block of the keyword belongs.

[0093] Since the fingerprint corresponding to the text content has an inverted index, the fingerprint block to which it belongs can be found through similar fingerprints, and then the similar fingerprint is used as a keyword to quickly find the corresponding similar text content from the fingerprint block.

[0094] Step S25: Calculate the similarity between the paper and the text content according to the similar text content, the paper unit and the total word count of the paper.

[0095] It should be noted that the calculation of the number of similar words involves the distance of feature vectors between texts, and the similarity between texts can be evaluated by the size of the distance. The calculation of the number of similar words is related to the similarity of the words between all similar text content and the corresponding thesis unit. Therefore, the number of similar words can be calculated through the similar text content and the corresponding thesis unit, and then combined with the total number of words in the paper. Find out the similarity between the paper and the text content.

[0096] This embodiment provides a specific method for fingerprint matching to obtain the similarity between the text content and the paper, and provides a technical basis for the implementation of the paper duplicate check. In addition, a multi-level index of fingerprints corresponding to the text content has been established, which helps to speed up the real-time response of the paper duplicate check and the result of the paper duplicate check.

[0097] See image 3 , Is a detailed flowchart of step S25 in the terminal-based paper duplicate checking method proposed by another embodiment of the present invention. The step S25 includes step S251 to step S255.

[0098] Step S251, searching for a similar paper unit corresponding to each similar text content from all the paper units according to the similar text content;

[0099] The similar paper unit in this embodiment may include: the sentence marked in the paper when the similarity displayed on the second paper document page exceeds the preset threshold, and the sentence in the paper when the similarity does not exceed the preset threshold, that is, all similar papers A unit is a collection of related content between text content and thesis.

[0100] Step S252: Perform word segmentation on each similar text content and corresponding similar paper units to obtain a text word segmentation set of each similar text content and a paper word segmentation set of each similar paper unit; wherein, a text word segmentation set consists of a similar text content It is composed of several words in the essay, and a paper segmentation set is composed of several words in a similar paper unit;

[0101] Since the determination of the number of similar words involves the calculation of the distance between the feature vectors of the text, and the calculation of the number of similar words is related to the similarity of the words between all the similar text content and the corresponding paper unit, it is necessary to compare the similar text content and the corresponding paper The unit performs word segmentation, and the specific word segmentation technology can refer to the existing word segmentation tools and word segmentation algorithms, which will not be repeated here. After segmentation of each similar text and the corresponding similar paper unit, a set of words after segmentation can be obtained, where a single similar paper unit is segmented to obtain a paper segmentation set, and a single similar text content is segmented to obtain text segmentation set. Jaccard similarity between similar paper units and similar text content can be calculated through the paper word segmentation set and text word segmentation set, or called the Jaccard similarity coefficient.

[0102] Step S253: Obtain the text length of each similar text content and the text length of the corresponding similar paper unit;

[0103] It should be noted that the text length of similar text content or similar thesis unit represents the number of words corresponding to a similar text content or a similar thesis unit. The text length can determine the editing distance and editing similarity between similar paper units and similar text content.

[0104] Step S254: Calculate the number of similar words between the paper and the text content according to the text length of each similar text content, the text word segmentation set, the text length of each similar paper unit, and the paper word segmentation set;

[0105] Specifically, the step S254 may include:

[0106] Calculate the similarity between each similar text content and the corresponding similar paper unit by similar=factor*editSimilar+(1-factor)*jaccardSimilar; among them, similar is the similarity between each similar text content and the corresponding similar paper unit; factor is the preset weight factor between each similar text content and the corresponding similar paper unit, 0≤factor≤1; editSimilar is the editing similarity, editSimilar=1-editDistance(a,b)/max, a is the similar text content The length of the text, b is the text length of the similar thesis unit, editDistance is the edit distance; jaccardSimilar is the Jaccard similarity, jaccardSimilar=|A∩B|/|A∪B|, A is the text segmentation set, B is the paper segmentation set;

[0107] by Calculate the number of similar words between the paper and the text content, where S is the number of similar words, i is the i-th similar paper unit, n is the total number of similar paper units, and similar is the similarity of each similar text content to the corresponding one The similarity between the thesis units, b is the text length of the similar thesis unit.

[0108] It should be noted that Jaccard similarity is used to compare the similarity and difference between limited sample sets. The greater Jaccard similarity is, the higher the similarity between sample sets. In this embodiment, the calculation of the number of similar words is obtained by multiplying and adding Jaccard similarity, editing similarity and weighting factors respectively.

[0109] Step S255: The quotient obtained by dividing the number of similar words by the total number of words is used as the similarity between the paper and the text content.

[0110] It should be noted that since the paper includes table of contents, titles, formulas, charts and references, etc., when the total word count is recognized through the software, the above non-text parts are generally skipped, so the total word count of the actual paper will be greater than the detected The total number of words. Contents such as similar titles and directories are generally less relevant to the actual content of the paper. Excluding them from the total number of words is also conducive to the determination of the true similarity. This embodiment provides a calculation method for the number of similar words in the similarity calculation, which is beneficial to the specific implementation of the paper duplicate check.

[0111] See Figure 4 , Figure 4 It is a schematic structural diagram of a terminal proposed in an embodiment of the present invention, and the terminal includes:

[0112] The response module 10 is used to respond to the user's paper duplicate check request;

[0113] The determining module 20 is configured to perform fingerprint matching between the paper and the text content in a preset database to determine the similarity between the paper and the text content;

[0114] The display module 30 is configured to display at least three pages on the display interface at the same time; the at least three pages include: a first paper document page for receiving an editing and modifying instruction from the user, and is dedicated to displaying the similarity exceeding When the threshold is preset, the second paper document page corresponding to the marked sentence in the paper, and the page used for indexing and displaying the text content corresponding to the sentence.

[0115] Further, in another embodiment, a duplicate check button is also provided on the terminal display interface;

[0116] The terminal also includes:

[0117] The detection module 40 is configured to detect a user's click operation through the check button, and trigger to re-determine the similarity between the paper and the text content when the user's click operation is detected;

[0118] The update module 50 is configured to update the second paper document page according to the newly determined similarity.

[0119] Further, in another embodiment, the determining module 20 includes:

[0120] The acquiring unit 21 is configured to acquire fingerprints corresponding to all text contents in the preset database and the total number of words of the paper;

[0121] The calculation unit 22 is configured to segment the paper to form a paper unit, and use the paper unit as input data to calculate the paper unit fingerprint corresponding to each paper unit through a similarity hash algorithm;

[0122] The searching unit 23 is used to search for all fingerprints similar to the fingerprints of the paper unit from the fingerprints corresponding to all text contents;

[0123] The loading unit 24 is used to load the corresponding similar text content according to all the similar fingerprints found;

[0124] The calculation unit 22 is further configured to calculate the similarity between the paper and the text content according to the similar text content, the paper unit, and the total number of words of the paper.

[0125] Further, in another embodiment, the fingerprints corresponding to the text content are N-digit fingerprints;

[0126] The determining module 20 further includes:

[0127] The segmentation unit 25 is configured to divide the fingerprints corresponding to all text content into M blocks to form M fingerprint blocks, wherein each fingerprint block has an N/M fingerprint;

[0128] The establishment unit 26 is configured to use fingerprints with N/M bits as keywords to respectively establish inverted indexes for M fingerprint blocks;

[0129] The loading unit 24 includes:

[0130] The determining subunit 241 is used to determine the fingerprint block to which each similar fingerprint belongs;

[0131] The first searching subunit 242 is configured to use the similar fingerprint as a keyword to search for similar text content corresponding to the similar fingerprint from the inverted index where the fingerprint block belongs to the keyword.

[0132] Further, in another embodiment, the paper unit fingerprint is an N-digit fingerprint;

[0133] The searching unit 23 includes:

[0134] The molecular unit 231 is used to divide all the paper unit fingerprints into M blocks to form M paper fingerprint blocks, wherein each paper fingerprint block has an N/M bit fingerprint;

[0135] The second searching subunit 232 sequentially compares each paper fingerprint block with each fingerprint block to find all similar fingerprints.

[0136] Further, in another embodiment, the calculation unit 22 includes:

[0137] The third searching subunit 221 is configured to search for a similar paper unit corresponding to each similar text content from all the paper units according to the similar text content;

[0138] The word segmentation subunit 222 is used to segment each similar text content and corresponding similar paper units to obtain the text word segmentation set of each similar text content and the paper word segmentation set of each similar paper unit; among them, a text word segmentation set consists of A similar text content consists of several words, and a thesis word set consists of several words in a similar paper unit;

[0139] The obtaining subunit 223 is used to obtain the text length of each similar text content and the text length of the corresponding similar paper unit;

[0140] The calculation subunit 224 is used to calculate the number of similar words between the paper and the text content through the text length of each similar text content, the text word segmentation set, the text length of each similar paper unit, and the paper word segmentation set;

[0141] The calculation subunit 224 is configured to divide the quotient obtained by dividing the number of similar words by the total number of words as the similarity between the paper and the text content.

[0142] Further, in another embodiment, the calculation subunit 224 is further configured to calculate the similarity between each similar text content and the corresponding similar paper unit by similar=factor*editSimilar+(1-factor)*jaccardSimilar; wherein , Similar is the similarity between each similar text content and the corresponding similar paper unit; factor is the preset weight factor between each similar text content and the corresponding similar paper unit, 0≤factor≤1; editSimilar is the editing similarity , EditSimilar=1-editDistance(a,b)/max, a is the text length of similar text content, b is the text length of similar paper units, editDistance is the edit distance; jaccardSimilar is the Jaccard similarity, jaccardSimilar=|A∩ B|/|A∪B|, A is the text segmentation set, B is the thesis word segmentation set; and passed Calculate the number of similar words between the paper and the text content, where S is the number of similar words, i is the i-th similar paper unit, n is the total number of similar paper units, and similar is the similarity of each similar text content to the corresponding one The similarity between the thesis units, b is the text length of the similar thesis unit.

[0143] This embodiment also provides a terminal, the terminal comprising: a memory, a processor, and a computer program stored on the memory and capable of running on the processor, and the computer program is implemented when the processor is executed The steps of the terminal-based paper duplicate checking method as described above.

[0144] This embodiment also provides a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the terminal-based paper duplication check method described above are realized.

[0145] The foregoing embodiments describe the technical principles of the present invention. These descriptions are only for explaining the principles of the present invention, and cannot be construed as limiting the protection scope of the present invention in any way. Based on the explanation here, those skilled in the art can think of other specific implementation manners of the present invention without creative work, and these manners will fall within the protection scope of the present invention.