Text duplicate checking method and system and storage medium

Abstract

The invention discloses a text duplicate checking method and system and a storage medium, and belongs to the technical field of text information processing. The method comprises the following steps: S1, customizing an experiment report template according to an experiment project by an administrator; s2, a user generates an experiment report according to the experiment report template and uploads the experiment report; s3, the system extracts the text content needing duplicate checking in the experiment report according to the duplicate checking identifier, and performs duplicate checking by using an algorithm to obtain a plurality of similar fragment results; s4, screening similar fragment results by using a term library to obtain final similar fragments; and S5, positioning a final similar fragment, highlighting, calculating a repetition rate, and generating a final duplicate checking report. A better detection effect can be achieved on the condition that a plurality of locally similar fragments exist in the text; the duplicate checking area can be dynamically set, duplicate checking of related terms is avoided by adding the term library, and misjudgment of the repetition rate is better reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of text information processing, and in particular to a text duplication checking method, system and storage medium. Background Art

[0002] At present, text similarity checking technology has been applied to many scenarios, such as paper duplication checking in the academic field and intellectual property protection in the legal field.

[0003] The traditional experiment report similarity checking algorithm has the following problems: (1) Reliance on fragments with the same structure: Traditional methods generally first formulate certain rules or patterns, and then match similar fragments according to the established rules. If the original text structure changes slightly, it may cause the pre-established matching rules to become invalid, and the two originally similar fragments may be mistakenly judged as unrelated. (2) Unable to handle semantic similarity: Similar to (1), since traditional detection methods often only compare features such as character encoding or shape characteristics of text, it is impossible to detect fragments with similar semantics. (3) Unable to locate the position of repeated paragraphs: In general text similarity detection methods, either matching according to certain rules or calculating the similarity of the entire text based on word frequency statistics, it is often impossible to locate the position of repeated fragments in the original text. (4) The calculated repetition rate is often too high: Since students' experimental reports do not all need to be checked for plagiarism, the contents such as experimental introductions and experimental steps may be highly similar in multiple experimental reports. If they are also included in the scope of plagiarism checking, the final repetition rate result will often be too high. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a text duplication checking method, system and storage medium.

[0005] The object of the present invention is achieved through the following technical solutions: The first aspect of the present invention provides: a text duplication checking method, comprising the following steps: S1: The administrator customizes the experiment report template according to the experiment project; S2: The user generates a lab report based on the lab report template and uploads it; S3: The system extracts the text content that needs to be checked for plagiarism in the experimental report based on the plagiarism check mark, and uses the algorithm to check for plagiarism to obtain several similar fragment results; S4: Use the term library to filter the similar fragment results to obtain the final similar fragments; S5: Locate the final similar segments, highlight them, calculate the repetition rate, and generate the final duplication check report.

[0006] Preferably, the S1 further comprises the following steps: Enter the terminology set for this experimental project; Add the components of this experimental report; Define the repetition rate threshold of each component module; Generate the anti-counterfeiting mark of this experimental report template; Add text content range markers for the component modules that need to be checked for plagiarism; Generate the document directory of each component module and set it as non-editable to obtain the template of this experiment report.

[0007] Preferably, the S2 further comprises the following steps: The system performs a compliance check on the experimental report uploaded by the user, and executes S3 only after the check passes.

[0008] Preferably, the S3 further comprises the following steps: Enter the text content to be detected; Set algorithm parameters, including similarity score, gap penalty, similarity threshold, and maximum similar paragraph threshold; Use the Chinese word segmenter to segment the current Chinese text and initialize the score matrix; Create a cell instance object and fill it into the two-dimensional array of the score matrix; According to the scoring rules, each cell is scored from left to right and from top to bottom. When scoring, the Chinese word vector model is used to judge the semantic similarity of the words. At the same time, the maximum score in the global data and its cell position are updated. Backtracking is performed. When backtracking, the cell with the maximum score is found in the global data, and it is checked whether it is in the undetectable area. If it is, the backtracking is abandoned and the cell information is removed from the global data. If it is in the detectable area, backtracking is performed to obtain similar text fragments, and they are put into the similar fragment set in the global data, and the cell information is removed from the global data at the same time. According to the maximum similar paragraph threshold, from the similar segment set of global data, segments that meet the similarity length are taken out to obtain several similar segment results and output them.

[0009] Preferably, the S4 further comprises the following steps: The similar fragment results are compared with the terms in the term library, and the term density is calculated. Term density = number of terms in the similar fragment results / total number of words in the similar fragment results. Similar fragment results with term density greater than the preset threshold are deleted to obtain the final similar fragment.

[0010] Preferably, the scoring rule is as follows: the final score of each cell is taken from the maximum value of the similarity score of the current cell, the similarity score of the cell to the upper left of the current cell plus the similarity score of the current cell, the similarity score of the cell to the left of the current cell minus the gap penalty, and the similarity score of the cell above the current cell minus the gap penalty. If any calculated value is negative, the value is set to 0; Assume that the similarity score s(a,b)= ±N. If the word corresponding to the current cell is detected as similar, the score is +N, otherwise it is -N. Where a and b represent the two segmentation elements corresponding to the same cell in the segmentation matrix formed after the two segments of the text to be detected are segmented. The vacancy penalty W = W0- k Δ W, where W0 is the initial vacancy penalty; k is the number of consecutive matches to similarity, that is, starting from a cell, the number of consecutive positive similarity scores s(a,b) of its upper left cell. When k is greater than the preset maximum number, the preset maximum number is taken; Δ W is the linear reduction coefficient of the gap penalty obtained according to the similar number of consecutive matches.

[0011] The second aspect of the present invention provides: a text duplication checking system, used to implement any of the above-mentioned text duplication checking methods, comprising: Custom module, used to customize the experiment report template according to the experimental project; The experimental report generation module is used to generate and upload the experimental report according to the experimental report template; The duplicate checking module is used to extract the text content that needs to be checked for duplicates in the experimental report according to the duplicate checking mark, and use the algorithm to check for duplicates to obtain several similar fragment results; A term screening module is used to screen similar fragment results using a term library to obtain final similar fragments; The duplicate checking report generation module is used to locate the final similar fragments, highlight them, calculate the repetition rate, and generate the final duplicate checking report.

[0012] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned text duplication checking methods is implemented.

[0013] The beneficial effects of the present invention are: 1) The original Smith-Waterman algorithm is improved, and the Chinese word segmentation is used as the minimum comparison unit, and the final comparison result is obtained by comparing the semantics. This enables the algorithm to handle the semantic similarity of the text to a certain extent. And by adjusting the semantic similarity threshold, the semantic similarity of the text can be detected to different degrees.

[0014] 2) The calculation method of the gap penalty value in the original Smith-Waterman algorithm is improved, so that the improved algorithm can have a better detection effect when there are multiple locally similar segments in the text.

[0015] 3) The duplicate checking area can be set dynamically, and by adding a term library, the duplicate checking of related terms can be avoided, thus better reducing the misjudgment of the repetition rate.

[0016] 4) In the matching results of similar text fragments, their position index in the original text is maintained, so similar areas can be highlighted in the original text, thereby generating a more intuitive duplicate checking report. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of the text duplication checking method; Figure 2 Configure flowcharts for custom templates; Figure 3 Flowchart for algorithm duplication checking; Figure 4 Schematic diagram of initializing the score matrix; Figure 5 This is a schematic diagram of the score matrix after the score is calculated; Figure 6 This is a schematic diagram of scoring matrix backtracking; Figure 7 This is a schematic diagram of the non-repeatable detection of the score matrix; Figure 8 Schematic diagram of cell storage content; Fig. 9 This is a schematic diagram of the global data storage content; Fig.10 This is a flowchart for the algorithm duplication checking program; Fig.11 This is a flowchart for term comparison. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1-Figure 11 The first aspect of the present invention provides: a text duplication checking method, comprising the following steps: S1: The administrator customizes the experiment report template according to the experiment project; S2: The user generates a lab report based on the lab report template and uploads it; S3: The system extracts the text content to be checked in the experimental report according to the duplicate check flag, and uses an algorithm to perform duplicate check to obtain several similar segment results; S4: Use the term library to screen the similar segment results to obtain the final similar segments; S5: Locate the final similar segments, highlight them and calculate the duplication rate to generate the final duplicate check report.

[0020] In this embodiment, the present invention uses and improves the algorithm proposed by Temple Smith and Michael Waterman in 1981. The original design of this algorithm was initially applied in the biological field to detect local similarity regions of base pairs in DNA. The present invention improves the original algorithm to make it have better effects on Chinese text similarity check.

[0021] The present invention makes the following improvements to the Smith-Waterman algorithm: (1) Chinese words as comparison units: The original Smith-Waterman algorithm is designed to compare local similarities on a long chain of base pairs. A single base position is only represented as one character (A, T, C, G), so the comparison unit is a single character. However, in Chinese characters, it is meaningless to take a single character as the smallest comparison unit. Therefore, after a long sentence is segmented, each segment should be used as a comparison unit respectively. For example, for the sentence "I love China", after segmentation, it should be disassembled into ["I / noun", "love / verb", "China / noun"], and then enter the subsequent comparison process, rather than simply splitting each character: ["I", "love", "China", "country"].

[0022] (2) Use semantic similarity comparison: The traditional Smith-Waterman algorithm compares whether characters are the same, that is, G-G is the same, and A-T is different. In this improvement, according to semantic similarity analysis techniques such as word vector analysis, two Chinese words within the threshold range are both defined as the same, adding a semantic dimension to the algorithm. For example, the similarity between the words "running" and "sprinting" is 95%. When the similarity threshold is set to 95% or above, they are considered the same.

[0023] (3) Improve the calculation method of the gap penalty value: The gap penalty value is an input parameter in the original algorithm (introduced in the subsequent chapter). Its function is to deduct points from the unit that generates a gap in the score matrix. There are two assignment methods for this value in the original Smith-Waterman algorithm: One is to set it as a constant, that is, set it to a fixed value from the beginning; the other is extended scoring, that is, the more the gap is extended, the greater the penalty value. In this solution, a method of dynamically reducing the gap penalty value is adopted, which can better adapt to the matching ability of multiple local similar paragraphs in a longer text.

[0024] (4) Multiple detection: In the traditional Smith-Waterman algorithm, the algorithm stops after detecting the most similar segment. However, in the scenario of detecting students’ lab reports, there may be more than one similarity. Therefore, this solution improves the above problem by adding a multiple matching mechanism and a non-repeatable matching area, so that multiple text segments that meet the similarity requirements can be matched.

[0025] In some embodiments, the S1 further comprises the following steps: Enter the terminology set for this experimental project; Add the components of this experimental report; Define the repetition rate threshold of each component module; Generate the anti-counterfeiting mark of this experimental report template; Add text content range markers for the component modules that need to be checked for plagiarism; Generate the document directory of each component module and set it as non-editable to obtain the template of this experiment report.

[0026] In this embodiment, if Figure 2 As shown in the figure, for different experimental projects, teachers can configure different experimental report templates. First, teachers need to enter the experimental terms in this experiment. For example, "acidity" and "PH value" are likely to appear frequently in students' experimental reports. Such terms should not be judged as repeated between reports. This step is not necessary. If it is not configured (that is, the term set is an empty set), an unlimited text duplication check is performed. Add experimental report components: Add the components of this experimental report, such as experimental purpose, experimental steps, experimental conclusions, discussion questions, etc. Define the thresholds for each part: For example, the following settings [Experimental purpose: 0%] [Experimental steps: 0%%] [Experimental conclusions: 30%] [Discussion questions: 50%] 0% means that there is no need to check for duplicates, because perhaps the experimental purpose and experimental steps are filled in by the teacher, or the content is highly similar in all students' reports, so there is no need to check for duplicates. The experimental conclusions, discussion questions, etc. are completed by students independently, so it is necessary to check for duplicates. 30% means that in the experimental conclusion part, if it is detected that another report has more than 30% of the same content, it is considered suspected plagiarism.

[0027] Generate template file: The system generates a report template document based on the above configuration. The generated template contains a custom anti-counterfeiting mark (which is transparent and invisible to students) to detect whether the lab report uploaded by students later is from the template file. The template also includes non-editable component modules and the range mark of the duplicate checking part (which is also invisible to students).

[0028] In some embodiments, the S2 further comprises the following steps: The system performs a compliance check on the experimental report uploaded by the user, and executes S3 only after the check passes.

[0029] In this embodiment, after the student downloads the template and completes the lab report, the report is uploaded to the system, and the system will check the compliance of the lab report (whether it comes from the template, whether any required content is blank, etc.). After the check is passed, the duplicate checking process will be started after the report reaches the upload deadline.

[0030] In some embodiments, the S3 further comprises the following steps: Enter the text content to be detected; Set algorithm parameters, including similarity score, gap penalty, similarity threshold, and maximum similar paragraph threshold; Use the Chinese word segmenter to segment the current Chinese text and initialize the score matrix; Create a cell instance object and fill it into the two-dimensional array of the score matrix; According to the scoring rules, each cell is scored from left to right and from top to bottom. When scoring, the Chinese word vector model is used to judge the semantic similarity of the words. At the same time, the maximum score in the global data and its cell position are updated. Backtracking is performed. When backtracking, the cell with the maximum score is found in the global data, and it is checked whether it is in the undetectable area. If it is, the backtracking is abandoned and the cell information is removed from the global data. If it is in the detectable area, backtracking is performed to obtain similar text fragments, and they are put into the similar fragment set in the global data, and the cell information is removed from the global data at the same time. According to the maximum similar paragraph threshold, from the similar segment set of global data, segments that meet the similarity length are taken out to obtain several similar segment results and output them.

[0031] In this embodiment, the algorithm duplication checking process is as follows: Figure 3 As shown, it includes: ① Comparison text input: input two paragraphs of text to be detected.

[0032] ② Algorithm parameter settings: Some necessary parameters are required before the algorithm starts. 1) Similarity score s(a,b): It is determined according to the similarity threshold of words. For example, when the similarity threshold is set to 95%, a = "running" and b = "jogging", a and b are similar, and the s value is positive. If the similarity threshold is set to 99.9%, then a and b are no longer similar, and the s value is negative. 2) Gap penalty Wk (k is the length of the gap penalty): Since the two texts being compared usually have different lengths, the length difference of the texts to be compared is considered to be caused by multiple character insertions or deletions in one of the texts. In other words, two texts of the same original length, due to several character deletions or insertions, resulting in different lengths, are called gaps. If two texts of the same length are compared, there are no gaps. The appearance of a gap requires a corresponding deduction of points, which is the gap penalty. 3) Similarity threshold: The threshold for determining whether the meanings of two words are similar. For example, if this value is set to 95%, it means that the similarity of the two words reaches 95% to be considered the same semantics. 4) Maximum similar paragraph threshold: In the detection results, the detection paragraphs greater than or equal to the maximum similar paragraph threshold will be output as similar segments (for example, in some paper plagiarism detection systems, it is default that 13 consecutive identical characters are considered a repetition, and less than this number of characters is not considered a repetition). For example, [After school, I was running on campus…] and [After class, I ran out of the classroom…], in the paragraphs, [I was running] and [I ran] were detected as similar segments, but only 3 characters are similar, which does not reach the maximum similar number of characters and is not defined as similar.

[0033] ③ Text tokenization: Tokenize the input text. For example, tokenize: [After school, the students walked out of the classroom one after another.] and [When the class bell rang, the students all ran out of the classroom.]: ["After school", "the", "students", "walked", "out", "of", "the", "classroom", "."] ["When the class bell", "rang", "the", "students", "all", "ran", "out", "of", "the", "classroom", "."] ④ Initialize the score matrix: Still using the above example, form a matrix with the two tokenized texts, as Figure 4 shown. The first row and the first column of the matrix are initialized to 0, only as boundaries and not counted for scores. The other rows and columns are also initialized to 0, but the scores are updated through subsequent calculations. In addition, there are no negative scores in the Smith-Waterman algorithm. When the score in a cell is negative, it is still counted as 0.

[0034] ⑤ Calculate scores: In the initialized score matrix, starting from the cell (1,1), from left to right and from top to bottom, calculate the scores of each cell in turn. The scoring rules are as follows: Let the coordinates of the current cell be [i,j].

[0035] The final score for each cell is the maximum of the following values: 1) Similarity score s(a,b) of the current cell: Assume that the current setting s(a,b)=±3, so if the word corresponding to the current cell is detected as similar, the score is 3, otherwise it is -3.

[0036] 2) The score of the cell to the upper left of the current cell is H i-1 , j-1 +The similarity score of the current cell s(a,b) 3) The scores of the cells to the left and above the current cell are Max{H i,j-1 - Gap Penalty W , H i-1,j - gap penalty W}, for the position of the gap penalty, the formula proposed in the present invention is: W = W0- k Δ W (when k>20, k=20) where W0 is the initial vacancy penalty and k is the number of consecutive matches. Continuous matches refer to the number of consecutive positive similarity scores s(a,b) of the cell to its upper left starting from a cell. Assuming that W0 is currently set to 4, Δ W is set to 0.1.

[0037] 4) After calculating the above scores, take the maximum value as the current maximum score.

[0038] 5) When a value is calculated to be negative, set the value to 0.

[0039] According to the above rules, the score matrix is ​​calculated, such as Figure 5As shown. In the figure, three typical cell scores are marked in red: At ①, the score of this position is calculated. First, the similarity score s(a,b) of the current cell is calculated. Since they are the same, it is recorded as 3 points. Then the scores of the left and top of the current cell are calculated minus the corresponding gap penalty. Since the scores of the left and top are both 0, the gap penalty of 4 is negative. Finally, the similarity score of this position is calculated plus the score of the upper left cell, which is still 3 points. The maximum value is 3 points, so the score of this cell is 3. At ②, the other calculation methods remain unchanged, but since the upper left of this position is successfully matched, k=1 in the gap penalty formula when calculating this position, because the top is successfully matched once. Therefore, the gap penalty W of this position is 4 - 1*0.1, which is 3.9. However, since the scores of the left and top of this position are both 0, the gap penalty of 3.9 is negative. Since the word at this position is not matched, the score is -3. Then calculate the score of the upper left room of this position plus the score of the current position, which is 3+(-3)=0. After taking the maximum value, the score of this position is 0. At ③, calculate the maximum value in the same way as above. First, calculate the score of this position after the vacancy penalty based on the cells on the left and above, which is 15-4=11 points on the left and 4-4=0 points on the right. Then calculate the sum of the score of the upper left cell and the current score, which is 8+(-3)=5 points. Therefore, take the maximum value of 11 points as the score of this position.

[0040] ⑥ Backtracking: Through calculation, we get a score matrix with scores. In this matrix, we start from the cell with the highest score and traverse to the upper left in sequence according to the size of the score until we stop at 0, and finally get the fragment with the greatest local similarity. Figure 6 As shown, after the search stops, the upper left corner is the starting position of the similar segment, and the lower right corner is the ending position. The two similar segments are: [, the students walked out of the classroom one after another. ], [, the students all ran out of the classroom. ] It should be noted that the similar segments in this example are searched from the lower right corner to the upper left corner (because the maximum score happens to appear in the diagonal direction) until the end. This situation shows that there is no space between the segments, that is, there is no operation of adding or deleting characters. If during the search process, the search arrow moves up or to the left, it means that the word at that position has not been matched, and the position is supplemented with "-". For example, if the above paragraph is modified to [After school, the students walked out of the classroom happily one after another. ], the final similar paragraph output is: [, the students walked out of the classroom happily one after another. ], [, the students all- ran out of the classroom. ] In addition, if you want to match multiple paragraphs with local similarities in two long texts, you can loop the above search method. But it should be noted that if the position has been matched, it cannot be matched again. As Figure 7 As shown, the region has been detected to have similar fragments, and when detected again, the region cannot be detected repeatedly.

[0041] The data structure of a cell is as follows Figure 8 As shown, each cell needs to record two data: Number of consecutive matches: As mentioned in the above description, this number is initially 0. The size of this value depends on whether the cell itself is successfully matched, and the size of the value in the cell to its upper left. If the cell itself is successfully matched, it is 1, and if it is not matched, it is 0. The value of the value of the cell to its upper left plus the size of the cell to its upper left is the number of consecutive matches of the current cell. In the above description, it is also introduced that the number of consecutive matches is generated in order to calculate the vacancy penalty value. It is a parameter proposed and used in this scheme, and there is no such value in the original algorithm. Score: the score of the cell.

[0042] The score matrix is ​​a two-dimensional group, and each element is the above-mentioned cell structure. Therefore, the score matrix not only saves the data in the cell, but also saves the cell position information through its subscript.

[0043] The data structure of global data is as follows Fig. 9 As shown, during the operation of the algorithm, some global data needs to be saved to ensure the normal operation of the algorithm. Score and position: During the calculation of the score matrix, the size of the score and the subscript of the cell are dynamically recorded to facilitate the rapid finding of the location of the maximum score when backtracking. Undetectable area: Similar text fragments that have been detected, record their starting coordinates, and the rectangular area formed by their starting coordinates is the area that cannot be repeatedly detected in subsequent detections. Similar fragment collection: The detected similar fragments are temporarily stored and uniformly output after the algorithm ends. The program running process is as follows Fig.10 shown.

[0044] In some embodiments, the S4 further comprises the following steps: The similar fragment results are compared with the terms in the term library, and the term density is calculated. Term density = number of terms in the similar fragment results / total number of words in the similar fragment results. Similar fragment results with term density greater than the preset threshold are deleted to obtain the final similar fragment.

[0045] In some embodiments, the scoring rule is as follows: the final score of each cell is taken from the maximum value of the similarity score of the current cell, the similarity score of the cell to the upper left of the current cell plus the similarity score of the current cell, the similarity score of the cell to the left of the current cell minus the gap penalty, and the similarity score of the cell above the current cell minus the gap penalty. If any calculated value is negative, the value is set to 0; Assume that the similarity score s(a,b)= ±N. If the word corresponding to the current cell is detected as similar, the score is +N, otherwise it is -N. Where a and b represent the two segmentation elements corresponding to the same cell in the segmentation matrix formed after the two segments of the text to be detected are segmented. The vacancy penalty W = W0- k Δ W, where W0 is the initial vacancy penalty; k is the number of consecutive matches to similarity, that is, starting from a cell, the number of consecutive positive similarity scores s(a,b) of its upper left cell. When k is greater than the preset maximum number, the preset maximum number is taken; Δ W is the linear reduction coefficient of the gap penalty obtained according to the similar number of consecutive matches.

[0046] In this embodiment, after the algorithm process is executed, several similar fragment results will be obtained, but this is not the final result, because it is also possible that two fragments containing many professional terms are repeated. If this is the case, they cannot be finally judged as similar. Therefore, according to the results obtained by the duplicate checking algorithm, it is necessary to compare them with the terms in the term library and output the final result. Fig.11 As shown. The similar segments output by the algorithm are compared with the terms in the term library to calculate the term density. Term density = (number of terms in the segment) / (total number of words in the segment) If the term density is fixed at 50%, similar segments exceeding the threshold are discarded, and the remaining segments are considered to be the final similar segments and output. After the term library comparison, the final similar segments are obtained. Since their position information in the original report has been maintained before, their positions in the original text can be easily located, highlighted, and the repetition rate calculated to generate the final duplicate check report.

[0047] The second aspect of the present invention provides: a text duplication checking system, used to implement any of the above-mentioned text duplication checking methods, comprising: Custom module, used to customize the experiment report template according to the experimental project; The experimental report generation module is used to generate and upload the experimental report according to the experimental report template; The duplicate checking module is used to extract the text content that needs to be checked for duplicates in the experimental report according to the duplicate checking mark, and use the algorithm to check for duplicates to obtain several similar fragment results; A term screening module is used to screen similar fragment results using a term library to obtain final similar fragments; The duplicate checking report generation module is used to locate the final similar fragments, highlight them, calculate the repetition rate, and generate the final duplicate checking report.

[0048] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned text duplication checking methods is implemented.

[0049] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A text duplication checking method, characterized by: The following steps are involved: S1: The administrator customizes the experiment report template according to the experimental project; S2: The user generates and uploads the experimental report according to the experimental report template; S3: The system extracts the text content that needs to be checked for plagiarism in the experimental report based on the plagiarism check mark, and uses the algorithm to check for plagiarism to obtain several similar fragment results; S4: Use the term library to filter the similar fragment results to obtain the final similar fragments; S5: Locate the final similar segments, highlight them, calculate the repetition rate, and generate the final duplication check report.

2. The text duplication checking method according to claim 1, characterized in that: The S1 further comprises the following steps: Enter the terminology set for this experimental project; Add the components of this experimental report; Define the repetition rate threshold of each component module; Generate the anti-counterfeiting mark of this experimental report template; Add text content range markers for the component modules that need to be checked for plagiarism; Generate the document directory for each component module and set it as non-editable to obtain the report template for this experiment.

3. The text duplication checking method according to claim 1, characterized in that: The S2 further comprises the following steps: The system performs a compliance check on the experimental report uploaded by the user, and executes S3 only after the check passes.

4. The text duplication checking method according to claim 1, characterized in that: The S3 further comprises the following steps: Enter the text content to be detected; Set algorithm parameters, including similarity score, gap penalty, similarity threshold, and maximum similar paragraph threshold; Use the Chinese word segmenter to segment the current Chinese text and initialize the score matrix; Create a cell instance object and fill it into the two-dimensional array of the score matrix; According to the scoring rules, each cell is scored from left to right and from top to bottom. When scoring, the Chinese word vector model is used to judge the semantic similarity of the words. At the same time, the maximum score in the global data and its cell position are updated. Backtracking is performed. When backtracking, the cell with the maximum score is found in the global data, and it is checked whether it is in the undetectable area. If it is, the backtracking is abandoned and the cell information is removed from the global data. If it is in the detectable area, backtracking is performed to obtain similar text fragments, and they are put into the similar fragment set in the global data, and the cell information is removed from the global data at the same time. According to the maximum similar paragraph threshold, from the similar segment set of global data, segments that meet the similarity length are taken out to obtain several similar segment results and output them.

5. The text duplication checking method according to claim 1, characterized in that: The S4 further comprises the following steps: The similar fragment results are compared with the terms in the term library, and the term density is calculated. Term density = number of terms in the similar fragment results / total number of words in the similar fragment results. Similar fragment results with term density greater than the preset threshold are deleted to obtain the final similar fragment.

6. The text duplication checking method according to claim 4, characterized in that: The scoring rule is as follows: the final score of each cell is taken from the maximum value of the similarity score of the current cell, the similarity score of the cell to the upper left of the current cell plus the similarity score of the current cell, the similarity score of the cell to the left of the current cell minus the gap penalty, and the similarity score of the cell above the current cell minus the gap penalty. If any calculated value is negative, the value is set to 0; Assume that the similarity score s(a,b)= ±N. If the word corresponding to the current cell is detected as similar, the score is +N, otherwise it is -N. Where a and b represent the two segmentation elements corresponding to the same cell in the segmentation matrix formed after the two segments of the text to be detected are segmented. The vacancy penalty W = W0- k Δ W, where W0 is the initial vacancy penalty; k is the number of consecutive matches to similarity, that is, starting from a cell, the number of consecutive positive similarity scores s(a,b) of its upper left cell. When k is greater than the preset maximum number, the preset maximum number is taken; Δ W is the linear reduction coefficient of the gap penalty obtained according to the similar number of consecutive matches.

7. A text duplication checking system, characterized by: The method for implementing the text duplication checking method according to any one of claims 1 to 6 comprises: Custom module, used to customize the experiment report template according to the experimental project; The experimental report generation module is used to generate and upload the experimental report according to the experimental report template; The duplicate checking module is used to extract the text content that needs to be checked for duplicates in the experimental report according to the duplicate checking mark, and use the algorithm to check for duplicates to obtain several similar fragment results; A term screening module is used to screen similar fragment results using a term library to obtain final similar fragments; The duplicate checking report generation module is used to locate the final similar fragments, highlight them, calculate the repetition rate, and generate the final duplicate checking report.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the text duplication checking method as described in any one of claims 1-6 is implemented.

Images