A paper duplicate checking system and method based on text feature extraction technology

Abstract

The invention discloses a paper duplicate checking system and method based on text feature extraction technology, and relates to the technical field of data processing. The invention performs word segmentation operation on the paper to be checked for duplicate, so that the information in the paper to be checked for duplicate can be segmented, and a query is performed in a database document set according to the information, so that the queried documents have certain similarities in content with the paper to be checked for duplicate, and the weight of each word segmentation in the paper word segmentation set is determined, so that the calculated repetition degree is more accurate, and further analysis and judgment are performed, and the paper to be checked for duplicate and each paper in the database document set are deeply analyzed from the aspects of field, problem solved and solution method. When these three aspects are repeated with other papers, it is determined that the paper to be checked for duplicate is repeated, otherwise, the paper to be checked for duplicate is not repeated, thereby improving the accuracy of checking for duplicate.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic digital data processing, and in particular, relates to a paper duplication checking system and method based on text feature extraction technology. Background Art

[0002] Paper duplicate checking refers to the use of specialized software or algorithms to compare and analyze a paper with existing academic literature, online resources, and other relevant literature to detect parts in the paper that are similar or duplicated with content published by others. The main purpose of paper duplicate checking is to prevent academic misconduct, especially plagiarism. By checking papers for duplicate content, it can help ensure the originality and academic integrity of academic research and promote the healthy development of the academic community. Paper duplicate checking usually compares the similarity and repetition of texts. In general, the duplicate checking system will compare the paper to be tested with the literature in the database, detect similar text paragraphs, sentences or words, and give a similarity matching result. Paper duplicate checking is a very important part of academia and educational institutions. It helps maintain the seriousness and reliability of academia and protects the efforts and achievements of researchers.

[0003] Chinese patent application CN108897781A discloses a paper graphics duplication checking system, including: a paper database; an acquisition module, which acquires the graphics contained in each paper in the paper database, extracts the blank closed area of ​​the graphics, and further extracts the edge contour of the blank closed area, repeats the extraction operation, obtains the regional contour corresponding to each graphic, and establishes a regional contour database; the acquisition module also acquires the target graphics in the target paper, and uses the above method to extract the regional contour of the target graphics, and a comparison module, which compares the regional contour of the target graphics with all regional contours in the regional contour database, calculates the similarity, and if the similarity is higher than 0.6, the similarity is marked near the target graphics in the target paper. The present invention can quickly check the graphics of the target paper for duplication, and the accuracy of checking duplication is high.

[0004] Existing paper duplication checking tools mainly detect whether there are similar parts in the paper by comparing existing literature databases and content on the Internet, so that existing paper duplication checking tools mainly focus on detecting text similarity. However, even if two articles do not have similar sentences or paragraphs, their research fields, problems solved and ways of solving problems may be similar. Therefore, it is impossible to fully determine whether there is real duplication through existing paper duplication checking tools, and thus it is impossible to achieve a real duplication checking effect. Summary of the invention

[0005] In view of the problems in the related art, the present invention proposes a paper duplication checking system and method based on text feature extraction technology to overcome the above-mentioned technical problems existing in the existing related technology.

[0006] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] The present invention is a method for checking for duplicate papers based on text feature extraction technology, comprising the following steps:

[0008] S1. Input the paper to be checked for duplicates, perform word segmentation on the paper to be checked for duplicates, and obtain a paper word segmentation set; set a database document set, query the database according to the paper word segmentation set, obtain the word segmentation contained in the paper word segmentation set for each document in the database document set, and form a document containing word segmentation matrix;

[0009] S2. According to the paper word segmentation set, the weight value of the corresponding word is set to obtain the word segmentation weight set, and the document inclusion word segmentation weight matrix is ​​obtained according to the word segmentation weight set. According to the document inclusion word segmentation weight matrix, the repetition between the document and the paper to be checked for duplicates is calculated to obtain the document repetition set; the repetition threshold is set, and the document repetition set is compared and analyzed with the repetition threshold. When there is a document in the document repetition set with a repetition greater than or equal to the repetition threshold, it is determined that the paper to be checked for duplicates is duplicated with the papers in the database document set; otherwise, an initial domain SVM model, an initial problem SVM model and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model and a secondary optimized solution SVM model;

[0010] S3, using the twice-optimized domain SVM model, the twice-optimized problem SVM model, and the twice-optimized solution SVM model to classify the papers to be checked for duplicates and the database document set, to obtain a first classification result set and a second classification result set, respectively;

[0011] S4, comparing the first classification result set with the second classification result set, and determining whether the paper to be checked for duplicates is duplicated according to the comparison result;

[0012] First, the paper to be checked for duplicate content is segmented to obtain a paper segmentation set, so that the information in the paper to be checked for duplicate content can be divided. Based on this information, a query is performed in the database document set, so that the queried documents have a certain similarity in content with the paper to be checked for duplicate content. Since the importance of each word in the paper to the paper is different, the weight of each segmentation in the paper segmentation set is determined to obtain a segmentation weight set, and then the repetition degree of the paper to be checked for duplicate content is calculated, so that the calculated repetition degree is more accurate. When the repetition degree exceeds the repetition degree threshold, it is determined that the paper to be checked for duplicate content is repeated with the document in the database document set. When the repetition degree does not exceed the repetition degree threshold, since whether the text part is repeated cannot accurately reflect whether the paper to be checked for duplicate content is repeated, further analysis is required. and judgment, which improves the accuracy of duplicate checking; by constructing the initial domain SVM model, the initial problem SVM model and the initial solution SVM model, and performing primary optimization and secondary optimization, the secondary optimized domain SVM model, the secondary optimized problem SVM model and the secondary optimized solution SVM model are obtained. The papers to be checked for duplicates and each paper in the database document collection can be deeply analyzed in terms of the field, the problems solved and the solutions. When these three aspects are repeated in other papers, it is determined that the paper to be checked for duplicates is repeated, otherwise, the paper to be checked for duplicates is not repeated. Through the above-mentioned duplicate checking scheme, the paper can be deeply checked for duplicates, so that the results of the duplicate checking can better reflect the true situation of the paper, making the results more accurate and of reference value.

[0013] Preferably, the S1 comprises the following steps:

[0014] S11, perform word segmentation operation on the paper to be checked for duplicates, and obtain a paper word segmentation set c″={c″1,c″2,...,c″ i ,...,c″ c′};

[0015] S12, set the database document set c = {c1, c2, ..., c i ,...,c c″′},c i represents the i-th document in the database, c″′ represents the total number of documents in the database; according to the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′} Query the database and obtain the database document set c = {c1, c2, ..., c i ,...,c c″′ Each document in} contains the word set c″={c″1,c″2,...,c″ i ,...,c″ c′}, the word segmentation matrix d of the document is as follows:

[0016]

[0017] Among them, d ii Represents the database document set c = {c1, c2, ..., c i ,...,c c″ The i-th participle contained in the i-th document in};

[0018] By performing word segmentation on the papers to be checked for plagiarism, a data basis is provided for the subsequent first round of plagiarism checking.

[0019] Preferably, the S11 comprises the following steps:

[0020] S111, divide the paper to be checked for duplicate content into paragraphs to obtain a paper paragraph set a = {a1, a2, ..., a i ,...,a a′}, a i represents the ith paragraph divided from the paper to be checked for plagiarism, a′ represents the total number of paragraphs divided from the paper to be checked for plagiarism; the paper paragraph set a={a1, a2, ..., a i ,...,a a′} into strings, and obtain the paragraph string set a″={a″1,a″2,...,a″ i ,...,a″ a},a″ i ″ represents the string corresponding to the ith paragraph divided from the paper to be checked for duplicates, a″ ij Indicates a″ i The jth character in a, a″′ i Indicates a″ i The total number of characters in ;

[0021] S112, constructing a Chinese character index table, and searching for a" according to the Chinese character index table i0 Is it a single word? i0 For single-word words, replace a″ i0 into words, and then to a″ i1 Make a query judgment; when a″ i0 If it is not a single-word word, all characters starting with a″ are searched according to the Chinese character index table. i0 The word set with prefix is ​​used to get the first word set b j Indicates a″ i0 is the jth word of the prefix, Indicates a″ i0The total number of words with prefix;

[0022] S113, a″ i0 and the next character a″ i1 Construct string a″ i0 a″ i1 , change a″ i0 a″ i1 With the first word set Match the first two characters of all words in the string. If the first two characters match a" i0 a″ i1 Same and total length as a″ i0 a″ i1 If the string is the same, replace a″ i0 a″ i1 Otherwise, a″ i0 Word formation;

[0023] S114, for the paragraph character string set a″={a″1,a″2,...,a″ i ,...,a″ a′ Repeat the operations in S112 and S113 for each paragraph string in} to obtain the paragraph segmentation matrix b″, as follows:

[0024]

[0025] Among them, b″′ ij Indicates a″ i The jth word obtained after word segmentation, b″′ i Indicates a″ i The total number of segmented words obtained after segmentation;

[0026] S115, perform a deduplication operation on each word in the paragraph word segmentation matrix b″ to obtain a paragraph word segmentation matrix after deduplication; then merge the paragraph word segmentation matrices after deduplication to obtain a paper word segmentation set c″={c″1,c″2,...,c″ i ,...,c″ c′};

[0027] By comparing and matching each character of the paper to be checked for plagiarism with the Chinese character index table in the above manner, the paper to be checked for plagiarism can be accurately segmented, providing data support for subsequent queries.

[0028] Preferably, S2 comprises the following steps:

[0029] S21, according to the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′} Set the weight value of the corresponding word segmentation to obtain the word segmentation weight set. According to the word segmentation weight set d″′={d″′1,d″′2,...,d″′ i ,...,d″′ c″} and the document inclusion word segmentation matrix d to obtain the document inclusion word segmentation weight matrix e′, and calculate the duplication between the document and the paper to be checked for duplicates according to the document inclusion word segmentation weight matrix e′ to obtain a document duplication set;

[0030] S22, setting a duplication threshold d', performing comparative analysis based on the document duplication set and the duplication threshold d', when there is a document in the document duplication set with a duplication greater than or equal to d', it is determined that the paper to be checked for duplicates is duplicated with the papers in the database document set; otherwise, constructing an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model, and performing primary optimization and secondary optimization to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model, and a secondary optimized solution SVM model;

[0031] Since the importance of each word in the paper is different, the weight of each word in the paper word set is determined to obtain the word weight set, and then the duplication degree of the paper to be checked is calculated, so that the calculated duplication degree is more accurate, and the judgment is more accurate.

[0032] Preferably, the S21 comprises the following steps:

[0033] S211, according to the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′} Set the weight value of the corresponding word segmentation to obtain the word segmentation weight set d″′={d″′1,d″′2,...,d″′ i ,...,d″′ c″},d″′ i Represents the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′ The weight value of the i-th word in};

[0034] S212, according to the word segmentation weight set d″′={d″′1,d″′2,...,d″′ i ,...,d″′ c″} and the document inclusion word segmentation matrix d to obtain the document inclusion word segmentation weight matrix e′, as follows:

[0035]

[0036] Among them, e′ iiRepresents the database document set c = {c1, c2, ..., c i ,...,c c″ The weight value of the i-th word contained in the i-th document in};

[0037] S213, according to the word segmentation weight matrix e′ contained in the document, the duplication between the document and the paper to be checked for duplicates is calculated to obtain the document duplication set e={e1, e2, ..., e i ,...e c″′},e i Indicates the degree of duplication between the i-th document and the paper to be checked; the calculation formula is as follows:

[0038]

[0039] The degree of repetition between the paper to be checked for duplicate content and the document is calculated based on the weight of each segmentation word and the segmentation word situation contained in the database document set, so that the calculation result is more accurate and can more accurately reflect the actual repetition situation of the paper to be checked for duplicate content and the database document set.

[0040] Preferably, in S22, an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model, and a secondary optimized solution SVM model; specifically, the following steps are included:

[0041] S221, randomly collect paper training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′}; e″ i represents the i-th paper training sample in the paper training sample set, e″′ represents the total number of paper training samples in the paper training sample set; h″ i represents the i-th paper test sample in the paper test sample set;

[0042] From the paper training sample set e″={e″1,e″′2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′} randomly select f% of the paper training samples and paper test samples as the field training sample set And domain test sample set Then from the training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′}, respectively select f″′% of the paper training samples and paper test samples as the problem training sample set and problem test sample set Then from the training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′}, respectively select g′% of the paper training samples and paper test samples as the solution training sample set and solution test sample set e″ i , l i , f′ i , l″ i , g″ i and l″′ i They respectively represent the i-th domain training sample in the domain training sample set, the i-th domain test sample in the domain test sample set, the i-th problem training sample in the problem training sample set, the i-th problem test sample in the problem test sample set, the i-th solution training sample in the solution training sample set, and the i-th solution test sample in the solution test sample set; f″1, f″2, f″3, l′1, l′2, and l′3 respectively represent the total number of domain training samples in the domain training sample set, the total number of domain test samples in the domain test sample set, the total number of problem training samples in the problem training sample set, the total number of problem test samples in the problem test sample set, the total number of solution training samples in the solution test sample set, and the total number of solution training samples in the solution test sample set;

[0043] According to the domain training sample set Set the corresponding domain training sample label set According to the problem training sample set Set the corresponding problem training sample label set The sample set is trained according to the solution Set the corresponding solution training sample label set

[0044] Test sample sets according to the described domain Set the corresponding domain test sample label set Test the sample set according to the stated problem Set the corresponding problem test sample label set Test sample set according to the described solution Set the corresponding solution test sample label set

[0045] S222: construct an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model, and set the kernel functions of the initial SVM model, the initial problem SVM model, and the initial solution SVM model to be Gaussian kernel functions, and the function expressions are as follows:

[0046] m″′(n,n′)=exp(-α||nn′|| 2 )

[0047] Where: n and n′ represent two different data sample points, α represents the kernel parameter;

[0048] S223, set the first domain training error threshold n", the first problem training error threshold n"' and the first solution training error threshold o; And domain training sample label set Input into the initial domain SVM model for training to optimize the penalty factor parameter β in the initial domain SVM model and the distribution determination parameter χ after the data is mapped to the new feature space; when the training error of the domain training sample set and the domain training sample label set in the initial domain SVM model is less than the first domain training error threshold n″, stop training and obtain the optimized β, recorded as δ, the optimized χ, recorded as ε, and the optimized domain SVM model;

[0049] The problem training sample set and the problem training sample label set Input into the initial problem SVM model for training to optimize the penalty factor parameter β′ in the initial problem SVM model and the distribution determination parameter χ′ after the data is mapped to the new feature space; when the training error of the problem training sample set and the problem training sample label set in the initial problem SVM model is less than the first problem training error threshold n″′, stop training, and obtain the optimized β′, recorded as δ′, the optimized χ′, recorded as ε′, and the optimized problem SVM model;

[0050] The solution training sample set And solution training sample label set Input into the initial solution SVM model for training to optimize the penalty factor parameter β″ in the initial solution SVM model and the distribution determination parameter χ″ after the data is mapped to the new feature space; when the training error of the solution training sample set and the solution training sample label set in the initial solution SVM model is less than the first solution training error threshold o, stop training, and obtain the optimized β″, recorded as δ″, the optimized χ″, recorded as ε″ and the optimized solution SVM model;

[0051] S224: The field test sample set And the domain test sample label set m = {m1,m2,...,m i ,...,m l1′}Input into the domain SVM model after the primary optimization, perform secondary optimization on the domain SVM model after the primary optimization, and obtain the domain SVM model after the secondary optimization;

[0052] The problem test sample set And the problem test sample label set Input into the problem SVM model after the primary optimization, perform secondary optimization on the problem SVM model after the primary optimization, and obtain the problem SVM model after the secondary optimization;

[0053] The solution was tested on a sample set And solution test sample label set Input into the solution SVM model after the primary optimization, perform secondary optimization on the solution SVM model after the primary optimization, and obtain the solution SVM model after the secondary optimization;

[0054] By collecting paper training sample sets and paper test sample sets, and dividing these two sample sets into three parts, respectively used as training data for classification in terms of fields, training data for classification in terms of problems to be solved, training data for classification in terms of solutions, test data for classification in terms of fields, test data for classification in terms of problems to be solved, and test data for classification in terms of solutions, data support is provided for the subsequent primary and secondary optimization of the three SVM models to obtain a more accurate SVM model.

[0055] Preferably, the S224 comprises the following steps:

[0056] S2241. Constructing the first ant colony The second ant colony and the third ant colony o′ i ,o″ i and o″ idenote the i-th ant in the first ant colony, the i-th ant in the second ant colony, and the i-th ant in the third ant colony, respectively; p1, p2, and p3 denote the size of the first ant colony, the size of the second ant colony, and the size of the third ant colony, respectively;

[0057] At the same time, set the first maximum number of iterations p′, the second maximum number of iterations p″ and the third maximum number of iterations p″′; set the first current number of iterations r, the second current number of iterations r′ and the third current number of iterations r″;

[0058] Set the pheromone importance factor to r″′ and the heuristic function importance factor to s;

[0059] S2242: Setting the first ant colony population The initial position of 50% of the ants is δ, and the initial position of the other 50% of the ants is ε; set the second ant colony population The initial position of 50% of the ants is δ′, and the initial position of the other 50% of the ants is ε′; set the third ant colony population The initial position of 50% of the ants is δ″, and the initial position of the other 50% of the ants is ε″;

[0060] S2243, setting the initial concentration of pheromones released by each individual ant in the first ant colony, the second ant colony and the third ant colony to s″;

[0061] The calculation formula for the movement probability of the first ant colony in each iteration is set as follows:

[0062]

[0063] Among them, s″1(r) represents the movement probability of the first ant colony in the rth iteration, s″′ 1r It represents the average error value of the first ant colony classifying the domain test sample set and the domain test sample label set using the optimized domain SVM model in the rth iteration;

[0064] The calculation formula for the movement probability of the second ant colony in each iteration is set as follows:

[0065]

[0066] Among them, s″2(r′) represents the movement probability of the second ant colony in the r′th iteration, s″′ 2r′ It represents the average error value of the second ant colony classifying the problem test sample set and the problem test sample label set using the optimized problem SVM model in the r′th iteration;

[0067] The calculation formula for the movement probability of the third ant colony in each iteration is set as follows:

[0068]

[0069] s″3(r″) represents the movement probability of the third ant colony in the r″th iteration, s″′ 3r″ It represents the average error value of the third ant colony classifying the solution test sample set and the solution test sample label set using the optimized solution SVM model in the r″th iteration;

[0070] S2244, starting an iteration process for the first ant colony, the second ant colony, and the third ant colony according to the movement probability calculation formula of the first ant colony in each iteration, the movement probability calculation formula of the second ant colony in each iteration, and the movement probability calculation formula of the third ant colony in each iteration; specifically as follows:

[0071] The first ant colony population obtains the first optimal position and the second optimal position in each round of iteration, the first optimal position represents the optimal value of the penalty factor parameter in the domain SVM model, and the second optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the domain SVM model; the second ant colony population obtains the third optimal position and the fourth optimal position in each round of iteration, the third optimal position represents the optimal value of the penalty factor parameter in the problem SVM model, and the fourth optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the problem SVM model; the third ant colony population obtains the fifth optimal position and the sixth optimal position in each round of iteration, the fifth optimal position represents the optimal value of the penalty factor parameter in the solution SVM model, and the sixth optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the solution SVM model;

[0072] S2245. When r≥p′, the first ant colony population stops iterating, and obtains a first final penalty factor parameter t and a distribution determination parameter u after the first final data is mapped to a new feature space;

[0073] When r′≥p″, the second ant colony population stops iterating, and obtains the second final penalty factor parameter t′ and the distribution determination parameter u′ after the second final data is mapped to the new feature space;

[0074] When r″≥p″′, the third ant colony population stops iterating, and obtains a third final penalty factor parameter t″ and a distribution determination parameter u″ after the third final data is mapped to a new feature space;

[0075] S2246, using the first final penalty factor parameter t and the distribution determination parameter u after the first final data is mapped to the new feature space as parameters of the domain SVM model after the primary optimization, to obtain the domain SVM model after the secondary optimization;

[0076] The second final penalty factor parameter t′ and the distribution determination parameter u′ after the second final data is mapped to the new feature space are used as parameters of the problem SVM model after the primary optimization to obtain the problem SVM model after the secondary optimization;

[0077] The third final penalty factor parameter t″ and the distribution determination parameter u″ after the third final data is mapped to the new feature space are used as parameters of the solution SVM model after the primary optimization to obtain the solution SVM model after the secondary optimization;

[0078] By using the ant colony algorithm to simultaneously optimize the penalty factor parameters in the SVM model and the distribution determination parameters after the data is mapped to the new feature space, the speed of SVM optimization is improved, and the optimization process is made more accurate and less likely to fall into the local optimal situation.

[0079] Preferably, S3 comprises the following steps:

[0080] S31, the database document set c={c1,c2,...,c i ,...,c c″′ Each document in} is input into the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model for classification operation, and the first classification result set t″′={t″′1,t″′2,...,t″′ i ,...,t″′ c″′}, where t″′ i represents the result set of classifying the i-th document in the database document set, t″′ i ={t″′ i1 ,t″′ i2 ,t″′ i3}, t″′ i1 , t″′ i2 and t″′ i3 They represent the classification results of the i-th document in the database document set in the domain SVM model after quadratic optimization, the problem SVM model after quadratic optimization, and the solution SVM model after quadratic optimization respectively;

[0081] S32, inputting the papers to be checked for duplicates into the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model for classification operations, and obtaining a second classification result set u″′={u″′1,u″′2,u″′3};

[0082] The optimized SVM model is used to classify each document in the database document set and the papers to be checked for duplicates, making the classification more accurate and providing an accurate basis for subsequent comparative analysis.

[0083] Preferably, the process of S4 is as follows:

[0084] The first classification result set t″′={t″′1, t″′2, ..., t″′ i ,...,t″′ c″′} is compared with the second classification result set u″′={u″′1,u″′2,u″′3}. If the first classification result set t″′={t″′1,t″′2,...,t″′ i ,...,t″′ c″′} exists in t″′ i ={t″′ i1 ,t″′ i2 ,t″′ i3}, so that t″′ i1 =u″′1、t″′ i2 =u″′2 and t″′ i3 =u″′3, it is determined that the paper to be checked for duplicates is duplicated with the paper in the database; otherwise, it is determined that the paper to be checked for duplicates is not duplicated with the paper in the database;

[0085] By comparing the three aspects of the paper field, the problem the paper solves and the way the paper solves the problem, the comparison is more comprehensive and the comparison results are more accurate.

[0086] A paper duplication checking system based on text feature extraction technology, comprising a paper word segmentation module, a document duplication calculation module, a first paper duplication determination module, an SVM model primary optimization module, an SVM model secondary optimization module, a paper classification module and a second paper duplication determination module;

[0087] The paper segmentation module is used to perform a segmentation operation on the paper to be checked for duplicates;

[0088] The document duplication calculation module calculates the duplication between the document and the paper to be checked for duplicates according to the word segmentation weight matrix contained in the document, and obtains a document duplication set;

[0089] The first paper duplication determination module compares and analyzes the document duplication set with the duplication threshold to determine whether the paper to be checked for duplication is duplicated;

[0090] The SVM model primary optimization module is used to optimize the initial domain SVM model, the initial problem SVM model and the initial solution SVM model;

[0091] The SVM model secondary optimization module is used to perform secondary optimization on the domain SVM model after the primary optimization, the problem SVM model after the primary optimization, and the solution SVM model after the primary optimization;

[0092] The paper classification module is used to classify the papers to be checked for duplicates and the document set in the database by using the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model;

[0093] The second paper duplication determination module is used to compare the first classification result set with the second classification result set, and determine whether the paper to be checked for duplication is duplicated based on the comparison result.

[0094] The present invention has the following beneficial effects:

[0095] 1. In the present invention, by setting a paper segmentation module, a document duplication calculation module, a first paper duplication determination module, an SVM model primary optimization module, an SVM model secondary optimization module, a paper classification module and a second paper duplication determination module, firstly, a word segmentation operation is performed on the paper to be checked for duplicate copies to obtain a paper segmentation set, so that the information in the paper to be checked for duplicate copies can be divided, and a query is performed in the database document set based on this information, so that the queried documents and the paper to be checked for duplicate copies have a certain similarity in content. Since the importance of each word in the paper to the paper is different, the weight of each segmentation in the paper segmentation set is determined to obtain a segmentation weight set, and then the duplication degree of the paper to be checked for duplicate copies is calculated, so that the calculated duplication degree is more accurate. When the duplication degree exceeds the duplication degree threshold, it is determined that the paper to be checked for duplicate copies is duplicated with the document in the database document set. When the duplication degree does not exceed the duplication degree threshold, Since whether the text part is repeated cannot accurately reflect whether the paper to be checked for duplicate content is repeated, further analysis and judgment are needed to improve the accuracy of the duplicate check; by constructing the initial domain SVM model, the initial problem SVM model and the initial solution SVM model, and performing primary optimization and secondary optimization, the secondary optimized domain SVM model, the secondary optimized problem SVM model and the secondary optimized solution SVM model are obtained. The paper to be checked for duplicate content and each paper in the database document collection can be deeply analyzed in terms of the field, the problem solved and the solution. When these three aspects are repeated with other papers, it is determined that the paper to be checked for duplicate content is repeated. Otherwise, the paper to be checked for duplicate content is not repeated. Through the above-mentioned duplicate checking scheme, the paper can be deeply checked for duplicate content, so that the result of the duplicate checking can better reflect the true situation of the paper, making the result more accurate and of reference value.

[0096] 2. In the present invention, each character of the paper to be checked for duplicates is compared and matched with the Chinese character index table, so that the paper to be checked for duplicates can be accurately segmented, providing data support for subsequent queries. At the same time, by collecting paper training sample sets and paper test sample sets, and dividing the two sample sets into three parts, respectively serving as training data classified for field aspects, training data classified for solved problems, training data classified for solution methods, test data classified for field aspects, test data classified for solved problems and test data classified for solution methods, data support is provided for subsequent primary optimization and secondary optimization of three SVM models to obtain a more accurate SVM model.

[0097] 3. In the present invention, the penalty factor parameters in the SVM model and the distribution determination parameters after the data is mapped to the new feature space are optimized simultaneously by using the ant colony algorithm, thereby improving the speed of SVM optimization and making the optimization process more accurate and less likely to fall into the local optimal situation.

[0098] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, they can also obtain drawings based on these drawings without paying creative work.

[0100] Figure 1 The present invention is a schematic diagram of the process of checking for plagiarism in a paper by a paper plagiarism checking system based on text feature extraction technology. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the invention to clearly and completely describe the technical solutions in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the invention.

[0102] In the description of the present invention, it is necessary to understand that the terms "opening", "upper", "lower", "top", "middle", "inside" and the like indicating orientation or positional relationship are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the invention.

[0103] The present invention is a method for checking for duplicate papers based on text feature extraction technology, comprising the following steps:

[0104] S1. Input the paper to be checked for duplicates, perform word segmentation on the paper to be checked for duplicates, and obtain a paper word segmentation set; set a database document set, query the database according to the paper word segmentation set, obtain the word segmentation contained in the paper word segmentation set for each document in the database document set, and form a document containing word segmentation matrix;

[0105] The S1 comprises the following steps:

[0106] S11, perform word segmentation operation on the paper to be checked for duplicates, and obtain a paper word segmentation set c″={c″1,c″2,...,c″i ,...,c″ c′};

[0107] The S11 comprises the following steps:

[0108] S111, divide the paper to be checked for duplicate content into paragraphs to obtain a paper paragraph set a = {a1, a2, ..., a i ,...,a a′}, a i represents the ith paragraph divided from the paper to be checked for plagiarism, a′ represents the total number of paragraphs divided from the paper to be checked for plagiarism; the paper paragraph set a={a1, a2, ..., a i ,...,a a′} into strings, and obtain the paragraph string set a″={a″1,a″2,...,a″ i ,...,a″ a′},a″ i Indicates the string corresponding to the ith paragraph divided from the paper to be checked for duplicates, a″ ij Indicates a″ i The jth character in a, a″′ i Indicates a″ i The total number of characters in ;

[0109] S112, constructing a Chinese character index table, and searching for a" according to the Chinese character index table i0 Is it a single word? i0 For single-word words, replace a″ i0 into words, and then to a″ i1 Make a query judgment; when a″ i0 If it is not a single-word word, all characters starting with a″ are searched according to the Chinese character index table. i0 The word set with prefix is ​​used to get the first word set b j Indicates a″ i0 is the jth word of the prefix, Indicates a″ i0 The total number of words with prefix;

[0110] S113, a″ i0 and the next character a″ i1 Construct string a″ i0 a″ i1 , change a″ i0 a″ i1 With the first word set Match the first two characters of all words in the string. If the first two characters match a" i0 a″i1 Same as a″ i0 a″ i1 If the string is the same, replace a″ i0 a″ i1 Otherwise, a″ i0 into words;

[0111] S114, for the paragraph character string set a″={a″1,a″2,...,a″ i ,...,a″ a′ Repeat the operations in S112 and S113 for each paragraph string in} to obtain the paragraph segmentation matrix b″, as follows:

[0112]

[0113] Among them, b″ ij Indicates a″ i The jth word obtained after word segmentation, b″′ i Indicates a″ i The total number of segmented words obtained after segmentation;

[0114] S115, perform a deduplication operation on each word in the paragraph word segmentation matrix b″ to obtain a paragraph word segmentation matrix after deduplication; then merge the paragraph word segmentation matrices after deduplication to obtain a paper word segmentation set c″={c″1,c″2,...,c″ i ,...,c″ c′};

[0115] S12, set the database document set c = {c1, c2, ..., c i ,...,c c″′}, c i represents the i-th document in the database, c″′ represents the total number of documents in the database; according to the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′} Query the database and obtain the database document set c = {c1, c2, ..., c i ,...,c c″′ Each document in} contains the word set c″={c″1,c″2,...,c″ i ,...,c″ c′}, the word segmentation matrix d of the document is as follows:

[0116]

[0117] Among them, d iiRepresents the database document set c = {c1, c2, ..., c i ,...,c c″ The i-th participle contained in the i-th document in};

[0118] S2. According to the paper word segmentation set, the weight value of the corresponding word is set to obtain the word segmentation weight set, and the document inclusion word segmentation weight matrix is ​​obtained according to the word segmentation weight set. According to the document inclusion word segmentation weight matrix, the repetition between the document and the paper to be checked for duplicates is calculated to obtain the document repetition set; the repetition threshold is set, and the document repetition set is compared and analyzed with the repetition threshold. When there is a document in the document repetition set with a repetition greater than or equal to the repetition threshold, it is determined that the paper to be checked for duplicates is duplicated with the papers in the database document set; otherwise, an initial domain SVM model, an initial problem SVM model and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model and a secondary optimized solution SVM model;

[0119] The S2 comprises the following steps:

[0120] S21, according to the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′} Set the weight value of the corresponding word segmentation to obtain the word segmentation weight set. According to the word segmentation weight set d″′={d″′1,d″′2,...,d″′ i ,...,d″′ c′} and the document inclusion word segmentation matrix d to obtain the document inclusion word segmentation weight matrix e′, and calculate the duplication between the document and the paper to be checked for duplicates according to the document inclusion word segmentation weight matrix e′ to obtain a document duplication set;

[0121] The S21 comprises the following steps:

[0122] S211, according to the paper word set c″={c″1,c″′2,...,c″ i ,...,c″ c′} Set the weight value of the corresponding word segmentation to obtain the word segmentation weight set d″′={d″′1,d″′2,...,d″′ i ,...,d″′ c′},d″′ i Represents the paper word set c″={c″1,c″2,...,c″ i ,...,c″ c′ The weight value of the i-th word in};

[0123] S212, according to the word segmentation weight set d″′={d″′1,d″′2,...,d″′ i ,...,d″′ c′} and the document inclusion word segmentation matrix d to obtain the document inclusion word segmentation weight matrix e′, as follows:

[0124]

[0125] Among them, e i ' i Represents the database document set c = {c1, c2, ..., c i ,...,c c″ The weight value of the i-th word contained in the i-th document in};

[0126] S213, according to the word segmentation weight matrix e′ contained in the document, the duplication between the document and the paper to be checked for duplicates is calculated to obtain the document duplication set e={e1, e2, ..., e i ,...e c″′},e i Indicates the degree of duplication between the i-th document and the paper to be checked; the calculation formula is as follows:

[0127]

[0128] S22, setting a duplication threshold d', performing comparative analysis based on the document duplication set and the duplication threshold d', when there is a document in the document duplication set with a duplication greater than or equal to d', it is determined that the paper to be checked for duplicates is duplicated with the papers in the database document set; otherwise, constructing an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model, and performing primary optimization and secondary optimization to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model, and a secondary optimized solution SVM model;

[0129] In S22, an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model, and a secondary optimized solution SVM model; specifically, the following steps are included:

[0130] S221, randomly collect paper training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′}; e″ irepresents the i-th paper training sample in the paper training sample set, e″′ represents the total number of paper training samples in the paper training sample set; h″ i represents the i-th paper test sample in the paper test sample set;

[0131] From the paper training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′} randomly select f% of the paper training samples and paper test samples as the domain training sample set Domain Test Sample Set Then from the training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″ i ,...,h″ h″′}, respectively take f″′% of the paper training samples and paper test samples as the problem training sample set and problem test sample set Then from the training sample set e″={e″1,e″2,...,e″ i ,...,e″ e″′} and the test sample set h″={h″1,h″2,...,h″i,...,h″ h″′}, respectively select g′% of the paper training samples and paper test samples as the solution training sample set and solution test sample set l i 、f i ', l" i , g″ i and l″′ iThey respectively represent the i-th domain test sample in the domain test sample set, the i-th domain training sample in the domain training sample set, the i-th problem training sample in the problem test sample set, the i-th solution training sample in the solution training sample set, and the i-th solution test sample in the solution test sample set; f″1, f″2, f″3, l′1, l′2, and l′3 respectively represent the total number of domain training samples in the domain training sample set, the total number of problem training samples in the problem training sample set, the total number of solution training samples in the solution training sample set, the total number of domain test samples in the domain test sample set, the total number of problem test samples in the problem test sample set, and the total number of solution test samples in the solution test sample set;

[0132] According to the domain training sample set Set the corresponding field training sample label set According to the problem training sample set Set the corresponding problem training sample label set The sample set is trained according to the solution Set the corresponding solution training sample label set

[0133] Test sample sets according to the described domain Set the corresponding domain test sample label set Test the sample set according to the stated problem Set the corresponding problem test sample label set Test sample set according to the described solution Set the corresponding solution test sample label set

[0134] S222: construct an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model, and set the kernel functions of the initial SVM model, the initial problem SVM model, and the initial solution SVM model to be Gaussian kernel functions, and the function expressions are as follows:

[0135] m″′(n,n′)=exp(-α||nn′|| 2 )

[0136] Where: n and n′ represent two different data sample points, α represents the kernel parameter;

[0137] S223, set the first domain training error threshold n", the first problem training error threshold n"' and the first solution training error threshold o; And domain training sample label set Input into the initial domain SVM model for training to optimize the penalty factor parameter β in the initial domain SVM model and the distribution determination parameter χ after the data is mapped to the new feature space; when the training error of the domain training sample set and the domain training sample label set in the initial domain SVM model is less than the first domain training error threshold n″, stop training and obtain the optimized β, recorded as δ, the optimized χ, recorded as ε, and the optimized domain SVM model;

[0138] The problem training sample set and the problem training sample label set Input into the initial problem SVM model for training to optimize the penalty factor parameter β′ in the initial problem SVM model and the distribution determination parameter χ′ after the data is mapped to the new feature space; when the training error of the problem training sample set and the problem training sample label set in the initial problem SVM model is less than the first problem training error threshold n″′, stop training, and obtain the optimized β′, recorded as δ′, the optimized χ′, recorded as ε′, and the optimized problem SVM model;

[0139] The solution training sample set And solution training sample label set Input into the initial solution SVM model for training to optimize the penalty factor parameter β″ in the initial solution SVM model and the distribution determination parameter χ″ after the data is mapped to the new feature space; when the training error of the solution training sample set and the solution training sample label set in the initial solution SVM model is less than the first solution training error threshold o, stop training, and obtain the optimized β″, recorded as δ″, the optimized χ″, recorded as ε″ and the optimized solution SVM model;

[0140] S224: The field test sample set And domain test sample label set Input into the domain SVM model after the primary optimization, perform secondary optimization on the domain SVM model after the primary optimization, and obtain the domain SVM model after the secondary optimization;

[0141] The problem test sample set And the problem test sample label set Input into the problem SVM model after the primary optimization, perform secondary optimization on the problem SVM model after the primary optimization, and obtain the problem SVM model after the secondary optimization;

[0142] The solution was tested on a sample set And solution test sample label set Input into the solution SVM model after the primary optimization, perform secondary optimization on the solution SVM model after the primary optimization, and obtain the solution SVM model after the secondary optimization;

[0143] The S224 comprises the following steps:

[0144] S2241. Constructing the first ant colony The second ant colony and the third ant colony o i ', o i ″ and o i ″′ respectively represent the i-th ant in the first ant colony, the i-th ant in the second ant colony, and the i-th ant in the third ant colony; p1, p2, and p3 respectively represent the size of the first ant colony, the size of the second ant colony, and the size of the third ant colony;

[0145] At the same time, set the first maximum number of iterations p′, the second maximum number of iterations p″ and the third maximum number of iterations p″′; set the first current number of iterations r, the second current number of iterations r′ and the third current number of iterations r″;

[0146] Set the pheromone importance factor to r″′ and the heuristic function importance factor to s;

[0147] S2242: Setting the first ant colony population The initial position of 50% of the ants is δ, and the initial position of the other 50% of the ants is ε; set the second ant colony population The initial position of 50% of the ants is δ′, and the initial position of the other 50% of the ants is ε′; set the third ant colony population The initial position of 50% of the ants is δ″, and the initial position of the other 50% of the ants is ε″;

[0148] S2243, setting the initial concentration of pheromones released by each individual ant in the first ant colony, the second ant colony and the third ant colony to s″;

[0149] The calculation formula for the movement probability of the first ant colony in each iteration is set as follows:

[0150]

[0151] Among them, s″1(r) represents the movement probability of the first ant colony in the rth iteration, s″′ 1r It represents the average error value of the first ant colony classifying the domain test sample set and the domain test sample label set using the optimized domain SVM model in the rth iteration;

[0152] The calculation formula for the movement probability of the second ant colony in each iteration is set as follows:

[0153]

[0154] Among them, s″2(r′) represents the movement probability of the second ant colony in the r′th iteration, s″′ 2r′ It represents the average error value of the second ant colony classifying the problem test sample set and the problem test sample label set using the optimized problem SVM model in the r′th iteration;

[0155] The calculation formula for the movement probability of the third ant colony in each iteration is set as follows:

[0156]

[0157] s″3(r″) represents the movement probability of the third ant colony in the r″th iteration, s″′ 3r″ It represents the average error value of the third ant colony classifying the solution test sample set and the solution test sample label set using the optimized solution SVM model in the r″th iteration;

[0158] S2244, starting an iteration process for the first ant colony, the second ant colony, and the third ant colony according to the movement probability calculation formula of the first ant colony in each iteration, the movement probability calculation formula of the second ant colony in each iteration, and the movement probability calculation formula of the third ant colony in each iteration; specifically as follows:

[0159] The first ant colony population obtains the first optimal position and the second optimal position in each round of iteration, the first optimal position represents the optimal value of the penalty factor parameter in the domain SVM model, and the second optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the domain SVM model; the second ant colony population obtains the third optimal position and the fourth optimal position in each round of iteration, the third optimal position represents the optimal value of the penalty factor parameter in the problem SVM model, and the fourth optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the problem SVM model; the third ant colony population obtains the fifth optimal position and the sixth optimal position in each round of iteration, the fifth optimal position represents the optimal value of the penalty factor parameter in the solution SVM model, and the sixth optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the solution SVM model;

[0160] S2245. When r≥p′, the first ant colony population stops iterating, and obtains a first final penalty factor parameter t and a distribution determination parameter u after the first final data is mapped to a new feature space;

[0161] When r′≥p″, the second ant colony population stops iterating, and obtains the second final penalty factor parameter t′ and the distribution determination parameter u′ after the second final data is mapped to the new feature space;

[0162] When r″≥p″′, the third ant colony population stops iterating, and obtains a third final penalty factor parameter t″ and a distribution determination parameter u″ after the third final data is mapped to a new feature space;

[0163] S2246, using the first final penalty factor parameter t and the distribution determination parameter u after the first final data is mapped to the new feature space as parameters of the domain SVM model after the primary optimization, to obtain the domain SVM model after the secondary optimization;

[0164] The second final penalty factor parameter t′ and the distribution determination parameter u′ after the second final data is mapped to the new feature space are used as parameters of the problem SVM model after the primary optimization to obtain the problem SVM model after the secondary optimization;

[0165] The third final penalty factor parameter t″ and the distribution determination parameter u″ after the third final data is mapped to the new feature space are used as parameters of the solution SVM model after the primary optimization to obtain the solution SVM model after the secondary optimization;

[0166] S3, using the twice-optimized domain SVM model, the twice-optimized problem SVM model, and the twice-optimized solution SVM model to classify the papers to be checked for duplicates and the database document set, to obtain a first classification result set and a second classification result set, respectively;

[0167] The S3 comprises the following steps:

[0168] S31, the database document set c={c1,c2,...,c i ,...,c c″′ Each document in} is input into the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model for classification operation, and the first classification result set t″′={t″′1,t″′2,...,t″′ i ,...,t″′ c″′}, where t″′ i represents the result set of classifying the i-th document in the database document set, t″′ i ={t″′ i1 ,t″′ i2 ,t″′ i3}, t″′ i1 , t″′ i2and t″′ i3 They represent the classification results of the i-th document in the database document set in the domain SVM model after quadratic optimization, the problem SVM model after quadratic optimization, and the solution SVM model after quadratic optimization respectively;

[0169] S32, inputting the papers to be checked for duplicates into the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model for classification operations, and obtaining a second classification result set u″′={u″′1,u″′2,u″′3};

[0170] S4, comparing the first classification result set with the second classification result set, and determining whether the paper to be checked for duplicates is duplicated according to the comparison result;

[0171] The process of S4 is as follows:

[0172] The first classification result set t″′={t″′1, t″′2, ..., t″′ i ,...,t″′ c″′} is compared with the second classification result set u″′={u″′1,u″′2,u″′3}. If the first classification result set t″′={t″′1,t″′2,...,t″′ i ,...,t″′ c″′} exists t″′ i ={t″′ i1 ,t″′ i2 ,t″′ i3}, so that t″′ i1 =u″′1、t″′ i2 =u″′2 and t″′ i3 =u″′3, it is determined that the paper to be checked for duplicates is a duplicate of the paper in the database; otherwise, it is determined that the paper to be checked for duplicates is not a duplicate of the paper in the database.

[0173] A paper duplication checking system based on text feature extraction technology, comprising a paper word segmentation module, a document duplication calculation module, a first paper duplication determination module, an SVM model primary optimization module, an SVM model secondary optimization module, a paper classification module and a second paper duplication determination module;

[0174] The paper segmentation module is used to perform a segmentation operation on the paper to be checked for duplicates;

[0175] The document duplication calculation module calculates the duplication between the document and the paper to be checked for duplicates according to the word segmentation weight matrix contained in the document, and obtains a document duplication set;

[0176] The first paper duplication determination module compares and analyzes the document duplication set with the duplication threshold to determine whether the paper to be checked for duplication is duplicated;

[0177] The SVM model primary optimization module is used to optimize the initial domain SVM model, the initial problem SVM model and the initial solution SVM model;

[0178] The SVM model secondary optimization module is used to perform secondary optimization on the domain SVM model after the primary optimization, the problem SVM model after the primary optimization, and the solution SVM model after the primary optimization;

[0179] The paper classification module is used to classify the papers to be checked for duplicates and the document set in the database by using the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model;

[0180] The second paper duplication determination module is used to compare the first classification result set with the second classification result set, and determine whether the paper to be checked for duplication is duplicated based on the comparison result.

[0181] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0182] The preferred embodiments of the invention disclosed above are only used to help explain the invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the invention, so that those skilled in the art can understand and use the invention well.

Claims

1. A paper duplication checking method based on text feature extraction technology, characterized in that: The following steps are involved: S1. Input the paper to be checked for duplicates, perform word segmentation on the paper to be checked for duplicates, and obtain a paper word segmentation set; set a database document set, query the database according to the paper word segmentation set, obtain the word segmentation contained in the paper word segmentation set for each document in the database document set, and form a document containing word segmentation matrix; S2. According to the paper word segmentation set, the weight value of the corresponding word is set to obtain the word segmentation weight set, and the document inclusion word segmentation weight matrix is ​​obtained according to the word segmentation weight set. According to the document inclusion word segmentation weight matrix, the repetition between the document and the paper to be checked for duplicates is calculated to obtain the document repetition set; the repetition threshold is set, and the document repetition set is compared and analyzed with the repetition threshold. When there is a document in the document repetition set with a repetition greater than or equal to the repetition threshold, it is determined that the paper to be checked for duplicates is duplicated with the papers in the database document set; otherwise, an initial domain SVM model, an initial problem SVM model and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model and a secondary optimized solution SVM model; S3, using the twice-optimized domain SVM model, the twice-optimized problem SVM model, and the twice-optimized solution SVM model to classify the papers to be checked for duplicates and the database document set, to obtain a first classification result set and a second classification result set, respectively; S4, comparing the first classification result set with the second classification result set, and determining whether the paper to be checked for duplicates is duplicated according to the comparison result; The S2 comprises the following steps: S21. According to the word set of the paper Set the weight value of the corresponding word segmentation to obtain the word segmentation weight set. And the document contains the word segmentation matrix Get the document containing the word segmentation weight matrix , according to the document containing the word segmentation weight matrix Calculate the duplication between the document and the paper to be checked for duplicates, and obtain the document duplication set; Represents the word set of the paper Middle The weight of a word segment, Indicates the first participle, Indicates the total number of segmentations in the paper segmentation set; S22. Set the repetition threshold , according to the literature repetition set and repetition threshold Comparative analysis is performed. When the literature duplication concentration contains literature with a duplication greater than or equal to If the paper to be checked for duplicates is found to be duplicated with the papers in the document set of the database, then it is determined that the paper to be checked for duplicates is duplicated; otherwise, an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model, and a secondary optimized solution SVM model; The S21 comprises the following steps: S211. According to the word set of the paper Set the weight value of the corresponding word segmentation to obtain the word segmentation weight set ; S212: According to the word segmentation weight set And the document contains the word segmentation matrix Get the document containing the word segmentation weight matrix ,as follows: ； in, Represents a collection of documents in a database The The literature contains The weight of a word segment, Indicates the first Documents S213, according to the document contains the word segmentation weight matrix Calculate the duplication between the document and the paper to be checked, and obtain the document duplication set , Indicates The degree of duplication between the document and the paper to be checked, Represents the total number of documents in the database; the calculation formula is as follows: ； In S22, an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model are constructed, and primary optimization and secondary optimization are performed to obtain a secondary optimized domain SVM model, a secondary optimized problem SVM model, and a secondary optimized solution SVM model; specifically, the following steps are included: S221, randomly collected paper training sample set And the paper test sample set ; Indicates the first Paper training samples, Represents the total number of paper training samples in the paper training sample set; Indicates the first Paper test samples; From the training sample set of the paper And the paper test sample set Randomly select The proportion of paper training samples and paper test samples are used as field training sample sets Domain test sample set ; Then from the training sample set of the paper And the paper test sample set Take The proportion of paper training samples and paper test samples are used as the problem training sample set and problem test sample set ; Then from the training sample set of the paper And the paper test sample set Select The proportion of paper training samples and paper test samples are used as solution training sample sets and solution test sample set ; , , , and They represent the first domain test samples, domain training samples The first domain training sample and the first problem test sample in the The problem training sample, the solution training sample set The solution training sample and the solution test sample set Solution test samples; , , , , and They respectively represent the total number of domain training samples in the domain training sample set, the total number of problem training samples in the problem training sample set, the total number of solution training samples in the solution training sample set, the total number of domain test samples in the domain test sample set, the total number of problem test samples in the problem test sample set, and the total number of solution test samples in the solution test sample set; According to the domain training sample set Set the corresponding domain training sample label set ; Training sample set according to the problem Set the corresponding problem training sample label set ; Training sample set according to the solution Set the corresponding solution training sample label set ; Test sample sets according to the described domain Set the corresponding domain test sample label set ; Test sample set according to the problem Set the corresponding problem test sample label set ; Test sample set according to the solution Set the corresponding solution test sample label set ; S222: construct an initial domain SVM model, an initial problem SVM model, and an initial solution SVM model, and set the kernel functions of the initial SVM model, the initial problem SVM model, and the initial solution SVM model to be Gaussian kernel functions, and the function expressions are as follows: ； Where: and represents two different data sample points, represents the kernel parameters; S223, setting the first domain training error threshold , the first question training error threshold And the first solution training error threshold ; The field training sample set And domain training sample label set Input into the initial domain SVM model for training to adjust the penalty factor parameters in the initial domain SVM model The distribution parameters after the data is mapped to the new feature space are determined by Perform an optimization; when the training error of the domain training sample set and the domain training sample label set in the initial domain SVM model is less than the first domain training error threshold When , stop training and get an optimized , denoted as , after an optimization , denoted as And the domain SVM model after one optimization; The problem training sample set and the problem training sample label set Input into the initial problem SVM model for training to adjust the penalty factor parameters in the initial problem SVM model The distribution parameters after the data is mapped to the new feature space are determined by Perform an optimization; when the training error of the problem training sample set and the problem training sample label set in the initial problem SVM model is less than the first problem training error threshold When , stop training and get an optimized , denoted as , after an optimization , denoted as And the problem SVM model after one optimization; The solution training sample set And solution training sample label set Input into the initial solution SVM model for training to adjust the penalty factor parameters in the initial solution SVM model The distribution parameters after the data is mapped to the new feature space are determined by Perform an optimization; when the training error of the solution training sample set and the solution training sample label set in the initial solution SVM model is less than the first solution training error threshold When , stop training and get an optimized , denoted as , after an optimization , denoted as And the solution SVM model after one optimization; S224: The field test sample set And domain test sample label set Input into the domain SVM model after the primary optimization, perform secondary optimization on the domain SVM model after the primary optimization, and obtain the domain SVM model after the secondary optimization; The problem test sample set And the problem test sample label set Input into the problem SVM model after the primary optimization, perform secondary optimization on the problem SVM model after the primary optimization, and obtain the problem SVM model after the secondary optimization; The solution was tested on a sample set And solution test sample label set Input into the solution SVM model after the primary optimization, perform secondary optimization on the solution SVM model after the primary optimization, and obtain the solution SVM model after the secondary optimization; The S3 comprises the following steps: S31, the database document collection The documents in the dataset are input into the domain SVM model after quadratic optimization, the problem SVM model after quadratic optimization and the solution SVM model after quadratic optimization for classification operation to obtain the first classification result set. ,in Indicates the first The result set of the classification of documents is , , and Respectively represent the first The classification results of the literature in the domain SVM model after quadratic optimization, the problem SVM model after quadratic optimization, and the solution SVM model after quadratic optimization; S32, inputting the duplicate papers to be checked into the twice-optimized domain SVM model, the twice-optimized problem SVM model, and the twice-optimized solution SVM model for classification operation, to obtain a second classification result set ; The process of S4 is as follows: The first classification result set With the second classification result set For comparison, if the first classification result set Existence , so that , and , then it is determined that the paper to be checked for duplicates is a duplicate of the paper in the database; otherwise, it is determined that the paper to be checked for duplicates is not a duplicate of the paper in the database.

2. According to claim 1, a method for checking for duplicate papers based on text feature extraction technology is characterized in that: The S1 comprises the following steps: S11. Perform word segmentation on the paper to be checked for duplicate content to obtain a word segmentation set of the paper. ; S12. Set up database document collection , Indicates the first Literature, Indicates the total number of documents in the database; according to the paper segmentation set Search the database to obtain the document set of the database Each document in the paper contains the word set The word segmentation matrix in the document contains ,as follows: ； in, Represents a collection of documents in a database The The literature contains A participle.

3. A method for checking for duplicate papers based on text feature extraction technology according to claim 2, characterized in that: The S11 comprises the following steps: S111, dividing the paper to be checked for duplicate content into paragraphs to obtain a collection of paper paragraphs , Indicates the first paragraphs, Indicates the total number of paragraphs divided into the paper to be checked for duplicates; Convert each paragraph in the string to get the paragraph string set , Indicates the first The string corresponding to the paragraph, , express The characters, express The total number of characters in ; S112, constructing a Chinese character index table, and searching according to the Chinese character index table Is it a single word? For single-word words, Words, then Make inquiries and judgments; when If it is not a single-word word, all words with the same character are searched according to the Chinese character index table. The word set with prefix is ​​used to get the first word set , Indicates Prefix words, Indicates The total number of words with prefix; S113, will and the next character String ,Will With the first word set Match the first two characters of all words in Same and total length If the string is the same, Otherwise, into words; S114: the paragraph character string set Repeat the operations in S112 and S113 for each paragraph string in to obtain the paragraph segmentation matrix ,as follows: ； in, Express After word segmentation, the participle, Express The total number of segmented words obtained after segmentation; S115: segment the paragraph word matrix The words in the paragraph are deduplicated to obtain a paragraph segmentation matrix after deduplication; the paragraph segmentation matrix after deduplication is then merged to obtain a paper segmentation set .

4. The method for checking for duplicate papers based on text feature extraction technology according to claim 1 is characterized in that: The S224 comprises the following steps: S2241. Constructing the first ant colony , the second ant colony and the third ant colony ; , and Respectively represent the first ant colony population Ants, the second ant colony ants and the third ant colony Only ants; , and They represent the size of the first ant colony, the size of the second ant colony, and the size of the third ant colony respectively; At the same time, set the first maximum number of iterations , the second maximum number of iterations and the third maximum number of iterations ; Set the first current iteration number , Second current iteration number and the third current iteration number ; Set the pheromone importance factor to And the heuristic function importance factor is ; S2242: Setting the first ant colony population middle The initial position of the ant is ,in addition The initial position of the ant is ; Set the second ant colony population middle The initial position of the ant is ,in addition The initial position of the ant is ; Set the third ant colony population middle The initial position of the ant is ,in addition The initial position of the ant is ; S2243, setting the initial concentration of pheromones released by each ant individual in the first ant colony, the second ant colony, and the third ant colony to be ; The calculation formula for the movement probability of the first ant colony in each iteration is set as follows: ； in, Indicates that the first ant colony is in The probability of moving in a round iteration, Indicates that the first ant colony is in The average error value of classifying the domain test sample set and the domain test sample label set using the optimized domain SVM model in the round iteration; The calculation formula for the movement probability of the second ant colony in each iteration is set as follows: ； in, Indicates that the second ant colony is in The probability of moving in a round iteration, Indicates that the second ant colony is in The average error value of classifying the problem test sample set and the problem test sample label set using the optimized problem SVM model in the round iteration; The calculation formula for the movement probability of the third ant colony in each iteration is set as follows: ； Indicates that the third ant colony is in The probability of moving in a round iteration, Indicates that the third ant colony is in The average error value of classifying the solution test sample set and the solution test sample label set using the optimized solution SVM model in the round iteration; S2244, starting an iteration process for the first ant colony, the second ant colony, and the third ant colony according to the movement probability calculation formula of the first ant colony in each iteration, the movement probability calculation formula of the second ant colony in each iteration, and the movement probability calculation formula of the third ant colony in each iteration; specifically as follows: The first ant colony population obtains the first optimal position and the second optimal position in each round of iteration, the first optimal position represents the optimal value of the penalty factor parameter in the domain SVM model, and the second optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the domain SVM model; the second ant colony population obtains the third optimal position and the fourth optimal position in each round of iteration, the third optimal position represents the optimal value of the penalty factor parameter in the problem SVM model, and the fourth optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the problem SVM model; the third ant colony population obtains the fifth optimal position and the sixth optimal position in each round of iteration, the fifth optimal position represents the optimal value of the penalty factor parameter in the solution SVM model, and the sixth optimal position represents the optimal value of the distribution determination parameter after the data is mapped to the new feature space in the solution SVM model; S2245, when When the first ant colony stops iterating, the first final penalty factor parameter is obtained. And the distribution parameters after the first final data is mapped to the new feature space ; when When the second ant colony stops iterating, the second final penalty factor parameter is obtained. And the distribution of the second final data after mapping to the new feature space determines the parameters ; when When the third ant colony stops iterating, the third final penalty factor parameter is obtained. And the third distribution parameter after the final data is mapped to the new feature space ; S2246: Set the first final penalty factor parameter And the distribution parameters after the first final data is mapped to the new feature space As the parameters of the domain SVM model after the first optimization, the domain SVM model after the second optimization is obtained; The second final penalty factor parameter And the distribution of the second final data after mapping to the new feature space determines the parameters As the parameters of the problem SVM model after the first optimization, the problem SVM model after the second optimization is obtained; The third final penalty factor parameter And the third distribution parameter after the final data is mapped to the new feature space As the parameters of the solution SVM model after the first optimization, the solution SVM model after the second optimization is obtained.

5. A system for implementing the paper duplication checking method based on text feature extraction technology as described in any one of claims 1 to 4, characterized in that: It includes a paper segmentation module, a document duplication calculation module, a first paper duplication determination module, an SVM model primary optimization module, an SVM model secondary optimization module, a paper classification module and a second paper duplication determination module; The paper segmentation module is used to perform a segmentation operation on the paper to be checked for duplicates; The document duplication calculation module calculates the duplication between the document and the paper to be checked for duplicates according to the word segmentation weight matrix contained in the document, and obtains a document duplication set; The first paper duplication determination module compares and analyzes the document duplication set with the duplication threshold to determine whether the paper to be checked for duplication is duplicated; The SVM model primary optimization module is used to optimize the initial domain SVM model, the initial problem SVM model and the initial solution SVM model; The SVM model secondary optimization module is used to perform secondary optimization on the domain SVM model after the primary optimization, the problem SVM model after the primary optimization, and the solution SVM model after the primary optimization; The paper classification module is used to classify the papers to be checked for duplicates and the document set in the database by using the twice-optimized domain SVM model, the twice-optimized problem SVM model and the twice-optimized solution SVM model; The second paper duplication determination module is used to compare the first classification result set with the second classification result set, and determine whether the paper to be checked for duplication is duplicated based on the comparison result.

Images