Paper duplicate checking method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a paper duplicate checking method and device, equipment and a storage medium. The method comprises the steps of obtaining text data of a paper to be subjected to duplicate checking; determining a text vector and a text segmented word corresponding to the text data of the paper to be subjected to duplicate checking; determining at least one target thesis matched with the thesis to be subjected to duplicate checking according to the text vector and the text segmented word; determining the repetition rate of the paper to be subjected to duplicate checking and the at least one target paper; determining duplicate checking suggestions between the paper to be subjected to duplicate checking and the at least one target paper through a large language model; displaying the at least one target paper according to the repetition rate of the paper to be subjected to duplicate checking and the at least one target paper and the duplicate checking suggestion; according to the method, semantic information can be captured through text vectorization, a text structure is processed in combination with a word segmentation technology, content similarity and keyword matching are considered, paper literatures with relatively high similarity can be efficiently screened out, and the duplicate checking accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method, apparatus, device, and storage medium for paper plagiarism detection. Background Art

[0002] With the rapid development of project topics and paper research, the number of related papers has increased sharply, and the problems of plagiarism and duplicate publication have become increasingly serious. Traditional plagiarism detection methods are based on hash tables and sorting search technologies, mainly relying on simple text matching technologies, which are difficult to meet the requirements in complex retrieval scenarios, and the retrieval accuracy is not high. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a method, apparatus, device, and storage medium for paper plagiarism detection that overcome the above problems or at least partially solve the above problems.

[0004] To solve the above problems, embodiments of the present invention disclose a method for paper plagiarism detection, the method comprising:

[0005] Obtaining text data of the paper to be detected for plagiarism;

[0006] Determining a text vector and text segmentation corresponding to the text data of the paper to be detected for plagiarism;

[0007] Determining at least one target paper that matches the paper to be detected for plagiarism according to the text vector and text segmentation;

[0008] Determining the duplication rate between the paper to be detected for plagiarism and the at least one target paper;

[0009] Determining a plagiarism detection suggestion for the paper to be detected for plagiarism and the at least one target paper through a large language model;

[0010] Displaying the at least one target paper according to the duplication rate between the paper to be detected for plagiarism and the at least one target paper and the plagiarism detection suggestion.

[0011] Optionally, the displaying the at least one target paper according to the duplication rate between the paper to be detected for plagiarism and the at least one target paper and the plagiarism detection suggestion includes:

[0012] Sorting the at least one target paper in descending order of the duplication rate to generate a first sorted list;

[0013] Adding a first identifier to the target papers with a duplication rate higher than a first threshold in the first sorted list, adding a second identifier to the target papers with a duplication rate between a second threshold and the first threshold, and hiding the target papers with a duplication rate lower than a third threshold to obtain a second sorted list;

[0014] Add corresponding duplicate check suggestions to the at least one target paper in the second row list and display them.

[0015] Optionally, the duplicate check suggestions include at least one of content expansion suggestions, format modification suggestions, citation specification verification suggestions, logic modification suggestions, and semantic paraphrasing suggestions.

[0016] Optionally, when the duplicate check suggestions include semantic paraphrasing suggestions, determining the duplicate check suggestions for the paper to be checked against the at least one target paper through a large language model includes:

[0017] Determine the duplicate text in the paper to be checked and the target paper.

[0018] Extract that the duplicate text is key semantic information; the key semantic information includes logical relationships and core content.

[0019] Input the key semantic information into the large language model to obtain various expression forms and language structures that match the key semantic information.

[0020] Generate duplicate check suggestions for the duplicate text of the paper to be checked and the target paper according to the various expression forms and language structures.

[0021] Optionally, determining at least one target paper that matches the paper to be checked according to the text vector and text tokenization includes:

[0022] Determine the vector scores of each paper in the retrieval database and the paper to be checked according to the text vector.

[0023] Determine the token scores of each paper in the retrieval database and the paper to be checked according to the text tokenization.

[0024] Determine the retrieval scores of each paper and the paper to be checked according to the vector scores of each paper and the paper to be checked and the token scores of each paper and the paper to be checked.

[0025] Determine the papers in the papers whose retrieval scores with the paper to be checked are greater than the preset score as target papers.

[0026] Optionally, determining the duplication rate of the paper to be checked and the at least one target paper includes:

[0027] Process the retrieval scores of the at least one target paper and the paper to be checked through a Sigmoid function to obtain the duplication rate of the paper to be checked and the at least one target paper.

[0028] Optionally, the text vector includes a title vector and a paper content vector; the determining of the vector scores of each paper in the retrieval database and the paper to be checked for duplication according to the text vector includes:

[0029] Obtain a first weighting parameter;

[0030] According to the first weighting parameter and the title vector, determine the cosine similarity scores of the title vectors of each paper and the paper to be checked for duplication;

[0031] Obtain a second weighting parameter;

[0032] According to the second weighting parameter and the paper content vector, determine the cosine similarity scores of the paper content of each paper and the paper to be checked for duplication;

[0033] According to the cosine similarity scores of the title vectors of each paper and the paper to be checked for duplication and the cosine similarity scores of the paper content of each paper and the paper to be checked for duplication, determine the vector scores of each paper and the paper to be checked for duplication.

[0034] Optionally, the text segmentation includes title segmentation and paper content segmentation, and the determining of the segmentation scores of each paper in the retrieval database and the paper to be checked for duplication according to the text segmentation includes:

[0035] Determine a first retrieval score corresponding to the title segmentation of each paper and the paper to be checked for duplication and a second retrieval score corresponding to the paper content segmentation;

[0036] Obtain a third weighting parameter for the first retrieval score and a fourth weighting parameter for the second retrieval score;

[0037] According to the first retrieval score, the second retrieval score, the third weighting parameter and the fourth weighting parameter, determine the segmentation scores of each paper and the paper to be checked for duplication.

[0038] Optionally, it further includes:

[0039] Obtain the text data of historical papers;

[0040] Vectorize the text data of the historical papers, and store the vectorized text data of the historical papers in the retrieval database;

[0041] The determining of the vector scores of each paper in the retrieval database and the paper to be checked for duplication according to the text vector includes:

[0042] According to the text vector, determine the vector scores of the text vectors of each historical paper and the paper to be checked for duplication in the retrieval database.

[0043] Optionally, it further includes;

[0044] Performing text segmentation on the text data of the historical paper to obtain keywords of the text data of the historical paper;

[0045] Storing the keywords of the text data of the historical paper into the retrieval database;

[0046] Determining the word segmentation scores of each paper in the retrieval database and the paper to be checked for duplication according to the text segmentation, including:

[0047] Determining the word segmentation scores of the text segmentation of each historical paper and the paper to be checked for duplication in the retrieval database according to the text segmentation.

[0048] Optionally, determining the corresponding text vector according to the text data of the paper to be checked for duplication includes:

[0049] Performing preprocessing on the text data to obtain preprocessed text data;

[0050] Using a vectorization model to process the preprocessed text data to obtain the corresponding text vector.

[0051] Optionally, the vectorization model is a multi-modal hybrid embedding model.

[0052] Optionally, determining the corresponding text segmentation according to the text data of the paper to be checked for duplication includes:

[0053] Performing preprocessing on the text data to obtain preprocessed text data;

[0054] Using a text segmenter to perform word segmentation on the preprocessed text data to obtain the text segmentation.

[0055] Optionally, the text segmenter is an IK text segmenter.

[0056] Optionally, it further includes:

[0057] Obtaining the feature information of the at least one target paper; the feature information includes at least one of title information, author information, and publication unit information;

[0058] Displaying the feature information of the at least one target paper.

[0059] The present invention also discloses a paper duplication checking device, and the device includes:

[0060] An acquisition module, configured to acquire the text data of the paper to be checked for duplication;

[0061] The first determination module is used to determine the corresponding text vector and text segmentation according to the text data of the paper to be checked for plagiarism.

[0062] The second determination module is used to determine at least one target paper that matches the paper to be checked for plagiarism according to the text vector and text segmentation.

[0063] The third determination module is used to determine the repetition rate of the paper to be checked for plagiarism and the at least one target paper.

[0064] The fourth determination module is used to determine the plagiarism checking suggestions for the paper to be checked for plagiarism and the at least one target paper through a large language model.

[0065] The display module is used to display the at least one target paper according to the repetition rate of the paper to be checked for plagiarism and the at least one target paper and the plagiarism checking suggestions.

[0066] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor is used to execute the steps of the paper plagiarism checking method as described above.

[0067] The present invention also discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the paper plagiarism checking method as described above are implemented.

[0068] The embodiments of the present invention include the following advantages:

[0069] The present invention discloses a paper plagiarism checking method, device, equipment, and storage medium. The present invention can capture semantic information through text vectorization, combine the word segmentation technology to process the text structure, take into account both content similarity and keyword matching, can efficiently screen out paper documents with higher similarity, improve the accuracy of plagiarism checking, and can display the target paper through the plagiarism checking suggestions and repetition rate generated by the large model, which can help users quickly locate plagiarism problems and improve the plagiarism checking efficiency for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flowchart of the steps of a paper plagiarism checking method provided by an embodiment of the present invention;

[0071] Figure 2 is a structural block diagram of a paper plagiarism checking device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0073] One of the core concepts of the embodiments of the present invention is that the present invention can capture semantic information through text vectorization, process the text structure in combination with the word segmentation technology, take into account content similarity and keyword matching, can efficiently screen out paper documents with relatively high similarity, improve the accuracy of duplicate checking, and can display the target paper through the duplicate checking suggestions and duplication rates generated by the large model, which can help users quickly locate duplicate checking problems and improve the duplicate checking efficiency for users.

[0074] Referring Figure 1 , a step flowchart of a paper duplicate checking method provided by an embodiment of the present invention is shown. The method may include the following steps:

[0075] Step 101, obtain the text data of the paper to be checked for duplicates.

[0076] In the embodiments of the present invention, the text data of the paper to be checked for duplicates may be in a pure text format, or may be text information extracted from document formats such as Word and PDF. For example, when a user uploads the paper they wrote in the form of a Word document to the duplicate checking system, the duplicate checking system can extract the text content in the document to form text data for subsequent processing.

[0077] Step 102, determine the text vector and text segmentation corresponding to the text data of the paper to be checked for duplicates.

[0078] In the embodiments of the present invention, a text vector is a numerical vector representation form that converts text data into a form that can be understood and processed by a computer. Usually, some natural language processing technologies are used to map each word in the text to a high-dimensional vector space. In this way, each word can be represented by a vector, and the entire text can be represented by the combination of these word vectors (such as averaging, weighting, etc.). For example, for the sentence "An apple is a kind of fruit", through the word embedding technology, "apple", "is", "a kind of", and "fruit" will all be converted into corresponding vectors, and then these vectors are processed to obtain the text vector of the entire sentence. In this way, the text is converted from the original character form into a numerical vector form, which is convenient for subsequent similarity calculation and other operations. <>

[0079] Text segmentation is the process of splitting continuous text data into individual words or phrases according to certain rules. For example, for Chinese text, common word segmentation tools (such as Jieba word segmentation, etc.) will split sentences into individual words according to the word library and algorithms. For English text, there are also corresponding methods to split sentences into words. For example, "The dogis running" will be split into "The", "dog", "is", "running". Through word segmentation, the text is decomposed into finer-grained units, which is convenient for subsequent text analysis and comparison.

[0080] Step 103: Determine at least one target paper that matches the paper to be checked for plagiarism based on the text vector and text segmentation.

[0081] In the embodiments of the present invention, in the paper retrieval database of the plagiarism detection system, similar text vector and text segmentation processing have been performed on each paper in advance. Then, by calculating the similarity between the text vector of the paper to be checked for plagiarism and the text vectors of other papers in the paper database, a similarity ranking can be obtained. At the same time, in combination with the results of text segmentation, further matching and screening of the specific content of the text are carried out. By comprehensively considering the similarity of text vectors and the matching situation of text segmentation, at least one paper with a relatively high similarity to the paper to be checked for plagiarism is selected from the paper database as the target paper. These target papers are considered to be papers that may have duplicate content with the paper to be checked for plagiarism.

[0082] Step 104: Determine the duplication rate between the paper to be checked for plagiarism and at least one target paper.

[0083] In the embodiments of the present invention, for each determined target paper, the text segmentation results and text vectors of the paper to be checked for plagiarism and the target paper can be compared respectively, so as to obtain the duplication rate between the paper to be checked for plagiarism and at least one target paper, and then the duplication degree between the two papers can be measured according to this duplication rate.

[0084] Step 105: Determine the plagiarism detection suggestions for the paper to be checked for plagiarism and at least one target paper through a large language model.

[0085] In the embodiments of the present invention, the large language model is a model trained with a large amount of text data. It can understand and generate natural language. Inputting the relevant information of the paper to be checked for plagiarism and the target paper into the large language model, the model will analyze the duplication situation of the two papers according to the knowledge and patterns it has learned and give corresponding plagiarism detection suggestions. These plagiarism detection suggestions may include specific analysis of the duplicate parts, directions for improvement, etc. For example, the large language model may point out that a certain section of the paper to be checked for plagiarism is highly similar to a certain section of the target paper, and it is recommended to reduce the duplication rate by reorganizing the language, adding one's own analysis and insights, etc.

[0086] It should be noted that the large language model can be one of the deepseek model, the Wenxin Yiyan model, and the Qianwen model, and is not limited herein.

[0087] Step 106: Display at least one target paper according to the duplication rate and plagiarism detection suggestions between the paper to be checked for plagiarism and at least one target paper.

[0088] In the embodiments of the present invention, the duplicate checking system can display the duplication rate of each target paper and the paper to be checked in an intuitive manner, for example, in the form of a percentage. At the same time, the duplicate checking suggestions given by the large language model will also be displayed. In this way, the user can clearly understand the duplication situation between the paper to be checked and each target paper, as well as the specific suggestions for these duplication situations. The display form can be to list the basic information of the target paper, the duplication rate value, and the specific content of the duplicate checking suggestions in an interface, which is convenient for the user to view and analyze, so as to make corresponding processing decisions, such as modifying the paper.

[0089] The present invention discloses a paper duplicate checking method, which can capture semantic information through text vectorization, process the text structure in combination with the word segmentation technology, take into account both content similarity and keyword matching, can efficiently screen out paper documents with relatively high similarity, improve the accuracy of duplicate checking, and can display the target paper through the duplicate checking suggestions and duplication rate generated by the large model, which can help users quickly locate duplicate checking problems and improve the duplicate checking efficiency for users.

[0090] In an embodiment mode of the present invention, according to the duplication rate and duplicate checking suggestions of the paper to be checked and at least one target paper, displaying at least one target paper includes: sorting at least one target paper in descending order of the duplication rate to generate a first sorted list; adding a first identifier to the target paper with a duplication rate higher than the first threshold in the first sorted list, adding a second identifier to the target paper with a duplication rate between the second threshold and the first threshold, and hiding the target paper with a duplication rate lower than the third threshold to obtain a second sorted list; adding the corresponding duplicate checking suggestions to at least one target paper in the second sorted list and displaying them.

[0091] In the embodiments of the present invention, after determining the duplication rate and duplicate checking suggestions of the paper to be checked and each target paper, these target papers can be sorted according to the numerical value of the duplication rate. For example, there are target papers A, B, and C, and their duplication rates with the paper to be checked are 30%, 15%, and 20% respectively. Then the first sorted list generated after sorting is target paper A, target paper C, and target paper B (because 30% > 20% > 15%). Through this sorting method, the user can first see the target paper with the highest duplication rate with the paper to be checked, which helps the user quickly locate the paper that may have relatively large problems.

[0092] The first threshold is a preset value. For example, it can be set to 20%. If the repetition rate of the target paper is higher than this first threshold (such as 25%), a first identifier will be added to the target paper. This identifier can be a specific color (such as red), a symbol (such as an asterisk), or other marks that can attract the user's attention. The purpose of this is to highlight those target papers with a high repetition rate compared to the paper to be checked for plagiarism, which may have serious plagiarism or high similarity issues.

[0093] The second threshold is also a preset value and is less than the first threshold. For example, it is 10%. When the repetition rate of the target paper is between the second threshold (10%) and the first threshold (20%) (such as 13%), a second identifier will be added to it. The second identifier is different from the first identifier. For example, it can be another color (such as yellow) or other different marks. This is to distinguish those target papers with a relatively low repetition rate but still requiring the user's attention.

[0094] The third threshold is also preset and is generally less than the second threshold. Suppose it is 5%. If the repetition rate of the target paper is lower than this third threshold (such as 3%), it will be hidden, that is, not shown in the display list. This is because target papers with too low a repetition rate may have less impact on judging whether the paper to be checked for plagiarism has problems. Hiding them can reduce the information interference that users see and allow users to focus more on target papers with a higher repetition rate.

[0095] Through the above operations, the second row list is obtained from the first row list, and this list classifies and filters the target papers more clearly.

[0096] For each target paper in the second row list, in the previous steps, the plagiarism checking suggestions for the target paper and the paper to be checked for plagiarism have been obtained through the large language model. This step is to add these plagiarism checking suggestions to the corresponding position next to the target paper, and then display the second row list with the added plagiarism checking suggestions. The display method can be presenting on the web interface, generating a report document, etc. In this way, when users view the target paper, they can not only see the repetition rate and identifier but also directly obtain the specific plagiarism checking suggestions for the target paper, which is convenient for users to analyze and process the repetition situation of the paper.

[0097] In the present invention, by sorting according to the repetition rate, users can first focus on the target papers with high repetition rates, without the need to search for and compare the repetition rates one by one among numerous target papers, saving the users' time and energy. By adding different identifiers, users can quickly distinguish the degree of repetition of the target papers, pay key attention to the papers with high repetition rates, and also have a certain understanding of the papers with relatively low repetition rates, making the presentation of the problem clearer and more straightforward. By hiding the target papers with too low repetition rates, it avoids the interference of these information that has little impact on judging plagiarism issues on the users' line of sight, enabling users to focus more on the target papers that may have problems. By adding plagiarism checking suggestions, users can obtain specific processing guidance while seeing the target papers and repetition rates, which helps users modify and improve the papers more effectively, and enhances the originality and quality of the papers.

[0098] In an embodiment of the present invention, the plagiarism checking suggestions include at least one of content expansion suggestions, format modification suggestions, citation standard verification suggestions, logic modification suggestions, and semantic paraphrasing suggestions.

[0099] In the embodiment of the present invention, the content expansion suggestions refer to that when there is a repetition situation between the paper to be checked for plagiarism and the target paper, it may be because the content of the paper to be checked for plagiarism is not rich enough and simply repeats existing viewpoints. The content expansion suggestions will guide the author to further deepen, refine or supplement new viewpoints, cases, data, etc. on the basis of the existing content, making the paper more in-depth and comprehensive.

[0100] For example, if the paper to be checked for plagiarism only briefly mentions a certain social phenomenon during the discussion, based on the target paper and professional knowledge, the plagiarism checking system may suggest that the author further collect relevant statistical data, typical cases, or analyze the phenomenon from different disciplinary perspectives to enrich the content of the paper.

[0101] The format modification suggestions refer to that different academic fields and journals may have different format requirements. The format modification suggestions will put forward modification opinions for the parts of the paper to be checked for plagiarism that do not conform to the specifications, including aspects such as font, font size, line spacing, paragraph format, and chart layout.

[0102] For example, if the numbering and title format of the charts in the paper to be checked for plagiarism do not meet the requirements, or the citation format of the references is inconsistent with the specified academic norms, the plagiarism checking system will clearly point out these problems and provide correct format examples for the author to refer to.

[0103] The citation standard verification suggestions will remind the author to check the accuracy and integrity of the cited content in the paper, ensure that the citation marks conform to academic norms, and the sources of the cited literature are reliable.

[0104] For example, if there are cited contents in the paper to be checked for plagiarism without indicating the source, or the cited information is incomplete, such as missing the author, year, journal name, etc., the plagiarism detection system will suggest the author to carefully check the cited contents, supplement the complete cited information, and mark it according to the standard citation format.

[0105] Logical modification suggestions will analyze the logical relationships among the contents of each part in the paper to be checked for plagiarism, such as whether the connection between paragraphs is natural, whether the argumentation process is reasonable, whether the conclusion is consistent with the previous discussion, etc., and put forward corresponding modification suggestions; for example, if the paper to be checked for plagiarism has contradictory viewpoints before and after when discussing a certain issue, or the argumentation process lacks coherence, the plagiarism detection system will suggest the author to reorganize the logical structure of the paper, adjust the paragraph order, and supplement necessary transitional sentences to make the argumentation of the paper more rigorous.

[0106] When there is a semantic similarity between the paper to be checked for plagiarism and the target paper, the semantic paraphrasing suggestions will guide the author to rephrase the repeated parts by changing the expression way, replacing synonyms, adjusting the sentence structure, etc., so as to reduce the repetition rate while keeping the original meaning unchanged.

[0107] For example, if there is a paragraph in the paper to be checked for plagiarism that is semantically similar to the target paper, the plagiarism detection system may provide several different paraphrasing schemes to help the author express the same meaning in different languages.

[0108] The present invention can help the author comprehensively improve the quality of the paper by integrating these plagiarism detection suggestions, and reduce the repetition rate at the same time.

[0109] In an embodiment of the present invention, when the plagiarism detection suggestions include semantic paraphrasing suggestions, the plagiarism detection suggestions for the paper to be checked for plagiarism and at least one target paper are determined through a large language model, including: determining the repeated texts in the paper to be checked for plagiarism and the target paper; extracting the key semantic information of the repeated texts; the key semantic information includes logical relationships and core contents; inputting the key semantic information into the large language model to obtain various expression ways and language structures that match the key semantic information; generating the plagiarism detection suggestions for the repeated texts of the paper to be checked for plagiarism and the target paper according to the various expression ways and language structures.

[0110] In the embodiment of the present invention, when the plagiarism detection suggestions include semantic paraphrasing suggestions, the text comparison technology can be used first to find out the same or highly similar text parts between the paper to be checked for plagiarism and the target paper. Usually, the methods adopted include string matching algorithms, such as the longest common subsequence (LCS) algorithm, edit distance algorithm, etc. These algorithms will compare the text contents of the two papers word by word and mark the repeated segments.

[0111] For example, if there is a sentence "Technological innovation is an important driving force for economic development" in the paper to be checked for plagiarism and the same sentence also exists in the target paper, then this sentence will be identified as a repeated text.

[0112] The key semantic information mainly covers logical relationships and core content. By leveraging natural language processing techniques such as part-of-speech tagging and named entity recognition, the core nouns, verbs, and key concepts in the text can be determined. For example, for the repeated text "Due to the intensification of environmental pollution, many rare animals are facing the risk of extinction", the core content is "the intensification of environmental pollution" and "rare animals facing the risk of extinction".

[0113] Logical relationship analysis: Identify the logical connections between the core contents in the text. Common logical relationships include causal relationships, parallel relationships, progressive relationships, etc. In the above example, "Due to..." indicates a causal logical relationship between "the intensification of environmental pollution" and "rare animals facing the risk of extinction".

[0114] Large language models, having been trained on a large amount of text data, can understand and generate natural language. After inputting the extracted key semantic information into the model, the model will generate various different expression forms and language structures based on the language patterns and knowledge it has learned, while maintaining the key semantics unchanged.

[0115] For example, when inputting the key semantic information "Because of fierce market competition, corporate profits have declined", the large language model may generate various expressions such as "Given the increasingly fierce market competition, the profits of enterprises have declined" and "Due to the intensification of market competition, corporate profits show a downward trend".

[0116] Based on the various expressions and language structures generated by the large language model, specific paraphrasing suggestions are generated for the repeated text in the paper to be checked for plagiarism. These suggestions will take into account the fluency, accuracy, and professionalism of the language to help the author modify the repeated text without changing the original meaning.

[0117] For example, if the repeated text is "With the progress of technology, people's lifestyles have changed greatly", the generated plagiarism checking suggestions may be "Along with the continuous progress of technology, people's lifestyles have undergone significant changes". The present invention can provide multiple semantic paraphrasing methods, and users can choose appropriate expressions to replace the repeated text, thereby significantly reducing the repetition rate of the paper and making the paper to be checked for plagiarism more in line with academic norms and requirements.

[0118] In an embodiment of the present invention, determining at least one target paper that matches the paper to be checked for plagiarism includes: determining the vector scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text vector; determining the token scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text tokens; determining the retrieval scores of each paper and the paper to be checked for plagiarism according to the vector scores of each paper and the paper to be checked for plagiarism and the token scores of each paper and the paper to be checked for plagiarism; and determining the papers with retrieval scores greater than a preset score as target papers.

[0119] In an embodiment of the present invention, the text vector represents the text as a vector in a multi-dimensional space. The similarity between texts can be measured by calculating the similarity between vectors. Common methods for calculating vector similarity include cosine similarity, Euclidean distance, etc. The vector score is a value obtained based on these similarity calculation methods and is used to represent the similarity degree of two papers in the vector space.

[0120] Furthermore, each paper in the paper to be checked for plagiarism and the retrieval database can be tokenized to obtain their token lists. The number of identical words in the token lists of the paper to be checked for plagiarism and a certain paper in the database can be counted, which can be achieved by set operations. For example, calculating the number of elements in the intersection of the two token lists. The token score can be calculated in various ways. For example, simply taking the number of identical words as the score, or calculating the proportion of the number of identical words to the total number of tokens in the paper to be checked for plagiarism as the score.

[0121] The vector score reflects the similarity degree of papers at the semantic level, and the token score reflects the similarity degree of papers at the word level. In order to comprehensively consider the information in these two aspects, the vector score and the token score can be weighted and combined to obtain a comprehensive retrieval score.

[0122] The preset score is a pre-set threshold used to screen out papers with a relatively high similarity to the paper to be checked for plagiarism. Only papers with a retrieval score greater than this threshold are considered target papers that match the paper to be checked for plagiarism.

[0123] By comprehensively considering the information of text vectors and text tokens in the present invention, both the semantic similarity of papers and the similarity at the word level are considered, so that target papers that match the paper to be checked for plagiarism can be found more accurately.

[0124] In an embodiment of the present invention, determining the plagiarism rate of the paper to be checked for plagiarism and at least one target paper includes: processing the retrieval scores of at least one target paper and the paper to be checked for plagiarism through the Sigmoid function to obtain the plagiarism rate of the paper to be checked for plagiarism and at least one target paper.

[0125] In an embodiment of the present invention, after calculating the retrieval scores of the target paper and the paper to be checked for plagiarism, the Sigmoid function can be used to process the retrieval scores of the target paper and the paper to be checked for plagiarism to obtain the duplication rate of the paper to be checked for plagiarism and the target paper, as shown in formula (1):

[0126]

[0127] Among them, similarity is used to represent the duplication rate between the target paper and the paper to be checked for plagiarism, finalScore represents the retrieval scores of the target paper and the paper to be checked for plagiarism, and k is a parameter that controls the slope of the function and can be set according to the needs of users.

[0128] The duplication rate of each target paper and the paper to be checked for plagiarism can be calculated through formula (1).

[0129] In an embodiment of the present invention, the text vector includes a title vector and a paper content vector; according to the text vector, the vector scores of each paper in the retrieval database and the paper to be checked for plagiarism are determined, including: obtaining a first weighting parameter; according to the first weighting parameter and the title vector, determining the cosine similarity score of the title vectors of each paper and the paper to be checked for plagiarism; obtaining a second weighting parameter; according to the second weighting parameter and the paper content vector, determining the cosine similarity score of the paper content of each paper and the paper to be checked for plagiarism; according to the cosine similarity score of the title vectors of each paper and the paper to be checked for plagiarism and the cosine similarity score of the paper content of each paper and the paper to be checked for plagiarism, determining the vector scores of each paper and the paper to be checked for plagiarism.

[0130] In an embodiment of the present invention, the first weighting parameter is used to measure the importance of the title vector in the overall vector score, and the second weighting parameter is used to measure the importance of the paper content vector in the overall vector score. First, the cosine similarity score of the title vectors of each paper and the paper to be checked for plagiarism can be calculated, as shown in formula (2):

[0131]

[0132] Among them, query_title_vector represents the vectorized value of the title of the paper to be checked for plagiarism, title_vector represents the vectorized value of the title of the paper in the retrieval database,[[]] represents the cosine similarity scoring function for the title, K1 represents the first weighting parameter, and Score1 represents the cosine similarity score of the title vector.

[0133] Furthermore, according to the second weighting parameter and the paper content vector, the cosine similarity score of the paper content of each paper and the paper to be checked for plagiarism can be calculated, as shown in formula (3):

[0134]

[0135] Among them, Score2 represents the cosine similarity score between the paper to be checked for duplication and the paper content of the paper, K2 represents the second weighting parameter, query_content_vector represents the vectorized value of the paper content of the paper to be checked for duplication, and content_vector represents the vectorized value of the paper content of the papers in the retrieval database. Represents the cosine similarity scoring function for the paper content.

[0136] In summary, the vector score of the paper and the paper to be checked for duplication can be calculated according to formulas (2) and (3), that is, Score3 = Score1 + Score2. Further, the vector scores of each paper and the paper to be checked for duplication can be calculated.

[0137] By calculating the similarity of the title vector and the paper content vector respectively and assigning different weights, the present invention can comprehensively consider the information of both the paper title and the content, and more comprehensively evaluate the similarity degree between papers.

[0138] In an embodiment of the present invention, text tokenization includes title tokenization and paper content tokenization. Determining the tokenization scores of each paper in the retrieval database and the paper to be checked for duplication includes: determining the first retrieval score corresponding to the title tokenization of each paper and the paper to be checked for duplication and the second retrieval score corresponding to the paper content tokenization; obtaining the third weighting parameter of the first retrieval score and the fourth weighting parameter of the second retrieval score; and determining the tokenization scores of each paper and the paper to be checked for duplication according to the first retrieval score, the second retrieval score, the third weighting parameter, and the fourth weighting parameter.

[0139] In the embodiment of the present invention, the third weighting parameter is used to measure the importance of the first retrieval score in the final tokenization score, and the fourth weighting parameter is used to measure the importance of the second retrieval score in the final tokenization score.

[0140] First, the first retrieval score corresponding to the title tokenization of each paper and the paper to be checked for duplication can be determined. The title of the paper to be checked for duplication can be tokenized to obtain a title tokenization list, and at the same time, the title of each paper in the retrieval database is also tokenized to obtain a corresponding title tokenization list. Then, according to the title tokenization list of the paper to be checked for duplication and the title tokenization list of each paper in the retrieval database, the similarity is calculated by the Jaccard similarity method to obtain the first retrieval score.

[0141] Further, the content of the paper to be checked for plagiarism can be segmented to obtain a list of segmented words of the paper content, denoted as, and the content of each paper in the retrieval database is segmented to obtain a corresponding list of segmented words of the paper content. Then, according to the list of segmented words of the paper content of the paper to be checked for plagiarism and the list of segmented words of the paper content of each paper in the retrieval database, the similarity is calculated by the Jaccard similarity method to obtain the second retrieval score;

[0142] Specifically, the segmentation score can be calculated by formula (4):

[0143]

[0144] Among them, K3 refers to the third weighting parameter of the first retrieval score, K4 refers to the fourth weighting parameter of the second retrieval score, titleScore refers to the first retrieval score, contentScore refers to the second retrieval score, and Score3 refers to the segmentation score of the paper and the paper to be checked for plagiarism.

[0145] Further, the retrieval score finalScore of the paper to be checked for plagiarism and the paper can be calculated according to formulas (2)-(4), finalScore =

[0146]

[0147] It should be noted that the sum of K1 + K2 + K3 + K4 is 1, and the specific values of K1, K2, K3, and K4 can be set according to the needs of users and are not limited here.

[0148] The present invention can comprehensively consider the similarity of the paper title and content at the word level by calculating the retrieval scores of the title segmentation and the paper content segmentation respectively and performing weighted combination.

[0149] In an embodiment of the present invention, it further includes: obtaining the text data of historical papers; vectorizing the text data of historical papers, and storing the vectorized text data of historical papers in the retrieval database; determining the vector scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text vectors, including: determining the vector scores of the text vectors of each historical paper and the paper to be checked for plagiarism in the retrieval database according to the text vectors.

[0150] In the embodiments of the present invention, thesis documents can be downloaded from academic databases first, and then the plain text content can be extracted from the documents (such as PDF, Word, etc.) through text extraction technology. Alternatively, relevant thesis texts can be obtained from the internal thesis repositories of institutions. For example, the thesis plagiarism detection system of a certain university will obtain the thesis text data submitted by students over the years from the school's thesis database for degree theses, and download academic journal papers in related fields from mainstream academic databases as historical thesis data.

[0151] After obtaining the text data of historical theses, the text data can be vectorized. Specifically, common vectorization methods include the Bag of Words model, word embeddings (such as Word2Vec, GloVe, FastText, etc.), Doc2Vec, etc. Taking word embeddings as an example, each word can be mapped to a point in a high-dimensional vector space, so that words with similar semantics are also close in the vector space. Through these methods, each word or the entire document in the historical theses is transformed into a corresponding vector representation, thus converting the text information into a form that can be mathematically operated and compared by a computer.

[0152] Furthermore, the vectorized text data of historical theses can be stored in a retrieval database. The retrieval database can be a relational database (such as MySQL, PostgreSQL, etc.), a non-relational database (such as MongoDB, Redis, etc.), or a database specifically designed for vector retrieval (such as Milvus, etc.). When storing, a unique identifier (such as a thesis ID) will be assigned to each historical thesis, and this identifier will be stored in the retrieval database together with the corresponding text vector and other relevant metadata (such as the thesis title, author, publication time, field of study, etc.) so that the relevant information of each thesis can be retrieved quickly and accurately later.

[0153] After storing the text data of historical theses in the retrieval database, when it is necessary to retrieve the thesis to be checked for plagiarism, the vector scores of the text vectors of each historical thesis and the thesis to be checked for plagiarism can be determined in the retrieval database according to the text vector of the thesis to be checked for plagiarism.

[0154] By pre-vectorizing historical theses and storing them in the retrieval database, the present invention can directly obtain text vectors from the database for similarity calculation during plagiarism detection, avoiding the repeated vectorization process and greatly improving the efficiency of plagiarism detection. Especially for large-scale thesis plagiarism detection tasks, this pre-processing and storage method can significantly reduce the calculation time.

[0155] In an embodiment of the present invention, it further includes: performing word segmentation on the text data of historical papers to obtain keywords of the text data of historical papers; storing the keywords of the text data of historical papers in a retrieval database; and determining the word segmentation scores of each paper in the retrieval database and the paper to be checked for duplication according to the word segmentation, including: determining the word segmentation scores of the text segmentation of each historical paper in the retrieval database and the paper to be checked for duplication according to the word segmentation.

[0156] In the embodiment of the present invention, word segmentation is a basic task in natural language processing. Its purpose is to divide continuous text data into individual words or phrases according to certain rules. For the text data of historical papers, whether it is the title, abstract or body part, word segmentation processing is required. For example, for Chinese texts, tools such as Jieba Segmentation and THULAC can be used. They can cut sentences into individual words according to the word library and algorithms. For English texts, tools such as NLTK (Natural Language Toolkit) and spaCy can be used for word segmentation.

[0157] Based on the word segmentation, keywords are extracted from the word segmentation results through certain algorithms or rules. Common keyword extraction methods include methods based on word frequency statistics (such as TF-IDF, that is, term frequency-inverse document frequency), methods based on text topic models (such as LDA, Latent Dirichlet Allocation), etc. Through these methods, keywords that can represent the core content of the paper can be extracted from the text data of historical papers.

[0158] After the keywords are extracted, they can be stored in a retrieval database. The retrieval database can be a relational database (such as MySQL, Oracle), or a non-relational database (such as MongoDB, Redis). When storing, usually the unique identifier of each historical paper (such as paper number, DOI, etc.) is associated and stored with the corresponding keywords. For example, in a relational database, a table containing paper identifier and keyword fields can be created, and each row record corresponds to a historical paper and its keyword information; in a non-relational database, the information of each historical paper can be stored in the form of a document, which contains an array of keywords. In this way, in the subsequent retrieval process, the corresponding keyword data can be quickly obtained according to the paper identifier.

[0159] In the retrieval database, the keyword data of each historical paper is traversed, and the word segmentation score is calculated by comparing the keywords of the paper to be checked for duplicates with the keywords of each historical paper. Common calculation methods include: counting the number of identical keywords, that is, counting the number of keywords that appear in the paper to be checked for duplicates and the historical paper. The more the number, the higher the similarity between the two papers at the word level; or using Jaccard Similarity, which measures the similarity by calculating the ratio of the number of elements in the intersection of two keyword sets to the number of elements in the union. The closer the ratio is to 1, the higher the keyword similarity between the two papers. The values obtained according to these calculation methods are the word segmentation scores of each historical paper and the paper to be checked for duplicates. The score reflects the degree of similarity between the two papers at the text segmentation level.

[0160] In one embodiment of the present invention, a corresponding text vector is determined based on the text data of the paper to be checked for duplicates, including: preprocessing the text data to obtain preprocessed text data; and processing the preprocessed text data using a vectorization model to obtain a corresponding text vector.

[0161] In an embodiment of the present invention, the process of detecting and preprocessing text data may include the following steps:

[0162] Text cleaning: The text data of the paper to be checked for duplicate content may contain some noise information, such as special characters (such as @, #, $, etc.), HTML tags (if the text is obtained from a web page), redundant blank characters, etc. The purpose of text cleaning is to remove this noise information and make the text cleaner and more standardized. For example, for a paper text captured from a web page, there may be a large number of HTML tags. These tags can be removed through methods such as regular expressions, leaving only the text content.

[0163] Case conversion: Convert all letters in the text to uppercase or lowercase. This is because in many cases, the case of a word does not affect its semantics. Unifying the case can reduce data diversity and improve the consistency of subsequent processing. For example, convert "Apple" and "apple" to "apple".

[0164] Word segmentation: Split continuous text into independent words or phrases according to specific rules. For Chinese text, tools such as Jieba Word Segmenter and THULAC can be used; for English text, tools such as NLTK and spaCy can be used. For example, for the Chinese sentence "I like natural language processing," Jieba Word Segmenter might segment it into "I, like, natural language, processing."

[0165] Remove stop words: Stop words refer to words that appear frequently in the text but contribute little to the semantic expression of the text, such as "的", "了", "在" in Chinese, and "the", "and", "is" in English. Removing stop words can reduce redundant information in the text and improve the efficiency and accuracy of text processing. By maintaining a stop word list, stop words in the text can be removed.

[0166] Lemmatization (for English text): This method restores different word forms (such as the past tense, past participle, present participle of verbs, and the plural form of nouns) to their base form. For example, "running" and "ran" are restored to "run," and "books" are restored to "book." This allows words with the same semantics to be represented uniformly, reducing the dimensionality of the data.

[0167] After preprocessing, the preprocessed text data can be vectorized and converted into corresponding text vectors through a vectorization model. The present invention provides a basis for subsequent paper duplication checking and analysis by converting the text data of the paper to be checked for duplicate content into a text vector form that can be understood and processed by a computer. These text vectors can reflect the characteristics and semantic information of the text from different angles, which helps to improve the accuracy and efficiency of duplication checking.

[0168] In one embodiment of the present invention, the vectorized model is a multimodal hybrid embedding model.

[0169] In an embodiment of the present invention, a multimodal hybrid embedding model is generally composed of multiple modality-specific encoders and a fusion module. The encoder of each modality is responsible for converting the data of the modality into a low-dimensional feature vector, and then the feature vectors of these different modalities are fused through the fusion module to obtain a multimodal embedding vector. This embedding vector not only contains the information of each modality, but also reflects the relationship between different modalities.

[0170] It should be noted that the vectorized model may also adopt other models, and the specific one to be adopted can be set according to user needs and is not limited here.

[0171] In one embodiment of the present invention, corresponding text segmentation is determined based on the text data of the paper to be checked for duplicates, including: preprocessing the text data to obtain preprocessed text data; and using a text segmenter to perform segmentation processing on the preprocessed text data to obtain text segmentation.

[0172] In an embodiment of the present invention, the process of preprocessing text data may include:

[0173] Removing special characters and noise: The text data of the paper to be checked for duplication may contain various special characters (such as punctuation marks, mathematical symbols, emojis, etc.) and some meaningless noise (such as typesetting marks, HTML tags, etc.). For example, for the paper text obtained from the web, there may be such as 、 These HTML tags are of no practical help for text segmentation and subsequent analysis. They need to be removed through methods such as regular expressions to make the text cleaner and more standardized.

[0174] Case conversion: To reduce data diversity, all letters in a text are usually converted to uppercase or lowercase. In natural language processing, the case of a word often does not affect its semantics or part of speech. Unifying the case can make subsequent processing more consistent. For example, converting "Apple" and "apple" to "apple" facilitates operations such as word segmentation and vocabulary statistics.

[0175] Stop word removal: Stop words are words that appear frequently in a text but contribute little to the text's semantics. Different languages have their own stop word lists, such as "the," "and," and "is" in English, and "的," "了," and "在" in Chinese. Maintaining a stop word list and removing these stop words from a text can reduce redundant information, improve word segmentation efficiency and accuracy, and help highlight key information.

[0176] Furthermore, a text word segmenter can be used to perform word segmentation on the preprocessed text data to obtain text word segmentations. Commonly used word segmenters include Jieba word segmenter, THULAC, and IK text word segmenter.

[0177] In one embodiment of the present invention, the text word segmenter is an IK text word segmenter.

[0178] In an embodiment of the present invention, the IK word segmenter provides two word segmentation modes, namely, the intelligent word segmentation mode and the fine-grained word segmentation mode. The intelligent word segmentation mode will segment the text in a relatively coarse-grained manner according to the common collocation and semantic understanding of words, and identify complete words as much as possible. It is suitable for scenarios where texts need to be quickly understood and processed. The fine-grained word segmentation mode will segment the text into smaller word units, which can more accurately reflect the semantic structure of the text and is suitable for scenarios where higher requirements are placed on the accuracy of text analysis.

[0179] The IK tokenizer is primarily dictionary-driven and uses a forward maximum matching algorithm for token segmentation. This algorithm starts at the beginning of the text and searches the dictionary for the longest matching term from left to right. For example, for the text "computer science and technology," the tokenizer will first try to match "computer science and technology" as a whole in the dictionary. If it finds a match, it will use that as the token segmentation result. If not, the tokenizer will gradually shorten the match length, trying "computer science" and "computer" separately, until a matching term is found or no further matches can be found.

[0180] The IK word segmenter not only integrates dictionary matching but also analyzes Chinese semantics and grammar. It can recognize some common Chinese grammatical structures and semantic combinations. For example, it can determine the relationship between auxiliary words such as "的", "地", and "得" and the words before and after them, as well as some fixed word combinations, so as to perform word segmentation more accurately. For example, for "beautiful flowers", it can correctly segment "美丽" as an adjective phrase and "花" as a noun, rather than simply separating "美丽" and "的开花".

[0181] It should be noted that the vocabulary in the present invention has a regular update strategy, which continuously introduces domain-specific vocabulary and new vocabulary to improve the practicality of the word segmentation results and the applicability to specific fields.

[0182] In one embodiment of the present invention, the method further includes: obtaining characteristic information of at least one target paper; the characteristic information includes at least one of title information, author information, and publishing unit information; and displaying the characteristic information of at least one target paper.

[0183] In an embodiment of the present invention, the target paper is a paper that is determined to have a certain degree of similarity with the paper to be checked for duplicate copies in a previous duplicate copy checking process after comparing the text vectors, text segmentation, etc. with the paper to be checked for duplicate copies.

[0184] The method of obtaining feature information depends on the storage location and format of this information. If these target papers are stored in an academic database, then relevant information can be obtained through database query statements (such as SQL statements). For example, from a relational database that stores academic papers, use a SELECT statement to extract title information, author information, and publishing unit information.

[0185] The title information is a high-level summary of the core content of the paper and can usually directly reflect the theme of the paper. The author information includes the name of the person who wrote the paper, which helps to understand the creative subject of the paper. The publishing unit information indicates the institution to which the paper belongs and may be related to the background and resources of the research. Obtaining this characteristic information can provide rich content for subsequent analysis and presentation.

[0186] There are many ways to display this feature information. The most common way is to present it in the form of tables, lists, etc. on the user interface. For example, when displayed in a table, each row represents a target paper, and each column corresponds to the title information, author information, and publishing unit information.

[0187] A more intuitive visualization method can also be adopted. For example, in an information panel, the title is highlighted in a larger font, and the author information and the information of the publishing unit are arranged in a smaller font below the title. During the display process, according to the user's needs, these information can be further processed, such as providing links for the user to click to view the full text or more detailed information of the target paper.

[0188] By presenting the characteristic information of the target papers, users can quickly understand the basic situation of these papers, including the research topics, the academic backgrounds of the authors, etc. This has important reference value for users to judge the relationship between the papers to be checked for plagiarism and the target papers, as well as to evaluate the innovation and originality of the papers to be checked for plagiarism.

[0189] The present invention discloses a method for checking paper plagiarism. It can capture semantic information through text vectorization, combine word segmentation technology to process the text structure, take into account content similarity and keyword matching, and can efficiently screen out paper documents with relatively high similarity, improving the accuracy of plagiarism checking. It can display the target papers through the plagiarism checking suggestions and repetition rates generated by a large model, which can help users quickly locate plagiarism problems and improve the efficiency of plagiarism checking for users.

[0190] Refer to Figure 2 , which shows the structural block diagram of a paper plagiarism checking device provided by an embodiment of the present invention. The device may include the following modules:

[0191] A text acquisition module 201, configured to acquire the text data of the paper to be checked for plagiarism;

[0192] A first determination module 202, configured to determine the corresponding text vector and text word segmentation according to the text data of the paper to be checked for plagiarism;

[0193] A second determination module 203, configured to determine at least one target paper that matches the paper to be checked for plagiarism according to the text vector and text word segmentation;

[0194] A third determination module 204, configured to determine the repetition rate between the paper to be checked for plagiarism and at least one target paper;

[0195] A fourth determination module 205, configured to determine the plagiarism checking suggestions for the paper to be checked for plagiarism and at least one target paper through a large language model;

[0196] A display module 206, configured to display at least one target paper according to the repetition rate and plagiarism checking suggestions between the paper to be checked for plagiarism and at least one target paper.

[0197] The present invention discloses a paper plagiarism detection device, which can capture semantic information through text vectorization, process the text structure in combination with the word segmentation technology, take into account content similarity and keyword matching, and can efficiently screen out paper documents with high similarity, improving the accuracy of plagiarism detection. It can display the target paper through the plagiarism detection suggestions generated by the large model and the repetition rate, which can help users quickly locate plagiarism problems and improve the plagiarism detection efficiency for users.

[0198] In an embodiment of the present invention, the display module may include:

[0199] A sorting sub-module for sorting at least one target paper in descending order of the repetition rate to generate a first sorted list;

[0200] An adding sub-module for adding a first identifier to the target papers with a repetition rate higher than the first threshold in the first sorted list, adding a second identifier to the target papers with a repetition rate between the second threshold and the first threshold, and hiding the target papers with a repetition rate lower than the third threshold to obtain a second sorted list;

[0201] A display sub-module for adding corresponding plagiarism detection suggestions to at least one target paper in the second sorted list and displaying them.

[0202] In an embodiment of the present invention, the plagiarism detection suggestions include at least one of content expansion suggestions, format modification suggestions, citation specification verification suggestions, logic modification suggestions, and semantic paraphrasing suggestions.

[0203] In an embodiment of the present invention, when the plagiarism detection suggestions include semantic paraphrasing suggestions, the fourth determination module may include:

[0204] A first determination sub-module for determining the repeated text in the paper to be detected and the target paper;

[0205] An extraction sub-module for extracting that the repeated text is key semantic information; the key semantic information includes logical relationships and core content;

[0206] An input sub-module for inputting the key semantic information into a large language model to obtain various expression forms and language structures that match the key semantic information;

[0207] A generation sub-module for generating plagiarism detection suggestions for the repeated text of the paper to be detected and the target paper according to the various expression forms and language structures.

[0208] In an embodiment of the present invention, the second determination module may include:

[0209] A first score determination sub-module for determining the vector scores of each paper in the retrieval database and the paper to be detected according to the text vectors;

[0210] A second scoring determination sub-module, configured to determine the word segmentation scores of each paper in the retrieval database and the paper to be checked for duplication according to the word segmentation of the text;

[0211] A third scoring determination sub-module, configured to determine the retrieval scores of each paper and the paper to be checked for duplication according to the vector scores of each paper and the paper to be checked for duplication and the word segmentation scores of each paper and the paper to be checked for duplication;

[0212] A second determination sub-module, configured to determine the papers in which the retrieval scores of the papers and the paper to be checked for duplication are greater than a preset score as target papers.

[0213] In an embodiment of the present invention, the third determination module includes:

[0214] A third determination sub-module, configured to process the retrieval scores of at least one target paper and the paper to be checked for duplication through a Sigmoid function to obtain the duplication rate of the paper to be checked for duplication and at least one target paper.

[0215] In an embodiment of the present invention, the text vector includes a title vector and a paper content vector; the first scoring determination sub-module includes:

[0216] A first acquisition unit, configured to acquire a first weighting parameter;

[0217] A first determination unit, configured to determine the cosine similarity scores of the title vectors of each paper and the paper to be checked for duplication according to the first weighting parameter and the title vector;

[0218] A second acquisition unit, configured to acquire a second weighting parameter;

[0219] A second determination unit, configured to determine the cosine similarity scores of the paper contents of each paper and the paper to be checked for duplication according to the second weighting parameter and the paper content vector;

[0220] A third determination unit, configured to determine the vector scores of each paper and the paper to be checked for duplication according to the cosine similarity scores of the title vectors of each paper and the paper to be checked for duplication and the cosine similarity scores of the paper contents of each paper and the paper to be checked for duplication.

[0221] In an embodiment of the present invention, the text word segmentation includes title word segmentation and paper content word segmentation, and the second scoring determination sub-module includes:

[0222] A fourth determination unit, configured to determine a first retrieval score corresponding to the title word segmentation of each paper and the paper to be checked for duplication and a second retrieval score corresponding to the paper content word segmentation;

[0223] A third acquisition unit, configured to acquire a third weighting parameter for the first retrieval score and a fourth weighting parameter for the second retrieval score;

[0224] A fifth determination unit, configured to determine the word segmentation scores of each thesis and the thesis to be checked for plagiarism according to the first retrieval score, the second retrieval score, the third weighting parameter, and the fourth weighting parameter.

[0225] In an embodiment of the present invention, it further includes:

[0226] A first acquisition module, configured to acquire the text data of historical theses;

[0227] A vectorization module, which is applied to vectorize the text data of historical theses and store the vectorized text data of historical theses in a retrieval database;

[0228] A first scoring determination sub-module, including:

[0229] A first scoring determination unit, configured to determine the vector scores of the text vectors of each historical thesis and the thesis to be checked for plagiarism in the retrieval database according to the text vectors.

[0230] In an embodiment of the present invention, it further includes:

[0231] A word segmentation module, configured to perform text word segmentation on the text data of historical theses to obtain the keywords of the text data of historical theses;

[0232] A storage module, configured to store the keywords of the text data of historical theses in the retrieval database;

[0233] A second scoring determination sub-module, including:

[0234] A second scoring determination unit, configured to determine the word segmentation scores of the text word segmentations of each historical thesis and the thesis to be checked for plagiarism in the retrieval database according to the text word segmentation.

[0235] In an embodiment of the present invention, the first determination module includes:

[0236] A first preprocessing sub-module, configured to preprocess the text data to obtain the preprocessed text data;

[0237] A first processing sub-module, configured to process the preprocessed text data by using a vectorization model to obtain the corresponding text vectors.

[0238] In an embodiment of the present invention, the vectorization model is a multi-modal hybrid embedding model.

[0239] In an embodiment of the present invention, the first determination module includes:

[0240] A second preprocessing sub-module, which preprocesses the text data to obtain the preprocessed text data;

[0241] A second processing sub-module, configured to perform word segmentation on the preprocessed text data by using a text word segmenter to obtain text word segments.

[0242] In an embodiment of the present invention, the text word segmenter is an IK text word segmenter.

[0243] In an embodiment of the present invention, it further includes:

[0244] A second acquisition module, configured to acquire feature information of at least one target paper; the feature information includes at least one of title information, author information, and publication unit information;

[0245] A feature information display module, configured to display the feature information of at least one target paper.

[0246] The present invention discloses a paper duplicate checking device, which can capture semantic information through text vectorization, process the text structure in combination with the word segmentation technology, take into account content similarity and keyword matching, can efficiently screen out paper documents with relatively high similarity, improve the accuracy of duplicate checking, can display the target paper through the duplicate checking suggestions and repetition rate generated by the large model, can help users quickly locate duplicate checking problems, and improve the duplicate checking efficiency for users.

[0247] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, refer to the partial description of the method embodiment.

[0248] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor is configured to execute the steps of the paper duplicate checking method as described above.

[0249] The present invention also discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the paper duplicate checking method as described above are implemented.

[0250] Each embodiment in this specification is described in a progressive manner, and the key point of each embodiment is to illustrate the differences from other embodiments. For the same and similar parts among the embodiments, refer to each other.

[0251] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0252] Embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0253] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0254] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0255] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

[0256] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent in such process, method, article or terminal device. Without further limitation, an element defined by the statement "including one..." does not exclude the presence of additional identical elements in the process, method, article or terminal device including the said element.

[0257] The above has introduced in detail a paper search method, device, equipment and storage medium provided by the present invention. Specific examples are used in this text to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A paper duplication checking method, characterized in that, The method includes: Obtaining the text data of the paper to be checked for plagiarism; Determining the text vector and text segmentation corresponding to the text data of the paper to be checked for plagiarism; Determining at least one target paper that matches the paper to be checked for plagiarism according to the text vector and text segmentation; Determining the plagiarism rate between the paper to be checked for plagiarism and the at least one target paper; Determining a plagiarism check suggestion for the paper to be checked for plagiarism and the at least one target paper through a large language model; Displaying the at least one target paper according to the plagiarism rate between the paper to be checked for plagiarism and the at least one target paper and the plagiarism check suggestion.

2. The method according to claim 1, wherein The displaying the at least one target paper according to the plagiarism rate between the paper to be checked for plagiarism and the at least one target paper and the plagiarism check suggestion includes: Sorting the at least one target paper in descending order of the plagiarism rate to generate a first sorted list; Adding a first identifier to the target papers with a plagiarism rate higher than a first threshold in the first sorted list, adding a second identifier to the target papers with a plagiarism rate between a second threshold and the first threshold, and hiding the target papers with a plagiarism rate lower than a third threshold to obtain a second sorted list; Adding corresponding plagiarism check suggestions to the at least one target paper in the second sorted list and displaying them.

3. The method according to claim 1, wherein The plagiarism check suggestion includes at least one of a content expansion suggestion, a format modification suggestion, a citation specification verification suggestion, a logic modification suggestion, and a semantic paraphrasing suggestion.

4. The method according to claim 3, characterized in that, When the plagiarism check suggestion includes a semantic paraphrasing suggestion, the determining a plagiarism check suggestion for the paper to be checked for plagiarism and the at least one target paper through a large language model includes: Determining the repeated text between the paper to be checked for plagiarism and the target paper; Extracting the key semantic information of the repeated text; the key semantic information includes logical relationships and core content; Inputting the key semantic information into the large language model to obtain various expression forms and language structures that match the key semantic information; Generating a plagiarism check suggestion for the repeated text between the paper to be checked for plagiarism and the target paper according to the various expression forms and language structures.

5. The method according to claim 1, wherein The determining at least one target paper that matches the paper to be checked for plagiarism according to the text vector and text segmentation includes: Determining the vector score of each paper in the retrieval database and the paper to be checked for plagiarism according to the text vector; Determining the segmentation score of each paper in the retrieval database and the paper to be checked for plagiarism according to the text segmentation; Determining the retrieval score of each paper and the paper to be checked for plagiarism according to the vector score of each paper and the paper to be checked for plagiarism and the segmentation score of each paper and the paper to be checked for plagiarism; Determining the papers with a retrieval score greater than a preset score between the papers and the paper to be checked for plagiarism as target papers.

6. The method according to claim 5, wherein The determining the plagiarism rate between the paper to be checked for plagiarism and the at least one target paper includes: Processing the retrieval score of the at least one target paper and the paper to be checked for plagiarism through a Sigmoid function to obtain the plagiarism rate between the paper to be checked for plagiarism and the at least one target paper.

7. The method according to claim 5, characterized in that, The text vector includes a title vector and a paper content vector; determining the vector scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text vector includes: Obtain a first weighting parameter; Determine the cosine similarity scores of the title vectors of each paper and the paper to be checked for plagiarism according to the first weighting parameter and the title vector; Obtain a second weighting parameter; Determine the cosine similarity scores of the paper content of each paper and the paper to be checked for plagiarism according to the second weighting parameter and the paper content vector; Determine the vector scores of each paper and the paper to be checked for plagiarism according to the cosine similarity scores of the title vectors of each paper and the paper to be checked for plagiarism and the cosine similarity scores of the paper content of each paper and the paper to be checked for plagiarism.

8. The method according to claim 5, wherein The text segmentation includes title segmentation and paper content segmentation. Determining the segmentation scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text segmentation includes: Determine a first retrieval score corresponding to the title segmentation of each paper and the paper to be checked for plagiarism and a second retrieval score corresponding to the paper content segmentation; Obtain a third weighting parameter for the first retrieval score and a fourth weighting parameter for the second retrieval score; Determine the segmentation scores of each paper and the paper to be checked for plagiarism according to the first retrieval score, the second retrieval score, the third weighting parameter, and the fourth weighting parameter.

9. The method according to claim 5, wherein It further includes: Obtain the text data of historical papers; Vectorize the text data of the historical papers and store the vectorized text data of the historical papers in the retrieval database; Determining the vector scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text vector includes: Determine the vector scores of the text vectors of each historical paper and the paper to be checked for plagiarism in the retrieval database according to the text vector.

10. The method according to claim 9, wherein It further includes: Perform text segmentation on the text data of the historical papers to obtain the keywords of the text data of the historical papers; Store the keywords of the text data of the historical papers in the retrieval database; Determining the segmentation scores of each paper in the retrieval database and the paper to be checked for plagiarism according to the text segmentation includes: Determine the segmentation scores of the text segmentations of each historical paper and the paper to be checked for plagiarism in the retrieval database according to the text segmentation.

11. The method according to claim 1, wherein Determining the corresponding text vector according to the text data of the paper to be checked for plagiarism includes: Preprocess the text data to obtain preprocessed text data; Process the preprocessed text data using a vectorization model to obtain the corresponding text vector.

12. The method according to claim 11, wherein, The vectorization model is a multi-modal hybrid embedding model.

13. The method according to claim 1, characterized in that, Determining the corresponding text segmentation according to the text data of the paper to be checked for plagiarism includes: Preprocess the text data to obtain preprocessed text data; Perform word segmentation on the preprocessed text data using a text segmenter to obtain the text segmentation.

14. The method according to claim 13, wherein The text segmenter is an IK text segmenter.

15. The method according to claim 1, characterized in that, It further includes: Obtain the feature information of the at least one target paper; The feature information includes at least one of title information, author information, and publication unit information; Display the feature information of the at least one target paper.

16. A paper duplication checking device, characterized in that, The device includes: An acquisition module, configured to acquire the text data of the paper to be checked for duplication; A first determination module, configured to determine the corresponding text vector and text segmentation according to the text data of the paper to be checked for duplication; A second determination module, configured to determine at least one target paper that matches the paper to be checked for duplication according to the text vector and text segmentation; A third determination module, configured to determine the duplication rate between the paper to be checked for duplication and the at least one target paper; A fourth determination module, configured to determine a duplication check suggestion for the paper to be checked for duplication and the at least one target paper through a large language model; A display module, configured to display the at least one target paper according to the duplication rate between the paper to be checked for duplication and the at least one target paper and the duplication check suggestion.

17. An electronic device, characterized in that, Includes: A processor, a memory, and a computer program stored on the memory and capable of running on the processor, where when the computer program is executed by the processor, the steps of the paper duplication check method described in any one of claims 1-15 are implemented.

18. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the paper duplication check method described in any one of claims 1-15 are implemented.

Images