Method for detecting Chinese paper module generated by large language model

Abstract

The invention discloses a method for detecting a Chinese paper module generated by a large language model. The method comprises the following steps: aiming at paper modules such as abstracts, citations and conclusions of Chinese papers, particularly humanity and social science academic papers, collecting sample data written by human beings, generating the sample data through a large language model, integrating the two kinds of sample data, extracting text structure features and text vector features of the sample data as multi-dimensional feature input, and extracting the text structure features and the text vector features of the sample data as multi-dimensional feature input; the variable of whether the large language model is generated or not is used as a classification label for multi-model classification training, and the detection accuracy is improved through multi-model fusion. The method can be widely applied to detection of each structure module of the academic paper, the academic paper module generated by the large language model can be accurately recognized, and the method has high application value.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and artificial intelligence, and in particular to a method for detecting a Chinese paper module generated by a large language model. Background Art

[0002] With the rapid development of artificial intelligence and machine learning, large language models such as GPT, Bard, ERNIE Bot, and PanGu have achieved remarkable achievements in natural language processing (NLP) tasks. These models are able to generate content that is almost indistinguishable from human-written text, including academic papers. However, such progress has also brought a series of problems. Since large language models can generate high-quality academic paper content, some people may use this to generate and submit untrue research results, which not only threatens the fairness and authenticity of the academic community, but also may lead to inaccurate research conclusions and the spread of misleading information. In the face of such problems, there are already a number of model tools that specialize in detecting content generated by large language models, such as AI Content Detector, GPTZero, Winston AI, etc. However, these AI detection tools are not trained for Chinese text generated by large language models, and do not take into account the structural and semantic characteristics of Chinese text, so most of them cannot detect Chinese, and even if they can detect, the effect is not as good as English text.

[0003] There have been some studies on the detection of content generated by large language models for Chinese academic papers. Wang Yibo et al. conducted a classification study on the abstracts of 100 papers in five journals in the library field based on TFIDF characterization technology, and further analyzed the differences in dimensions such as abstract length, number of sentences, vocabulary features, and common collocations (Detection and Difference Comparative Study of Chinese Paper Abstracts Generated by AI and Written by Scholars [J]. Journal of Intelligence, 2023, 42(09): 127-134.). Taking the introductions of eight journals in information science as an example, Guo Xin et al. first analyzed the text structure features of each word granularity, sentence granularity, and chapter granularity, integrated these features with text vector features to detect multiple classification models, and performed interpretability analysis on the models with good classification effects (ChatGPT generates Chinese academic content analysis - taking the field of information science as an example [J / OL]. Library Forum: 1-12 [2023-10-02]. https: / / link.cnki.net / urlid / 44.1306.G2.20230921.1040.004). However, previous research on content detection generated by large language models of Chinese academic papers can only detect a specific structural part of the paper due to the limitation of training data, resulting in the limitation of being limited to specific module detection. Therefore, it is an urgent need to develop a method that can accurately identify the modules of the entire Chinese academic paper generated by a large language model.

[0004] Academic papers in the humanities and social sciences usually explore social, cultural, philosophical, and historical issues. Their writing style is more narrative and explanatory. Compared with science and engineering papers, they may use more subjective and explanatory language, and rely less on numerical data, charts, and statistical analysis. Large language models can help researchers in the humanities and social sciences complete more text content. At the same time, academic papers are long in overall length and cannot be generated by large language models in one go. Instead, they require multiple question-and-answer interactions to generate different module contents. Academic papers mainly include conventional modules such as title, abstract, introduction, literature review, methods, discussion, and conclusion. Taking into account the semantic continuity of content generated by large language models and length limitations, the modules of the paper are more likely to be generated by the large language model as a whole, becoming the "hardest hit area" of content generated by large language models. It is more valuable to detect whether the content of these modules is generated by machines. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a detection method for Chinese paper modules generated by a large language model, especially academic paper modules in humanities and social sciences, so as to accurately identify academic paper modules generated by a large language model.

[0006] The present invention provides a method for detecting a Chinese paper module generated by a large language model, comprising the following steps:

[0007] 1) Collect sample data of Chinese paper modules written by humans and divide them into training set, test set and example set;

[0008] 2) Using the example set as a reference sample, the large language model is used to generate corresponding paper modules for the paper titles in the training set and the test set, respectively, to obtain sample data generated by the large language model;

[0009] 3)Build domain dictionary;

[0010] 4) Integrate the sample data written by humans collected in step 1) and the sample data generated by the large language model in step 2) into three columns of data, namely, the paper title, the paper module, and whether it was generated by the large language model;

[0011] 5) Use natural language processing tools to load the domain dictionary constructed in step 3) and perform word segmentation and part-of-speech tagging on the text content of all paper modules;

[0012] 6) Extracting text structure features of the paper module, wherein the text structure features include chapter granularity, paragraph granularity, sentence granularity and vocabulary granularity;

[0013] 7) Extract text vector features from the paper module;

[0014] 8) Select P classification algorithm models, use the training set data, input the text structure features and text vector features as multi-dimensional features, and use the variable whether the paper module is generated by the large language model as the classification label for model training, and obtain 2P trained classification models;

[0015] 9) Input the test set data into the trained 2P classification models respectively, and calculate the classification accuracy of each classification model based on the classification labels, using A1, A2, ..., A 2p To represent; if the classification accuracy of a certain classification model is lower than the set R%, the classification model will be abandoned, otherwise the classification model will be retained, thereby screening out Q models with better classification effects, and their classification accuracy rates are represented by B1, B2, ... B Q To represent, the highest classification accuracy is represented by B max Indicates; using R% accuracy as the lower limit, the extreme value processing method is used to calculate the weight of each classification model. The formula is as follows:

[0016]

[0017] Among them, W i represents the weight of the i-th classification model, B i Represents the classification accuracy of the i-th classification model;

[0018] 10) For the paper module to be tested, first perform word segmentation and part-of-speech tagging according to step 5), and then perform text structure feature extraction and text vector feature extraction according to steps 6) and 7) respectively; input the text structure features and text vector features of the paper module to be tested into the Q classification models retained in step 9), and record the outputs of the models as Y1, Y2, ..., Y Q , Y i represents the output of the i-th classification model, and its value range is between 0 and 1, “1” marks the paper module generated by the large language model, and “0” marks the paper module written by humans; if ∑W i *Y i >0.5*∑W i , then the detected paper module is determined to be generated by a large language model, otherwise it is determined to be written by humans.

[0019] The present invention is mainly aimed at Chinese academic papers in the humanities and social sciences, including philosophy, history, linguistics, economics, management, law, sociology, anthropology, psychology, etc., and the paper module includes but is not limited to the modules such as the abstract, introduction, and conclusion of the paper. In step 1), a literature database with a relatively complete collection of papers in the humanities and social sciences is selected as the data source, and core journal papers published in the past M years are retrieved, and arranged in descending order according to the citation frequency, and the N highly cited papers in the front are selected to extract the modules such as the abstract, introduction, or conclusion of each paper. The collected paper module data is divided into three parts: a training set, a test set, and a sample set, wherein the sample set is further divided into two parts, respectively in step 2), for constructing the prompts for the input large language model for the training set and the test set. For example, 0.6N of the N highly cited papers collected are classified as a training set, 0.2N are classified as a test set, and 0.2N are classified as a sample set. 0.15N articles in the example set are used for constructing prompts for the training set, and 0.05N articles are used for constructing prompts for the test set, to ensure that the reference contents of the training set and the test set do not overlap and interfere with each other.

[0020] In step 2), a large language model, such as GPT, Bard, ERNIE Bot, Wenxinyiyan, etc., is used to generate the content of a specific paper module based on the paper title, with reference to the paper module samples from the example set. Each paper title in the training set and the test set generates a paper module through the large language model, thereby obtaining sample data generated by the large language model that is equal to the number of sample data written by humans.

[0021] Step 3) Build a domain dictionary based on the keyword data of the paper. The keyword data of papers in related fields can be exported in batches from the literature database, and the domain dictionary can be obtained after merging and removing duplicates.

[0022] Step 4) Integrate the paper modules written by humans and generated by the large language model into three columns of data. In the third column, use "1" to identify the paper modules generated by the large language model, and use "0" to identify the paper modules written by humans. In the subsequent classification model training and application, "1" and "0" are used as classification label outputs.

[0023] In step 5), use natural language processing tools (such as Python's jieba package) to load the domain dictionary, perform word segmentation and part-of-speech tagging on the text content of all paper modules, annotate the word segmentation results as nouns, prepositions, pronouns, auxiliary words, quantifiers, etc., and delete all punctuation marks during the word segmentation process.

[0024] The chapter granularity, paragraph granularity, and sentence granularity described in step 6) are all based on the extraction of the paper module text before word segmentation, wherein the chapter granularity includes text length and time features; the paragraph granularity includes average paragraph length and average number of sentences in a paragraph; the sentence granularity includes average sentence length, sentence pauses, proportion of long sentences, degree of sentence sentiment change, average sentence readability score, and sentence semantic similarity; the vocabulary granularity is based on the extraction of the paper module text after word segmentation, including average vocabulary length, vocabulary diversity, proportion of specific part-of-speech vocabulary, and proportion of stop words.

[0025] For each sentence, a sentiment analysis tool (such as SnowNLP, etc.) is used to generate a sentiment score, and the standard deviation of the sentiment score of the entire paper module is calculated as an indicator of sentiment change, that is, the degree of sentiment change of the sentence. The average readability score of all sentences is taken to get the average sentence readability score, where the readability score of each sentence can be calculated using the Flesch readability score, which is calculated as follows:

[0026] Flesch readability score = 206.835 - (1.015*ASL) - (84.6*ASW)

[0027] Among them, ASL is the average number of words in a sentence, which is equal to the total number of words divided by the number of sentences; ASW is the average number of syllables per word, which is equal to the total number of syllables divided by the total number of words. The total number of words refers to the number of words obtained after word segmentation, and each Chinese character represents a syllable.

[0028] In step 7), a text vector feature extraction model is selected, and the text content is input to obtain the corresponding high-dimensional vector.

[0029] The vectorized representation of text is a method of encoding text information into a high-dimensional space. Such a representation can reveal the importance of words in the text and the semantics behind them. Commonly used text vectorization models include Doc2Vec, ELMo, and BERT. Select a text vector feature extraction model (such as SentenceBERT based on Python's SentenceTransformers library), input text data into the model, and output the L (for example: 768) dimensional vector corresponding to the text. In this way, the 1.2N paper modules in the training set can be converted into a 1.2N×L two-dimensional matrix, and the 0.4N paper modules in the test set can be converted into a 0.4N×L two-dimensional matrix.

[0030] In step 8), select P (preferably, P≥4) classic classification algorithm models such as support vector machine, random forest, gradient boosting machine (such as XGBoost), multi-layer perceptron, etc. In the training of the classification model, in order to avoid the problem of overfitting, the training set data is further divided into a training set and a validation set, and the K-fold cross-validation method is used to calculate the model evaluation index, and the calculated indicators include precision, recall rate and F1-Score value; based on the first training result of the model, the F1-Score value is used as the evaluation index for hyperparameter selection, and the hyperparameter with the best performance is selected from the predefined hyperparameter range through a random search method, and the models trained with text structure features and text vector features are respectively tuned a certain number of times; different hyperparameters need to be optimized for different models.

[0031] The paper module generated by the large language model is marked with "1", and the paper module written by humans is marked with "0". The classification label output by the classification model trained in step 8) is "1" or "0". i Represents the output of the i-th classification model, and its value is 1 or 0. 1 means that the model determines that the input paper module is generated by the large language model, and 0 means that the model determines that the input paper module is written by humans. If ∑W i *Y i >0.5*∑W i , then the detected paper module is determined to be generated by a large language model, otherwise it is determined to be written by humans.

[0032] The present invention proposes a detection method for Chinese paper modules generated by a large language model based on the fusion of text structure features and text vector features, which can be applied to servers, computer devices, smart mobile terminals, etc. Compared with the existing detection methods, the main innovations and contents of the present invention are as follows:

[0033] 1. Comprehensive training data: We select core journal papers in the humanities and social sciences as data sources, and collect data in detail from databases such as China National Knowledge Infrastructure to ensure the representativeness and quality of the samples.

[0034] 2. Detailed feature construction: Extract rich structural and semantic features at different granularity levels from vocabulary, sentences, paragraphs to chapters. For example: introduce advanced sentence vectorization methods to capture deep semantic information; introduce sentiment analysis tools to construct sentiment change features of the paper module.

[0035] 3. Multi-model classification training and fusion: Select a variety of classic classification models for training, such as support vector machine, random forest

[0036] And gradient boosting machine, etc. Through model fusion technology, the advantages of multiple models are combined to further improve the accuracy of detection.

[0037] 4. Wide application of the model: This method is not limited to the detection of specific modules, but can be widely applied to various structural modules of academic papers and has high application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the detection method of the academic paper module generated by the large language model according to the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and technical effect of the present invention clearer, the specific implementation process of the present invention is described in detail in the form of an embodiment in combination with the accompanying drawings. Obviously, what is described here is only an exemplary embodiment of the present invention, not all embodiments, and is not intended to limit the present invention.

[0040] A detection method for a humanities and social sciences academic paper module generated by a large language model comprises the following steps:

[0041] Step 1: Sample data collection

[0042] 1.1 Select the scope of sample paper collection. Select a literature database with a relatively complete collection of papers in the humanities and social sciences as the data source website, select the humanities and social sciences related fields in the database document classification catalog, select core journal papers published in the past M years for retrieval, and sort them in descending order according to the frequency of citation. The content ranked in the front is the highly cited papers in the humanities and social sciences in the past M years. Taking China National Knowledge Infrastructure (https: / / www.cnki.net / ) as an example, select "Philosophy and Humanities", "Social Sciences I", "Social Sciences II", and "Economics and Management Sciences" in the document classification catalog in the "Journal" tab, and select "Core Journals", "CSSCI" and "CSCD" as the journal source type to retrieve highly cited papers in the humanities and social sciences in the past M years.

[0043] 1.2 Download sample papers and parse the contents of each module. Download the first N papers (such as 1,000) in sequence according to the search results, and manually extract the abstract, introduction or conclusion module of each paper. The abstract module will be clearly indicated in each paper. There are two cases for the introduction module: for papers whose main text starts with the chapter title, the introduction module corresponds to the content of the first chapter; for papers whose main text starts directly from the paragraph content, the introduction module corresponds to the content from the beginning of the text to before the title of the first chapter. The conclusion module corresponds to the content of the last chapter of the main text of the paper. Through the above structural features, the text content of each module of N papers can be parsed from the paper.

[0044] 1.3 Division of sample data set. The N collected papers are divided into three parts: the first part is the training set of the classifier, which is used to train the classification model, including 0.6N papers in total; the second part is the test set of the classifier, which is used to evaluate the generalization ability of the model, including 0.2N papers in total; the third part is the example set when writing the large model, which is used to construct the prompts input to the large model, including 0.2N papers in total, of which 0.15N are used for the prompt construction of the training set, and 0.05N are used for the prompt construction of the test set, to ensure that the reference content of the training set and the test set do not overlap and do not interfere with each other.

[0045] 1.4 Large language model sample data collection. For the training set papers and test set papers, call the large language model interface (such as gpt-3.5-turbo) and require the large language model to generate the content of a specific module according to the specified format based on the title and reference examples of each paper, where the reference examples come from the example set. The content format of the prompt transmitted by calling the interface is as follows:

[0046] "I want you to play the role of a researcher in the humanities and social sciences, and you need to help me complete the abstract (or introduction, conclusion, the same below) of an academic paper in the humanities and social sciences. Before giving a specific topic, I will first give you two sample paper titles and corresponding abstracts for reference, and then give you a specific paper title. You can imitate the language style of these samples to write.

[0047] Example of the abstract of the first paper: {"Title":"XXX(Title of reference paper 1)","Abstract":"(Abstract of reference paper 1)"}

[0048] Example of abstract of the second paper: {"Title":"XXX(Title of reference paper 2)","Abstract":"(Abstract of reference paper 2)"}

[0049] Next, you need to complete the writing of a paper abstract according to the above requirements. The title of the paper is: XXXXXX (paper with abstract to be completed)"

[0050] Thus, we obtain the summary content generated by the large language model for the 0.6N training set paper titles and the summary content generated by the large model for the 0.2N test set paper titles.

[0051] 1.6 Construction of field dictionary. Using the same paper search method as in Section 1.1, we batch-exported the keyword data of the first Z papers (e.g., 5,000 papers) through the "Export Literature" function provided by CNKI, and then merged and deduplicated to obtain a dictionary in the field of humanities and social sciences.

[0052] Step 2: Sample data feature extraction

[0053] 2.1 Sample data integration. After completing data acquisition, the abstracts (or introductions, conclusions, the same below) written by humans and generated by the large language model are integrated into three columns of data, namely the title of the paper, the abstract of the paper, and whether it is generated by the large language model (1 represents machine-generated, 0 represents human-written). 0.8N paper titles correspond to 1.6N abstracts, of which 1.2N are training sets and 0.4N are test sets. The ratio of positive and negative samples in the two sets is equal.

[0054] 2.2 Word segmentation and part-of-speech tagging of summary text. Use natural language processing tools (such as Python's jieba package), load the domain dictionary built in Section 1.6, perform word segmentation and part-of-speech tagging on all summary text content, annotate the word segmentation results as nouns, verbs, prepositions, etc., and delete all punctuation marks during the word segmentation process.

[0055] 2.3 Extraction of structural features of sample data. Text structural features are defined based on the statistical and structural information of the text. Features are extracted from four granularities: chapter, paragraph, sentence, and vocabulary. These features reflect the structure and style of the text. The following 18 features are used to calculate the feature values ​​of the summary texts of the training set and the test set:

[0056] (1) Chapter granularity (based on text extraction before word segmentation)

[0057] ① Text length: the number of characters in the summary text. Each Chinese character is considered as a character, each English word is considered as a character, and each half-width or full-width punctuation mark is considered as a character. The same applies below.

[0058] ② Time feature: the number of strings in the form of "XXXX year XX month" in the article (using regular expressions to count the number of times this form appears in the text).

[0059] (2) Paragraph granularity (based on text extraction before word segmentation)

[0060] ① Average paragraph length: text length divided by the total number of text paragraphs. Each line break is considered to start a new paragraph.

[0061] ② Average number of sentences in a paragraph: the total number of sentences in the text divided by the total number of paragraphs in the text. A sentence ending with a period, question mark, or exclamation mark is considered a sentence, and a new paragraph begins after a line break.

[0062] (3) Sentence granularity (based on text extraction before word segmentation, a sentence ending with a period, question mark, or exclamation mark is considered a sentence)

[0063] ①Average sentence length: text length divided by the total number of sentences.

[0064] ② Pauses in sentences: For each sentence, count the number of commas, semicolons, and periods, and then divide by the total number of words in the sentence to get the frequency of punctuation marks. Use the standard deviation of the punctuation mark frequencies of all sentences to calculate pauses in sentences.

[0065] ③The proportion of long sentences: the number of sentences with more than 100 characters divided by the total number of sentences.

[0066] ④ Degree of sentiment change of sentences: For each sentence, use sentiment analysis tools (such as SnowNLP, etc.) to generate a sentiment score. Calculate the standard deviation of the sentiment score of the entire paper module as an indicator of sentiment change.

[0067] ⑤ Average sentence readability score: Take the average of the readability scores of all sentences. The score of each sentence can use a readability score indicator (for example: Flesch readability score). The Flesch readability score calculation formula is as follows:

[0068] Flesch readability score = 206.835 - (1.015*ASL) - (84.6*ASW)

[0069] In the above formula, ASL (Average Sentence Length) is the average number of words in a sentence, which is equal to the total number of words divided by the number of sentences; ASW (Average Syllables per Word) is the average number of syllables per word, which is equal to the total number of syllables divided by the total number of words. The Flesch readability score is usually applied to English texts. When applied to Chinese texts, it needs to be processed through word segmentation to obtain the number of words, where each Chinese character represents a syllable.

[0070] ⑥ Sentence semantic similarity: In order to capture the semantic information of sentences more deeply, we use a sentence vectorization method to generate its corresponding semantic vector representation (for example: SentenceBERT model). This will transform each sentence into a fixed-size numerical vector that captures the semantic content of the sentence. For each sentence in the paper (except the first and last sentences), calculate its cosine similarity with the previous and next sentences. For the first sentence, only calculate its similarity with the second sentence; for the last sentence, only calculate its similarity with the second to last sentence. For each sentence, take the average of its similarities with the previous and next sentences as the similarity value of the sentence. Calculate the average of the similarity values ​​of all sentences in the paper. This will give the sentence similarity of the entire paper.

[0071] (4) Lexical granularity (text extraction based on word segmentation)

[0072] ① Average word length: the total number of characters in all words divided by the total number of words;

[0073] ② Lexical diversity: the number of words after all the words in the text are removed divided by the total number of words;

[0074] ③ The proportion of words of a specific part of speech: the number of words of a certain part of speech divided by the total number of words. A total of five parts of speech were selected, namely pronouns, nouns, quantifiers, auxiliary words, and conjunctions, to construct five features;

[0075] ④ The proportion of stop words: Use a common Chinese stop word list (for example, the Harbin Institute of Technology stop word list) and count the number of stop words divided by the total number of words.

[0076] 2.4 Sample data vector feature extraction. The vectorized representation of text is a method of encoding text information into a high-dimensional space. Such a representation can reveal the importance of words in the text and the semantics behind them. There are currently a variety of text vectorization models, such as Doc2Vec, ELMo, and BERT. Select a text vector feature extraction model (such as SentenceBERT based on Python's SentenceTransformers library). We can input the training set into the model and output the L (for example: 768) dimensional vector corresponding to the text. In this way, we can convert the 1.2N abstracts in the training set into a 1.2N*L two-dimensional matrix, and convert the 0.4N abstracts in the test set into a 0.4N*L two-dimensional matrix.

[0077] Step 3: Training the classification model

[0078] 3.1 Model selection and indicator calculation. Select P (e.g. 5) classic classification algorithm models such as support vector machine, random forest, gradient boosting machine (such as XGBoost), multi-layer perceptron, etc., use text structure features and text vector features as multi-dimensional feature inputs respectively, and use the variable whether the abstract of the paper is generated by a large language model as the classification label for model training. In order to avoid overfitting problems, the training set data needs to be further divided into a training set and a validation set, and the K-fold cross-validation method is used to calculate the model evaluation indicators. The calculated indicators include precision, recall, and F1-Score value.

[0079] 3.2 Model hyperparameter tuning. Based on the results of the first model training, the F1-Score value is used as the evaluation indicator for hyperparameter selection. The best performing hyperparameter is selected from the predefined hyperparameter range through a random search method. Similarly, the models trained with text structure features and text vector features are tuned a certain number of times (e.g., 100 times). Different hyperparameters need to be optimized for different models. Taking the support vector machine as an example, the hyperparameters that need to be tuned include the kernel function type (kernel) and the regularization parameter (C).

[0080] Step 4: Testing and fusion of classifiers

[0081] 4.1 Classifier test. The test set data is input into the trained 2P models (P classification algorithms × 2 text features), and the accuracy of each classification effect is calculated by combining the classification labels, using A1, A2, ..., A 2p To represent the generalization ability of each model.

[0082] 4.2 Classifier weight calculation. For the classification accuracy of 2P models A1~A 2p If the classification accuracy of a model is lower than R% (such as 90%), the model is discarded. Otherwise, the model is retained. Q models with better classification effects can be selected from them. Their classification accuracy rates are represented by B1, B2, ..., B Q To represent, the model with the highest classification accuracy is represented by B max Next, the power coefficient method is used to weight the model, and the extreme value processing method is used to calculate the weight of each model with the accuracy of R% as the lower limit. The formula is as follows:

[0083]

[0084] Among them, W i represents the weight of the i-th classification model, B i Represents the classification accuracy of the i-th classification model, i is an integer from 1 to Q, and R% is the set lower limit of the classification accuracy.

[0085] 4.3 Model application. When testing a new paper module, first complete the word segmentation of the new paper module data according to the content of Section 2.2, then extract the text structure features according to the indicator calculation method defined in Section 2.3, and calculate the text vector according to the text vector feature extraction model trained in Section 2.4. Then, input the text structure features and text vector features into the Q models that are still retained, and record the output of the model as Y1, Y2, ..., Y Q If ∑W i *Y i >0.5*∑W i , then the module being tested is determined to be generated by a large language model, otherwise it is determined that the content being tested is written by humans, where Y i Represents the output of the i-th classification model, and its value range is between 0 and 1.

[0086] Finally, it should be noted that the purpose of publishing the embodiments is to help technicians further understand the technical solutions and effects of the present invention, rather than to limit the scope of the present invention. Those skilled in the art should understand that, according to the needs of practical applications, various modifications, combinations and substitutions can be made to the specific implementation plans, and any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention, and the scope of protection of the present invention shall be based on the scope defined by the attached claims.

Claims

1. A method for detecting a Chinese paper module generated by a large language model, comprising the following steps: 1) Collect sample data of Chinese paper modules written by humans and divide them into training set, test set and example set; 2) Using the example set as a reference sample, the large language model is used to generate corresponding paper modules for the paper titles in the training set and the test set, respectively, to obtain sample data generated by the large language model; 3)Build domain dictionary; 4) Integrate the sample data written by humans collected in step 1) and the sample data generated by the large language model in step 2) into three columns of data, namely, the paper title, the paper module, and whether it was generated by the large language model; 5) Use natural language processing tools to load the domain dictionary constructed in step 3) and perform word segmentation and part-of-speech tagging on the text content of all paper modules; 6) Extracting text structure features of the paper module, wherein the text structure features include chapter granularity, paragraph granularity, sentence granularity and vocabulary granularity; 7) Extract text vector features from the paper module; 8) Select P classification algorithm models, use the training set data, input the text structure features and text vector features as multi-dimensional features, and use the variable whether the paper module is generated by the large language model as the classification label for model training, and obtain 2P trained classification models; 9) Input the test set data into the trained 2P classification models respectively, and calculate the classification accuracy of each classification model based on the classification labels, using A1, A2, ..., A 2p To represent; if the classification accuracy of a certain classification model is lower than the set R%, the classification model will be abandoned, otherwise the classification model will be retained, thereby screening out Q models with better classification effects, and their classification accuracy rates are represented by B1, B2, ... B Q To represent, the highest classification accuracy is represented by B max express; The weight of each classification model is calculated using the extreme value processing method with R% accuracy as the lower limit. The formula is as follows: Among them, W i represents the weight of the i-th classification model, B i Represents the classification accuracy of the i-th classification model; 10) For the paper module to be tested, first perform word segmentation and part-of-speech tagging according to step 5), and then perform text structure feature extraction and text vector feature extraction according to steps 6) and 7) respectively; input the text structure features and text vector features of the paper module to be tested into the Q classification models retained in step 9), and record the outputs of the models as Y1, Y2, ..., Y Q , whose value range is between 0 and 1, "1" marks the paper module generated by the large language model, and "0" marks the paper module written by humans; if ΣW i *Y i >0.5*∑W i , then the detected paper module is determined to be generated by a large language model, otherwise it is determined to be written by humans.

2. The detection method according to claim 1, characterized in that The Chinese paper module refers to the academic paper module in the humanities and social sciences, including but not limited to: abstract module, introduction module and conclusion module.

3. The detection method according to claim 1, characterized in that The sample data in step 1) are collected from the highly cited papers in the humanities and social sciences in the past M years. The first N papers are downloaded in descending order of citation frequency, and the abstract, introduction or conclusion module of each paper is extracted; then they are divided into three parts: training set, test set and example set. The example set is further divided into two parts, which are used to construct prompts for the input large language model for the training set and the test set in step 2).

4. The detection method according to claim 1, characterized in that In step 2), a paper module is generated by the large language model for each paper title in the training set and the test set, thereby obtaining sample data generated by the large language model that is equal to the number of sample data written by humans.

5. The detection method according to claim 1, characterized in that Step 3) Export keyword data of papers in related fields in batches from the literature database, and obtain a field dictionary after merging and removing duplicates.

6. The detection method according to claim 1, characterized in that The chapter granularity in step 6) is based on the text extraction before word segmentation, including text length and time features; the paragraph granularity is based on the text extraction before word segmentation, including the average paragraph length and the average number of sentences in a paragraph; the sentence granularity is based on the text extraction before word segmentation, including the average sentence length, sentence pauses, the proportion of long sentences, the degree of sentence sentiment change, the average sentence readability score, and the sentence semantic similarity; the vocabulary granularity is based on the text extraction after word segmentation, including the average vocabulary length, vocabulary diversity, the proportion of specific part-of-speech vocabulary, and the proportion of stop words.

7. The detection method according to claim 6, characterized in that In step 6), for each sentence, a sentiment analysis tool is used to generate a sentiment score, and the standard deviation of the sentiment score of the entire paper module is calculated as an indicator of sentiment change; the readability scores of all sentences are averaged to obtain the average sentence readability score, where the readability score of each sentence uses the Flesch readability score, which is calculated as follows: Flesch readability score = 206.835 - (1.015*ASL) - (84.6*ASW) Among them, ASL is the average number of words in a sentence, which is equal to the total number of words divided by the number of sentences; ASW is the average number of syllables per word, which is equal to the total number of syllables divided by the total number of words. The total number of words refers to the number of words obtained after word segmentation, and each Chinese character represents a syllable.

8. The detection method according to claim 1, characterized in that Step 7) Select a text vector feature extraction model and input the text content to obtain the corresponding high-dimensional vector.

9. The detection method according to claim 1, characterized in that: In step 8), in order to avoid the overfitting problem, the training set data is further divided into a training set and a validation set, and the K-fold cross-validation method is used to calculate the model evaluation indicators, including precision, recall and F1-Score value; based on the first training results of the model, the F1-Score value is used as the evaluation indicator for hyperparameter selection, and the hyperparameter with the best performance is selected from the predefined hyperparameter range through a random search method. Similarly, the models trained on text structure features and text vector features are tuned a certain number of times respectively; different hyperparameters need to be optimized for different models.

10. The detection method according to claim 1, characterized in that: In step 4), "1" is used to identify the paper module generated by the large language model, and "0" is used to identify the paper module written by humans.

Images