A detection method for a large language model generating a Chinese thesis module

Abstract

The application discloses a detection method of a large language model generated Chinese thesis module. For the Chinese thesis, especially the abstract, introduction, conclusion and other thesis modules of the humanities and social science academic thesis, sample data written by human are collected, and the sample data are generated through a large language model. After the two kinds of sample data are integrated, the text structure features and the text vector features of the sample data are extracted as multi-dimensional feature inputs. Whether generated by the large language model is taken as a classification label for multi-model classification training. The accuracy of detection is improved through multi-model fusion. The method can be widely applied to detection of various structural modules of academic theses, can accurately identify academic thesis modules generated by a large language model, and has high application value.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and artificial intelligence, and in particular to a method for detecting Chinese paper modules generated by a large language model. Background Art

[0002] With the rapid development of artificial intelligence and machine learning, large language models such as GPT, Bard, ERNIE Bot, and PanGu have achieved remarkable success in natural language processing (NLP) tasks. These models are capable of generating content that is nearly indistinguishable from human-written text, including academic papers. However, this progress also presents a series of challenges. Because large language models can generate high-quality academic content, some may exploit this to generate and submit fraudulent research results. This not only threatens the fairness and authenticity of academia but also may lead to inaccurate research conclusions and the dissemination of misleading information. To address this issue, a number of tools have emerged specifically designed to detect content generated by large language models, such as AI Content Detector, GPTZero, and Winston AI. However, these AI detection tools are not trained for Chinese text generated by large language models and do not consider the structural and semantic characteristics of Chinese text. As a result, most of them cannot detect Chinese text, and even if they can, their performance is inferior to that of English text.

[0003] There have been some studies on content detection generated by large language models for Chinese academic papers. Wang Yibo et al. conducted a classification study on the abstracts of 100 papers in 5 journals in the library field based on TFIDF characterization technology, and further analyzed the differences in dimensions such as abstract length, number of sentences, vocabulary features, and common collocations (Detection and Difference Comparative Study of AI-Generated and Scholar-Written Chinese Paper Abstracts [J]. Intelligence Magazine, 2023, 42(09): 127-134.). Using the introductions of eight information science journals as examples, Guo Xin et al. first analyzed text structural features at the word, sentence, and chapter levels. They then integrated these features with text vector features to test multiple classification models and performed interpretability analysis on models with good classification results (Analysis of Chinese Academic Content Generated by ChatGPT—A Case Study of Information Science [J / OL]. Library Forum: 1-12 [2023-10-02]. https: / / link.cnki.net / urlid / 44.1306.G2.20230921.1040.004). However, previous research on content detection generated by large language models in Chinese academic papers has been limited to detecting specific structural components of the papers due to training data limitations, resulting in the limitation of only detecting specific modules. Therefore, there is an urgent need to develop a method that can accurately identify modules of entire Chinese academic papers generated by large language models.

[0004] Academic papers in the humanities and social sciences typically explore social, cultural, philosophical, and historical issues, often with a more narrative and explanatory writing style. Compared to papers in science and engineering, they may use more subjective and explanatory language and rely less on numerical data, charts, and statistical analysis. Large language models can help researchers in the humanities and social sciences complete more textual content. At the same time, academic papers are generally long and cannot be generated in one go by large language models. Instead, they require multiple question-and-answer interactions to generate different modules. Academic papers primarily include conventional modules such as title, abstract, introduction, literature review, methods, discussion, and conclusion. Given the semantic continuity and length limitations of content generated by large language models, these modules are more likely to be generated en masse by large language models, making them a particularly vulnerable area for large language model-generated content. Detecting whether these modules are machine-generated is therefore of particular value. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a detection method for Chinese paper modules generated by large language models, especially academic paper modules in the humanities and social sciences, to accurately identify academic paper modules generated by large language models.

[0006] The present invention provides a method for detecting a Chinese paper module generated by a large language model, comprising the following steps:

[0007] 1) Collect sample data of Chinese paper modules written by humans and divide them into training set, test set and example set;

[0008] 2) Using the example set as a reference sample, the large language model generates corresponding paper modules for the paper titles in the training set and test set, respectively, to obtain sample data generated by the large language model;

[0009] 3) Build domain dictionary;

[0010] 4) Integrate the human-written sample data collected in step 1) and the sample data generated by the large language model in step 2) into three columns of data: the paper title, the paper module, and an indicator of whether it was generated by the large language model;

[0011] 5) Use natural language processing tools to load the domain dictionary constructed in step 3) and perform word segmentation and part-of-speech tagging on the text content of all paper modules;

[0012] 6) Extracting text structure features from the paper module, including chapter granularity, paragraph granularity, sentence granularity, and vocabulary granularity;

[0013] 7) Extract text vector features from the paper module;

[0014] 8) Select P classification algorithm models, use the training set data, input text structure features and text vector features as multi-dimensional features, and use the variable whether the paper module is generated by the large language model as the classification label for model training, to obtain 2P trained classification models;

[0015] 9) Input the test set data into the trained 2P classification models respectively, and calculate the classification accuracy of each classification model based on the classification labels, using A1, A2...A 2p To express; if the classification accuracy of a classification model is lower than the set R%, the classification model will be abandoned, otherwise the classification model will be retained, thereby screening out Q models with better classification effects, and their classification accuracy rates are represented by B1, B2...B Q To represent, the highest classification accuracy is represented by B max Indicates that the weight of each classification model is calculated using the extreme value processing method with the accuracy of R% as the lower limit. The formula is as follows:

[0016]

[0017] Among them, W i Represents the weight of the i-th classification model, B i Represents the classification accuracy of the i-th classification model;

[0018] 10) For the paper module to be tested, first perform word segmentation and part-of-speech tagging according to step 5), then perform text structure feature extraction and text vector feature extraction according to steps 6) and 7) respectively; input the text structure features and text vector features of the paper module to be tested into the Q classification models retained in step 9), and record the output of the model as Y1, Y2...Y Q , Y i Represents the output of the i-th classification model, and its value range is between 0-1. “1” marks the paper module generated by the large language model, and “0” marks the paper module written by humans. If ∑W i *Y i >0.5*∑W i , then the detected paper module is determined to be generated by a large language model, otherwise it is determined to be written by humans.

[0019] The present invention is mainly aimed at Chinese academic papers in the humanities and social sciences, including philosophy, history, linguistics, economics, management, law, sociology, anthropology, psychology, etc. The paper module includes but is not limited to the paper's abstract, introduction, conclusion and other modules. In step 1), a literature database with a relatively complete collection of papers in the humanities and social sciences is selected as the data source, and core journal papers published in the past M years are retrieved and sorted in descending order according to the frequency of citation. The top N highly cited papers are selected and the modules such as the abstract, introduction or conclusion of each paper are extracted. The collected paper module data is divided into three parts: a training set, a test set and an example set. The example set is further divided into two parts, which are used in step 2) to construct prompts for the input large language model for the training set and the test set. For example, of the N highly cited papers collected, 0.6N papers are divided into a training set, 0.2N papers are divided into a test set, and 0.2N papers are divided into an example set. 0.15N articles in the example set are used to construct prompts for the training set, and 0.05N articles are used to construct prompts for the test set, to ensure that the reference content of the training set and the test set do not overlap and interfere with each other.

[0020] In step 2), a large language model (e.g., GPT, Bard, ERNIE Bot, Wenxin Yiyan, etc.) is used to generate the content of a specific paper module based on the paper title, using the paper module examples from the example set as a reference. For each paper title in the training and test sets, a paper module is generated using the large language model. This yields a sample data set generated by the large language model that is equal to the sample data written by humans.

[0021] Step 3) Build a domain dictionary based on the keyword data of the paper. The keyword data of the papers in the related fields can be exported in batches from the literature database, and the domain dictionary can be obtained after merging and removing duplicates.

[0022] Step 4) Combine the human-written and large language model-generated paper modules into three columns. In the third column, use "1" to identify the large language model-generated paper modules and "0" to identify the human-written paper modules. In subsequent classification model training and application, "1" and "0" are used as the output classification labels.

[0023] In step 5), use natural language processing tools (such as Python's jieba package) to load the domain dictionary, perform word segmentation and part-of-speech tagging on the text content of all paper modules, mark the word segmentation results as nouns, prepositions, pronouns, auxiliary words, quantifiers, etc., and delete all punctuation marks during the word segmentation process.

[0024] The chapter granularity, paragraph granularity, and sentence granularity described in step 6) are all based on the paper module text extraction before word segmentation, wherein the chapter granularity includes text length and time features; the paragraph granularity includes average paragraph length and average number of sentences in a paragraph; the sentence granularity includes average sentence length, sentence pauses, proportion of long sentences, degree of sentence sentiment change, average sentence readability score, and sentence semantic similarity; the vocabulary granularity is based on the paper module text extraction after word segmentation, including average vocabulary length, vocabulary diversity, proportion of specific part-of-speech vocabulary, and proportion of stop words.

[0025] For each sentence, a sentiment analysis tool (such as SnowNLP) is used to generate a sentiment score. The standard deviation of the sentiment score for the entire paper module is calculated as an indicator of sentiment change, that is, the degree of sentiment change in the sentences. The readability scores of all sentences are averaged to obtain the average sentence readability score. The readability score of each sentence can be calculated using the Flesch readability score, which is calculated as follows:

[0026] Flesch readability score = 206.835 - (1.015 * ASL) - (84.6 * ASW)

[0027] Among them, ASL is the average number of words in a sentence, which is equal to the total number of words divided by the number of sentences; ASW is the average number of syllables per word, which is equal to the total number of syllables divided by the total number of words. The total number of words refers to the number of words obtained after word segmentation, and each Chinese character represents a syllable.

[0028] In step 7), a text vector feature extraction model is selected and the text content is input to obtain the corresponding high-dimensional vector.

[0029] Text vectorization is a method of encoding textual information into a high-dimensional space. This representation can reveal the importance of words in a text and the underlying semantics. Commonly used text vectorization models include Doc2Vec, ELMo, and BERT. A text vector feature extraction model (such as SentenceBERT, based on the Python SentenceTransformers library) is selected. The model inputs text data and outputs an L-dimensional vector (e.g., 768) corresponding to the text. This allows the 1.2N paper modules in the training set to be converted into a 1.2N×L two-dimensional matrix, and the 0.4N paper modules in the test set to be converted into a 0.4N×L two-dimensional matrix.

[0030] In step 8), P (preferably, P >= 4) classical classification algorithm models such as support vector machine, random forest, gradient boosting machine (such as XGBoost), multilayer perception, etc. are selected. In the training of the classification model, to avoid overfitting problem, the training set data is further divided into training set and validation set, and K-fold cross-validation method is used to calculate the model evaluation index, and the calculated index includes precision, recall and F1-Score value; on the basis of the first training result of the model, F1-Score value is taken as the evaluation index of hyperparameter selection, and the optimal hyperparameters are selected from the pre-defined hyperparameter range by random search method, and the models trained by the text structure features and the text vector features are optimized for a certain number of times; different models need to optimize different hyperparameters.

[0031] The paper module generated by the large language model is marked with "1", and the paper module written by the human is marked with "0". The classification label output by the classification model trained in step 8) is "1" or "0". In step 10), Y i represents the output of the i-th classification model, and the value is 1 or 0, 1 represents that the model determines that the input paper module is generated by a large language model, and 0 represents that the model determines that the input paper module is written by a human, if ∑W i *Y i > 0.5 * ∑W i , it is determined that the detected paper module is generated by a large language model, otherwise it is determined to be written by a human.

[0032] The present application proposes a kind of based on text structure feature and text vector feature fusion The detection method of the Chinese paper module generated by large language model, it can be applied to server, computer equipment, intelligent mobile terminal etc..Compared with the existing detection method, the main innovation points and contents of the present application are as follows:

[0033] 1. Comprehensive training data: select the core periodical paper of humanities and social sciences field as data source, collect in detail from China knowledge network etc.

[0034] 2. Detailed feature construction: from vocabulary, sentence, paragraph to chapter different granularity level, extract rich structure and semantic features.For example: introduce advanced sentence vectorization method, capture deep semantic information; introduce sentiment analysis tool to construct paper module sentiment change feature.

[0035] 3. Multi-model classification training and fusion: select multiple classical classification models for training, such as support vector machine, random forest

[0036] and gradient boosting machine, etc. Through model fusion technology, the advantages of multiple models are combined to further improve the accuracy of detection.

[0037] 4. The model can be widely applied to various structural modules of academic papers, not limited to detection of specific modules, and has high application value. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A flowchart of the detection method for the academic paper module generated by the large language model. DETAILED DESCRIPTION

[0039] To make the purpose, technical solutions and technical effects of the present application clearer, the specific implementation process of the present application will be described in detail below in the form of examples combined with the drawings. Obviously, only one exemplary embodiment of the present application is described here, rather than all embodiments, and is not intended to limit the present application.

[0040] A detection method for a humanities and social science academic paper module generated by a large language model, comprising the following steps:

[0041] Step 1: Sample data collection

[0042] 1.1 Select sample paper collection range. Select a literature database with a complete collection of papers in the humanities and social science field as the data source website, and select the relevant fields of humanities and social sciences in the database literature classification directory. Select core journal papers published in the last M years in descending order of citation frequency, and the content ranked in the front is the highly cited papers in the humanities and social science field in the last M years. For example, in the literature classification directory in the "Journal" tab of China Knowledge Network (https: / / www.cnki.net / ), select "Philosophy and Humanities", "Social Science I", "Social Science II", and "Economic and Management Science". Select "core journals", "CSSCI", and "CSCD" as the journal source type to retrieve highly cited papers in the humanities and social science field in the last M years.

[0043] 1.2 Download sample papers and analyze the content of each module. According to the retrieval results, download the first N papers (such as 1000) in turn, and manually extract the abstract, introduction or conclusion module of each paper. The abstract module will be clearly marked in each paper. The introduction module has two cases: for papers whose main text starts from the chapter title, the introduction module corresponds to the content of the first chapter; for papers whose main text starts directly from the paragraph content, the introduction module corresponds to the content from the beginning of the main text to the content before the first chapter title. The conclusion module corresponds to the content of the last chapter of the main text. Through the above structural characteristics, the text content of each module of the N papers can be parsed from the papers.

[0044] 1.3 Sample Dataset Division. The N collected papers were divided into three parts: the first part is the training set of the classifier, which is used to train the classification model and includes a total of 0.6N papers; the second part is the test set of the classifier, which is used to evaluate the generalization ability of the model and includes a total of 0.2N papers; the third part is the example set used when writing the large model, which is used to construct prompts for the large model input and includes a total of 0.2N papers, of which 0.15N papers are used for constructing prompts for the training set and 0.05N papers are used for constructing prompts for the test set. This ensures that the reference content of the training set and the test set do not overlap and do not interfere with each other.

[0045] 1.4 Large Language Model Sample Data Collection. For training and test set papers, call the large language model interface (such as gpt-3.5-turbo) and ask the large language model to generate specific module content according to the specified format based on the title of each paper and reference examples, where the reference examples come from the example set. The prompt content format transmitted by calling the interface is as follows:

[0046] "I want you to play the role of a researcher in the humanities and social sciences. You need to help me complete the abstract (or introduction, conclusion, etc.) of an academic paper in the humanities and social sciences. Before giving me a specific topic, I will first give you two sample paper titles and corresponding abstracts for reference. After that, I will give you a specific paper title. You can imitate the language style of these samples when writing.

[0047] Example of the abstract of the first paper: {"Title":"XXX(Title of Reference Paper 1)","Abstract":"(Abstract of Reference Paper 1)"}

[0048] Example of abstract for the second paper: {"Title":"XXX(Title of Reference Paper 2)","Abstract":"(Abstract of Reference Paper 2)"}

[0049] Next, you need to complete the writing of a paper abstract according to the above requirements. The title of the paper is: XXXXXX (paper with the abstract to be completed)"

[0050] As a result, we obtain the summary content generated by the large language model for the 0.6N training set paper titles and the summary content generated by the large model for the 0.2N test set paper titles.

[0051] 1.6 Domain Dictionary Construction: Using the same paper search method as in Section 1.1, we batch-export keyword data from the first Z papers (e.g., 5,000) through the "Export Literature" function provided by CNKI. After merging and removing duplicates, we obtain a dictionary for the humanities and social sciences.

[0052] Step 2: Sample data feature extraction

[0053] 2.1 Sample Data Integration. After data acquisition, we integrated the human-written and large language model-generated abstracts (or introductions, conclusions, etc.) into three columns: the paper title, the paper abstract, and an indicator indicating whether it was generated by the large language model (1 for machine-generated, 0 for human-written). There were 0.8N paper titles corresponding to 1.6N abstracts, of which 1.2N were the training set and 0.4N were the test set. The ratio of positive and negative samples in both sets was equal.

[0054] 2.2 Segmentation and part-of-speech tagging of the abstract text. Use a natural language processing tool (such as the Python package jieba), load the domain dictionary built in Section 1.6, and perform segmentation and part-of-speech tagging on all abstract text content. Label the segmentation results as nouns, verbs, prepositions, etc., and remove all punctuation during the segmentation process.

[0055] 2.3 Extracting Structural Features of Sample Data. Text structural features are defined based on the statistical and structural information of the text. Features are extracted at four granularities: chapter, paragraph, sentence, and vocabulary. These features reflect the structure and style of the text. The following 18 features are used to calculate the feature values ​​of the summary texts of the training and test sets:

[0056] (1) Chapter granularity (based on text extraction before word segmentation)

[0057] ① Text length: The number of characters in the summary text. Each Chinese character is considered as a character, each English word is considered as a character, and each half-width or full-width punctuation mark is considered as a character. The same applies below.

[0058] ② Time feature: the number of strings in the form of "XXXX年XX月" in the article (use regular expressions to count the number of times this form appears in the text).

[0059] (2) Paragraph granularity (based on text extraction before word segmentation)

[0060] ① Average paragraph length: text length divided by the total number of text paragraphs. Each line break is considered to start a new paragraph.

[0061] ② Average number of sentences in a paragraph: the total number of sentences in the text divided by the total number of paragraphs in the text. A sentence ending with a period, question mark, or exclamation mark is considered a sentence, and a new paragraph is considered to begin after a line break.

[0062] (3) Sentence granularity (based on text extraction before word segmentation, a sentence ending with a period, question mark, or exclamation mark is considered a sentence)

[0063] ①Average sentence length: text length divided by the total number of sentences.

[0064] ② Sentence Pauses: For each sentence, count the number of commas, semicolons, and periods, and then divide by the total number of words in the sentence to get the frequency of punctuation marks. Use the standard deviation of the punctuation mark frequencies across all sentences to calculate sentence pauses.

[0065] ③Ratio of long sentences: the number of sentences with more than 100 characters divided by the total number of sentences.

[0066] ④ Sentence sentiment change: For each sentence, use sentiment analysis tools (such as SnowNLP) to generate a sentiment score. Calculate the standard deviation of the sentiment score for the entire paper module as an indicator of sentiment change.

[0067] ⑤ Average sentence readability score: Take the average of the readability scores of all sentences. The score of each sentence can use a readability score indicator (for example, Flesch readability score). The Flesch readability score is calculated as follows:

[0068] Flesch readability score = 206.835 - (1.015 * ASL) - (84.6 * ASW)

[0069] In the above formula, ASL (Average Sentence Length) is the average number of words in a sentence, which is equal to the total number of words divided by the number of sentences; ASW (Average Syllables per Word) is the average number of syllables per word, which is equal to the total number of syllables divided by the total number of words. The Flesch readability score is typically applied to English text. When applied to Chinese text, word segmentation is required to determine the vocabulary count, where each Chinese character represents a syllable.

[0070] ⑥ Sentence semantic similarity: In order to capture the semantic information of the sentence more deeply, we adopted a sentence vectorization method to generate its corresponding semantic vector representation (for example: SentenceBERT model). This will transform each sentence into a fixed-size numerical vector that captures the semantic content of the sentence. For each sentence in the paper (except the first and last sentences), calculate its cosine similarity with the previous and next sentences. For the first sentence, only calculate its similarity with the second sentence; for the last sentence, only calculate its similarity with the penultimate sentence. For each sentence, take the average of its similarities with the previous and next sentences as the similarity value of the sentence. Calculate the average of the similarity values ​​of all sentences in the paper. This will give the sentence similarity of the entire paper.

[0071] (4) Lexical granularity (text extraction based on word segmentation)

[0072] ① Average word length: the total number of characters in all words divided by the total number of words;

[0073] ② Lexical diversity: the number of words in the text after removing all duplicate words divided by the total number of words;

[0074] ③ The proportion of words of a specific part of speech: the number of words of a certain part of speech divided by the total number of words. A total of five parts of speech were selected, namely pronouns, nouns, quantifiers, auxiliary words, and conjunctions, to construct five features;

[0075] ④ The proportion of stop words: Use a common Chinese stop word list (for example, the Harbin Institute of Technology stop word list) and count the number of stop words divided by the total number of words.

[0076] 2.4 Sample Data Vector Feature Extraction. The vectorized representation of text is a method of encoding text information into a high-dimensional space. Such a representation can reveal the importance of words in the text and the semantics behind them. There are currently a variety of text vectorization models, such as Doc2Vec, ELMo, and BERT. We select a text vector feature extraction model (such as SentenceBERT based on the Python SentenceTransformers library). We can input the training set into the model and output the L (for example: 768) dimensional vector corresponding to the text. In this way, we can convert the 1.2N abstracts in the training set into a 1.2N*L two-dimensional matrix, and convert the 0.4N abstracts in the test set into a 0.4N*L two-dimensional matrix.

[0077] Step 3: Training the classification model

[0078] 3.1 Model Selection and Indicator Calculation. Select P (e.g., 5) classic classification algorithm models, including support vector machines, random forests, gradient boosting machines (e.g., XGBoost), and multi-layer perceptrons. Use text structure features and text vector features as multi-dimensional feature inputs, respectively. Use the variable (whether the paper's abstract was generated by a large language model) as the classification label for model training. To avoid overfitting, the training set data needs to be further divided into a training set and a validation set, and K-fold cross-validation is used to calculate model evaluation indicators, including precision, recall, and F1-score.

[0079] 3.2 Model Hyperparameter Tuning. Based on the initial model training results, the F1-Score value is used as the evaluation metric for hyperparameter selection. A random search method is used to select the best performing hyperparameters from a predefined hyperparameter range. Similarly, the models trained on text structure features and text vector features are tuned a certain number of times (e.g., 100 times). Different hyperparameters need to be optimized for different models. Taking the support vector machine as an example, the hyperparameters that need to be tuned include the kernel function type (kernel) and the regularization parameter (C).

[0080] Step 4: Testing and fusion of classifiers

[0081] 4.1 Classifier Test. The test set data is input into the trained 2P models (P classification algorithms × 2 text features), and the accuracy of each classification effect is calculated based on the classification labels, using A1, A2, ..., A 2p To represent the generalization ability of each model.

[0082] 4.2 Classifier weight calculation. For the classification accuracy of 2P models A1~A 2p If the classification accuracy of a model is lower than R% (such as 90%), the model is discarded, otherwise the model is retained. Q models with better classification effects can be screened out, and their classification accuracy is expressed as B1, B2...B Q To express it, the accuracy of the model with the highest classification accuracy is expressed as B max Next, we use the efficacy coefficient method to weight the model, and use the extreme value processing method with the R% accuracy as the lower limit to calculate the weight of each model. The formula is as follows:

[0083]

[0084] Among them, W i Represents the weight of the i-th classification model, B i Represents the classification accuracy of the i-th classification model, i is an integer from 1 to Q, and R% is the set lower limit of the classification accuracy.

[0085] 4.3 Model Application. When testing a new paper module, first complete the word segmentation of the new paper module data according to the content of Section 2.2, then extract the text structure features according to the indicator calculation method defined in Section 2.3, and calculate the text vector according to the text vector feature extraction model trained in Section 2.4. Then, input the text structure features and text vector features into the Q models that are still retained, and record the output of the model as Y1, Y2, ..., Y Q If ∑W i *Y i >0.5*∑W i , then the module being tested is determined to be generated by a large language model, otherwise it is determined that the content being tested is written by humans, where Y i Represents the output of the i-th classification model, and its value range is between 0 and 1.

[0086] Finally, it should be noted that the purpose of publishing the embodiments is to help technicians further understand the technical solutions and effects of the present invention, rather than to limit the scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made to the specific implementation plans according to the needs of actual applications. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the scope of the present invention, and the scope of protection of the present invention shall be based on the scope defined by the appended claims.

Claims

1. A method for detecting a Chinese paper module generated by a large language model, comprising the following steps: 1) Collect sample data of Chinese paper modules written by humans and divide them into training set, test set and example set; 2) Using the example set as a reference sample, the large language model generates corresponding paper modules for the paper titles in the training set and test set, respectively, to obtain sample data generated by the large language model; 3) Build a domain dictionary; 4) Integrate the human-written sample data collected in step 1) and the sample data generated by the large language model in step 2) into three columns of data: the paper title, the paper module, and an indicator of whether it was generated by the large language model; 5) Use natural language processing tools to load the domain dictionary constructed in step 3) and perform word segmentation and part-of-speech tagging on the text content of all paper modules; 6) Extracting text structure features from the paper module, including chapter granularity, paragraph granularity, sentence granularity, and vocabulary granularity; 7) Extract text vector features from the paper module; 8) Select P classification algorithm models, use the training set data, input text structure features and text vector features as multi-dimensional features, and use the variable whether the paper module is generated by the large language model as the classification label for model training, to obtain 2P trained classification models; 9) Input the test set data into the trained 2P classification models respectively, and calculate the classification accuracy of each classification model based on the classification labels, using A1, A2...A 2p To express; if the classification accuracy of a classification model is lower than the set R%, the classification model will be abandoned, otherwise the classification model will be retained, thereby screening out Q models with better classification effects, and their classification accuracy rates are represented by B1, B2...B Q To represent, the highest classification accuracy is represented by B max express; The weight of each classification model is calculated using the extreme value processing method with the accuracy of R% as the lower limit. The formula is as follows: Among them, W i Represents the weight of the i-th classification model, B i Represents the classification accuracy of the i-th classification model; 10) For the paper module to be tested, first perform word segmentation and part-of-speech tagging according to step 5), then perform text structure feature extraction and text vector feature extraction according to steps 6) and 7) respectively; input the text structure features and text vector features of the paper module to be tested into the Q classification models retained in step 9), and record the output of the model as Y1, Y2...Y Q , its value range is between 0-1, "1" marks the paper module generated by the large language model, and "0" marks the paper module written by humans; if ΣW i *Y i >0.5*∑W i , then the detected paper module is determined to be generated by a large language model, otherwise it is determined to be written by humans.

2. The detection method according to claim 1, wherein The Chinese paper module refers to the academic paper module in the humanities and social sciences, including but not limited to: abstract module, introduction module and conclusion module.

3. The detection method according to claim 1, wherein The sample data in step 1) are collected from highly cited papers in the humanities and social sciences over the past M years. The first N papers are downloaded in descending order of citation frequency, and the abstract, introduction, or conclusion module of each paper is extracted. They are then divided into three parts: a training set, a test set, and an example set. The example set is further divided into two parts, which are used in step 2) to construct prompts for the input large language model for the training set and the test set.

4. The detection method according to claim 1, wherein In step 2), a paper module is generated for each paper title in the training set and the test set through the large language model, thereby obtaining sample data generated by the large language model that is equal to the number of sample data written by humans.

5. The detection method according to claim 1, wherein Step 3) Export keyword data of papers in related fields in batches from the literature database, merge and remove duplicates to obtain a field dictionary.

6. The detection method according to claim 1, wherein The chapter granularity in step 6) is based on the text extraction before word segmentation, including text length and time features; the paragraph granularity is based on the text extraction before word segmentation, including average paragraph length and average number of sentences in a paragraph; the sentence granularity is based on the text extraction before word segmentation, including average sentence length, sentence pauses, proportion of long sentences, degree of sentence sentiment change, average sentence readability score, and sentence semantic similarity; the vocabulary granularity is based on the text extraction after word segmentation, including average vocabulary length, vocabulary diversity, proportion of specific part-of-speech vocabulary, and proportion of stop words.

7. The detection method according to claim 6, wherein In step 6), for each sentence, a sentiment analysis tool is used to generate a sentiment score, and the standard deviation of the sentiment score of the entire paper module is calculated as an indicator of sentiment change; the readability scores of all sentences are averaged to obtain the average sentence readability score, where the readability score of each sentence is calculated using the Flesch readability score, which is calculated as follows: Flesch readability score = 206.835 - (1.015 * ASL) - (84.6 * ASW) Among them, ASL is the average number of words in a sentence, which is equal to the total number of words divided by the number of sentences; ASW is the average number of syllables per word, which is equal to the total number of syllables divided by the total number of words. The total number of words refers to the number of words obtained after word segmentation, and each Chinese character represents a syllable.

8. The detection method according to claim 1, wherein Step 7) Select a text vector feature extraction model and input the text content to obtain the corresponding high-dimensional vector.

9. The detection method according to claim 1, wherein In step 8), in order to avoid the overfitting problem, the training set data is further divided into a training set and a validation set, and the K-fold cross-validation method is used to calculate the model evaluation indicators, including precision, recall rate and F1-Score value; based on the first training results of the model, the F1-Score value is used as the evaluation indicator for hyperparameter selection, and the best performing hyperparameter is selected from the predefined hyperparameter range through a random search method. Similarly, the models trained on text structure features and text vector features are tuned a certain number of times respectively; different hyperparameters need to be optimized for different models.

10. The detection method according to claim 1, wherein In step 4), "1" is used to identify the paper module generated by the large language model, and "0" is used to identify the paper module written by humans.

Images