A machine-generated natural language detection method based on multiple features

Abstract

This invention relates to the field of natural language processing technology and proposes a machine-generated natural language detection method based on multiple features. The method includes: acquiring the text to be detected and preprocessing it to obtain preprocessed text; inputting the preprocessed text into a word segmenter of a pre-trained language model for word segmentation and converting the segmented text into a sequence of tokens recognizable by the pre-trained language model; inputting the token sequence into the pre-trained language model for computation to obtain the unnormalized prediction probability of each token; calculating 11 features of the text based on the token sequence and the unnormalized prediction probability of each token; concatenating the 11 features into an 11-dimensional vector and inputting it into a machine learning classification algorithm for supervised learning to obtain a machine-generated natural language detector; inputting the text to be detected into the machine-generated natural language detector to obtain the detection result. This invention offers fast detection speed, high detection accuracy, and high reliability and transparency of the detection method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and specifically discloses a machine-generated natural language detection method based on multiple features. BACKGROUND

[0002] With the vigorous evolution of large language models (LLM), their outstanding performance has made it a great challenge for ordinary users to distinguish between machine-generated text (MGT) and human-written text (HWT). Therefore, the application of large language models has also caused many problems. In actual application scenarios, criminals frequently use large language models to carry out phishing, spread false information, commit academic fraud, and maliciously generate spam emails, etc. For example, by means of large language models, fraudulent product reviews are fabricated, which seriously misleads consumers' purchasing decisions; in the academic field, plagiarism and cheating phenomena occur frequently, which greatly damages the academic integrity environment and disrupts the normal order of society. At the same time, when faced with texts of unknown origin, Internet users often lack effective means of discrimination and are difficult to make accurate judgments, which provides an opportunity for malicious use of information, thereby posing a potential threat to social stability.

[0003] Current machine-generated natural language detection methods mainly include: statistical feature-based detection methods, language model-based detection methods, and watermark algorithm-based detection methods, which have the following defects: for statistical feature-based detection methods, although they extract statistical features with pre-trained language models and use machine learning classification models for supervised learning, it is extremely difficult to obtain model internal weight parameters and network structures when facing unopened language models, which seriously limits their promotion and use in actual applications; language model-based detection methods achieve text classification tasks by modifying the output layer of open-source large language models. This method belongs to black-box detection and has weak generalization ability and is prone to overfitting. Once the text to be detected exceeds the field range of the training data or involves language replacement, the detection effect will be greatly reduced. Especially in the field of education, this method often misjudges human-written texts as machine-generated texts, which seriously affects the accuracy of detection; watermark algorithm-based detection methods detect text by manipulating model generation behavior or embedding markers, but this process negatively affects the quality of generated text. And when attacked, this method has poor robustness, for example, when the watermark sampling algorithm is modified or the watermark information is cracked, the detection mechanism will fail. In addition, for commercial and unopened models, this method is not suitable.

[0004] In summary, existing detection methods cannot meet actual needs, and there is an urgent need for a machine-generated natural language detection method that is efficient, accurate, and has strong generalization ability and robustness. SUMMARY

[0005] The application provides a machine-generated natural language detection method based on multiple features.

[0006] The application provides a machine-generated natural language detection method based on multiple features, which comprises the following steps:

[0007] S1. Obtain a text to be detected, and pre-process the text to be detected to obtain a pre-processed text;

[0008] S2. Input the pre-processed text obtained in step S1 into a tokenizer of a pre-trained language model to perform text segmentation and convert the segmentation into a wordpiece sequence recognizable by the pre-trained language model, input the wordpiece sequence into the pre-trained language model to perform operation, and obtain an unnormalized prediction probability of each wordpiece;

[0009] S3. Calculate 11 features of the text according to the wordpiece sequence and the unnormalized prediction probability of each wordpiece obtained in step S2, wherein the 11 features include log-likelihood, rank, log-rank, perplexity, ratio of rank and log-likelihood, surprise variance, mean of adjacent wordpiece surprise difference, information entropy, vocabulary density, variance of log-rank, and result of supervised detection;

[0010] S4. Combine the 11 features obtained in step S3 into an 11-dimensional vector, and input the 11-dimensional vector into a machine learning classification algorithm to perform supervised learning, and obtain a machine-generated natural language detector;

[0011] S5. Input the text to be detected into the machine-generated natural language detector obtained in step S4 to obtain a detection result.

[0012] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the application, the pre-processing in step S1 comprises removing punctuation, spaces, URLs, special characters and emoticons in the text to be detected, converting the vocabulary of the text to be detected into lowercase vocabulary, and restoring the part of speech.

[0013] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the application, the pre-trained language model in step S2 is GPT2 XL, and the tokenizer parameters max_length is set to 512 and truncation is set to True.

[0014] According to the multi-feature-based machine-generated natural language detection method of some embodiments of the present application, the step S3 of calculating the log-likelihood of the text comprises: calculating the normalized value of the unnormalized prediction probability of each word unit in the text, that is, performing Softmax function operation on the value of the unnormalized prediction probability to obtain word unit probability, and taking the logarithm of the word unit probability to obtain the log-likelihood negative mean value of the text word unit, that is, the log-likelihood of the text S, as shown in formula (1):

[0015] LL(S) = -E(log(p(x))) (1)

[0016] wherein LL(S) represents the log-likelihood of the text S, S represents the text, x represents the word unit in the text S, p(·) represents the probability function, and E(·) represents the mean function;

[0017] The rank of the text is calculated by: calculating the normalized value of the unnormalized prediction probability of each word unit, that is, performing Softmax function operation on the value of the unnormalized prediction probability to obtain word unit probability, and obtaining the rank of the word unit probability, and obtaining the rank mean value of the text word unit by the mean function, which is simply referred to as the rank, as shown in formula (2):

[0018] R(S) = E(rank(p(x))) (2)

[0019] wherein R(S) represents the rank of the text S, and rank represents the function of ranking the probability of the word unit x in the pre-trained language model prediction candidate word;

[0020] The log-rank of the text can be directly calculated by taking the logarithm value of R(S), as shown in formula (3):

[0021] LR(S) = log(R(S)) (3)

[0022] wherein LR(S) represents the log-rank of the text S;

[0023] The perplexity of the text is calculated, as shown in formula (4):

[0024]

[0025] wherein PPL(S) represents the perplexity of the text S, n represents the number of word units in the text, i∈(1,2,3, …, n), and exp(·) represents the exponential function;

[0026] The ratio of the rank and the log-likelihood of the text is calculated, as shown in formula (5):

[0027]

[0028] wherein, Ratio(S) represents the ratio of rank and log-likelihood of text S;

[0029] The calculation of the surprise variance of the text comprises: first calculating the surprise value of the unnormalized prediction probability of each word unit, so as to obtain the surprise variance of the text, as shown in formula (6):

[0030]

[0031] wherein, UID(S) represents the surprise variance of text S, s(x i ) represents the surprise of the i th word unit, x i represents the i th word unit, s(x i )=-log(p(x i |x <i )), and s(S) represents the surprise of text S;

[0032] The calculation of the mean of the adjacent word unit surprise difference of the text is shown in formula (7):

[0033]

[0034] wherein, UID'(S) represents the mean of the adjacent word unit surprise difference of text S, x i-1 represents the i-1 th word unit;

[0035] The calculation of the information entropy of the text is shown in formula (8):

[0036]

[0037] wherein, H(S) represents the information entropy of text S;

[0038] The calculation of the lexical density of the text is shown in formula (9):

[0039]

[0040] wherein, LD(S) represents the lexical density of text S, and V represents the number of vocabulary used in the text;

[0041] The calculation of the variance of the log-rank of the text is shown in formula (10):

[0042]

[0043] wherein, VR(S) represents the variance of the log-rank of text S;

[0044] The calculation of the supervised detection result of the text for the supervised detection model is shown in formula (11):

[0045] SD(S) = F(S) (11)

[0046] wherein SD(S) represents the result of supervised detection of the text S on the supervised detection model, and F represents the supervised detection model.

[0047] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the present application, the supervised detection model is AI generated text detection and roberta large openai detector, and the supervised detection model tokenizer parameter max_length is set to 512 and the truncation is set to True.

[0048] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the present application, the machine learning classification algorithm in step S4 is the extreme gradient boosting tree machine learning algorithm.

[0049] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the present application, in the supervised learning in step S4, the ratio of the training set to the test set data is 9:1, the learning rate is 0.01, the maximum depth of the weak classifier is set to 29, and the number of weak classifiers is set to 1800.

[0050] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the present application, the loss function of the supervised learning in step S4 is binary logistic regression loss, as shown in formula (12):

[0051]

[0052] wherein, represents the binary logistic regression loss, y k represents the true probability of the kth sample, k∈(1,2,3, …,N), represents the predicted probability of the kth sample, and N represents the number of samples.

[0053] According to the machine-generated natural language detection method based on multiple features according to some embodiments of the present application, the detection result in step S5 includes machine-generated text and human-written text.

[0054] The machine-generated natural language detection method based on multiple features provided by the present application uses eleven features in the detection process, so that the prediction result has good interpretability, and the credibility and transparency of the detection method are enhanced. Moreover, the present method uses a pre-trained language model with a small number of parameters, which has a significant resource-friendly feature. Even in an environment with relatively scarce computing resources, the detection function can still be efficiently realized, greatly expanding its application range and scenarios. The present application can comprehensively consider the source problem of the text to be detected, so as to more accurately distinguish between machine-generated text and human-written text, and significantly improve the accuracy of text source detection. Compared with the existing baseline method, the present application can achieve superior performance. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 FIG. 1 is a flowchart of a machine-generated natural language detection method based on multiple features according to the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application will be further described in detail below in combination with the drawings and examples. The following examples are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0057] Example 1 provides a machine-generated natural language detection method based on multiple features, as shown in FIG. 1, which includes the following steps: Figure 1

[0058] S1. Obtain the text to be detected, and pre-process the text to be detected to obtain a pre-processed text;

[0059] S2. Input the pre-processed text obtained in step S1 into the word segmenter of the pre-trained language model to perform text segmentation and convert the segmentation into a wordpiece sequence recognizable by the pre-trained language model, and input the wordpiece sequence into the pre-trained language model to perform operation, to obtain the unnormalized prediction probability of each wordpiece;

[0060] S3. Calculate 11 features of the text according to the wordpiece sequence and the unnormalized prediction probability of each wordpiece obtained in step S2, the 11 features including: log-likelihood, rank, log-rank, perplexity, ratio of rank and log-likelihood, surprise variance, mean of adjacent wordpiece surprise difference, information entropy, vocabulary density, variance of log-rank, and result of supervised detection;

[0061] S4. Combine the 11 features obtained in step S3 into an 11-dimensional vector, and input the 11-dimensional vector into a machine learning classification algorithm for supervised learning to obtain a machine-generated natural language detector;

[0062] S5. Input the text to be detected into the machine-generated natural language detector obtained in step S4 to obtain a detection result.​

[0063] Embodiment 2 provides a multi-feature-based machine-generated natural language detection method, comprising the following steps:

[0064] S1. Obtain the text to be detected, and pre-process the text to be detected to obtain a pre-processed text;

[0065] As a preferred embodiment, the pre-processing specifically includes removing punctuation, spaces, URLs, special characters and emoticons in the text to be detected, converting the vocabulary of the text to be detected to lowercase vocabulary and restoring the part of speech, thereby improving data quality, reducing noise and providing a good data foundation for subsequent steps;

[0066] S2. Input the pre-processed text obtained in step S1 into the tokenizer of the pre-trained language model to perform text segmentation and convert the segmentation into a wordpiece sequence recognizable by the pre-trained language model, input the wordpiece sequence into the pre-trained language model to perform calculation, and obtain the unnormalized prediction probability of each wordpiece;

[0067] As a preferred embodiment, the pre-trained language model is GPT2 XL, and the tokenizer is the tokenizer corresponding to GPT2 XL, which ensures the adaptation of text processing and the model, and improves the accuracy and effectiveness of feature extraction.

[0068] The tokenizer parameter max_length is set to 512 and truncation is set to True. That is, the length of the imported word vector is 1024 at most, and the vector is truncated beyond that, ensuring that the data length meets the range that the model can accept, and at the same time, the data dimension size within the same batch is consistent. At the same time, check whether the length of the word vector is less than 50, if the length is less than 50, remove the text. Because shorter texts are more difficult to detect, and the significance of detection is not great.

[0069] S3. Based on the word sequence obtained in step S2 and the unnormalized predicted probability of each word, calculate 11 features of the text. These 11 features include: log-likelihood, rank, log-rank, perplexity, the ratio of rank to log-likelihood, surprise variance, the mean of the surprise difference between adjacent words, information entropy, vocabulary density, the variance of log-rank, and the results of supervised detection. Among these, log-likelihood, rank, log-rank, perplexity, and the ratio of rank to log-likelihood are features related to the decoding strategy; surprise variance and the mean of the surprise difference between adjacent words are features based on the principle of uniform information density; information entropy is a feature based on information entropy; and vocabulary density and the variance of log-rank are features based on dataset research. In natural language generation tasks, pre-trained language models typically need to predict each word sequentially to generate text sequences until a termination condition is reached. During the generation process, the pre-trained language model generates probability scores for all words in the vocabulary based on the current state. The decoding strategy of a pre-trained language model involves how it selects output words based on probabilities. Researching this decoding strategy can uncover statistical features useful for detection. Typically, pre-trained language models select words with higher probability scores for output. Therefore, the higher the probability scores represented by these feature values ​​in the text to be detected, the more likely the text is to be machine-generated; conversely, it may be human-written.

[0070] As a preferred embodiment, specifically, calculating the log-likelihood of the text includes: calculating the normalized value of the unnormalized predicted probability of each word in the text, that is, performing a Softmax function operation on the value of the unnormalized predicted probability to obtain the word probability, and taking the logarithm of the word probability, and obtaining the negative mean of the log-likelihood of the text words by taking the mean function, which is the log-likelihood of the text S, as shown in formula (1):

[0071] LL(S)=-E(log(p(x))) (1)

[0072] Where LL(S) represents the log-likelihood of text S, S represents the text, x represents the word in text S, p(·) represents the probability function, and E(·) represents the mean function.

[0073] The negative mean of the log-likelihood of word probabilities is used to amplify the differences between the probabilities of different word probabilities. Therefore, a logarithm is introduced; smaller probabilities are represented by higher values, and larger probabilities by lower values, with the final result rounded positive. This transforms linear numerical changes into exponential changes, thus amplifying numerical differences. A high log-likelihood value suggests the text may be machine-generated, while a low value suggests it may be human-written.

[0074] Calculating the rank of a text involves: calculating the normalized value of the unnormalized predicted probability of each word, i.e., performing a Softmax function operation on the unnormalized predicted probability value to obtain the word probability, and then calculating the rank of the word probability. The mean rank of the text words is obtained by using a mean function, referred to as the rank, as shown in formula (2), which is used to calculate the mean rank of the word in the probability distribution of the vocabulary.

[0075] R(S)=E(rank(p(x))) (2)

[0076] Where R(S) represents the rank of text S, and rank represents the function that ranks the probability of a word x among the predicted candidate words of the pre-trained language model. It calculates the mean rank of the word x in the probability distribution of the vocabulary, similar to the idea of ​​log-likelihood, but a higher probability corresponds to a higher rank in the vocabulary, i.e., a lower rank. A smaller rank value indicates that the text may be machine-generated, while a larger rank value indicates that it may be human-written.

[0077] The logarithm-rank of a text can be calculated by directly taking the logarithm of R(S), as shown in formula (3), which is the logarithm used to calculate the rank:

[0078] LR(S)=log(R(S)) (3)

[0079] Wherein, LR(S) represents the log-rank of text S. In the operation, the logarithmic value of R(S) can be directly taken. Since the vocabulary of the language model is large, the logarithmic form of the rank value can be used, which has a better detection effect. If the log-rank value is small, the text may be machine-generated text, while if the log-rank value is large, it may be human-written text.

[0080] The probability for the text is shown in formula (4):

[0081]

[0082] Among them, w i Let i represent words in the text, n represent the length of the text, and i ∈ (1, 2, 3, ..., n).

[0083] The perplexity of the text is calculated as shown in formula (5):

[0084]

[0085] Where PPL(S) represents the perplexity of text S, and exp(·) represents the exponential function. In the training process of a language model, perplexity is often used as an indicator to evaluate the performance of the language model. According to the calculation of perplexity in the above formula, perplexity is based on the model's estimation of the probability distribution of the data, reflecting the model's adaptability to real data. Since the detection is performed using a pre-trained language model, given its superior performance, the smaller the perplexity feature value of the text to be detected, the more likely it is to be machine-generated text; conversely, the larger the perplexity feature value, the more likely it is to be human-written text.

[0086] The ratio of the rank to the log-likelihood of the text is calculated as shown in Formula (6) to highlight differences and aid classification:

[0087]

[0088] Here, Ratio(S) represents the ratio of the rank to the log-likelihood of text S. Combining the rank and log-likelihood, as described above, for the same output term of the model, a higher probability corresponds to a lower rank, and vice versa. Therefore, by calculating the ratio of the log-rank to the log-likelihood, this difference can be amplified, enabling machine-generated natural language detectors to better identify the differences between humans and machines.

[0089] The Uniform Information Density Principle (UID) posits that for efficient human communication, information in language should be distributed as evenly as possible. Furthermore, according to information theory, language can be viewed as a communication system, with each language unit carrying a different amount of information, which can be quantified using the surprise factor in information theory. For the i-th language unit u... i In this regard, surprising(s(·)) is u i The negative logarithm of the conditional probability of the model is given by equation (7):

[0090]

[0091] This definition implies that the lower the probability of a word appearing, the greater its information content. Based on the characteristic of uniform information distribution in human-written text according to the UID principle, the dispersion of word information can be calculated to derive its distribution characteristics, which can then be used as a standard for detection.

[0092] Calculating the surprise variance of the text includes: calculating the surprise value of the unnormalized predicted probability of each word, thereby obtaining the surprise variance of the text, as shown in formula (8):

[0093]

[0094] Where UID(S) represents the variance of the surprise of text S, s(x i ) represents the degree of surprise of the i-th word, x i Let s(x) represent the i-th word element. i )=-log(p(x i |x <i s(S) represents the surprise of text S, examining the dispersion of the surprise of the text with respect to the model. According to the UID principle, the information content of human-written text is more evenly distributed. If the surprise variance value is small, that is, the dispersion is low and the information content is more evenly distributed, the text is more likely to be human-written text; conversely, if the surprise variance value is large, that is, the dispersion is high and the information content is more chaotic, the text is more likely to be machine-generated text.

[0095] The mean of the difference in surprise between adjacent words in the text is calculated as shown in formula (9):

[0096]

[0097] Where UID'(S) represents the mean difference in the degree of surprise between adjacent terms in text S, x i-1 Let UID represent the (i-1)th lexical unit. Another interpretation of the UID principle is that UID acts as a resistance, preventing a rapid transition from information-dense to information-sparse parts. The transition between the sparse and dense components of the signal should be smooth. Examining the mean difference in the surprise of adjacent lexical units with respect to the model, according to the UID principle, the information content changes more smoothly in human-written text. If the mean difference in the surprise of adjacent lexical units is small, the information content changes more smoothly, and the text is more likely to be human-written; conversely, if the mean difference in the surprise of adjacent lexical units is large, the information content changes drastically, and the text is more likely to be machine-generated.

[0098] The information entropy of the text is calculated as shown in formula (10):

[0099]

[0100] Here, H(S) represents the information entropy of text S; information entropy is a fundamental concept in information theory. It describes the uncertainty of possible events occurring in an information source, representing the average amount of information after eliminating redundancy. Information entropy measures the uncertainty or unpredictability contained in an information source. The greater the information entropy, the greater the uncertainty in the information source, and the more information there is; conversely, the smaller the information entropy, the smaller the uncertainty in the information source, and the less information there is. In language models, information entropy represents the amount of information in the text, and the model's uncertainty about the text. The calculation method is the same as for information entropy in information theory, where the probability uses the model's predicted probability for each word. Since information entropy describes the uncertainty of the text, and the probability corresponding to the text generated by the model should be higher, the uncertainty is lower, and the information entropy is lower. Another interpretation is that for the text generated by the model, it is data "within the model's distribution," therefore its probability is higher, uncertainty is lower, and information entropy is lower. The mean of the information entropy of each word in the text is calculated. If the information entropy feature value is small, the text may be machine-generated; if the information entropy feature value is large, the text may be human-written.

[0101] Human-generated text and human-written text datasets have been studied, revealing that machine-generated text uses a smaller vocabulary than human-written text, indicating that humans use a richer vocabulary. Therefore, calculating the vocabulary size in text can also serve as a metric for detection. The vocabulary density of text is calculated as shown in formula (11):

[0102]

[0103] Where LD(S) represents the lexical density of text S, and V represents the number of words used in the text. The lexical density of a text is calculated. A high lexical density suggests that the text was written by humans, while a low lexical density suggests that the text was generated by a machine.

[0104] The variance of the log-rank of the text is calculated as shown in formula (12):

[0105]

[0106] Here, VR(S) represents the variance of the log-rank of text S, combining language model decoding strategies and lexical size research methods. Therefore, it calculates the dispersion of the rank values ​​of lexical units. Since machine-generated text has more concentrated choices, its log-rank variance is lower; while human-written text has more concentrated choices, resulting in a higher log-rank variance.

[0107] The result of the supervised detection of the text for the supervised detection model is calculated as shown in Equation (13):

[0108] SD(S)=F(S) (13)

[0109] Here, SD(S) represents the result of the supervised detection of text S by the supervised detection model, and F represents the supervised detection model. In the detection of machine-generated natural language text, in addition to feature-based detection, there are also methods based on pre-trained language models. Typically, this involves modifying the network structure of a pre-trained language model, freezing the network parameters except for the output layer, and replacing the output layer with a classification layer. It is then trained under supervision with a large amount of data to enable classification and detection. The result obtained by the pre-trained language model for machine-generated natural language detection, after inputting the text to be detected, is usually the probability that the text is machine-generated or human-written.

[0110] The supervised detection models are AI-generated text detection and Roberta large openaidetector. The word segmentation parameters max_length and truncation are set to True in the supervised detection models.

[0111] S4. Combine the 11 features obtained in step S3 into an 11-dimensional vector, and input the 11-dimensional vector into a machine learning classification algorithm for supervised learning to obtain a machine-generated natural language detector.

[0112] As a preferred embodiment, the machine learning classification algorithm is the extreme gradient boosting tree machine learning algorithm. In supervised learning, the ratio of training set to test set data is 9:1, the learning rate is 0.01, the maximum depth of the weak classifier is set to 29, and the number of weak classifiers is set to 1800. At the same time, it is stipulated that the label of machine-generated text is 1, and the label of human-written text is 0.

[0113] The loss function for supervised learning is the binary logistic regression loss, as shown in formula (14):

[0114]

[0115] in, y represents the binary logistic regression loss. k Let k represent the true probability of the k-th sample, k∈(1,2,3,…,N). Let N represent the predicted probability of the k-th sample, and N represent the number of samples.

[0116] More preferably, the 11 features need to be standardized before supervised learning. After data standardization, the model can be created. During the parameter tuning phase, grid search can be used to select parameters. After multiple iterations of searching and testing, the model with the best performance metrics is selected as the classification model for the detection algorithm.

[0117] S5. Input the text to be detected into the machine-generated natural language detector obtained in step S4 to obtain the detection result;

[0118] As a preferred embodiment, specifically, the detection results include whether the text to be detected is machine-generated text or human-written text.

[0119] To verify the detection effectiveness of the natural language detector in this embodiment, the following experiment was conducted:

[0120] To test the performance of the natural language detector in this embodiment, nine human-written text datasets and one human ChatGPT contrastive corpus (HC3) dataset were collected. Based on the human-written text datasets, corresponding machine-generated texts were generated using multiple language models (T5-3B, Davinci, GPT series). The nine human-written text datasets, the corresponding machine-generated texts, and HC3 were then merged to form the dataset for this experiment.

[0121] To cover diverse domains and generation methods, nine human-written text datasets were used as source data, and various large-scale language models were employed to construct machine-generated text. The machine-generated text construction methods for the datasets described below are as follows: Method 1 involves selecting the first 30 tokens from the source text and inputting them into a large-scale language model to generate more text. The source text is human-written, and the text generated by the language model is the machine-generated text. Method 2 requires the large-scale language model to generate machine-generated text based on a specific topic, such as an argument, news headline, story theme, or article theme.

[0122] The following provides an overview of the training and testing datasets:

[0123] 1. Statements and restates from the r / ChangeMyView (CMV) Reddit subcommunity.

[0124] 2. Views from the Yelp review dataset.

[0125] 3. News articles from XSum and TLDR_news (TLDR).

[0126] 4. Question and answer text from the ELI5 dataset.

[0127] 5. Story text based on the RedditWritingPrompts (WP) dataset.

[0128] 6. Story text from ROCStories Corpora (ROC).

[0129] 7. Common sense reasoning text from HellaSwag (HS).

[0130] 8. Reading comprehension dataset from SQuAD.

[0131] 9. Scientific article abstracts from SciGen (SGen).

[0132] 10. Human ChatGPT Contrastive Corpus (HC3) dataset.

[0133] Table 1 shows the statistical information of human-written text data and machine-generated text data from various sources.

[0134] Table 1 Dataset Details

[0135]

[0136] HWT represents human-written text, totaling 178,000 entries; MGT represents machine-generated text, totaling 318,000 entries, for a total of 497,000 entries.

[0137] This embodiment employs various metrics for evaluating the performance of classification models, including: Accuracy (Acc), Precision (Pre), Recall (Rec), F1 Score (F1), Confusion Matrix, AUC, False Positive Rate (FPR), and False Negative Rate (FNR). The metrics for classification problems are defined as follows:

[0138] TN (True Negative): The number of samples that the model correctly predicted as negative.

[0139] FN (False Negative): The number of actual positive samples that the model incorrectly predicted as negative.

[0140] TP (True Positive): The number of samples that the model correctly predicts as positive.

[0141] FP (False Positive): The number of actual negative samples that the model incorrectly predicted as positive.

[0142] The definitions of each evaluation indicator are as follows:

[0143] The accuracy is shown in formula (15):

[0144]

[0145] The accuracy is shown in formula (16):

[0146]

[0147] The recall rate is shown in formula (17):

[0148]

[0149] The F1 score is shown in formula (18):

[0150]

[0151] The false positive rate is shown in formula (19):

[0152]

[0153] The false negative rate is shown in formula (20):

[0154]

[0155] In the experiments of this embodiment, eight algorithms are mainly used to compare with the natural language detection method of this embodiment.

[0156] 1. Log Likelihood: This method evaluates the average log probability of a text. The principle is that paragraphs with a higher average log probability are more likely to have been generated by a machine.

[0157] 2. Rank Method: This method evaluates the average rank of each word in a text. The principle is that paragraphs with lower average rank are more likely to have been generated by a machine.

[0158] 3. Log Rank: This method does not directly use the rank number, but instead evaluates the average log rank of each word in the text. Paragraphs with a lower average observed log rank are more likely to have been generated by a machine.

[0159] 4. Entropy Method: This method is inspired by the assumption that machine-generated text is more likely to have an overconfident (and therefore low-entropy) predictive distribution. This method uses a higher average entropy as a signal of machine-generated text.

[0160] 5. DetectGPT: DetectGPT is a perturbation-based detection method. Specifically, this method demonstrates that text sampled from large language models often lies in the negative curvature region of the model's logarithmic probability function. Therefore, DetectGPT examines the average probability change of text after perturbation.

[0161] 6. Supervised Detector Based on Large Language Model: This is a classifier based on a large language model. It takes text as input and outputs the probability that the text belongs to machine-generated text. Hereinafter referred to as supervised detection.

[0162] 7. DetectLLM-LRR: DetectLLM-LRR is a decoding-based method. This method uses the log-likelihood and log-rank ratio as features. By leveraging the numerical properties of these two features, it amplifies feature information, making detection more efficient.

[0163] 8. DetectLLM-NPR: DetectLLM-NPR is also a perturbation-based detection method. However, DetectLLM-NPR examines the average rank change of the text after it has been perturbed.

[0164] The experimental results of this embodiment are shown in Table 2:

[0165] Table 2. Experimental results of the six comparative algorithms and the algorithm of this invention on the entire dataset.

[0166]

[0167]

[0168] In this experiment, a total of 400,000 data points were first used for training and testing. Experimental results were obtained for the multi-feature machine-generated natural language detection method (MFE-Detector) and four baseline methods. Table 2 shows that the MFE-Detector algorithm achieved an accuracy of 93.95%, an F1 score of 0.9383, and an AUC of 0.9396. Compared to the comparison algorithms, the accuracy of the MFE-Detector algorithm improved by an average of approximately 60%. The experiment revealed that the MFE-Detector algorithm misclassified human text as machine-generated text (false positive rate) by 3.790%, while misclassifying machine-generated text as human-written text (false negative rate) by 8.295%. This indicates a slightly higher false positive rate, meaning the MFE-Detector algorithm is more likely to classify human-written text as machine-generated text.

[0169] Compared to single-feature detection methods such as entropy, log-likelihood, and rank methods, the MFE-Detector algorithm introduces multiple features, including probabilistic features based on decoding strategies, features based on information entropy, features based on the difference between human and machine vocabulary, features based on the principle of uniform information density, and features based on the results of supervised detectors. It covers a more comprehensive range, using multiple features to comprehensively consider the source of the text to be detected, determining the text source from multiple perspectives, thus making the results more accurate. This demonstrates the necessity of introducing multi-feature classification. Therefore, the detection accuracy is improved.

[0170] This embodiment was then compared with a perturbation-based contrast algorithm on the HC3 dataset, and the results are shown in Table 3:

[0171] Table 3. Experimental results of the four comparison algorithms and the algorithm of this invention on the HC3 dataset.

[0172]

[0173] Experimental results show that DetectLLM-LRR outperforms other baseline methods, while DetectLLM-NPR's AUC value is close to 0.5, indicating complete task failure. Therefore, the unusually high recall observed in DetectLLM-NPR is indeed unusually high. In contrast, the proposed method, MFE-Detector, achieves state-of-the-art detection performance across various metrics on the HC3 dataset. The superior performance of MFE-Detector on the HC3 dataset is likely due to the fact that the HC3 dataset is constructed using ChatGPT, while MFE-Detector employs GPT2 XL as its feature extraction model. These two homologous models can extract features more effectively, thus improving performance on the HC3 dataset. This finding suggests that selecting a suite of models as feature extractors can better extract text features from diverse sources.

[0174] This embodiment also conducted experiments on the impact of language model selection on detection accuracy. This embodiment investigated the effect of different pre-trained language models on detection performance while keeping other conditions constant. The aim was to evaluate whether model selection affects detection results. The tested models included RoBERTa-large, GPT2 Medium, Mamba, and GPT2 XL. The experimental results are shown in Table 4.

[0175] Table 4. The impact of different language models as embedded computing models on performance.

[0176]

[0177] As shown in Table 4, for the GPT model, the performance of the MFE-Detector algorithm in this embodiment slightly decreases across all metrics as the model size (number of parameters) decreases. This indicates that detection performance decreases with decreasing model size. In contrast, RoBERTa requires less computation time but has lower accuracy. This suggests that RoBERTa may be more suitable for resource-constrained environments. Mamba, a promising new model architecture across different domains, has less robust detection performance, possibly due to lower pre-training strength or a suboptimal model structure. In summary, the MFE-Detector algorithm uses GPT2 XL as the computational model for text embedding, satisfying both the model's detection performance requirements and its low computational resource requirements, thus meeting the requirements of resource conservation.

[0178] In this embodiment, selecting an effective classification model is crucial. Several classification algorithms were evaluated experimentally, including K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Fully Connected Neural Network (FCNN), and Extreme Gradient Boosting Tree (XGBoost). The experimental results are shown in Table 5. The XGBoost algorithm exhibits better performance; therefore, XGBoost was chosen as the classification model in this invention.

[0179] Table 5. Impact of different algorithms as classification models on performance.

[0180]

[0181] Tables 6 and 7 are implementation examples of this embodiment.

[0182] Table 6 Implementation Case 1 of this Embodiment

[0183]

[0184] Table 7 Implementation Case 2 of this Example

[0185]

[0186] Tables 6 and 7 show the predicted labels of the present invention and the comparison method. As can be seen from the tables, the machine-generated natural language detection method based on multiple features in this embodiment can correctly predict the source of the text, while other comparison algorithms may not be able to detect the source correctly.

[0187] This invention presents a multi-feature-based machine-generated natural language detection method. It utilizes statistical analysis of a series of feature data distributions, including log-likelihood, rank, log-rank, perplexity, the ratio of rank to log-likelihood, variance of surprise, mean of the difference in surprise between adjacent words, information entropy, lexical density, variance of log-rank, and the output of supervised detection. In practice, machine-generated text typically exhibits the following characteristics: low log-likelihood, low rank, low log-rank, low ratio of rank to log-likelihood, low perplexity, high variance of surprise, high mean of the difference in surprise between adjacent words, low information entropy, low variance of log-rank, low lexical density, and a large output from the supervised detector. Conversely, human-written text exhibits the opposite characteristics. By performing multi-dimensional analysis of these features, this invention comprehensively considers the source of the text to be detected, thereby more accurately distinguishing between machine-generated and human-written text, significantly improving the accuracy of text source detection. Compared to existing baseline methods, this invention achieves superior performance.

[0188] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A multi-feature based machine-generated natural language detection method, characterized in that, The method comprises the following steps: S1. Obtain a text to be detected, and pre-process the text to be detected to obtain a pre-processed text; S2. Input the pre-processed text obtained in step S1 into a tokenizer of a pre-trained language model to perform text segmentation and convert the segmentation into a wordpiece sequence recognizable by the pre-trained language model, input the wordpiece sequence into the pre-trained language model to perform calculation, and obtain an unnormalized prediction probability of each wordpiece; S3. Calculate 11 features of the text according to the wordpiece sequence and the unnormalized prediction probability of each wordpiece obtained in step S2, the 11 features including: log-likelihood, rank, log-rank, perplexity, ratio of rank to log-likelihood, surprise variance, mean of adjacent wordpiece surprise difference, information entropy, vocabulary density, variance of log-rank, and result of supervised detection; wherein the result of supervised detection is the result of supervised detection of the text on a supervised detection model; S4. splice the 11 features obtained in step S3 into an 11-dimensional vector, and input the 11-dimensional vector into a machine learning classification algorithm for supervised learning to obtain a machine-generated natural language detector; S5. Input the text to be detected into the machine-generated natural language detector obtained in step S4 to obtain a detection result.

2. The method of claim 1, wherein, The pre-processing in step S1 includes removing punctuation, spaces, URLs, special characters and emoticons in the text to be detected, converting the vocabulary of the text to be detected into lowercase vocabulary and restoring the part of speech. 3.The method of claim 1, wherein, In step S2, the pre-trained language model is GPT2 XL, the tokenizer parameter max_length is set to 512, and the truncation is set to True.

4. The method of claim 1, wherein, In step S3, the log-likelihood of the text is calculated as shown in formula (1): wherein denotes the log-likelihood of the text S, S denotes the text, denotes the wordpiece in the text S, p(·) denotes the probability function, E(·) denotes the mean function; The rank of the text is calculated as shown in formula (2): wherein, denotes a rank number of the text S, denotes a function of ranking the probability of the word piece x in the pre-trained language model to predict the candidate word. The log-rank of the text is calculated as shown in formula (3): wherein, denotes the log-rank statistic for the text S; The perplexity of the text is calculated as shown in formula (4): wherein, denotes the perplexity of the text S, denotes the number of wordpieces in the text, exp(·) denotes the exponential function; The ratio of the rank to the log-likelihood of the text is calculated as shown in formula (5): wherein, represents the ratio of the rank number and the log-likelihood of the text S; The surprise variance of the text is calculated as shown in formula (6): wherein, denotes the variance of surprise of the text S, denotes the surprise of the i-th token, denotes the i-th token, , denotes the surprise of the text S; The mean of the adjacent wordpiece surprise difference of the text is calculated as shown in formula (7): wherein, represents the mean of the adjacent wordpiece surprise difference values for the text S, represents the i-1th wordpiece; The information entropy of the text is calculated as shown in formula (8): wherein, denotes the information entropy of the text S; The vocabulary density of the text is calculated as shown in formula (9): wherein, represents the lexical density of the text S, represents the number of lexical items used in the text; The variance of the log-rank of the text is calculated as shown in formula (10): wherein, denotes the variance of the log-rank of the text S; The result of the supervised detection of the text on the supervised detection model is calculated as shown in formula (11): wherein, represents a result of supervised detection of the text S by the supervised detection model, represents a supervised detection model.

5. The method of claim 4, wherein, The supervised detection model is AI generated text detection and roberta large openai detector, the tokenizer parameter max_length of the supervised detection model is set to 512, and the truncation is set to True.

6. The method of claim 1, wherein the method further comprises: In step S4, the machine learning classification algorithm is an extreme gradient boosting tree machine learning algorithm.

7. The method of claim 1, wherein the method further comprises: In the step S4, the ratio of the training set to the test set is 9:1, the learning rate is 0.01, the maximum depth of the weak classifier is set to 29, and the number of weak classifiers is set to 1800.

8. The method of claim 1, wherein, In the step S4, the loss function of the supervised learning is a binary logistic regression loss, as shown in formula (12): where, denotes the binary logistic loss, denotes the true probability of the kth sample, , denotes the predicted probability of the kth sample, denotes the number of samples.

9. The method of claim 1, wherein, In the step S5, the detection result includes that the text to be detected is machine-generated text and the text to be detected is human-written text.

Images