Machine generated text tracing method and system based on feature fusion

Abstract

The invention provides a feature fusion-based machine-generated text traceability method and a feature fusion-based machine-generated text traceability system aiming at the problem that a single feature in the existing machine-generated text traceability technology is difficult to accurately trace the source of a machine-generated text, a new machine-generated text traceability model is constructed, and the traceability of the machine-generated text is improved by combining statistical features and semantic features of the text. And richer feature representation is provided for the model. The method comprises the following steps of: firstly, converting an input text into a semantic vector by using a word embedding technology, and calculating a prediction probability, a cumulative probability and an information entropy of the text by using a GPT-2 model to serve as a statistical vector of the text; then, further extracting features through GRU and Transform encoders, performing linear transformation on the features through a linear layer to obtain semantic features and statistical features of the text, and integrating the semantic features and the statistical features through a feature fusion strategy; and finally, after the fused feature vectors are analyzed by a classifier, outputting a specific source of the text.

Description

Technical Field

[0001] The present invention relates to a method and system for tracing the origin of machine-generated text based on feature fusion, and belongs to the technical field of machine-generated text detection and tracing. Background Art

[0002] At present, machine-generated text detection technology is generally devoted to determining whether a text is machine-generated, while there are relatively few studies on the detailed source of the text, that is, the identification of the specific language model that generates these texts. The task of identifying whether a piece of text is machine-generated text or human-written text, and if it is machine-generated, further determining which specific language model it is generated by, is called machine-generated text tracing, which is a fine-grained machine-generated text detection. With the emergence of many large language models (LLM), the problem of accurately tracing the source of text has become increasingly prominent, which is of great significance for ensuring the authenticity and transparency of information and tracing the source of responsibility. However, the existing machine-generated text tracing model uses a single statistical feature to judge the source of the text. With the continuous improvement of LLM text generation capabilities, the quality of generated text is getting higher and higher, which makes it more difficult to use a single feature to accurately trace the source of the text. Therefore, it is particularly important to study more effective machine-generated text tracing technology. Summary of the invention

[0003] In order to cope with the challenges faced in the process of machine-generated text tracing, the present invention proposes a machine-generated text tracing model (Feature Fusion Model, FFM) based on feature fusion to solve this problem. Compared with the existing model, FFM calculates two features of text, statistical features and semantic features, and combines two different features to identify the source of the text. The statistical features and semantic features of the text provide information of different dimensions. By combining these two features, the model can extract the feature information of the text more comprehensively. A single statistical feature cannot capture the semantic and contextual information of the text. Some texts have more overlaps in statistical features, but the differences in semantic features are obvious. The addition of semantic features can improve the model's recognition accuracy of the text source. Specifically, FFM first converts the input sentence into a semantic vector representation through word embedding technology to capture the semantic information of the text. At the same time, the model also calculates the predicted probability, cumulative probability and information entropy of the input text, etc., which together constitute the statistical vector of the text. On this basis, the semantic features and statistical features of the text are obtained through feature extraction, and then FFM effectively integrates the lexical features and statistical features through the feature fusion strategy to form a comprehensive feature vector. Finally, the fused feature vector is fed into a classifier, which is responsible for judging whether the input text is human-written or comes from a certain LLM based on the content of the feature vector. The flexibility and scalability of this model enable it to adapt to the emerging new language models, providing a new perspective and method for the development of machine-generated text tracing technology.

[0004] The present invention discloses a method and system for tracing the origin of machine-generated text based on feature fusion, comprising the following steps:

[0005] Step 1: Obtain a text tracing dataset including training data and corresponding category labels;

[0006] Step 2: Use word embedding technology to convert the text traceability dataset in step 1 into a semantic vector. Use the GPT-2 model to convert the text traceability dataset in step 1 into a statistical vector.

[0007] Step 3: Use the text semantic vector and statistical vector of step 2 to train a machine-generated text traceability model based on feature fusion. The machine-generated text traceability detection model based on feature fusion includes a feature extraction layer, a feature fusion layer, and a classification layer. After the training is completed, the trained machine-generated text traceability model based on feature fusion is obtained;

[0008] Step 4: Deploy the trained feature fusion-based machine-generated text traceability model into the system;

[0009] Step 5: Take the received text to be traced as input, and use the trained machine-generated text detection model based on feature fusion to perform classification prediction to obtain the traceability result;

[0010] Step 6. The system returns the traceability result to the user. The user obtains the traceability result, which is the category label to which the input text belongs. If there is a traceability error, the user can upload the error traceability information through the front-end page.

[0011] A further improvement of the present invention is that the present invention uses word embedding technology to convert text into a semantic vector form, and further calculates the predicted probability of the text, the cumulative probability of the text and the information entropy of the text as the statistical vector of the text through the GPT-2 model. The specific steps include:

[0012] Step 2.1. The text in the input text tracing dataset is decomposed into a series of tokens through the BERT model. It maps each word of the input text to a high-dimensional vector space, so that the model can capture the semantic information of each token and the relationship between words. These embedding vectors not only contain the information of the token itself, but also incorporate its position information in the text, that is, position embedding. By adding position information to word embedding, the model can understand the position of each token in the sentence. In addition, the BERT model also uses paragraph embedding, which is particularly important for processing paired sentences. Paragraph embedding helps the model distinguish tokens in different sentences, so as to better understand and process the interactions between text pairs. By combining word embedding, position embedding, and paragraph embedding, BERT can generate a comprehensive vector for each token, and input these comprehensive embedding representations into the encoding layer for further processing to obtain the final embedding representation, which not only enriches the semantic representation of the token, but also gives the model a deep understanding of the text structure;

[0013] Step 2.2, calculate the statistical vector of the text through the GPT-2 model as the input of the subsequent steps. Specifically, the present invention achieves this goal by calculating the predicted probability of the input text, the cumulative probability of the text, and the information entropy of the text. The invention uses a specific language model M (GPT-2 model) to process the input text. For an input text consisting of n words W = {w1, w2, ..., w i ,…,w n} and the size of the vocabulary of the language model M (the number of tokens) l, the text prediction probability is calculated as follows:

[0014] The predicted probability of the i-th word in the input text of the language model M The calculation formula is as follows:

[0015]

[0016] In the formula, S M (w i |w1,w2,…,w i-1 ) means that all the previous words are (w1,w2,…,w i-1 ), the current word is predicted to be w i Score. M (positioni k |w1,w2,…,w i-1 ) means that all words before the given are (w1,w2,…,w i-1 ), the scores of all tokens in the vocabulary are summed;

[0017] Step 2.3, calculate the cumulative probability of the text, the calculation method is as follows:

[0018] For the language model M, the cumulative probability C of the i-th word in the input text is M (w i ) is calculated as follows:

[0019]

[0020] The formula calculates all the probabilities greater than or equal to Pro at the i-th position of the input text M (w i After concatenating the cumulative probabilities of each word in the input text, we can get the cumulative probability C of the text. M (W)=[C M (w1),C M (w2),…,C M (w n )].

[0021] Step 2.4: Calculate the text information entropy as follows:

[0022] Information entropy is a measure of the uncertainty of the probability distribution of all tokens in the vocabulary at a specific position. For the i-th position in the text, define E M (i) is the information entropy of the probability distribution of all tokens in the vocabulary at that position, and the specific calculation method is shown in the following formula:

[0023]

[0024] This formula calculates the probability of each token in the vocabulary appearing given the previous text and the 4 negative log-likelihood of the probability by summing them up, and obtains the information entropy of the i-th word by taking the product. Then, the information entropy E of the entire input text can be calculated M (W)=[E M(1),E M (2),…,E M (n)], taking it as part of the statistical features of the input text;

[0025] Step 2.5: The three statistical features of text prediction probability, cumulative probability and information entropy calculated by the GPT-2 model are integrated into a feature vector.

[0026] A further improvement of the present invention is that the present invention adopts a feature fusion strategy to integrate the semantic features and statistical features of the text, and its steps specifically include:

[0027] Step 3.1, for the feature extraction layer, its input is the semantic vector and text statistical vector of the text, which are processed by GRU and Transformer encoder respectively. GRU receives the semantic vector of the text as input, which is extracted from the text data and contains rich semantic information. The output of GRU is a tensor that incorporates the contextual information of each time step in the sequence, and its dimensions include batch size, sequence length, and feature dimension. By updating the gate and resetting the gate, GRU can effectively capture long-term dependencies in the sequence data and provide the model with a detailed time series feature representation. The specific calculation process is as follows:

[0028] After the text of the BERT model, each element of the text sequence becomes an embedding vector. Next, these vectors are passed as input to GRU. The goal of GRU is to capture the features and time dependencies in the sequence.

[0029] At each time step t, the GRU updates its hidden state h t The update process depends on the current input e t , the hidden state h of the previous time step t-1 and update gate z t and reset gate r t .

[0030] Update gate z t Controls the previous hidden state h t-1 To the current hidden state h t The information flow of the reset gate r t Determines how much "past information" should be ignored. The calculation method of these two gates is as follows:

[0031] z t =σ(W z [e t ;h t-1 ]+b z )

[0032] rt =σ(W r [e t ;h t-1 ]+b r )

[0033] In the formula, W z and W r is the weight matrix of the update gate and reset gate, b z and b r is the corresponding bias term, σ is the sigmoid activation function, e t Represents the current input vector.

[0034] Using the outputs of the update gate and the reset gate, GRU calculates the candidate hidden state h t ′, which is obtained by converting the current input e t With reset gate r t The previous hidden state h after calculation t-1 Combined with the nonlinear activation function tanh, the specific calculation method is shown in the formula:

[0035] h t ′=tanh(W[r t ·h t-1 ]+Ue t +b)

[0036] Among them, W and U are the weight matrices of GRU, b is the bias term, and tanh represents the activation function. Finally, update gate z t Used to combine the hidden state h of the previous time step t-1 and the current candidate hidden state h t ′ to generate the final hidden state h of the current time step t , the specific calculation method is shown in the formula:

[0037] h t =(1-z t )h t-1 +z t h t '

[0038] This process is repeated at each time step of the sequence, and finally the hidden state sequence H = (h1, h2, ..., h n ). In this way, GRU is able to provide useful feature representations.

[0039] The statistical vector calculated by the GPT-2 model for the input text is processed by the Transformer encoder, which encodes the input sequence and outputs a tensor containing global dependency information. The sequence length of the output tensor of the Transformer encoder is the same as the input, in which each element is encoded and contains position information and global dependency information.

[0040] The Transformer encoder receives a vector as input and converts it into a series of position-related representations that capture the position information of the input vector. The Transformer encoder consists of an input layer, a multi-head attention layer, a feedforward network, a residual connection, and a normalization layer. The input layer converts the input sequence into a format that the model can handle, while the multi-head attention layer calculates the attention weights of the input sequence. The feedforward network performs further linear and nonlinear transformations on the output of the multi-head attention layer, and the residual connections and normalization layers help stabilize the training process and avoid the gradient vanishing problem in deep networks. The specific calculation method is as follows:

[0041] The encoder first position-encodes the input vector. After position encoding, the input sequence generates three different vector sets: query, key, and value through three different weight matrices. This step is the basis of the multi-head attention layer, which allows the model to calculate the impact of each element on other elements in subsequent steps. The calculation of the output vectors of different parts in the multi-head attention layer is achieved through the query vector, key vector, value vector, the dimension of the key vector and the corresponding weight matrix. The specific calculation method is shown in the following formula:

[0042]

[0043] In the formula, Q, K, and V represent the query vector, key vector, and value vector, respectively. i Q , and is the corresponding weight matrix, d k is the dimension of the key vector. This formula first calculates QW i Q and The dot product of k The square root ratio of is used to obtain an attention score matrix, and then these scores are converted into probabilities through the Softmax function. Finally, a weighted output vector is obtained. This output vector is a transformation of the input vector, which contains the contribution of each element in the input vector to the current element. The final output of the multi-head attention layer is calculated by the following formula:

[0044] MultiHead(Q,K,V)=Concat(Head1,Head2,…,Headh)W O

[0045] Here, Concat means vector concatenation, W O Represents a weight matrix used to linearly transform features after concatenation, and MultiHead(Q,K,V) represents the output vector.

[0046] After that, the output of the multi-head attention layer will pass through a feed-forward network (FFN) after residual connection and normalization. This is a neural network that contains two linear transformations and a nonlinear activation function. The calculation formula of the feed-forward network is as follows:

[0047] FFN(x)=max(0,xW1+b1)W2+b2

[0048] In the formula, W1 and W2 are the weight matrices of the feedforward network, x represents the input of the previous layer, and b1 and b2 are bias terms. This network performs further linear and nonlinear transformations on the output of the previous layer, increasing the expressive power of the model. Finally, this output is added to the input vector, and residual connections and normalization are performed to obtain the final output of the encoder;

[0049] Step 3.2: The outputs of the GRU and Transformer encoders are transformed linearly through a linear layer to obtain the statistical features and semantic features of the text, which are used as the input of the feature fusion layer.

[0050] Step 3.3: The role of the feature fusion layer is to integrate the information of different features so that the classification model can understand the text data from different perspectives. In this way, the model can not only capture the semantic content of the text, but also take into account the statistical information of the text sequence, providing a richer feature representation for the subsequent classification task.

[0051] Specifically, after the feature extraction layer, two new feature vectors S and P are obtained. The purpose of the feature fusion layer is to merge these two sets of features into a single feature vector so that the subsequent classification layer can use this comprehensive information to make decisions.

[0052] Next, feature fusion is performed through the concatenation operation, that is, S and P are connected in element order to form a longer feature vector F:

[0053] F = [S; P]

[0054] In the formula, the semicolon ";" indicates the concatenation operation of the vectors. In this way, the fused feature vector F contains information about the text semantic features and text statistical features, and its dimension is the sum of the S and P dimensions;

[0055] Step 3.4: In the classification layer, the input feature vector is first linearly transformed, and then a Dropout layer is used to randomly "discard" the output of some neurons to reduce the network's dependence on specific neurons. Next, the output of each neuron is nonlinearly transformed through the ReLU activation function, so that the network can learn nonlinear relationships. Then, a fully connected layer is used to perform a linear transformation on the input features by applying a weight matrix and a bias vector to generate a score vector whose length is the same as the number of target categories. Each element in the vector represents the confidence of the model's prediction that the input text belongs to a specific category. Then, the Softmax function is used to calculate the probability that the input feature belongs to each category; BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 System architecture diagram for machine-generated text tracing method and system based on feature fusion

[0057] Figure 2 System flow chart of the machine-generated text tracing method and system based on feature fusion

[0058] Figure 3 Model structure diagram for machine-generated text tracing method based on feature fusion DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific implementation methods.

[0060] It should be noted here that in order to avoid unnecessary details that would obscure important parts of the present invention, the drawings only show structures or processing steps that are closely related to the present invention, while ignoring other details that are not very relevant to the present invention.

[0061] In addition, it should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

[0063] The present invention discloses a machine-generated text tracing method and system based on feature fusion, the system architecture is as follows: Figure 1 As shown in the figure, the system architecture adopts a layered design, mainly including the data layer, business logic layer and presentation layer. The data layer is responsible for storing user information, log information, error feedback information and model parameters. The relational database management system MySQL is used to store structured data. The business logic layer is the core of the system, responsible for handling user operations, user management, input text detection and traceability tasks. In this layer, the Django backend framework is used to manage the interaction with the database, perform user authentication, process data, and communicate effectively with the front end through the API. Django's efficient routing mechanism and rich libraries enable the system to handle complex business rules and data flows, while ensuring the security of the system and the integrity of the data. The business logic layer uses the deep learning framework PyTorch to build the algorithm model. The presentation layer is responsible for interacting with the user and providing a friendly user interface. The Vue framework is used to build the user interface to achieve dynamic data interaction and a good user experience, and to ensure the compatibility and ease of use of the system on different devices. Figure 2 The process of machine-generated text tracing method and system based on feature fusion is demonstrated, including the following steps:

[0064] Step 1. Obtain a text tracing dataset including training data and corresponding category labels. The human text comes from multiple public datasets, such as XSum, WritingP, SQuAD, and Yelp datasets. In addition, machine text is generated from parallel human text to study the problem of tracing the origin of text. In this study, machine text was generated by multiple LLMs, including GPT-2, GPT-3, GPT-J, GPT-Neo, ChatGPT, PaLM, and LLaMA models. The present invention uses three types of prompts to guide the text generation of LLM: (1) Continuation prompt: Require LLM to continue generating text based on the first 20 words of the original human text. (2) Topic prompt: Use prompts such as arguments, news headlines, and story themes to guide LLM to generate text. (3) Specified prompt: Information provided to LLM containing specific text style requirements to guide the model to produce text that meets preset features. Text written by humans is classified and labeled as "human", and the specific source of machine-generated text is marked, that is, which LLM generated the machine-generated text;

[0065] Step 2: Use word embedding technology to convert the text tracing dataset in step 1 into a word vector. Use the GPT-2 model to convert the text tracing dataset in step 1 into a statistical vector. The specific steps are as follows:

[0066] Step 2.1. The text in the input text tracing dataset is decomposed into a series of tokens through the BERT model. It maps each word of the input text to a high-dimensional vector space, so that the model can capture the semantic information of each token and the relationship between words. These embedding vectors not only contain the information of the token itself, but also incorporate its position information in the text, that is, position embedding. By adding position information to word embedding, the model can understand the position of each token in the sentence. In addition, the BERT model also uses paragraph embedding, which is particularly important for processing paired sentences. Paragraph embedding helps the model distinguish tokens in different sentences, so as to better understand and process the interactions between text pairs. By combining word embedding, position embedding, and paragraph embedding, BERT can generate a comprehensive vector for each token, and input these comprehensive embedding representations into the encoding layer for further processing to obtain the final embedding representation, which not only enriches the semantic representation of the token, but also gives the model a deep understanding of the text structure;

[0067] Step 2.2: Use the GPT-2 model to calculate the predicted probability of each word in the text, thereby forming the predicted probability of the entire text;

[0068] Step 2.3: Use the GPT-2 model to calculate the cumulative probability of each word in the text, thereby forming the cumulative probability of the entire text;

[0069] Step 2.4: Use the GPT-2 model to calculate the information entropy of each word in the text, thereby forming the information entropy of the entire text;

[0070] Step 2.5: Input the text and calculate the text prediction probability, cumulative probability and information entropy through the GPT-2 model, and integrate these three into a statistical vector;

[0071] Step 3: Figure 2 As shown, the text word vector and statistical vector of step 2 are used to train the machine-generated text traceability model based on feature fusion. The machine-generated text traceability detection model based on feature fusion includes a feature extraction layer, a feature fusion layer and a classification layer. After the training is completed, the trained machine-generated text traceability model based on feature fusion is obtained. The specific steps are as follows:

[0072] Step 3.1, for the feature extraction layer, its input is the semantic vector and text statistical vector of the text, which are processed by GRU and Transformer encoder respectively. GRU receives the semantic features of the text as input, which are extracted from the text data and contain rich semantic information. The output of GRU is a tensor that incorporates the contextual information of each time step in the sequence, and its dimensions include batch size, sequence length, and feature dimension. By updating the gate and resetting the gate, GRU can effectively capture long-term dependencies in sequence data and provide the model with a detailed representation of time series features;

[0073] Step 3.2: The outputs of the GRU and Transformer encoders are transformed linearly through a linear layer to obtain the semantic features and statistical features of the text, which are used as the input of the feature fusion layer.

[0074] Step 3.3, feature fusion layer integration Through the feature fusion strategy, the semantic features and statistical features of the text are integrated, so that the classification model can understand the text data from different perspectives. In this way, the model can not only capture the semantic content of the text, but also take into account the statistical information of the text sequence, providing richer feature representation for the subsequent classification task;

[0075] Step 3.4, in the classification layer, first perform a linear transformation on the input feature vector, and then use a Dropout layer to randomly "discard" the output of some neurons to reduce the network's dependence on specific neurons. Next, use the ReLU activation function to perform a nonlinear transformation on the output of each neuron, so that the network can learn nonlinear relationships. Then use a fully connected layer to perform a linear transformation on the input features by applying a weight matrix and a bias vector to generate a score vector whose length is the same as the number of target categories. Each element in the vector represents the confidence of the model's prediction that the input text belongs to a specific category. Then, through the Softmax function, the probability of the input feature belonging to each category;

[0076] Step 4: Deploy the trained feature fusion-based machine-generated text traceability model into the system. The specific steps are as follows:

[0077] Step 4.1. Export the trained FFM model as an ONNX format file to ensure that it can run in a production environment.

[0078] Step 4.2: Configure the server environment to support HTTP requests;

[0079] Step 4.3, upload the exported model file to the production server and integrate it into the predetermined service. Next, upload the model file to the server via FTP, SCP and other protocols. Then set up the API interface routing and database connection initialization. Finally, start the model service and perform interface testing to verify whether the model receives input, performs predictions, and returns results normally;

[0080] Step 5: Take the received text to be traced as input, and use the trained machine-generated text detection model based on feature fusion to perform classification prediction to obtain the traceability result;

[0081] Step 6. The system returns the traceability result to the user. The user obtains the traceability result, which is the category label to which the input text belongs. If there is a traceability error, the user can upload the error traceability information through the front-end page.

Claims

1. A machine-generated text tracing method and system based on feature fusion, characterized in that: The following steps are involved: Step 1: Obtain a text tracing dataset including training data and corresponding category labels; Step 2: Use word embedding technology to convert the text tracing dataset in step 1 into a semantic vector; use the GPT-2 model to convert the text tracing dataset in step 1 into a statistical vector. Step 3: Use the text semantic vector and statistical vector of step 2 to train a machine-generated text traceability model based on feature fusion. The machine-generated text traceability model based on feature fusion includes a feature extraction layer, a feature fusion layer, and a classification layer. After the training is completed, the trained machine-generated text traceability model based on feature fusion is obtained. Step 4: Deploy the trained feature fusion-based machine-generated text traceability model into the system; Step 5: Take the received text to be traced as input, and use the trained machine-generated text detection model based on feature fusion to perform classification prediction to obtain the traceability result; Step 6. The system returns the traceability result to the user. The user obtains the traceability result, which is the category label to which the input text belongs. If there is a traceability error, the user can upload the error traceability information through the front-end page.

2. According to claim 1, the method of converting input text into semantic vectors and statistical vectors by using word embedding technology and GPT-2 model is characterized in that: The word embedding technology is used to convert the text into a semantic vector form. In addition, the prediction probability of the text, the cumulative probability of the text, and the information entropy of the text are calculated through the GPT-2 model as the statistical vector of the text. The specific steps include: Step 2.1: The text in the input text tracing dataset is decomposed into a series of tokens through the BERT model. By mapping each word of the input text to a high-dimensional vector space, the model can capture the semantic information of each token and the relationship between words. These embedding vectors not only contain the information of the token itself, but also incorporate its position information in the text, namely position embedding. By adding position information to word embedding, the model can understand the position of each token in the sentence. In addition, the BERT model also uses paragraph embedding, which is particularly important for processing paired sentences. Paragraph embedding helps the model distinguish tokens in different sentences, so as to better understand and process the interaction between text pairs. By combining word embedding, position embedding and paragraph embedding, BERT can generate a comprehensive vector for each token, and input these comprehensive embedding representations into the encoding layer for further processing to obtain the final embedding representation, which not only enriches the semantic representation of the token, but also gives the model a deep understanding of the text structure. Step 2.2, calculate the statistical vector of the text through the GPT-2 model as the input of the subsequent steps; specifically, the present invention achieves this goal by calculating the predicted probability of the input text, the cumulative probability of the text and the information entropy of the text; the present invention uses a specific language model M (GPT-2 model) to process the input text, for an input text consisting of n words W = {w1, w2, ..., w i , ..., w n } and the size of the vocabulary of the language model M (the number of tokens) l, the text prediction probability is calculated as follows: The predicted probability of the i-th word in the input text of the language model M The calculation formula is as follows: In the formula, S M (w i |w1,w2,...,w i-1 ) means that all the previous words are (w1, w2, ..., w i-1 ), the current word is predicted to be w i Score; S M (positioni k |w1,w2,...,w i-1 ) means that all words before the given are (w1, w2, ..., w i-1 ), the scores of all tokens in the vocabulary are summed; Step 2.3, calculate the cumulative probability of the text, the calculation method is as follows: For the language model M, the cumulative probability C of the i-th word in the input text is u (w i ) is calculated as follows: The formula calculates all the probabilities greater than or equal to Pro at the i-th position of the input text M (w i ) token; after concatenating the cumulative probabilities of each word in the input text, we can get the cumulative probability C of the text M (W)=[C M (w1), C M (w2), ..., C M (w n )]; Step 2.4: Calculate the text information entropy as follows: Information entropy is a measure of the uncertainty of the probability distribution of all tokens in the vocabulary at a specific position; for the i-th position in the text, define E M (i) is the information entropy of the probability distribution of all tokens in the vocabulary at that position, and the specific calculation method is shown in the following formula: This formula calculates the probability of each token in the vocabulary appearing given the previous text and the 4 negative log-likelihood of the probability by summing them up, and obtains the information entropy of the i-th word by taking the product; then, the information entropy E of the entire input text can be calculated M (W)=[E M (1), E M (2), ..., E M (n)], taking it as part of the statistical features of the input text; Step 2.5: The text prediction probability, cumulative probability, and information entropy calculated by the GPT-2 model for the input text are integrated into a statistical vector.

3. The machine-generated text tracing model based on feature fusion according to claim 1 is characterized in that: The present invention uses a feature extraction layer to extract features, specifically including: For the feature extraction layer, its input is the semantic vector and text statistical vector of the text, which are processed by GRU and Transformer encoder respectively; GRU receives the semantic vector of the text as input, which is extracted from the text data and contains rich semantic information; the output of GRU is a tensor that integrates the context information of each time step in the sequence, and its dimensions include batch size, sequence length and feature dimension; through the update gate and reset gate, GRU can effectively capture the long-term dependencies in the sequence data and provide the model with a detailed time series feature representation. The specific calculation process is as follows: After the text of the BERT model, each element of the text sequence becomes an embedding vector; next, these vectors are passed as input to the GRU; the goal of the GRU is to capture the features and time dependencies in the sequence; At each time step t, the GRU updates its hidden state h t ; The update process depends on the current input e t , the hidden state h of the previous time step t-1 and update gate z t and reset gate r t ; Update gate z t Controls the previous hidden state h t-1 To the current hidden state h t The information flow of the reset gate r t Determines how much "past information" should be ignored; these two gates are calculated as follows: With t =σ(W z [e t ;h t-1 ]+b z ) r t =σ(W r [e t ;h t-1 ]+b r ) In the formula, W z and W r is the weight matrix of the update gate and reset gate, b z and b r is the corresponding bias term, σ is the sigmoid activation function, e t It represents the current input vector; Using the outputs of the update gate and the reset gate, GRU calculates the candidate hidden state h t ′, which is obtained by converting the current input e t With reset gate r t The previous hidden state h after calculation t-1 Combined with the nonlinear activation function tanh, the specific calculation method is shown in the formula: h t ′=tanh(W[r t ·h t-1 ]+Ue t +b) Among them, W and U are the weight matrices of GRU, b is the bias term, and tanh represents the activation function; finally, update gate z t Used to combine the hidden state h of the previous time step t-1 and the current candidate hidden state h t ′ to generate the final hidden state h of the current time step t , the specific calculation method is shown in the formula: h t =(1-z t )h t-1 +z t h t ′ This process is repeated at each time step of the sequence, and finally the hidden state sequence H = (h1, h2, ..., h n ). In this way, GRU can provide useful feature representation; The statistical vector calculated by the GPT-2 model of the input text is processed by the Transformer encoder, which encodes the input sequence and outputs a tensor containing global dependency information. The sequence length of the output tensor of the Transformer encoder is the same as the input, in which each element is encoded and contains position information and global dependency information. The Transformer encoder receives a vector as input and converts it into a series of position-related representations that can capture the position information of the input vector. The Transformer encoder consists of an input layer, a multi-head attention layer, a feedforward network, a residual connection, and a normalization layer. The input layer converts the input sequence into a format that the model can handle, while the multi-head attention layer calculates the attention weights of the input sequence. The feedforward network performs further linear and nonlinear transformations on the output of the multi-head attention layer, and the residual connection and normalization layer help stabilize the training process and avoid the gradient vanishing problem in deep networks. The specific calculation method is as follows: The encoder first position-encodes the input vector. After position encoding, the input sequence generates three different vector sets: query, key, and value through three different weight matrices. This step is the basis of the multi-head attention layer, which allows the model to calculate the impact of each element on other elements in subsequent steps; the calculation of the output vector of different parts in the multi-head attention layer is achieved through the query vector, key vector, value vector, the dimension of the key vector and the corresponding weight matrix. The specific calculation method is shown in the following formula: In the formula, Q, K, and V represent the query vector, key vector, and value vector, respectively. i Q , W i K and W i V is the corresponding weight matrix, d k is the dimension of the key vector; this formula first calculates QW i Q and KW i K The dot product of k The square root ratio of is used to obtain an attention score matrix, and then these scores are converted into probabilities through the Softmax function. Finally, a weighted output vector is obtained; this output vector is a transformation of the input vector, which contains the contribution of each element in the input vector to the current element; the final output of the multi-head attention layer is calculated by the following formula: MultiHead(Q,K,V)=Concat(Head1,Head2,…,Head h )W O Here, Concat means vector concatenation, W O It represents a weight matrix, which is used to linearly transform the features after concatenation, and MultiHead(Q, K, V) represents the output vector; After that, the output of the multi-head attention layer will pass through a feed-forward network (FFN) after residual connection and normalization. This is a neural network containing two linear transformations and a nonlinear activation function; the calculation formula of the feed-forward network is as follows: FFN(x)=max(0,xW1+b1)W2+b2 In the formula, W1 and W2 are the weight matrices of the feedforward network, x represents the input of the previous layer, and b1 and b2 are bias terms. This network further performs linear and nonlinear transformations on the output of the previous layer, increasing the expressive power of the model; finally, this output is added to the input vector, and residual connections and normalization are performed to obtain the final output of the encoder; The outputs of the GRU and Transformer encoders are respectively transformed linearly through a linear layer to obtain the semantic features and statistical features of the text, which are used as the input of the feature fusion layer.

4. The machine-generated text tracing model based on feature fusion according to claim 1 is characterized in that: The present invention uses a fusion layer to perform feature fusion on the extracted features, specifically including: The function of the feature fusion layer is to integrate the information of different features so that the classification model can understand the text data from different perspectives. In this way, the model can not only capture the semantic content of the text, but also take into account the statistical information of the text sequence, providing richer feature representation for the subsequent classification task. Specifically, after the feature extraction layer, two new feature vectors S and P are obtained. The purpose of the feature fusion layer is to merge these two sets of features into a single feature vector so that the subsequent classification layer can use this comprehensive information to make decisions; Next, feature fusion is performed through the concatenation operation, that is, S and P are connected in element order to form a longer feature vector F: F = [S; P] In the formula, the semicolon ";" represents the concatenation operation of the vectors; thus, the fused feature vector F contains information about text semantic features and text statistical features, and its dimension is the sum of the S and P dimensions.

5. The machine-generated text tracing model based on feature fusion according to claim 1 is characterized in that: The present invention classifies the fused feature vectors in the classification layer, specifically including: In the classification layer, the input feature vector is first transformed linearly, and then a Dropout layer is used to randomly "discard" the output of some neurons to reduce the network's dependence on specific neurons; next, the output of each neuron is transformed nonlinearly through the ReLU activation function, so that the network can learn nonlinear relationships; then a fully connected layer is used to perform a linear transformation on the input features by applying a weight matrix and a bias vector to generate a score vector whose length is the same as the number of target categories; each element in the vector represents the model's confidence in predicting that the input text belongs to a specific category; then the Softmax function is used to calculate the probability that the input feature belongs to each category.

Images