Method and system for tracing machine-generated text based on feature fusion

Abstract

In view of the problem that a single feature is difficult to accurately trace the source of machine-generated text in the existing machine-generated text tracing technology, the present application proposes a machine-generated text tracing method and system based on feature fusion, constructs a new machine-generated text tracing model, and provides the model with richer feature representation by combining the statistical features and semantic features of the text. The present application first converts the input text into a semantic vector by using word embedding technology to capture the semantic information of the text, and uses the GPT-2 model to calculate the prediction probability, cumulative probability and information entropy of the text as the statistical vector of the text to capture the statistical information of the text. Then, the features are further extracted by GRU and Transformer encoder, GRU effectively captures the long-term dependence in the sequence data through its update gate and reset gate mechanism, and the Transformer encoder captures the global dependence information by using the multi-head attention mechanism. The output of the GRU and the Transformer encoder is then linearly transformed by a linear layer to obtain the semantic features and statistical features of the text, and then the text semantic features and statistical features are integrated by a feature fusion strategy. Finally, the fused feature vector is analyzed by a classifier to determine whether the text is written by a human or generated by a specific language model.

Description

Technical Field

[0001] This invention relates to a method and system for tracing the source of machine-generated text based on feature fusion, belonging to the field of machine-generated text detection and source tracing technology. Background Technology

[0002] Currently, machine-generated text detection technologies primarily focus on determining whether text was generated by a machine, while research on identifying the detailed source of the text—specifically, the language model that generated it—is relatively limited. The task of identifying whether a piece of text is machine-generated or human-written, and if machine-generated, further determining which specific language model generated it, is called machine-generated text source tracing, a fine-grained approach to machine-generated text detection. With the emergence of numerous Large Language Models (LLMs), accurately tracing the source of text has become increasingly prominent, crucial for ensuring the authenticity and transparency of information and tracing the source of responsibility. However, existing machine-generated text source tracing models rely on single statistical features to determine the source of text. As LLM text generation capabilities continue to improve, the quality of generated text becomes increasingly higher, making it more difficult to accurately trace the source of text using a single feature. Therefore, researching more effective machine-generated text source tracing techniques is particularly critical. Summary of the Invention

[0003] To address the challenges of source tracing in machine-generated text, this invention proposes a Feature Fusion Model (FFM) for source tracing. Compared to existing models, FFM calculates both statistical and semantic features of the text, combining these two different features to identify the text's source. Statistical and semantic features provide information from different dimensions; by combining these two features, the model can extract more comprehensive feature information from the text. A single statistical feature cannot capture the semantic and contextual information of the text. Some texts have significant overlap in statistical features, but their semantic features differ significantly. The inclusion of semantic features can improve the model's accuracy in identifying the text's source. Specifically, FFM first converts the input sentence into a semantic vector representation using word embedding technology to capture the text's semantic information. Simultaneously, the model also calculates the predicted probability, cumulative probability, and information entropy of the input text; these three together constitute the text's statistical vector. Based on this, semantic and statistical features of the text are obtained through feature extraction. Then, FFM effectively integrates lexical and statistical features through a feature fusion strategy to form a comprehensive feature vector. Finally, the fused feature vectors are fed into a classifier, which is responsible for determining whether the input text is human-written or comes from an LLM (Local Language Model) based on the content of the feature vectors. The model's flexibility and scalability allow it to adapt to emerging new language models, providing a new perspective and method for the development of machine-generated text source tracing technology.

[0004] This invention discloses a machine-generated text tracing method and system based on feature fusion, comprising the following steps:

[0005] Step 1: Obtain the text source tracing dataset, which includes training data and corresponding category labels;

[0006] Step 2: Convert the text source dataset from Step 1 into semantic vectors using word embedding technology. The GPT-2 model is used to convert the text source dataset from Step 1 into statistical vectors.

[0007] Step 3: Use the text semantic vectors and statistical vectors from Step 2 to train the machine-generated text source tracing model based on feature fusion. The machine-generated text source tracing detection model based on feature fusion includes a feature extraction layer, a feature fusion layer, and a classification layer. After training, the trained machine-generated text source tracing model based on feature fusion is obtained.

[0008] Step 4: Deploy the trained feature fusion-based machine-generated text tracing model into the system;

[0009] Step 5: Take the received text to be traced as input, and use the trained machine-generated text detection model based on feature fusion to perform classification and prediction to obtain the traceability result.

[0010] Step 6: The system returns the tracing results to the user. The user obtains the tracing results, which is the category label of the input text. If there are tracing errors, the user can upload the error tracing information through the front-end page.

[0011] A further improvement of this invention lies in the following: This invention employs word embedding technology to convert text into semantic vector form. Furthermore, it uses the GPT-2 model to calculate the predicted probability, cumulative probability, and information entropy of the text as statistical vectors. Specific steps include:

[0012] Step 2.1: The text in the input text source dataset is decomposed into a series of tokens using the BERT model. By mapping each word in the input text to a high-dimensional vector space, the model can capture the semantic information of each token and the relationships between words. These embedding vectors not only contain information about the token itself but also incorporate its positional information within the text, i.e., positional embedding. By adding positional information to word embeddings, the model helps understand the position of each token in the sentence. Furthermore, the BERT model also uses paragraph embeddings, which are particularly important for processing paired sentences. Paragraph embeddings help the model distinguish tokens in different sentences, thereby better understanding and processing the interactions between text pairs. By combining word embeddings, positional embeddings, and paragraph embeddings, BERT can generate a comprehensive vector for each token. These comprehensive embedding representations are then input into the encoding layer for further processing to obtain the final embedding representation. This not only enriches the semantic representation of the tokens but also gives the model a deeper understanding of the text structure.

[0013] Step 2.2: Calculate the statistical vector of the text using the GPT-2 model as input for subsequent steps. Specifically, this invention achieves this goal by calculating the predicted probability of the input text, the cumulative probability of the text, and the information entropy of the text. The invention uses a specific language model M (GPT-2 model) to process the input text. For an input text W = {w1, w2, ..., w...} consisting of n words... i ,…,w n The text prediction probability is calculated as follows, along with the size of the language model M's vocabulary (number of tokens).

[0014] The predicted probability of the i-th word in the input text for language model M. The calculation formula is as follows:

[0015]

[0016] In the formula, S M (w i |w1,w2,…,w i-1 ) indicates that all preceding words are (w1, w2, ..., w i-1 In the case of ), predict the current word as w i S score. M (positioni k |w1,w2,…,w i-1 ) indicates that all preceding words are (w1, w2, ..., w i-1 In the case of ), sum the scores of all tokens in the vocabulary;

[0017] Step 2.3: Calculate the cumulative probability of the text, as follows:

[0018] For the cumulative probability C of the i-th word in the input text of the language model M... M (w i The calculation formula is as follows:

[0019]

[0020] The formula calculates all probabilities greater than or equal to Pro at the i-th position of the input text. M (w i The cumulative probability C of the text is the sum of the probabilities of the tokens. By concatenating the cumulative probabilities of each word in the input text, we can obtain the cumulative probability C of the text. M (W)=[C M (w1),C M (w2),…,C M (w n )).

[0021] Step 2.4: Calculate the text information entropy, as follows:

[0022] Information entropy is a metric that measures the uncertainty of the probability distribution of all tokens in a vocabulary at a specific location. For the i-th position in the text, we define E... M (i) is the information entropy of the probability distribution of all tokens in the vocabulary at this position, calculated as shown in the following formula:

[0023]

[0024] This formula calculates the probability of each token in the vocabulary given the preceding text, along with the negative log-likelihood of that probability, by summing the results. The information entropy of the i-th word is then obtained by product. Finally, the information entropy E of the entire input text can be calculated. M (W) = [E M(1),E M (2),…,E M (n)], which is used as part of the statistical features of the input text;

[0025] Step 2.5: The input text is processed by the GPT-2 model to obtain three statistical features: text prediction probability, cumulative probability, and information entropy. These three statistical features are then integrated into a single feature vector.

[0026] A further improvement of this invention lies in that: this invention employs a feature fusion strategy to integrate the semantic and statistical features of text, and its specific steps include:

[0027] Step 3.1: For the feature extraction layer, its input consists of semantic vectors and statistical vectors of the text. These inputs are processed by the GRU and Transformer encoder, respectively. The GRU receives semantic vectors of the text as input, which are extracted from the text data and contain rich semantic information. The output of the GRU is a tensor that incorporates the contextual information of each time step in the sequence. Its dimensions include batch size, sequence length, and feature dimension. Through update and reset gates, the GRU can effectively capture long-term dependencies in the sequence data and provide the model with a detailed time-series feature representation. The specific calculation process is as follows:

[0028] After passing through the BERT model, each element of the text sequence becomes an embedding vector. These vectors are then fed as input to the GRU. The goal of the GRU is to capture the features and temporal dependencies in the sequence.

[0029] At each time step t, the GRU updates its hidden state h. t The update process depends on the current input e. t The hidden state h of the previous time step t-1 And update gate z t and reset door r t .

[0030] Update Gate Z t Controlling the previous hidden state h t-1 To the current hidden state h t Information flow, and reset gate r t This determines how much "past information" should be ignored. The calculations for these two gates are as follows:

[0031] z t =σ(W z [e t h t-1 ]+b z )

[0032] rt =σ(W r [e t h t-1 ]+b r )

[0033] In the formula, W z and W r It is the weight matrix for the update gate and the reset gate, b z and b r These are the corresponding bias terms, σ is the sigmoid activation function, and e t This represents the current input vector.

[0034] Using the outputs of the update gate and the reset gate, GRU computes the candidate hidden state h. t ′, it is by passing the current input e t With Reset Gate r t The calculated previous hidden state h t-1 The formula is obtained by combining and adding the nonlinear activation function tanh. The specific calculation method is shown in the formula:

[0035] h t =tanh(W[r t ·h t-1 ]+Ue t +b)

[0036] Where W and U are the weight matrices of the GRU, b is the bias term, and tanh represents the activation function. Finally, the update gate z... t Used to combine the hidden state h from the previous time step t-1 and the current candidate hidden state h t ' to generate the final hidden state h at the current time step. t The specific calculation method is shown in the formula:

[0037] h t =(1-z) t )h t-1 +z t h t ′

[0038] This process is repeated at each time step of the sequence, and the final hidden state sequence output by the GRU is H = (h1, h2, ..., h...). n In this way, GRU is able to provide useful feature representations.

[0039] The statistical vector obtained from the input text by the GPT-2 model is processed by the Transformer encoder. The Transformer encoder encodes the input sequence and outputs a tensor containing global dependency information. The sequence length of the output tensor of the Transformer encoder is the same as that of the input, and each element is encoded, containing positional information and global dependency information.

[0040] The Transformer encoder takes a vector as input and transforms it into a series of position-dependent representations that capture the positional information of the input vector. The Transformer encoder consists of an input layer, a multi-head attention layer, a feedforward network, residual connections, and a normalization layer. The input layer converts the input sequence into a format that the model can process, while the multi-head attention layer computes the attention weights of the input sequence. The feedforward network performs further linear and non-linear transformations on the output of the multi-head attention layer, and the residual connections and normalization layers help stabilize the training process and avoid the vanishing gradient problem in deep networks. The specific calculation method is shown below:

[0041] The encoder first performs positional encoding on the input vector. The positionally encoded input sequence then generates three distinct vector sets: query, key, and value, using three different weight matrices. This step is fundamental to the multi-head attention layer, allowing the model to calculate the influence of each element on other elements in subsequent steps. The calculation of the output vectors for different parts in the multi-head attention layer is achieved using the query vector, key vector, value vector, the dimension of the key vector, and the corresponding weight matrix. The specific calculation method is shown in the following formula:

[0042]

[0043] In the formula, Q, K, and V represent the query vector, key vector, and value vector, respectively, and W... i Q , and It is the corresponding weight matrix, d k This is the dimension of the key vector. This formula first calculates QW. i Q and dot product with d k The ratio of the square roots of the input vectors yields an attention score matrix. These scores are then converted into probabilities using the Softmax function, resulting in a weighted output vector. This output vector is a transformation of the input vector, containing the contribution of each element in the input vector to the current element. The final output of the multi-head attention layer is calculated using the following formula:

[0044] MultiHead(Q,K,V)=Concat(Head1,Head2,…,Headh)W O

[0045] Here, Concat represents vector concatenation, W O This represents a weight matrix used to perform a linear transformation on the features after concatenation. MultiHead(Q,K,V) represents the output vector.

[0046] After passing through residual connections and normalization, the output of the multi-head attention layer then passes through a feed-forward network (FFN), a neural network containing two linear transformations and a non-linear activation function. The calculation formula for the feed-forward network is as follows:

[0047] FFN(x) = max(0, xW1+b1)W2+b2

[0048] In the formula, W1 and W2 are the weight matrices of the feedforward network, x represents the input of the previous layer, and b1 and b2 are bias terms. This network performs further linear and nonlinear transformations on the output of the previous layer, increasing the expressive power of the model. Finally, this output is added to the input vector, and residual connections and normalization are performed to obtain the final output of the encoder.

[0049] Step 3.2: The outputs of the GRU and Transformer encoders are each linearly transformed through a linear layer to obtain the statistical and semantic features of the text, which are then used as the input to the feature fusion layer.

[0050] Step 3.3: The function of the feature fusion layer is to integrate information from different features, enabling the classification model to understand text data from different perspectives. In this way, the model can not only capture the semantic content of the text, but also take into account the statistical information of the text sequence, providing richer feature representations for subsequent classification tasks.

[0051] Specifically, after the feature extraction layer, two new feature vectors, S and P, are obtained. The purpose of the feature fusion layer is to merge these two sets of features into a single feature vector so that the subsequent classification layer can use this combined information to make decisions.

[0052] Next, feature fusion is performed through concatenation, that is, S and P are concatenated element-wise to form a longer feature vector F:

[0053] F = [S; P]

[0054] In the formula, the semicolon ";" indicates a vector concatenation operation. Thus, the fused feature vector F contains information about both textual semantic features and textual statistical features, and its dimension is the sum of the dimensions of S and P.

[0055] Step 3.4: In the classification layer, the input feature vector is first linearly transformed. Then, a Dropout layer randomly "drops" some of the neuron outputs to reduce the network's dependence on specific neurons. Next, the ReLU activation function is used to perform a non-linear transformation on the output of each neuron, enabling the network to learn non-linear relationships. Then, a fully connected layer applies a weight matrix and bias vector to perform a linear transformation on the input features, generating a score vector with a length equal to the number of target categories. Each element in the vector represents the model's confidence that the input text belongs to a specific category. Finally, the Softmax function is used to calculate the probability that the input features belong to each category. Attached Figure Description

[0056] Figure 1 System architecture diagram of a machine-generated text tracing method and system based on feature fusion

[0057] Figure 2 System flowchart of a machine-generated text tracing method and system based on feature fusion.

[0058] Figure 3 Model structure diagram of a machine-generated text tracing method based on feature fusion Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] It should be noted that, in order to avoid unnecessary details from obscuring the important parts of the present invention, the accompanying drawings only show structures or processing steps closely related to the present invention, while omitting other details that are not closely related to the present invention.

[0061] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

[0063] This invention discloses a machine-generated text tracing method and system based on feature fusion, the system architecture of which is as follows: Figure 1 As shown, the system architecture adopts a layered design, mainly including a data layer, a business logic layer, and a presentation layer. The data layer is responsible for storing user information, log information, error feedback information, and model parameters. It uses the relational database management system MySQL to store structured data. The business logic layer is the core of the system, responsible for handling user operations, user management, and input text detection and tracing tasks. In this layer, the Django backend framework is used to manage interaction with the database, perform user authentication, process data, and communicate effectively with the frontend through APIs. Django's efficient routing mechanism and rich libraries enable the system to handle complex business rules and data flows, while ensuring system security and data integrity. The business logic layer uses the deep learning framework PyTorch to build algorithm models. The presentation layer is responsible for interacting with the user, providing a user-friendly interface. It uses the Vue framework to build the user interface to achieve dynamic data interaction and a good user experience, ensuring system compatibility and ease of use on different devices. Figure 2 The process of a machine-generated text tracing method and system based on feature fusion is demonstrated, including the following steps:

[0064] Step 1: Obtain a text source dataset including training data and corresponding category labels. Human text comes from multiple public datasets, such as XSum, WritingP, SQuAD, and Yelp datasets. In addition, machine text is generated from parallel human text to study the problem of tracing the origin of text. In this study, machine text was generated by multiple LLMs, including GPT-2, GPT-3, GPT-J, GPT-Neo, ChatGPT, PaLM, and LLaMA models. This invention uses three types of prompts to guide LLM text generation: (1) Continuation prompts: requiring LLM to continue generating text based on the first 20 words of the original human text. (2) Topic prompts: using prompts such as arguments, news headlines, and story themes to guide LLM text generation. (3) Specific prompts: providing LLM with information containing specific text style requirements to guide the model to produce text that conforms to preset features. Human-written text is categorized and labeled as "human", and machine-generated text is labeled with its specific source, i.e., which LLM generated the machine-generated text.

[0065] Step 2: Convert the text source dataset from Step 1 into word vectors using word embedding technology. The GPT-2 model is used to convert the text source dataset from Step 1 into statistical vectors. The specific steps are as follows:

[0066] Step 2.1: The text in the input text source dataset is decomposed into a series of tokens using the BERT model. By mapping each word in the input text to a high-dimensional vector space, the model can capture the semantic information of each token and the relationships between words. These embedding vectors not only contain information about the token itself but also incorporate its positional information within the text, i.e., positional embedding. By adding positional information to word embeddings, the model helps understand the position of each token in the sentence. Furthermore, the BERT model also uses paragraph embeddings, which are particularly important for processing paired sentences. Paragraph embeddings help the model distinguish tokens in different sentences, thereby better understanding and processing the interactions between text pairs. By combining word embeddings, positional embeddings, and paragraph embeddings, BERT can generate a comprehensive vector for each token. These comprehensive embedding representations are then input into the encoding layer for further processing to obtain the final embedding representation. This not only enriches the semantic representation of the tokens but also gives the model a deeper understanding of the text structure.

[0067] Step 2.2: Use the GPT-2 model to calculate the predicted probability of each word in the text, thereby forming the predicted probability of the entire text.

[0068] Step 2.3: Use the GPT-2 model to calculate the cumulative probability of each word in the text, thereby forming the cumulative probability of the entire text.

[0069] Step 2.4: Use the GPT-2 model to calculate the information entropy of each word in the text, thereby constructing the information entropy of the entire text.

[0070] Step 2.5: The text prediction probability, cumulative probability, and information entropy obtained by the input text through the GPT-2 model are integrated into a statistical vector;

[0071] Step 3, as follows Figure 2 As shown, the text word vectors and statistical vectors obtained in step 2 are used to train a machine-generated text source tracing model based on feature fusion. The machine-generated text source tracing detection model based on feature fusion includes a feature extraction layer, a feature fusion layer, and a classification layer. After training, the trained machine-generated text source tracing model based on feature fusion is obtained. The specific steps are as follows:

[0072] Step 3.1: For the feature extraction layer, its input consists of the semantic vector and text statistical vector of the text. These inputs are processed by the GRU and Transformer encoder, respectively. The GRU receives the semantic features of the text as input, which are extracted from the text data and contain rich semantic information. The output of the GRU is a tensor that incorporates the contextual information of each time step in the sequence. Its dimensions include batch size, sequence length, and feature dimension. Through update and reset gates, the GRU can effectively capture long-term dependencies in the sequence data and provide the model with a detailed time-series feature representation.

[0073] Step 3.2: The outputs of the GRU and Transformer encoders are respectively linearly transformed through a linear layer to obtain the semantic and statistical features of the text, which are then used as the input to the feature fusion layer.

[0074] Step 3.3: Feature Fusion Layer Integration. Through feature fusion strategies, the semantic and statistical features of the text are integrated, enabling the classification model to understand the text data from different perspectives. In this way, the model can not only capture the semantic content of the text but also consider the statistical information of the text sequence, providing richer feature representations for subsequent classification tasks.

[0075] Step 3.4: In the classification layer, the input feature vector is first linearly transformed. Then, a Dropout layer randomly "drops" some of the neuron outputs to reduce the network's dependence on specific neurons. Next, the ReLU activation function is used to perform a non-linear transformation on the output of each neuron, enabling the network to learn non-linear relationships. Then, a fully connected layer applies a weight matrix and bias vector to perform a linear transformation on the input features, generating a score vector with a length equal to the number of target categories. Each element in the vector represents the model's confidence that the input text belongs to a specific category. Finally, the Softmax function is used to calculate the probability that the input features belong to each category.

[0076] Step 4: Deploy the trained feature fusion-based machine-generated text source tracing model into the system. The specific steps are as follows:

[0077] Step 4.1: Export the trained FFM model as an ONNX format file to ensure it can run in a production environment;

[0078] Step 4.2: Configure a server environment that supports HTTP requests;

[0079] Step 4.3: Upload the exported model file to the production server and integrate it into the planned service. Next, upload the model file to the server using protocols such as FTP and SCP. Then, configure API interface routing and initialize the database connection. Finally, start the model service and perform interface tests to verify that the model's functions of receiving input, performing predictions, and returning results are working correctly.

[0080] Step 5: Take the received text to be traced as input, and use the trained machine-generated text detection model based on feature fusion to perform classification and prediction to obtain the traceability result.

[0081] Step 6: The system returns the tracing results to the user. The user obtains the tracing results, which is the category label of the input text. If there are tracing errors, the user can upload the error tracing information through the front-end page.

Claims

1. A machine-generated text source tracing method based on feature fusion, characterized in that, Includes the following steps: Step 1: Obtain the text source tracing dataset, which includes training data and corresponding category labels; Step 2: Convert the text source dataset from Step 1 into semantic vectors using word embedding technology; convert the text source dataset from Step 1 into statistical vectors using the GPT-2 model. Step 3: Use the text semantic vectors and statistical vectors from Step 2 to train the machine-generated text source tracing model based on feature fusion. The machine-generated text source tracing detection model based on feature fusion includes a feature extraction layer, a feature fusion layer, and a classification layer. After training, the trained machine-generated text source tracing model based on feature fusion is obtained. Step 4: Deploy the trained feature fusion-based machine-generated text tracing model into the system; Step 5: Take the received text to be traced as input, and use the trained machine-generated text detection model based on feature fusion to perform classification and prediction to obtain the traceability result. Step 6: The system returns the tracing results to the user. The user obtains the tracing results, which is the category label of the input text. If there are tracing errors, the user can upload the error tracing information through the front-end page. The text is converted into a semantic vector using word embedding technology. Furthermore, the predicted probability, cumulative probability, and information entropy of the text are calculated using the GPT-2 model, which serve as the text's statistical vector. Specific steps include: Step 2.1: The text in the input text source dataset is decomposed into a series of tokens using the BERT model. By mapping each word of the input text to a high-dimensional vector space, the model can capture the semantic information of each token and the relationships between words. These embedding vectors not only contain the information of the token itself, but also incorporate its positional information in the text, i.e., positional embedding. By adding positional information to word embeddings, the model can understand the position of each token in the sentence. In addition, the BERT model also uses paragraph embeddings, which help the model distinguish tokens in different sentences. By combining word embeddings, positional embeddings, and paragraph embeddings, BERT can generate a comprehensive vector for each token. These comprehensive embedding representations are then input into the encoding layer for further processing to obtain the final embedding representation. Step 2.2: Calculate the statistical vector of the text using the GPT-2 model as input. This is achieved by calculating the predicted probability, cumulative probability, and information entropy of the input text. A specific language model M is used to process the input text. For an input text W = {w1, w2, ..., w...} consisting of n words... i ,...,w n The text prediction probability is calculated as follows, along with the size l of the language model M's vocabulary: The predicted probability of the i-th word in the input text for language model M. The calculation formula is as follows: In the formula, S M (w i |w1, w2, ..., w i-1 ) indicates that all preceding words are (w1, w2, ..., w i-1 In the case of ), predict the current word as w i S score M (positioni k |w1,w2,...,w i-1 ) indicates that all preceding words are (w1, w2, ..., w i-1 In the case of ), sum the scores of all tokens in the vocabulary; Step 2.3: Calculate the cumulative probability of the text, as follows: For the cumulative probability C of the i-th word in the input text of the language model M... M (w i The calculation formula is as follows: The formula calculates all probabilities greater than or equal to Pro at the i-th position of the input text. M (w i The cumulative probability C of the text is obtained by summing the probabilities of the tokens and concatenating the cumulative probabilities of each word in the input text. M (W)=[C M (w1), C M (w2), ..., C M (w n )]; Step 2.4: Calculate the text information entropy, as follows: Information entropy is a metric that measures the uncertainty of the probability distribution of all tokens in a vocabulary at a specific location. For the i-th position in the text, E is defined as... M (i) is the information entropy of the probability distribution of all tokens in the vocabulary at this position, calculated as shown in the following formula: This formula calculates the probability of each token in the vocabulary given the preceding text, along with the negative log-likelihood of that probability, by summing the products. The information entropy of the i-th word is then obtained through product, and the information entropy E of the entire input text can then be calculated. M (W) = [E M (1), E M (2), ..., E M (n)], which is used as part of the statistical features of the input text; Step 2.5: The text prediction probability, cumulative probability, and information entropy obtained by the input text through the GPT-2 model are integrated into a statistical vector; Step 3.1: For the feature extraction layer, its input consists of semantic features and statistical features of the text. These input data are processed by the GRU and Transformer encoders, respectively. The GRU receives the semantic features of the text as input. These features are extracted from the text data and contain rich semantic information. The output of the GRU is a tensor that incorporates the contextual information of each time step in the sequence. Its dimensions include batch size, sequence length, and feature dimension. Through update and reset gates, the GRU can effectively capture long-term dependencies in the sequence data and provide the model with a detailed time-series feature representation. The specific calculation process is as follows: After passing through the BERT model, each element of the text sequence becomes an embedding vector. Next, these vectors are passed as input to GRU, whose goal is to capture the features and temporal dependencies in the sequence. At each time step t, the GRU updates its hidden state h. t The update process depends on the current input e t The hidden state h of the previous time step t-1 And update gate z t and reset door r t ; Update Gate Z t Controlling the previous hidden state h t-1 To the current hidden state h t Information flow, and reset gate r t The calculation method for these two gates determines how much "past information" should be ignored: With t =σ(W z [e t ;h t-1 ]+b z ) r t =σ(W r [e t ;h t-1 ]+b r ) In the formula, W z and W r It is the weight matrix for the update gate and the reset gate, b z and b r These are the corresponding bias terms, σ is the sigmoid activation function, and e t This represents the current input vector; Using the outputs of the update gate and the reset gate, GRU computes the candidate hidden state h. t ′, it is by passing the current input e t With Reset Gate r t The calculated previous hidden state h t-1 The formula is obtained by combining and adding the nonlinear activation function tanh. The specific calculation method is shown in the formula: h t ′=tanh(W[r t h t-1 ]+Ue t +b) Where W and U are the weight matrices of the GRU, b is the bias term, tanh represents the activation function, and finally, the update gate z... t Used to combine the hidden state h from the previous time step t-1 and the current candidate hidden state h t ' to generate the final hidden state h at the current time step. t The specific calculation method is shown in the formula: h t =(1-z t )h t-1 +z t h t ′ This process is repeated at each time step of the sequence, and the final hidden state sequence output by the GRU is H = (h1, h2, ..., h...). n In this way, GRU can provide useful feature representations; The statistical features obtained from the input text through statistical feature calculation are processed by the Transformer encoder. The Transformer encoder encodes the input sequence and outputs a tensor containing global dependency information. The sequence length of the output tensor of the Transformer encoder is the same as that of the input, and each element is encoded, containing positional information and global dependency information. The Transformer encoder takes a vector as input and transforms it into a series of position-dependent representations that capture the positional information of the input vector. The Transformer encoder consists of an input layer, a multi-head attention layer, a feedforward network, residual connections, and a normalization layer. The input layer converts the input sequence into a format that the model can process, while the multi-head attention layer calculates the attention weights of the input sequence. The feedforward network performs further linear and non-linear transformations on the output of the multi-head attention layer. The residual connections and normalization layers help stabilize the training process and avoid the vanishing gradient problem in deep networks. Their specific calculation methods are shown below: The encoder first performs positional encoding on the input vector. The positionally encoded input sequence then generates three distinct vector sets—Query, Key, and Value—using three different weight matrices. This step is fundamental to the multi-head attention layer, allowing the model to calculate the influence of each element on other elements in subsequent steps. The calculation of the output vectors for different parts in the multi-head attention layer is achieved using the dimensions of the query vector, key vector, and value vector, along with the corresponding weight matrices. The specific calculation method is shown in the following formula: In the formula, Q, K, and V represent the query vector, key vector, and value vector, respectively, and W... i Q , and It is the corresponding weight matrix, d k It is the dimension of the key vector. This formula first calculates QW. i Q and dot product with d k The ratio of the square roots of the input vectors is used to obtain an attention score matrix. These scores are then converted into probabilities using the Softmax function. Finally, a weighted output vector is obtained, which is a transformation of the input vector, containing the contribution of each element in the input vector to the current element. The final output of the multi-head attention layer is calculated using the following formula: MultiHead(Q,K,V)=Concat(Head1,Head2,...,Head h )W O Here, Concat represents vector concatenation, W O It represents a weight matrix used to perform a linear transformation on the features after concatenation, and MultiHead(Q, K, V) represents the output vector; After the output of the multi-head attention layer undergoes residual connection and normalization, it passes through a feed-forward network (FFN). This is a neural network containing two linear transformations and a non-linear activation function. The calculation formula for the feed-forward network is as follows: FFN(x) = max(0, xW1+b1)W2+b2 In the formula, W1 and W2 are the weight matrices of the feedforward network, x represents the input of the previous layer, and b1 and b2 are bias terms. This network performs further linear and nonlinear transformations on the output of the previous layer, increasing the expressive power of the model. Finally, this output is added to the input vector, and residual connections and normalization are performed to obtain the final output of the encoder. Step 3.2: The outputs of the GRU and Transformer encoders are respectively linearly transformed through a linear layer to obtain the semantic and statistical features of the text, which are then used as the input to the feature fusion layer. Step 3.3: The function of the feature fusion layer is to integrate information from different features, enabling the classification model to understand text data from different perspectives. Specifically, after the feature extraction layer, two new feature vectors S and P are obtained. The purpose of the feature fusion layer is to merge these two sets of features into a single feature vector so that the subsequent classification layer can use this comprehensive information to make decisions. Next, feature fusion is performed through concatenation, that is, S and P are concatenated element-wise to form a longer feature vector F: F = [S; P] In the formula, the semicolon ";" indicates the concatenation operation of vectors. The fused feature vector F contains information on text semantic features and text statistical features, and its dimension is the sum of the dimensions of S and P. Step 3.4: In the classification layer, the input feature vector is first linearly transformed. Then, a Dropout layer randomly "drops" some of the neuron outputs to reduce the network's dependence on specific neurons. Next, the ReLU activation function is used to perform a non-linear transformation on the output of each neuron, enabling the network to learn non-linear relationships. Then, a fully connected layer is used to perform a linear transformation on the input features by applying the weight matrix and bias vector to generate a score vector with the same length as the number of target categories. Each element in the vector represents the confidence level of the model's prediction that the input text belongs to a specific category. Finally, the Softmax function is used to calculate the probability that the input features belong to each category.

Images