Training Method for Machine-Generated Text Detection Model Based on Prefix Enhancement, Machine-Generated Text Detection Method and Device

Abstract

The present invention discloses a training method for a machine-generated text detection model based on prefix enhancement, a machine-generated text detection method and device. The training method for the machine-generated text detection model includes: inputting the original text into a prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation; inputting the original text into a word embedding module to obtain an original text feature vector; inputting the prefix enhancement prompt feature vector and the original text feature vector into a feature weighted average module to obtain a first fused feature vector; finally, through an output module, obtaining a prediction result for the original text; and training based on the degree of difference between the first fused feature vector, the prediction result, the current prediction result of the model and the true label. The present invention introduces the prefix enhancement technology, and through a specific prompt word generation algorithm, enhances the model's ability to understand the input text, thereby improving the accuracy of machine-generated text detection.

Description

Technical Field

[0001] The present invention relates to the field of machine-generated text detection, and particularly to a training method for a machine-generated text detection model based on prefix enhancement, a machine-generated text detection method, and an apparatus. Background Art

[0002] Today, large language models (LLMs) rely on their large parameter scales and extensive training data to excel in generating complex and coherent narratives. However, they sometimes introduce incorrect information or fictional details, making the generated text suffer from accuracy and originality issues. Therefore, how to develop effective detection techniques to evaluate the reliability of machine-generated text has become increasingly urgent.

[0003] In the existing field of machine-generated text detection, the zero-shot method performs most prominently in identifying machine-generated text. This method relies on the understanding of text generation patterns and language rules by general pre-trained language models, such as BERT and GPT, to identify machine-generated text without specific task training by using predefined labels or prompts. Since the similarity between the text estimated by the detection model and the machine-generated text in terms of language structure and semantics is high enough, the zero-shot detection method can work effectively.

[0004] However, there are some non-negligible defects in the existing zero-shot methods:

[0005] The effectiveness of the zero-shot method basically depends on the assumption that the language structure and semantic knowledge learned by the model during general pre-training are extensive and in-depth enough. In many previous studies, especially when dealing with early generation models, this assumption holds, as the text generation process of these models usually relies on a small number of initial tokens. However, the zero-shot method faces generalization difficulties when dealing with scenarios involving more advanced language models. These complex models usually go through additional training phases to make their outputs conform to human preferences and be consistent with task requirements, and can generate text with unique style elements, which deviate significantly from typical text generation patterns. This makes it difficult for the detection model to accurately distinguish machine-generated text from human-written text, thus reducing the effectiveness of the current zero-shot detection technology. Summary of the Invention

[0006] Aiming at the above defects or improvement requirements of the prior art, the present invention provides a training method for a machine-generated text detection model based on prefix enhancement, a machine-generated text detection method, and an apparatus, aiming to solve the problem of poor accuracy in detecting machine-generated text due to the lack of in-depth understanding of the text generation process in the prior art.

[0007] To achieve the above object, according to one aspect of the present invention, there is provided: a method for training a machine-generated text detection model, wherein the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module, and the method includes:

[0008] Input the original text into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation;

[0009] Input the original text into the word embedding module to obtain an original text feature vector;

[0010] Input the prefix enhancement prompt feature vector and the original text feature vector into the feature weighted average module to obtain a first fused feature vector;

[0011] Input the first fused feature vector into the output module to obtain a prediction result for the original text; and

[0012] Train the machine-generated text detection model based on the difference degree between the first fused feature vector, the prediction result for the original text, the current prediction result of the model and the true label.

[0013] In one implementation, the prefix enhancement mapping module includes a prefix enhancement tokenizer, a prefix enhancement mapping model, and a prefix enhancement prompt word adapter. Inputting the original text into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation includes:

[0014] Input the original text into the prefix enhancement mapping module, and perform word segmentation processing on the original text through the prefix enhancement tokenizer to obtain a token ID sequence of the prefix enhancement text;

[0015] Process the token ID sequence of the prefix enhancement text through the prefix enhancement mapping model to obtain a prefix enhancement prompt feature vector;

[0016] Process the prefix enhancement prompt feature vector through the prefix enhancement prompt word adapter to obtain a prefix enhancement prompt feature vector after linear adaptation.

[0017] In one implementation, the word embedding module includes a tokenizer and a word embedding matrix. Inputting the original text into the word embedding module to obtain an original text feature vector includes:

[0018] Input the original text into the word embedding module, and perform token processing through the tokenizer to obtain a text token ID sequence;

[0019] Process the text token ID sequence through the word embedding matrix to obtain an original text feature vector.

[0020] In one implementation, the prefix enhanced prompt feature vector and the original text feature vector are input into the feature weighted average module to obtain a fused feature vector, including:

[0021] The feature weighted average module performs weighted averaging of the prefix enhanced prompt feature vector and the original text feature vector along the feature dimension to obtain a weighted feature vector as the first fused feature vector.

[0022] In one implementation, based on the fused feature vector, the prediction result of the original text, and the degree of difference between the current prediction result of the model and the true label, the machine-generated text detection model is trained, including:

[0023] By creating an attention mask to indicate the region that the model should focus on when processing the fused features and ignoring padding or other irrelevant parts, the first fused feature vector and the attention mask are used as inputs, and forward propagation is performed in the model to output a data dictionary, where the data dictionary contains the predicted value calculated by the output module, the weighted feature vector calculated by the feature weighted average module, and the cross-entropy loss calculated based on the degree of difference between the current prediction result of the model and the true label.

[0024] Based on the same inventive concept, a second aspect of the present invention provides a machine-generated text detection method, where the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module, and the method includes:

[0025] Input the text to be detected into the prefix enhancement mapping module to obtain a second prefix enhanced prompt feature vector after linear adaptation;

[0026] Input the text to be detected into the word embedding module to obtain a text feature vector to be detected;

[0027] Input the prefix enhanced prompt feature vector and the text feature vector to be detected into the feature weighted average module to obtain a second fused feature vector;

[0028] Input the second fused feature vector into the output module to obtain a prediction result for the text to be detected,

[0029] where the machine-generated text detection model is a model obtained by using the training method of the machine-generated text detection model in the first aspect.

[0030] Based on the same inventive concept, a third aspect of the present invention provides a training device for a machine-generated text detection model, where the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module, and the device includes:

[0031] The first prefix-enhanced prompt feature vector acquisition module is configured to input the original text into the prefix-enhanced mapping module to obtain the first prefix-enhanced prompt feature vector after linear adaptation;

[0032] The original text feature vector acquisition module is configured to input the original text into the word embedding module to obtain the original text feature vector;

[0033] The first fused feature vector acquisition module is configured to input the prefix-enhanced prompt feature vector and the original text feature vector into the feature weighted average module to obtain the first fused feature vector;

[0034] The first prediction result output module is configured to input the first fused feature vector into the output module to obtain the prediction result for the original text; and

[0035] The forward propagation module is configured to train the machine-generated text detection model based on the difference degree between the first fused feature vector, the prediction result for the original text, the current prediction result of the model and the true label.

[0036] Based on the same inventive concept, a fourth aspect of the present invention provides a machine-generated text detection device. Wherein, the machine-generated text detection model includes a prefix-enhanced mapping module, a word embedding module, a feature weighted average module and an output module, and the device includes:

[0037] The second prefix-enhanced prompt feature vector acquisition module is configured to input the text to be detected into the prefix-enhanced mapping module to obtain the second prefix-enhanced prompt feature vector after linear adaptation;

[0038] The text to be detected feature vector acquisition module is configured to input the text to be detected into the word embedding module to obtain the text to be detected feature vector;

[0039] The second fused feature vector acquisition module is configured to input the prefix-enhanced prompt feature vector and the text to be detected feature vector into the feature weighted average module to obtain the second fused feature vector;

[0040] The second prediction result output module is configured to input the second fused feature vector into the output module to obtain the prediction result for the text to be detected,

[0041] wherein, the machine-generated text detection model is a model obtained by using the training device of the machine-generated text detection model in the third aspect.

[0042] Based on the same inventive concept, the fifth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the training method of the machine-generated text detection model described in the first aspect or the machine-generated text detection method described in the second aspect.

[0043] Based on the same inventive concept, the sixth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the training method of the machine-generated text detection model described in the first aspect or the machine-generated text detection method described in the second aspect.

[0044] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0045] For the training method of the machine-generated text detection model provided by the present invention, the prefix enhancement mapping module can perform prefix strengthening processing on the input original text to enhance the expression ability of text features. By repeating these embedding vectors to match the batch size of the input text, the sub-embedding module is used to label and embed the original text to obtain the original text feature vector, and weighted averaging is performed through the feature weighted average module, thereby integrating feature information from different sources. In the training mode, the output of the feature weighted average module will be used for the forward propagation of the model to generate prediction results. At the same time, if the true label is provided, the loss function will be calculated and backpropagation will be performed to update the model parameters. Therefore, a machine-generated text detection model with better generalization ability can be obtained. Using the trained machine-generated text detection model to detect machine-generated text can improve the prediction accuracy. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is a flowchart of the training method of the machine-generated text detection model in the embodiment of the present invention;

[0048] Figure 2 It is a flowchart of the machine-generated text detection method in the embodiment of the present invention;

[0049] Figure 3 It is a structural block diagram of the training device of the machine-generated text detection model in the embodiment of the present invention;

[0050] Figure 4 This is the structural block diagram of the machine-generated text detection device in the embodiment of the present invention.

[0051] Figure 5 This is the structural diagram of the machine-generated text detection model based on prefix enhancement in the embodiment of the present invention;

[0052] Figure 6 This is the detailed flowchart of the training method of the machine-generated text detection model in the embodiment of the present invention. Detailed implementation manners

[0053] A machine-generated text detection model based on prefix enhancement and its training method disclosed by the present invention. In the specific implementation process, it first obtains the GPT-J model, initializes the scoring model and the reference model and their respective tokenizers based on this model to obtain a set of preprocessing models, then obtains a data set, divides the data set into a training set and a validation set, and uses the preprocessing models, the validation set, and the data set to fine-tune and evaluate the preprocessing models. The present invention can solve the problem of poor accuracy in detecting machine-generated text due to the lack of in-depth understanding of the text generation process in the prior art.

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] Embodiment 1

[0056] The present invention discloses a training method for a machine-generated text detection model based on prefix enhancement. Among them, the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module. The method includes:

[0057] S110: Input the original text into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation;

[0058] S120: Input the original text into the word embedding module to obtain an original text feature vector;

[0059] S130: Input the prefix enhancement prompt feature vector and the original text feature vector into the feature weighted average module to obtain a first fused feature vector;

[0060] S140: Input the first fused feature vector into the output module to obtain a prediction result for the original text; and

[0061] Based on the difference degrees between the first fused feature vector, the prediction result of the original text, the current prediction result of the model and the true label, train the machine-generated text detection model.

[0062] Please refer to Figure 1 and Figure 5 , where Figure 1 is a flowchart of the training method of the machine-generated text detection model in an embodiment of the present invention, Figure 5 is a structural diagram of the machine-generated text detection model based on prefix enhancement in an embodiment of the present invention.

[0063] Among them, the prefix enhancement mapping module includes a prefix enhancement tokenizer, a prefix enhancement mapping model, and a prefix enhancement prompt adapter;

[0064] In the specific implementation process, the prefix enhancement tokenizer uses BertTokenizer to perform word segmentation on the original text, which is used to convert the text into a token sequence that the model can understand. The input of the prefix enhancement tokenizer is the original text sequence, and the output is the token ID sequence of the prefix enhancement text.

[0065] The prefix enhancement mapping model uses the Bert model to process the token sequence of the prefix enhancement text to generate the feature representation of the prefix enhancement text. The input of the prefix enhancement mapping model is the token ID sequence of the prefix enhancement text, and the output is the prefix enhancement prompt feature vector.

[0066] The prefix enhancement prompt adapter is a custom linear layer that adjusts the dimension of the feature through linear transformation. The input is the feature vector of the prefix enhancement prompt word, and the output is the prefix enhancement prompt feature vector after linear adaptation.

[0067] The word embedding module includes a tokenizer and a word embedding matrix, which processes the original text data into a text token sequence and then converts it into a continuous vector representation. The input is the original text, and the output is the original text feature vector.

[0068] The feature weighted average module is used to perform weighted averaging of the prefix enhancement prompt feature vector and the original text feature vector along the feature dimension. The input of the feature weighted average module is the prefix enhancement prompt feature vector and the original text feature vector, and the output is the weighted feature vector, that is, the first fused feature vector.

[0069] In one implementation, the forward propagation module is used to perform a forward propagation process with prompt prefix enhancement. By creating an attention mask to indicate the regions that the model should focus on when processing the fused features, the padding or other irrelevant parts are ignored. First, the first fused feature representation is obtained by weighted averaging or concatenating the prefix enhancement prompt feature vector and the original text feature vector. At the same time, when processing the input data, by analyzing the length of the input sequence and combining the actual valid token information, a binary matrix is generated as the attention mask. In the matrix, 1 represents the valid input part that the model should focus on, and 0 represents the padding part or irrelevant part that should be ignored, which is used to indicate the part that the model should focus on during the forward propagation process. This module takes the fused feature representation and the attention mask as inputs, performs forward propagation in the model, and outputs a data dictionary. The data dictionary contains the predicted values calculated by the tokenizer, the weighted feature vectors calculated by the feature weighted averaging module, and the cross-entropy loss calculated by the cross-entropy loss function. The specific form of the cross-entropy loss function is as follows:

[0070] L(y,p) = -[ylog(p)+(1 - y)log(1 - p)]

[0071] where L(y,p) is the obtained cross-entropy loss, y is the true label, p is the probability that the model predicts as the positive class (label 1), and L(y,p) is the loss value.

[0072] During the training process, an adaptive weight decay mechanism is adopted. The adaptive weight decay optimizer specifically uses the Adam optimization algorithm and weight decay for the efficient optimization of model parameters and controlling the generalization ability of the model. The inputs of the adaptive weight decay optimizer are the model weights and biases to be optimized, the gradient values calculated according to the loss function during the model forward propagation process, the initial learning rate, and the hyperparameter controlling the weight decay intensity. The outputs are the weights, learning rate, gradients, and loss value of the model after the gradient and learning rate are adjusted.

[0073] The cross-entropy loss evaluator is a module of the forward propagation module, which is a module used to calculate the cross-entropy loss in classification problems. The inputs include the prediction results of the model for the input data and the labels of the true classes of each sample, and the output is the cross-entropy loss representing the degree of difference between the current prediction results of the model and the true labels.

[0074] Preferably, the prefix enhancement mapping module first tokenizes the input text and converts it into a token sequence that the model can understand using T5Tokenizer. Then, the T5Model model is used to process the reversed text sequence to obtain an initial reversed feature representation. Through a custom linear layer, the reversed features are mapped to the target dimension to adapt to the input requirements of the main model. Then, the static prompt embedding vector is initialized to enhance the expressive ability of the text features. These embedding vectors are repeated to match the batch size of the input text.

[0075] The feature weighted average module first calculates the weighted average of the prefix enhancement prompt feature vector and the original text feature vector. This step first calculates the weighting coefficients and achieves the weighted average of the two feature vectors through element-wise multiplication and addition, thus integrating the feature information from different sources. Subsequently, the model uses these weighted feature vectors for subsequent processing without the need to construct an attention mask. Specifically, in the training mode, the output of the feature weighted average module will be used for the forward propagation of the model to generate prediction results. At the same time, if the true labels are provided, the loss function will be calculated and backpropagation will be performed to update the model parameters. In this process, the effect of the feature weighted average module directly affects the learning efficiency and final performance of the model. In the non-training mode, such as the validation or test phase, the feature weighted average module still processes the input text according to the same steps but does not perform parameter updates. At this time, the purpose of the technology is to stably provide the weighted features in order to evaluate the prediction accuracy and generalization ability of the model for new data.

[0076] In the training mode, the output of the feature weighted average module will be used for the forward propagation of the model to generate prediction results. At the same time, if the true labels are provided, the loss function will be calculated and backpropagation will be performed to update the model parameters. In this process, the effect of the feature weighted average module directly affects the learning efficiency and final performance of the model. In the non-training mode, such as the validation or test phase, the feature weighted average module still processes the input text according to the same steps but does not perform parameter updates. At this time, the purpose of the technology is to stably provide the weighted features in order to evaluate the prediction accuracy and generalization ability of the model for new data.

[0077] The forward propagation module includes the following sub-steps:

[0078] (S1) Tokenize the text using the prefix enhancement tokenizer to convert it into a sequence of token IDs of the reversed text that the model can process, and add padding and truncation with a maximum length of 20. Move the obtained sequence of token IDs of the reversed text as the input ID to the device.

[0079] (S2) Move the input embedding of the decoder to the device and repeat to match the batch size of the inverse mapping input ID (the token ID of the reversed text), ensuring that each input text has a corresponding embedding representation.

[0080] (S3) Using the input ID obtained in step (S1) and the decoder input feature vector obtained in step (S2) as inputs, perform a forward pass through the prefix enhancement mapping model to obtain the last hidden state of the prefix enhancement prompt word, then convert it to a floating point number and adapt it through the prefix enhancement prompt adapter. Output the adapted prefix enhancement prompt word feature vector and the length of the prefix enhancement prompt word feature vector.

[0081] (S4) Use the tokenizer to convert the text tokenized in step (S1) into a sequence of token IDs of the original text that can be processed by the model, add padding, do not return the token type ID, and move it to the device. Obtain the result of tokenizing the original text and the input ID after removing the first token.

[0082] (S5) Input the adapted prompt word feature vector in step (S3) and the tokenized result in step (S4), and obtain the original text feature vector through the word embedding transformation matrix. Secondly, concatenate the adapted prefix enhancement prompt word feature vector obtained through the prefix enhancement mapping model in step (S3) with the original text feature vector obtained after tokenizing the original text in step (S4), and convert it to an appropriate data type to obtain the fused feature vector.

[0083] (S6) Input the adapted prefix enhancement prompt word feature vector in step (S3) and the tokenized result in step (S4), create a tensor of all 1s with a shape consistent with the batch and length of the prefix enhancement prompt word feature vector, and concatenate it with the tokenized attention mask to obtain the fused attention mask.

[0084] (S7) Input the fused word embedding vector in step (S5) and the fused attention mask in step (S6), perform a forward pass through the scoring model to calculate the logits scores, and output the logits scores corresponding to each category of the scoring model.

[0085] (S8) Input the logits scores obtained in step (S7) and the input IDs after removing the first token in step (S4), and determine whether the scoring model used to calculate the logits scores is the same as the reference model used to calculate the reference logits, so as to ensure the consistency and comparability of the calculation results. Specifically, input the logits scores obtained in step (S7) and the input IDs after removing the first token in step (S4), and determine whether the scoring model used to calculate the logits scores is the same as the reference model used to calculate the reference logits, so as to ensure the consistency and comparability of the calculation results. If different reference models are used, tokenize the original text of the input model using the reference tokenizer, and perform a forward pass through the reference model to calculate the reference logits. Then calculate the original criterion using the discriminant criterion function. Output the logits of the reference model and the original criterion.

[0086] (S9) Input the tokenized result of the original text in step (S4), the logits of the reference model in step (S8), the logits scores obtained in (S7), and the input IDs after removing the first token in step (S4). Use the scoring model to obtain the logits of the last token, and make a prediction through the classifier to obtain a probability distribution representing the likelihood that the text belongs to a machine or a human. Output the probability distribution and the original criterion. If the true label gts is provided, calculate the loss, and finally output the prediction result and the loss.

[0087] The method of the present invention will be described below through specific examples.

[0088] According to another aspect of the present invention, a training method for a machine-generated text detection model based on prefix enhancement is provided. Please refer to Figure 6 , including the following steps:

[0089] (1) Obtain the GPT-J model and download the model file to the local directory.

[0090] (2) Use the pre-trained model obtained in step (1) as the scoring model and the reference model, initialize their tokenizers and models respectively, and define the loss function and the forward propagation function to obtain a complete model ready for training and validation.

[0091] (3) Obtain a data set, divide 80% of the data in the data set into a training set, and divide 20% of the data in the data set into a validation set.

[0092] (4) Use the training set obtained in step (3) to randomly select 1000 pieces of data to obtain a training subset.

[0093] (5) Use the preprocessing model obtained in step (2) (the complete model ready for training and validation), the validation set obtained in step (3), and the training subset obtained in step (4) to fine-tune and evaluate the preprocessing model, and obtain evaluation data.

[0094] It should be noted that the forward propagation function is a more general term, referring to the process (forward propagation process) in which the model calculates the output after receiving the input, passing through each layer and module. It involves feature extraction, fusion, and the generation of the final prediction result. The main task of the forward propagation function is to pass the input data through the network structure of the model to obtain the output of the model.

[0095] The forward loss function is a specific term, referring to a part of the forward propagation process, used to calculate the difference (loss) between the model output and the true label. In the present invention, it refers to the cross-entropy loss function. Its main task is to evaluate the error between the prediction result of the model and the true label, so as to perform backpropagation and parameter update during the training process.

[0096] Therefore, although both involve the forward propagation process of the model, the "forward propagation function" is a more general concept, while the "forward loss function" focuses on the specific implementation of calculating the loss.

[0097] Preferably, step (2) includes the following sub-steps:

[0098] (2-1) Load the basic components required for initializing the prefix enhancement mapping task.

[0099] (2-2) Use the model obtained in step (1) as the scoring model, set the dataset to 'xsum', and load the tokenizer loading module to obtain a scoring model tokenizer loaded onto the specified device.

[0100] (2-3) Use the model obtained in step (1) as the scoring model, and load the model loading module to obtain a pre-trained scoring model loaded onto the specified device.

[0101] (2-4) Determine whether to load the reference model, or whether the scoring model is different from the reference model. If so, go to step (2-5); if not, go to step (2-10).

[0102] (2-5) Use the model obtained in step (1) as the reference model, set the dataset to 'xsum', and load the tokenizer loading module to obtain a reference model tokenizer loaded onto the specified device.

[0103] (2-6) Use the model obtained in step (1) as the reference model, and load the model loading module to obtain a pre-trained reference model loaded onto the specified device.

[0104] (2-7) Register model components that do not participate in gradient updates.

[0105] (2-8) Use the components and models in steps (2-1) to (2-6) to complete initialization and prepare for training.

[0106] (2-9) Initialize the sampling bias and sampling bias analysis module for evaluating between two sets of logits. Define a linear classifier and move it to the specified device (CPU or GPU). This linear layer has 50257 input features and 2 output features for binary classification tasks.

[0107] (2-10) Initialize the forward propagation module for executing the prefix enhancement prompt generation algorithm.

[0108] Preferably, step (2-1) includes the following sub-steps:

[0109] (2-1-1) Load the pre-trained T5 tokenizer to convert text into a token sequence that the model can process. Load the pre-trained T5 model and run it with 8-bit precision. Initialize the basic components required for the prefix enhancement mapping task.

[0110] (2-1-2) Create a custom linear layer as an adapter layer to map the output of the prefix enhancement mapping model from a 512-dimensional feature space to a 2560-dimensional feature space.

[0111] (2-1-3) Set the maximum length of the prefix enhancement prompt to 20 and the length of the static prompt to 0.

[0112] (2-1-4) Initialize a learnable embedding vector of shape (20, 512) for the decoder input, filled with random numbers from a normal distribution.

[0113] Preferably, step (5) includes the following sub-steps:

[0114] (5-1) This step is for the model evaluation module. Use the validation set obtained in step (3) to evaluate the preprocessed model obtained in step (2) to verify the effectiveness and accuracy of the model.

[0115] (5-2) The training subset obtained in step (4) will be used for model training, and the validation set obtained in step (3) will be used for model evaluation and hyperparameter tuning.

[0116] (5-3) Obtain the initialized logger, data loader, total number of epochs, AdamW optimizer, cosine annealing learning rate scheduler, and gradient scaler.

[0117] (5-4) Move the model obtained in step (2) to the specified device.

[0118] (5-5) Set the gradient step count cumulative value to 1.

[0119] (5-6) Initialize the epoch loss value list to store the loss value for each epoch.

[0120] (5-7) Initialize the iteration counter i to 0.

[0121] (5-8) Initialize the loss tensor and move it to the specified device.

[0122] (5-9) Clear the gradients of the Adam optimizer obtained in step (5-3).

[0123] (5-10) Use the data loader obtained in step (5-3) to obtain the original text and the sampled text in batches. Combine them into mixed text and create corresponding labels.

[0124] (5-11) Update the learning rate scheduler obtained in step (5-3).

[0125] (5-12) Start automatic mixed-precision training. Use the text data obtained in step (5-10) to perform forward propagation through the model to obtain the output of the model. Calculate the loss for the current batch using the loss function and accumulate the loss of the current batch to the total loss.

[0126] (5-13) Generate a corresponding label list according to the lengths of the original text and the sampled text, where the label for the original text is 0 and the label for the sampled text is 1.

[0127] (5-14) Convert the logits output by the model to probabilities, perform binary classification on the probabilities using a threshold of 0.5 to obtain the prediction results. Calculate the sigmoid probabilities output by the model and add them to the probability list. Then, add the true labels for the current batch to the true label list.

[0128] (5-15) Determine whether the iteration counter i+1 has reached the gradient step count cumulative value in step (5-5). If so, go to step (5-16); if not, go to step (5-17).

[0129] (5-16) Perform backpropagation, update the model parameters and the gradient scaler, and clear the gradients of the optimizer.

[0130] (5-17) Record the loss for the current batch in the epoch loss value list.

[0131] (5-18) Increment the iteration counter obtained in step (5-7) by 1.

[0132] (5-19) Use the area under the ROC curve calculation module in step (5-1-11), input the probability list and true label list obtained in step (5-14), and obtain the area under the ROC curve (ROC AUC) of the current model.

[0133] (5-20) Use the area under the precision-recall curve calculation module in step (5-1-12), input the probability list and true label list obtained in step (5-14), and obtain the area under the precision-recall curve (PRAUC) of the current model.

[0134] (5-21) Convert the prediction result list and true label list in step (5-14) into tensors, and calculate their matching degree and average accuracy. Use the epoch loss value list updated in step (5-17) to obtain the average loss value of the current epoch.

[0135] (5-22) Use the logger obtained in step (5-3) to write the average loss value of the epoch obtained in the current step (5-21), the average accuracy obtained in step (5-21), the ROC AUC obtained in step (5-19), and the PR AUC obtained in step (5-20) into TensorBoard.

[0136] (5-23) Determine whether the current training epoch is a multiple of 4. If it is, go to step (5-24); if not, go to step (5-25).

[0137] (5-24) Call the evaluation module of the training model in step (5-1) to obtain the evaluation result of the current model on the performance of the validation dataset.

[0138] (5-25) Save the model parameters fine-tuned in the current training epoch to the specified directory, and print the save path.

[0139] (5-26) Close the logger in step (5-3).

[0140] (5-27) Return the fine-tuned model.

[0141] Preferably, step (5-1) is a model evaluation module, and this module includes the following sub-steps:

[0142] (5-1-1) Create an evaluation loader for loading the dataset batch by batch.

[0143] (5-1-2) Move the preprocessing model obtained in step (2) to the specified device and set the model to the evaluation mode.

[0144] (5-1-3) Initialize the loss value to 0.

[0145] (5-1-4) Create a data loader to load the validation set data obtained in step (3) batch by batch.

[0146] (5-1-5) Initialize an accuracy list to store the accuracy of each batch; initialize a probability list to store the predicted probabilities output by the model; initialize a true label list to store the true labels.

[0147] (5-1-6) Use a context manager so that gradients are not calculated within the context.

[0148] (5-1-7) Determine whether the lengths of the original text and the sampled text are equal. If they are equal, go to step (5-1-8); if not, report an error.

[0149] (5-1-8) Combine the original text and the sampled text into a mixed text, and generate a corresponding label list based on the original text and the sampled text.

[0150] (5-1-9) Pass the mixed text into the model for forward propagation to obtain the output.

[0151] (5-1-10) Add the loss value of the current batch to the cumulative loss. Add the true labels and predicted probabilities of the batch to the true label list and probability list respectively, and calculate and accumulate the accuracy.

[0152] (5-1-11) Calculate the PR AUC. Input the true labels and probability list of all samples updated in step (5-1-10), calculate the precision and recall. Then use the obtained precision and recall to calculate the area under the precision-recall curve to obtain the PR AUC.

[0153] (5-1-12) Calculate the ROC AUC using the true labels and probability list of all samples updated in step (5-1-10).

[0154] (5-1-13) Divide the loss accumulated during the entire epoch by the total number of samples to obtain the average loss.

[0155] (5-1-14) Return a dictionary containing the average accuracy, average loss, ROC AUC, and PR AUC.

[0156] Preferably, step (5-3) includes the following sub-steps:

[0157] (5-3-1) Initialize the logger. Obtain the current time and format it as a string.

[0158] (5-3-2) Initialize a logger object for TensorBoard and set the log path according to the obtained time string.

[0159] (5-3-3) Create a data loader, set the batch size to 1, load one batch of data using the training subset obtained in step (4), and perform a random shuffling operation.

[0160] (5-3-4) Set the total number of epochs for training.

[0161] (5-3-5) Initialize the parameters of the model using the Adam optimizer, set the learning rate, with a default value of 5e-5.

[0162] (5-3-6) Initialize a cosine annealing learning rate scheduler for dynamically adjusting the learning rate, with the period set to the product of the length of the data loader obtained in step (5-3-3) and the total number of epochs obtained in step (5-3-4).

[0163] (5-3-7) Define a cross-entropy loss function for calculating the loss during model training.

[0164] (5-3-8) Initialize a GradScaler gradient scaler for mixed-precision training.

[0165] Preferably, step (5-16) includes the following sub-steps:

[0166] (5-16-1) Use the gradient scaler in step (5-3) and the loss tensor updated in step (5-12), calculate the gradient of the loss and perform backpropagation.

[0167] (5-16-2) Use the gradient scaler in step (5-3) and the Adam optimizer in step (5-3) to update the parameters of the model and execute the step of the optimizer.

[0168] (5-16-3) Clear the gradients of the Adam optimizer in step (5-3).

[0169] (5-16-4) Update the scaling factor of the gradient scaler in step (5-3).

[0170] (5-16-5) Use the recorder in step (5-3) to write the loss updated in step (5-12) to TensorBoard for visualizing the loss during the training process.

[0171] (5-16-6) Add the loss tensor updated in step (5-12) to the list of epoch loss values in step (5-6).

[0172] (5-16-7) Reset the loss tensor and move it to the specified device.

[0173] The training method of the machine-generated text detection model based on prefix enhancement proposed in this solution has the following main differences compared with the general training method:

[0174] 1. Prefix enhancement technology: This method introduces prefix enhancement technology. Through a specific prompt word generation algorithm, it enhances the model's ability to understand the input text. This technology can help the model better capture context information, thereby improving the quality and accuracy of text generation.

[0175] 2. Model initialization and fine-tuning: In the method, the scoring model and the reference model are clearly distinguished, and the model is trained and evaluated by fine-tuning the preprocessing model. This method enables the model to better utilize existing knowledge during training and improve the performance of the model.

[0176] Embodiment 2

[0177] Based on the same inventive concept, this embodiment discloses a method for detecting machine-generated text. Among them, the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module. Please refer to Figure 2 , and the method includes:

[0178] S210: Input the text to be detected into the prefix enhancement mapping module to obtain a second prefix enhancement prompt feature vector after linear adaptation;

[0179] S220: Input the text to be detected into the word embedding module to obtain a text feature vector to be detected;

[0180] S230: Input the prefix enhancement prompt feature vector and the text feature vector to be detected into the feature weighted average module to obtain a second fused feature vector;

[0181] S240: Input the second fused feature vector into the output module to obtain a prediction result for the text to be detected,

[0182] wherein, the machine-generated text detection model is a model obtained by using the training method of the machine-generated text detection model in Embodiment 1.

[0183] Since the method introduced in Embodiment 2 of the present invention is a method implemented by using the model obtained by the training method of the machine-generated text detection model in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific implementation and application of this method, so it will not be elaborated here. Any implementation and application based on the model obtained by the method in Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0184] Embodiment 3

[0185] Based on the same inventive concept, the present invention provides a training device for a machine-generated text detection model based on prefix enhancement. Among them, the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module. Please refer to Figure 3 , the device includes:

[0186] The first prefix enhancement prompt feature vector obtaining module 301 is configured to input the original text into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation;

[0187] The original text feature vector obtaining module 302 is configured to input the original text into the word embedding module to obtain an original text feature vector;

[0188] The first fused feature vector obtaining module 303 is configured to input the prefix enhancement prompt feature vector and the original text feature vector into the feature weighted average module to obtain a first fused feature vector;

[0189] The first prediction result output module 304 is configured to input the first fused feature vector into the output module to obtain a prediction result for the original text; and

[0190] The forward propagation module 305 is configured to train the machine-generated text detection model based on the difference degree between the first fused feature vector, the prediction result for the original text, the current prediction result of the model, and the true label.

[0191] Since the device introduced in Embodiment 3 of the present invention is the device adopted for implementing the training method of the machine-generated text detection model in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated here. Any device adopted in the method of Embodiment 1 of the present invention belongs to the scope protected by the present invention.

[0192] Embodiment 4

[0193] Based on the same inventive concept, the present invention provides a machine-generated text detection device. Among them, the machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module, and an output module. Please refer to Figure 4 , the device includes:

[0194] The second prefix enhancement prompt feature vector obtaining module 401 is configured to input the text to be detected into the prefix enhancement mapping module to obtain a second prefix enhancement prompt feature vector after linear adaptation;

[0195] The text to be detected feature vector obtaining module 402 is configured to input the text to be detected into the word embedding module to obtain a text to be detected feature vector;

[0196] The second fused feature vector obtaining module 403 is configured to input the prefix enhanced prompt feature vector and the text feature vector to be detected into the feature weighted average module to obtain the second fused feature vector;

[0197] The second prediction result output module 404 is configured to input the second fused feature vector into the output module to obtain the prediction result for the text to be detected.

[0198] Wherein, the machine-generated text detection model is a model obtained by using the training device of the machine-generated text detection model of the third aspect.

[0199] Since the device introduced in the fourth embodiment of the present invention is the device used to implement the machine-generated text detection method in the second embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the device, so it will not be elaborated here. Any device used in the method of the first embodiment of the present invention falls within the scope of protection of the present invention.

[0200] Embodiment Five

[0201] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method described in Embodiment One.

[0202] Since the computer-readable storage medium introduced in the fifth embodiment of the present invention is the computer-readable storage medium used to implement the training method of the machine-generated text detection model or the machine-generated text detection method in the second embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the computer-readable storage medium, so it will not be elaborated here. Any computer-readable storage medium used in the method of the first or second embodiment of the present invention falls within the scope of protection of the present invention.

[0203] Embodiment Six

[0204] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the method described in Embodiment One.

[0205] Since the computer device described in the fourth embodiment of the present invention is the computer device used in the training method of the machine-generated text detection model in the first embodiment of the present invention or the machine-generated text detection method in the second embodiment of the present invention, based on the method described in the first embodiment of the present invention, those skilled in the art can understand the specific structure and variations of the computer device, so it will not be elaborated here. Any computer device used in the method of the first embodiment or the second embodiment of the present invention falls within the scope of protection of the present invention.

[0206] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0207] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0208] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention. Obviously, those skilled in the art can make various changes and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. A training method for a machine-generated text detection model based on prefix enhancement, characterized in that: in, The machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module and an output module, and the method includes: Inputting the original text into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation; Input the original text into the word embedding module to obtain the original text feature vector; Inputting the prefix enhancement prompt feature vector and the original text feature vector into the feature weighted average module to obtain the first fused feature vector; Inputting the first fused feature vector into the output module to obtain a prediction result of the original text; and Training the machine-generated text detection model based on the first fused feature vector, the prediction result of the original text, and the degree of difference between the current prediction result of the model and the true label; The prefix enhancement mapping module includes a prefix enhancement marker, a prefix enhancement mapping model and a prefix enhancement prompt word adapter. The original text is input into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation, including: The original text is input into the prefix enhancement mapping module, and the original text is segmented by the prefix enhancement tagger to obtain the tag ID sequence of the prefix enhancement text; The tag ID sequence of the prefix enhanced text is processed by the prefix enhancement mapping model to obtain the prefix enhancement prompt feature vector; The prefix enhancement prompt feature vector is processed by a prefix enhancement prompt word adapter to obtain a prefix enhancement prompt feature vector after linear adaptation.

2. The method for training a machine-generated text detection model as claimed in claim 1, characterized in that: The word embedding module includes a tagger and a word embedding matrix. The original text is input into the word embedding module to obtain the original text feature vector, including: The original text is input into the word embedding module and marked by the marker to obtain a text tag ID sequence; The text tag ID sequence is processed through the word embedding matrix to obtain the original text feature vector.

3. The training method of a machine-generated text detection model as claimed in claim 1, characterized in that: The prefix enhancement hint feature vector and the original text feature vector are input into the feature weighted average module to obtain the fused feature vector, including: The feature weighted averaging module performs weighted averaging on the prefix enhancement hint feature vector and the original text feature vector along the feature dimension to obtain a weighted feature vector as the first fused feature vector.

4. The method for training a machine-generated text detection model as claimed in claim 1, wherein: Based on the fused feature vector, the prediction result of the original text, and the degree of difference between the current prediction result of the model and the true label, the machine-generated text detection model is trained, including: By creating an attention mask to indicate the area that the model should focus on when processing the fused features, ignoring padding or other irrelevant parts, the first fused feature vector and the attention mask are used as input, forward propagation is performed in the model, and a data dictionary is output, where the data dictionary contains the predicted value calculated by the output module, the weighted feature vector calculated by the feature weighted average module, and the cross entropy loss calculated according to the degree of difference between the current prediction result of the model and the true label.

5. A method for detecting machine-generated text, characterized in that: in, The machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module and an output module, and the method includes: Input the text to be detected into the prefix enhancement mapping module to obtain a second prefix enhancement prompt feature vector after linear adaptation; Input the text to be detected into the word embedding module to obtain the feature vector of the text to be detected; Inputting the prefix enhancement prompt feature vector and the feature vector of the text to be detected into the feature weighted average module to obtain a second fused feature vector; The second fused feature vector is input into the output module to obtain the prediction result of the text to be detected. The machine-generated text detection model is a model obtained by using the training method of the machine-generated text detection model described in any one of claims 1 to 4.

6. A training device for a machine-generated text detection model, characterized in that: in, The machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module and an output module, and the device includes: A first prefix enhancement prompt feature vector acquisition module, used for inputting the original text into the prefix enhancement mapping module to obtain a first prefix enhancement prompt feature vector after linear adaptation; The original text feature vector acquisition module is used to input the original text into the word embedding module to obtain the original text feature vector; A first fused feature vector obtaining module, used for inputting the prefix enhancement prompt feature vector and the original text feature vector into a feature weighted average module to obtain a first fused feature vector; A first prediction result output module, used for inputting the first fused feature vector into the output module to obtain a prediction result of the original text; and A forward propagation module, used for training the machine-generated text detection model based on the first fused feature vector, the prediction result of the original text, and the degree of difference between the current prediction result of the model and the true label; The prefix enhancement mapping module includes a prefix enhancement marker, a prefix enhancement mapping model and a prefix enhancement prompt word adapter, and the first prefix enhancement prompt feature vector acquisition module is specifically used for: The original text is input into the prefix enhancement mapping module, and the original text is segmented by the prefix enhancement tagger to obtain the tag ID sequence of the prefix enhancement text; The tag ID sequence of the prefix enhanced text is processed by the prefix enhancement mapping model to obtain the prefix enhancement prompt feature vector; The prefix enhancement prompt feature vector is processed by a prefix enhancement prompt word adapter to obtain a prefix enhancement prompt feature vector after linear adaptation.

7. A machine-generated text detection device, characterized in that: in, The machine-generated text detection model includes a prefix enhancement mapping module, a word embedding module, a feature weighted average module and an output module, and the device includes: A second prefix enhancement prompt feature vector acquisition module, used for inputting the text to be detected into the prefix enhancement mapping module to obtain a second prefix enhancement prompt feature vector after linear adaptation; A module for obtaining a feature vector of the text to be detected is used to input the text to be detected into a word embedding module to obtain a feature vector of the text to be detected; A second fused feature vector obtaining module, used for inputting the prefix enhancement prompt feature vector and the feature vector of the text to be detected into a feature weighted average module to obtain a second fused feature vector; The second prediction result output module is used to input the second fused feature vector into the output module to obtain the prediction result of the text to be detected. The machine-generated text detection model is a model obtained by using the training method of the machine-generated text detection model described in any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the training method of the machine-generated text detection model as described in any one of claims 1 to 4 or the machine-generated text detection method as described in claim 5.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the training method of the machine-generated text detection model as described in any one of claims 1 to 4 or the machine-generated text detection method as described in claim 5 is implemented.

Images