Large language model generation text detection method based on semantic decoupling

Abstract

The invention discloses a semantic decoupling-based big language model generated text detection method, which comprises the following steps of: firstly, giving a candidate text, reconstructing a prompt by utilizing an auxiliary big language model according to the candidate text, and enabling the auxiliary big language model to obtain a regenerated text by utilizing the reconstructed prompt; comparing the similarity between the candidate text and the regenerated text; and judging whether the candidate text is generated by a machine or not according to a comparison result, and realizing detection of the text generated by the large language model. According to the method, semantic transformation attack and translation-back attack are effectively resisted through decoupling prompt and text regeneration, and the detection stability and reliability in a complex application scene in the real world are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a text detection method generated by a large language model based on semantic decoupling. Background Art

[0002] In recent years, large language models (LLMs) such as GPT-4, ChatGPT, and PaLM have demonstrated significant capabilities in understanding and generating natural language. These models have performed well in a variety of complex tasks, sparking widespread discussion about their potential as artificial general intelligence (AGI). At the same time, the capabilities of these models also bring potential risks, such as text generation, which may pose a threat to social, political, and academic integrity when used improperly.

[0003] Text generation technology uses deep learning networks, especially the Transformer architecture, and the model can generate fluent and coherent text. Text detection technology is to distinguish text generated by large language models (LLMs) from text created by humans. Researchers have developed supervised and zero-shot classifiers to improve the accuracy and generalization of detection. However, although traditional supervised classifiers perform well on data similar to the training set, their generalization ability is seriously insufficient when facing new fields or changing data types; and although zero-shot classifiers do not rely on a large amount of labeled data, when processing text generated by closed-source large language models (LLMs), due to the lack of understanding of the internal structure of the model, their detection performance is often unsatisfactory. In addition, existing technologies fail to effectively separate the prompt information in the generated text and the intrinsic characteristics of the content itself, which limits the accuracy and reliability of the detection system. Summary of the invention

[0004] The purpose of the present invention is to provide a large language model generation text detection method based on semantic decoupling, which effectively resists semantic transformation attacks and back translation attacks by decoupling prompts and text regeneration, and ensures detection stability and reliability in complex application scenarios in the real world.

[0005] The objective of the present invention is achieved through the following technical solutions: A method for generating text detection using a large language model based on semantic decoupling, the method comprising: Step 1: Given a candidate text, use the auxiliary large language model to reconstruct a prompt according to the candidate text, and use the reconstructed prompt to enable the auxiliary large language model to obtain a regenerated text; Step 2: Compare the similarity between the candidate text and the regenerated text; Step 3: Determine whether the candidate text is machine-generated based on the comparison result, thereby detecting the text generated by the large language model.

[0006] It can be seen from the technical solution provided by the present invention that the above method effectively resists semantic transformation attacks and back translation attacks by decoupling prompts and text regeneration, ensures detection stability and reliability in complex application scenarios in the real world, and significantly improves the detection accuracy of machine-generated text. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0008] Figure 1 A flowchart of a method for generating text detection using a large language model based on semantic decoupling provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0010] like Figure 1 The figure is a flow chart of a method for generating text detection based on a large language model with semantic decoupling according to an embodiment of the present invention, the method comprising: Step 1: Given a candidate text, use the auxiliary large language model to reconstruct a prompt according to the candidate text, and use the reconstructed prompt to enable the auxiliary large language model to obtain a regenerated text; In this step, given a candidate text and auxiliary large language model , obtained through the separation process in the prompt Under the semantic constraints of the candidate text Consistently regenerated text ,Right now: This formula represents the selection of all possible generated texts so that Generate text with the highest probability under the semantic constraints; Represents the process of sampling text from the predictive distribution; Since it is used to generate candidate text in the black box setting The hint is unknown to the detector, and the candidate text needs to be reconstructed first Tips: In the specific implementation, when the log-probability is hidden, the recently proposed hint recovery method based on probability inversion is not applicable, so we turn our attention to another similar task, namely question generation QG; Question generation (QG) involves generating natural questions based on a given sentence or paragraph. The QG task is defined as finding , is the original prompt of the text to be detected, such that:

[0011] This formula represents selecting the generated question with the highest probability under a given text from all possible generated questions, with the goal of generating natural questions that fit the context for subsequent analysis or detection; is a given input The prediction problem sequence The conditional log-likelihood of ; Note that the auxiliary large language model Performed very well in the QG task, so directly use Get candidate text Refactoring tips for ,Right now: in Indicates that it is used for query Refactored prompt templates for prompts; After obtaining the candidate text Refactoring tips for After that, use Based on refactoring hints Get regenerated text ,Right now: in For query the problem of regeneration; In the above process, It can be a black box for users. Users only need to input the text to be interpreted or regenerated to obtain the corresponding regenerated text. The separation process is formally expressed as: This formula represents the input of text and related constraints into the auxiliary large language model , and obtain the corresponding regenerated text .

[0012] Step 2: Compare the similarity between the candidate text and the regenerated text; In this step, the reconstruction hints are obtained through the separation process. Aligned regenerated text ; Then compare the candidate texts and regenerate text and use it as a discriminant feature. Since the two texts are consistent at the prompt level, the similarity only indicates the candidate text The inherent features included,the assumption of the comparison process is that:,machine-generated texts and human-authored texts exhibit inconsistent,inherent features; Specifically, a twin network is used to compare candidate texts and regenerate text The similarity of the candidate text is that the twin network is a neural network that processes two different input tensors with the same weights at the same time and produces comparable output tensors. and regenerate text Two subnetworks with shared structures, parameters, and weights are input respectively. For each subnetwork, a language model is used as an encoder to extract features from the text, as shown below: in represents the encoder; and Represents candidate texts and regenerate text The feature representation vector of The two vectors are input into the twin network for similarity judgment. The specific process is as follows: and are connected and input into a twin network classifier, represented as ; Different from traditional similarity calculation methods (such as cosine similarity and Euclidean distance), this application uses the Siamese network structure to compare candidate texts. and regenerate text The similarity between the two networks is directly judged by the learned weights, thus having higher flexibility and nonlinear expression ability. The twin network structure contains two sub-networks with shared parameters, which are used to extract the features of two inputs, and then use a binary classification layer to determine the similarity between the two inputs. In this framework, the concatenation of the input text and the concatenation operation after feature extraction enable the twin network to process two inputs at the same time and represent their similarity as a classification problem.

[0013] For example, using gte-Qwen1.5-7B-instruct (Alibaba NLP’s Qianwen model) as the encoder, which encodes up to 32K tokens of text into 4096-dimensional embeddings, the twin network classifier consists of three fully connected layers with ReLU functions, and the dimensions of the middle layer of the classifier are 1024 and 512 respectively; Freeze the weights of the twin network and use binary cross entropy loss to train the classifier. The loss function is formalized as:

[0014] in Indicates candidate text The true label of Represents the total number of samples in a dataset or batch; Represents the index of each sample, which is used to traverse each sample one by one during the summation process; The predicted label of the detector is expressed as: .

[0015] Step 3: Determine whether the candidate text is machine-generated based on the comparison result, thereby detecting the text generated by the large language model.

[0016] In this step, if the candidate text If it is machine generated, then the text is generated With candidate text have a higher similarity because the machine-generated texts have similar intrinsic features; on the contrary, if the candidate text If it is created by humans, then regenerate the text With candidate text The similarity will be lower.

[0017] In the specific implementation, in addition to using ChatGPT and Vicuna-7b as auxiliary large language models, other large language models such as GPT-4 or Claude3 can also be considered to adapt to different application requirements and optimize detection strategies.

[0018] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0019] In summary, the method described in the embodiment of the present invention has the following advantages: 1. Efficient text generation detection performance: This paper significantly improves the detection accuracy of machine-generated text by using auxiliary models (such as ChatGPT and Vicuna-7b) to decouple and regenerate candidate texts; 2. Generalization ability: Compared with the existing technology, the present invention shows better generalization ability on different data sets (such as news, story writing and biomedical research question answering) and different model-generated texts. This is due to the fact that when processing text, the present invention not only focuses on the surface text features, but also deeply explores the intrinsic characteristics of semantics and generative models through the decoupling process; 3. Robustness: The present invention effectively resists semantic transformation attacks such as DIPPER and back-translation attacks by decoupling the prompt and text regeneration methods, ensuring stability and reliability in complex application scenarios in the real world.

[0020] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0021] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A text detection method based on a large language model generated by semantic decoupling, characterized in that: The method comprises: Step 1: Given a candidate text, use the auxiliary large language model to reconstruct a prompt according to the candidate text, and use the reconstructed prompt to enable the auxiliary large language model to obtain a regenerated text; Step 2: Compare the similarity between the candidate text and the regenerated text; Step 3: Determine whether the candidate text is machine-generated based on the comparison result, thereby detecting the text generated by the large language model.

2. According to claim 1, the method for generating text detection based on a large language model with semantic decoupling is characterized in that: In step 1, given a candidate text and auxiliary large language model , obtained through the separation process in the prompt Under the semantic constraints of the candidate text Consistently regenerated text ,Right now: This formula represents the selection of all possible generated texts so that Generate text with the highest probability under the semantic constraints; Represents the process of sampling text from the predictive distribution; Since it is used to generate candidate text in the black box setting The hint is unknown to the detector, and the candidate text needs to be reconstructed first Tips: Direct use Get candidate text Refactoring Tips ,Right now: in Indicates that it is used for query Refactored prompt templates for prompts; After obtaining the candidate text Refactoring Tips After that, use Based on refactoring hints Get regenerated text ,Right now: in For query the problem of regeneration; The above separation process is formalized as: This formula represents the input of text and related constraints into the auxiliary large language model , and obtain the corresponding regenerated text .

3. According to claim 2, the method for generating text detection based on a large language model with semantic decoupling is characterized in that: In step 2, the reconstruction hint is obtained through the separation process Aligned regenerated text ; Then compare the candidate texts and regenerate text and use it as a discriminant feature. Since the two texts are consistent at the prompt level, the similarity only indicates the candidate text The inherent characteristics contained; Specifically, a twin network is used to compare candidate texts and regenerate text The similarity of the candidate text is that the twin network is a neural network that processes two different input tensors with the same weights at the same time and produces comparable output tensors. and regenerate text Two subnetworks with shared structures, parameters, and weights are input respectively. For each subnetwork, a language model is used as an encoder to extract features from the text, as shown below: in represents the encoder; and Represents candidate texts and regenerate text The feature representation vector of The two vectors are input into the twin network for similarity judgment. The specific process is as follows: and are connected and input into a twin network classifier, represented as ; Use the structure of the twin network to compare candidate texts and regenerate text The similarity between the two, the twin network directly judges the similarity through the learned weights; The twin network structure consists of two sub-networks with shared parameters, which are used to extract the features of two inputs, and then a binary classification layer is used to determine the similarity of the two inputs; In this framework, the concatenation of the input text and the concatenation operation after feature extraction enable the Siamese network to process two inputs simultaneously and represent their similarity as a classification problem.

Images