An artificial intelligence generated text detection method and system

Abstract

The present application provides an artificial intelligence generated text detection method, comprising: using an agent language model to select multiple characters from the text to be detected as representative characters; using a preselected language model to perform multiple generation operations on each of the representative characters in turn, generating a generated character set for each representative character from the first character of each generation result, and obtaining an estimated probability matrix for all representative characters from the generated character set; extracting probability features from the estimated probability matrix, and forming a context feature matrix using the context semantic representation of all representative characters, and obtaining enhanced features based on the context feature matrix and the probability features; and obtaining a detection result of the text to be detected by classifying and detecting the enhanced features. The present application also provides an artificial intelligence generated text detection system and a data processing device for detecting artificial intelligence generated text.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular to an artificial intelligence-generated text detection method and system based on probability estimation using a black-box model. Background Technology

[0002] Recent breakthroughs in large language models have significantly improved the quality of AI-generated text and further propelled its application across various scenarios. People can easily instruct services like ChatGPT to generate text that is virtually imperceptible to humans. While large language models offer many conveniences, their misuse also poses new social threats. To combat these threats, there is an urgent need for automated detection of AI-generated text.

[0003] AI-generated text detection can be defined as a binary or multi-class classification task. In a binary classification setting, the goal is to determine whether a given text was written by a human or generated by AI. In a multi-class classification setting, the goal is to determine whether a given text was written by a human or generated by a specific AI language model (assuming there are M candidate language models, it's an M+1 classification task). Based on whether the detector can access the internal state of the language model, existing methods can be categorized into white-box and black-box methods. White-box methods use fine-grained features reflected in the internal state, such as token probabilities, output by the language model to distinguish between large language models, generally offering good detection performance. However, since commercial large model services typically only provide the generated text and not the internal state, their applicability is limited. Black-box methods, on the other hand, only require the text to be detected for feature extraction. In principle, they can be applied to any large language model, offering a wider range of applications. However, their detection performance and generalization ability on domain texts not seen in the training set are usually weaker than white-box methods. This "performance-applicability" contradiction has become a key challenge hindering the application of AI-generated text detection technology in real-world scenarios. Summary of the Invention

[0004] In view of the above problems, the present invention proposes an artificial intelligence generated text detection method, comprising: obtaining the proxy probability of each character of the text to be detected by using a proxy language model, selecting a plurality of characters as representative characters of the text to be detected based on the proxy probability, and forming a representative character set of the text to be detected; performing multiple generation operations on each of the representative characters in sequence by using a preselected language model, wherein the generation operation executes "please continue the following text, starting from the next word: {x<i}"; generating a generated character set of the representative character with the first character of each generation result, and obtaining the estimated probability matrix of all the representative characters by using the generated character set; {x<i} represents the content of the text to be detected before the i-th character; extracting probability features from the estimated probability matrix, forming a context feature matrix by using the context semantic representations of all the representative characters, and obtaining enhanced features based on the context feature matrix and the probability features; performing classification detection on the enhanced features to obtain the detection result of the text to be detected.

[0005] In the artificial intelligence generated text detection method of the present invention, the step of obtaining the detection result specifically comprises: extracting the probability feature L' from the estimated probability matrix L by using a convolutional neural network CNN and a Transformer encoding network, where L'=Transformer(CNN(L)); inputting the text x to be detected into a pre-trained language model, and obtaining the character x of x i The corresponding last layer output is used as the context semantic representation E, E={e1,e2,…,e n}, e i is the semantic representation vector of x i ; mapping the context semantic representations of all the representative characters to d dimensions through a multi-layer perceptron network MLP map to form the context feature matrix C, C={MLP map (e i )|x i ∈S}, where s is the representative character set; obtaining the enhanced feature F from C and L' through a bidirectional cross-attention mechanism, represents a concatenation operation; performing classification detection on F using a multi-layer perceptron network MLP cls to obtain the detection result

[0006] In the artificial intelligence generated text detection method of the present invention, MLP is used cls to perform binary classification detection on F to obtain the detection result or use MLP cls to perform multi-class classification detection on F with the output layer dimension being M+1 to obtain the detection result θ iLet M be the i-th pre-selected language model, and M be the number of pre-selected language models.

[0007] The artificial intelligence-generated text detection method of the present invention includes the following steps in forming the representative character set S: performing inference operations on the text to be detected x using a proxy language model θ, including segmenting x into a word list using a word segmenter of θ, inputting the word list into θ to obtain a word probability list of the word list, aligning the word probability list to the character list of x, and processing characters x containing multiple words. i The joint probability of each word is used as the surrogate probability of the character; if x is obtained through N sampling operations... i Estimated probability Then its standard error p i For x i The true probability; selection p i The character x i List of generated word probabilities p θ ', from p θ Select k characters with the lowest estimated probabilities from x as representative characters of x, and generate S.

[0008] The artificial intelligence-generated text detection method of the present invention, wherein the generated character set is used... Estimated probability of obtaining the representative character by Generate an estimated probability matrix L for the text to be detected; N is the number of times the generation operation is performed on each representative character using the pre-selected language model. For indicator functions, if o j =x i ,but If o j ≠x i ,but

[0009] The artificial intelligence-generated text detection method of the present invention, wherein the generated character set is used... Estimated probability of obtaining the representative character by Generate an estimated probability matrix L of the text to be detected; {x i-b:i-1} represents the content of the b characters preceding the i-th character in the text to be detected, and N is the number of times the pre-selected language model is used to generate the data for each representative character. For indicator functions, if o j =x i ,but If o j ≠x i ,but

[0010] The artificial intelligence-generated text detection method of this invention uses a cross-entropy loss function L as the loss function for classifying and detecting the text x to be detected. y i Let x be the true label generated by the i-th pre-selected language model. If x is generated by the i-th pre-selected language model, then y i =1, otherwise y i =0.

[0011] This invention also proposes an artificial intelligence-generated text detection system, comprising: a representative character extraction module, used to obtain the proxy probability of each character in the text to be detected using a proxy language model, and select multiple characters as representative characters of the text to be detected based on the proxy probability, forming a representative character set of the text to be detected; and a probability estimation module, used to perform multiple generation operations on each representative character sequentially using a pre-selected language model, wherein the generation operation executes "Please continue the following text, starting from the next word: {x}". <i}”; Generate a set of generated characters for the representative character based on the first character of each generation result, and obtain the estimated probability matrix of all representative characters from this set of generated characters; {x <i} represents the content of the text to be detected before the i-th character; the classification and detection module is used to extract probability features from the estimated probability matrix, and form a context feature matrix using the context semantic representation of all representative characters, and obtain enhanced features based on the context feature matrix and the probability features; by performing classification and detection on the enhanced features, the detection result of the text to be detected is obtained.

[0012] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the artificial intelligence-generated text detection method described above is implemented.

[0013] The present invention also proposes a data processing apparatus, including a computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it realizes the detection of artificial intelligence-generated text. Attached Figure Description

[0014] Figure 1 This is a flowchart of the AI-generated text detection method based on black-box model probability estimation according to the present invention.

[0015] Figure 2 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0017] To address the poor performance of existing technologies in black-box AI-generated text detection tasks, this invention proposes an efficient resampling method guided by a proxy for black-box generated text detection. The core idea is to select a subset of words from all words in a given text that are more likely to indicate unique word usage characteristics of a large language model, and estimate the generation probability of words within this subset on a candidate black-box language model, thus obtaining a "pseudo-probability feature" to enhance black-box detection performance. The inventors discovered that words with lower probabilities are more likely to reflect the uniqueness of word usage in a large language model. Therefore, in the word selection process, this invention uses a proxy white-box language model with access to internal states to determine words with relatively low probabilities in a given text. By performing multiple resampling operations on the positions of these words on the target black-box language model, an estimated probability list is obtained, which serves as a pseudo-probability feature for AI-generated text detection tasks.

[0018] The AI-generated text detection method of this invention employs a representative word selection based on a surrogate model and error awareness. First, a surrogate language model selects low-probability words. To mitigate the negative impact of estimation errors on feature effectiveness, an error-aware selector is used to obtain a representative word set for the text to be detected. Based on the surrogate model and the error-aware selector, a word set representing the source characteristics of the target text can be obtained. Furthermore, a black-box language model probability estimation based on resampling is used. A specially designed prompt allows the black-box language model to continue writing the text. Through repeated sampling, the probability of words in the current context on the language model is estimated. Finally, the estimated probabilities of all representative words on each candidate model are obtained. This allows for efficient resampling to obtain the estimated probabilities of representative words on the large black-box language model, thus achieving robust AI-generated text detection. This invention also employs context-compensated AI-generated text detection. A pre-trained language model is used to extract contextual semantic features, and a cross-attention mechanism is used to compensate for the estimated probability features, thereby achieving even more robust AI-generated text detection. This method utilizes semantic features containing contextual information to compensate for probabilistic features, effectively improving the detector's detection performance.

[0019] Given a text x = (x1, x2, ..., xn) containing n characters n(For languages ​​like English that use words as the unit of meaning, this would be n words.) The AI-generated text detector needs to output the source y of text x. For binary classification, the detector only needs to determine whether the text was generated by AI, i.e., y∈{human, AI}; for multi-class classification, the detector also needs to further determine which generative language model the given text specifically comes from, i.e., y∈{human, θ1, θ2, ..., θ M}, where θ i This represents the i-th candidate generative model.

[0020] This invention achieves AI-generated text detection based on probability estimation using a black-box model. The overall process is as follows: Figure 1 As shown: (1) A small representative subset of words is formed from the text to be detected by selecting words with low probability and low estimation error. (2) Only the words in the subset are resampled multiple times to obtain pseudo-probability features. (3) The features are enhanced by compensating for contextual information, and then input into the classifier for final detection. The following is a detailed introduction to each module of the method.

[0021] 1. Selection of representative characters for error perception

[0022] This invention first uses a lightweight, internally state-accessible proxy language model θ (e.g., GPT-2) to perform inference operations on the text to be detected x. For the i-th character x... i Its proxy probability is θ with respect to x. i Given the first i-1 characters as input, generate x i Since the surrogate model θ is a white-box model, the surrogate probability of each word / character obtained from the inference operation of θ on x can be regarded as the true probability of each word / character. Specifically, the word segmenter of the surrogate model θ is used to segment the text into a list of words, which is then input into the surrogate model θ. The true probability of each word of x in θ is obtained through the above inference operation. Then, the list of word probabilities is aligned to the list of characters in the text (for English, it is aligned to "words", hereinafter referred to as "characters"). For characters containing multiple words, the joint probability of each word is calculated as the probability of the character, and finally, the character probability list of x on the surrogate model θ is obtained. Where n is the number of words in x.

[0023] To mitigate the negative impact of estimation errors on feature validity, this invention designs an error-aware representative word selector. For a true probability of p... i The character x i If x is obtained by resampling N times i The estimated probability is but The standard error is:

[0024]

[0025] Because the estimation error of characters with low probability is relatively large, this invention sets a lower probability limit for selected characters, requiring that the error of their estimated probability does not exceed Δ times, in order to filter out characters whose estimation error may be too large:

[0026]

[0027] By controlling the relative error, p is removed. θ The terms that do not meet the error requirements are obtained as p. θ '。 From p θ The k characters with the lowest estimated probabilities are selected as the characters representing the source characteristics of the text x to be detected (hereinafter referred to as representative characters), and finally a set of representative characters S is formed.

[0028] 2. Probability estimation of black-box candidate language models

[0029] Due to the pre-selected language model θ1-θ M As a black-box model, it is impossible to access the internal true probabilities of the pre-selected language model. The relative probabilities of the characters in x to the candidate language model θ are obtained during the inference operation of x by the candidate language model. i The estimated probability.

[0030] Given a text x to be detected, in order to obtain the character x i Estimated probability Use the following input to make the candidate black-box language model generate N times: "Please continue the following text, starting from the next word: {x}" <i}”, where {x <i} represents the content before the i-th character of x. Record the first character of each generated result to obtain the output set of generated characters. Then x i The estimated probability can be calculated as follows:

[0031]

[0032] in It is an indicator function, when o j =x i The value is 1 if it is active, and 0 otherwise.

[0033] The above resampling operation is performed on all M candidate models for k characters in set S. If higher sampling efficiency is required, only the first b characters of the current character can be input as context, and the estimated probability can be calculated. Finally, the estimated probability matrix of representative characters of the text to be detected x is obtained.

[0034] 3. Context-enhanced classification

[0035] First, a convolutional neural network (CNN) and a Transformer encoding network are used to extract probabilistic features from L. To enhance the representation:

[0036] L' = Transformer(CNN(L))

[0037] Since the representative characters in S are generally scattered in the original text, the above process cannot preserve the local contextual information provided by the adjacent characters of the representative characters, as well as their relative position information in the original text. To alleviate the above information loss, this invention introduces the contextual semantic representation of k representative characters to guide the probabilistic representation L' as context compensation. Specifically, the given text x is input into a pre-trained language model (e.g., RoBERTa), and the last layer output corresponding to each character is obtained as the contextual semantic representation E = {e1, e2, ..., e}. n}, where e i It is x i The semantic representation vector. Then, through a multilayer perceptron network (MLP) map Map the representations of k representative characters to d dimensions to form a context feature matrix.

[0038] C={MLP map (e i )|x i ∈S},

[0039] Then, a bidirectional cross-attention mechanism is used to enable interaction between C and L', and to output enhanced features.

[0040]

[0041]

[0042] in This represents a concatenation operation. Since the representation of each position in C implicitly contains contextual information, the above interactive operation can fuse relative positional information into the enhanced feature F.

[0043] Finally, F is input into another multilayer perceptron network (MLP). cls The MLP is used for classification. For binary classification detection tasks, MLP... cls The output layer dimension is 2; for multi-class detection tasks, MLP cls The output layer has a dimension of M+1, corresponding to human and M candidate language models. The predicted label for text x is obtained.

[0044]

[0045] The above network is optimized using the cross-entropy loss function as a supervision signal:

[0046]

[0047] Where y i To indicate whether x belongs to the true label of the i-th class, 1 is used if it belongs and 0 otherwise.

[0048] Specifically, the AI-generated text detection method based on black-box model probability estimation of the present invention includes:

[0049] Step S1: Use the proxy language model θ to obtain each character x of the text to be detected x. i The proxy probability is estimated, and multiple characters are selected as representative characters to represent the source characteristics of the text to be detected, forming a representative character set S of the text to be detected;

[0050] This includes: using a word segmenter of θ to segment x into a list of tokens, inputting this list of tokens into θ to obtain a list of token probabilities for this list of tokens, aligning this list of token probabilities to a list of characters of x, and processing characters x containing multiple tokens. i The joint probability of each word is used as the proxy probability of the character; let x be obtained through N sampling operations. i Estimated probability Then its standard error p i For x i The true probability; selection The character x i List of generated word probabilities p θ ', from p θ Select k characters with the lowest estimated probabilities from x as representative characters of x, and generate S.

[0051] Step S2: Using the pre-selected language model, perform multiple generation operations on each representative character sequentially. This generation operation executes the following: "Please continue the following text, starting from the next word: {x}" <i}”; Generate a set of generated characters for the representative character based on the first character of each generation result, and obtain the estimated probability matrix L of all the representative characters based on this set of generated characters;

[0052] Among them, the generated character set Estimated probability of obtaining representative characters {x <i} represents the content of the text to be detected before the i-th character; or {x i-b:i-1} represents the content of the b characters preceding the i-th character in the text to be detected;

[0053] by or Generate an estimated probability matrix L for the text to be detected; N is the number of times the generation operation is performed on each representative character using the pre-selected language model. For indicator functions, if o j =x i ,but If o j ≠x i ,but

[0054] Step S3: Extract probability features L' from L, and form a context feature matrix C using the context semantic representation E of all representative characters. Obtain enhanced features F based on C and L'. Obtain the detection result of x by performing classification detection on F.

[0055] In this process, a convolutional neural network (CNN) and a Transformer encoding network are used to extract L' from L, where L' = Transformer(CNN(L));

[0056] Input x into a pre-trained language model (e.g., RoBERTa) to obtain the words x. i The corresponding output of the last layer serves as the context semantic representation E, where E = {e1, e2, ..., e} n}, e i For x i The semantic representation vector;

[0057] Through Multilayer Perceptron Network (MLP) map Map the contextual semantic representations of all representative characters to d dimensions to form C = {MLP} map (e i )|x i ∈S};

[0058] F is obtained from C and L' through a bidirectional cross-attention mechanism. Indicates a splicing operation;

[0059] Where by using MLP cls Perform binary classification detection on F and obtain the detection results. Or use MLP cls Perform multi-class detection on F with an output layer dimension of M+1 to obtain the detection results. θ i Let M be the i-th pre-selected language model, and M be the number of pre-selected language models.

[0060] The above AI-generated text detection uses the cross-entropy loss function L as a supervision signal for optimization.

[0061]

[0062] Where y i Let x be the true label generated by the i-th pre-selected language model. If x is generated by the i-th pre-selected language model θ i If generated, then y i =1, if x is derived from the j-th pre-selected language model θ j (j≠i) If generated or written by humans, then y i =0.

[0063] Figure 2 This is a schematic diagram of the data processing apparatus of the present invention. Figure 2 As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The data processing apparatus includes a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, they achieve artificial intelligence-generated text detection based on probability estimation using a black-box model. Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by using integrated circuits to implement its corresponding function, or it can be implemented as a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.

[0064] Compared with existing AI-generated text detection methods, this invention improves both in-distribution detection performance and out-of-distribution generalization detection performance.

[0065] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A method for detecting artificial intelligence-generated text, characterized in that, include: Using the proxy language model The word segmenter will detect the text. The word is segmented into a list of lexical units, and this list of lexical units is input. To obtain the lexical probability list of this lexical list, and align this lexical probability list to... A list of words containing multiple lexical units. The joint probability of each word is used as the proxy probability of the character; if through The second sampling operation obtained The estimated probability is Then its standard error ; for The true probability; Select The word List of generated character probabilities ,from Selected from The character with the lowest estimated probability is used as... Representative characters, generate Representative character set ; This is the maximum error multiple; The pre-selected language model is used to generate multiple times for each representative character. This generation process involves writing "Please continue the following text, starting from the next word:" ”; The first character of each generation result is used to generate a set of generated characters for that representative character. This set of generated characters is then used to obtain the estimated probability matrix for all representative characters. ; This indicates the text to be detected. The content preceding the i-th character; Using convolutional neural networks and Transformer coding network from Extraction probability features , ;Will Input into a pre-trained language model to obtain The word The corresponding final layer output serves as the context semantic representation. , , for Semantic representation vector; through a multilayer perceptron network Represent the contextual semantics of all representative characters. Mapped to Dimension, forming the context feature matrix , ; through a bidirectional cross-attention mechanism to and Enhanced features are obtained , , This represents a splicing operation; a multilayer perceptron network is used. right Perform classification detection to obtain Test results , .

2. The artificial intelligence-generated text detection method as described in claim 1, characterized in that, use right Perform binary classification detection to obtain the detection result. Or use right The output layer dimension is Multi-class detection is used to obtain the detection results. , For the first A pre-selected language model, This represents the number of pre-selected language models.

3. The artificial intelligence-generated text detection method as described in claim 1, characterized in that, With this generated character set The estimated probability of obtaining the representative character. ;by Generate an estimated probability matrix for the text to be detected. ; This represents the number of times the generation operation is performed sequentially for each representative character using the pre-selected language model. For indicator functions, if ,but ,like ,but .

4. The artificial intelligence-generated text detection method as described in claim 1, characterized in that, With this generated character set The estimated probability of obtaining the representative character. ,by Generate an estimated probability matrix for the text to be detected. ; This indicates that the text to be detected begins before the i-th character. The content of each word This represents the number of times the generation operation is performed sequentially for each representative character using the pre-selected language model. For indicator functions, if ,but ,like ,but .

5. The artificial intelligence-generated text detection method as described in claim 2, characterized in that, Using the cross-entropy loss function As for the text to be detected The loss function for classification and detection, ; express For the first If the real labels generated by the pre-selected language model are... By the If a pre-selected language model is generated, then =1, otherwise... =0.

6. An artificial intelligence-generated text detection system, characterized in that, include: The representative character extraction module is used to form a set of representative characters for the text to be detected; among them, a proxy language model is used. The word segmenter will detect the text. The word is segmented into a list of lexical units, and this list of lexical units is input. To obtain the lexical probability list of this lexical list, and align this lexical probability list to... A list of words containing multiple lexical units. The joint probability of each word is used as the proxy probability of the character; if through The second sampling operation obtained The estimated probability is Then its standard error ; for The true probability; selection The word List of generated character probabilities ,from Selected from The character with the lowest estimated probability is used as... Representative characters, generate ; This is the maximum error multiple; The probability estimation module is used to perform multiple generation operations on each representative character using a pre-selected language model. This generation operation executes the command, "Please continue the following text, starting from the next word:..." "; Generate a set of generated characters for the representative character based on the first character of each generation result, and obtain the estimated probability matrix of all the representative characters from this set of generated characters." ; This indicates the text to be detected. The content preceding the i-th character; The classification and detection module is used to obtain the text to be detected. Test results Among them, convolutional neural networks are used. and Transformer coding network from Extraction probability features , ;Will Input into a pre-trained language model to obtain The word The corresponding final layer output serves as the context semantic representation. , , for Semantic representation vector; through a multilayer perceptron network Represent the contextual semantics of all representative characters. Mapped to Dimension, forming the context feature matrix , ; through a bidirectional cross-attention mechanism to and Enhanced features are obtained , , This represents a splicing operation; a multilayer perceptron network is used. right Perform classification detection to obtain , .

7. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the artificial intelligence-generated text detection method as described in any one of claims 1 to 5 is implemented.

8. A data processing apparatus comprising a computer-readable storage medium as claimed in claim 7, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it performs detection of artificial intelligence-generated text.

Images