Machine general text detection method based on maximum mean value difference

Abstract

The invention discloses a machine general text detection method based on maximum mean value difference, which comprises the following steps of: training a depth kernel of a measurement model by increasing an inter-class distance between a machine text feature and a human handwritten text feature and reducing an intra-class distance of the human handwritten text feature as an optimization target; the problem that an existing text detection method is insufficient in detection precision or universality is solved, whether the text is generated by the AI or not can be effectively detected, and an important guarantee is provided for credibility and safety of network information.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection, and in particular to a machine-general text detection method based on maximum mean difference. Background Art

[0002] With the development of large language models, the quality of text generated by language models is getting higher and higher, more natural and fluent, and extremely similar to human handwritten text, playing an important role in fields such as summarization, dialogue generation, and machine translation. However, the proliferation of generated text also brings many potential hazards: large language models may be used to generate fake news, spam, online fraud information and other content with factual errors or biases, which poses a great threat to the credibility and security of online information, and there is an urgent need for reliable detection technology for machine-generated text.

[0003] Existing machine-generated text detection methods are mainly divided into metric-based and model-based text detection methods. Among them, metric-based text detection methods use feature statistics to calculate the probability that the test text is machine-generated text, but due to the large distribution gap between the generation domains of different generation models, this type of method is characterized by insufficient accuracy; while model-based text detection methods train a classifier based on the generation model to distinguish machine-generated text from human handwritten text, but it is heavily dependent on the specific generation model and is difficult to adapt to the detection of text generated by other models. Summary of the invention

[0004] In view of the above problems, the present invention proposes a machine-wide text detection method based on maximum mean difference, which mainly solves the problem of insufficient detection accuracy or insufficient versatility of existing text detection methods.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A machine-generalized text detection method based on maximum mean difference comprises the following steps:

[0007] Step 1: extract features from the text generated by the language model and the human handwritten text, which are defined as machine text features and human handwritten text features respectively;

[0008] Step 2, establishing a measurement model based on maximum mean difference, training the deep kernel of the measurement model with the machine text features and the human handwritten text features, wherein the optimization goal of the training process includes increasing the inter-class distance between the machine text features and the human handwritten text features, and reducing the intra-class distance of the human handwritten text features, to obtain the trained MMD deep kernel;

[0009] Step 3, for the detection task of the paragraph text to be tested, using the MMD deep kernel to calculate the difference between the human handwritten text and the paragraph text to be tested, which is defined as a first difference value;

[0010] Step 4, randomly mixing the paragraph text to be tested with the human handwritten text, and splitting the mixed text into two sections, using the MMD deep kernel to calculate the difference between the two sections of the mixed text, and defining a second difference value;

[0011] Step 5, repeat step 4 according to a preset number of times to obtain multiple second difference values, calculate the differences between the multiple second difference values ​​and the first difference value, accumulate the number of the difference values ​​greater than 0, calculate the ratio between the number and the preset number of times, if the ratio is greater than the preset probability threshold, it indicates that the paragraph text to be tested is generated by AI.

[0012] In some embodiments, step 6 is also included. For the detection task of the single sentence text to be tested, the MMD deep kernel is used to calculate the difference between the single sentence text to be tested and the human handwritten text, which is defined as a first difference value, and whether the single sentence text to be tested is generated by AI is determined according to the first difference value.

[0013] In some implementations, the measurement model is:

[0014]

[0015] In the formula, are random samples from different distributions, represents a distribution of machine-generated text, represents a distribution of human handwritten text, is a Hilbert space with a reproducing kernel k, and f is a mapping of the distribution X or Y data to the real space. The function of is the set of functions f, and represents the expectation of two distributions sampled from machine text features and human handwritten text features, k(X,X′) represents the intra-class distance of machine text features, k(Y,Y′) represents the intra-class distance of human handwritten text features, and k(X,Y) represents the inter-class distance between machine text features and human handwritten text features.

[0016] In some embodiments, the optimization goal is:

[0017]

[0018] Among them are

[0019]

[0020] H ij :=k(x i ,x j )-k(x i ,y j )-k(y i ,x j )+k(y i ,y j ) (4);

[0021] In the formula, k ω represents the MMD kernel based on parameter set ω, express The coefficient of difference between the two distributions is It represents the test power of the optimization objective and characterizes the certainty of the distribution difference.

[0022] In some implementations, in step 2, the intra-class distance of the machine text feature is ignored.

[0023] In some implementations, a multi-population-aware optimization target with an MPP agent is established according to the optimization target, and the multi-population-aware optimization target includes:

[0024]

[0025] In some implementations, the detection process of the paragraph text to be tested is:

[0026]

[0027] In the formula, H ij :=k(x i ,x j )-k(x i ,y j )-k(y i ,x j )+k(y i ,y j ), is The Borel measure on , are independent and identically distributed observations.

[0028] In some implementations, the detection process of the single sentence text to be tested is:

[0029]

[0030] In the formula, represents a reference human handwritten text dataset, x i ,x jis a single sample in the dataset, To test a single sentence text.

[0031] The beneficial effects of the present invention are as follows: by increasing the inter-class distance between machine text features and human handwritten text features, and reducing the intra-class distance of human handwritten text features to optimize the deep kernel of the target training metric model, the problem of insufficient detection accuracy or insufficient versatility of existing text detection methods is solved, and it can effectively detect whether the text is generated by AI, providing important guarantees for the credibility and security of network information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a machine-general text detection method based on maximum mean difference disclosed in an embodiment of the present invention;

[0033] Figure 2 Experimental data of applying MMD-MP disclosed in the embodiment of the present invention to paragraph-level detection;

[0034] Figure 3 This is experimental data of applying the MMD-MP disclosed in the embodiment of the present invention to single sentence level detection. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the content of the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

[0036] This embodiment proposes a machine-general text detection method based on maximum mean difference, such as Figure 1 As shown, the following steps are included:

[0037] Step 1: Use a fixed feature extractor to extract features from the language model generated text and human handwritten text, which are defined as machine text features and human handwritten text features, respectively.

[0038] Step 2: Establish a measurement model based on maximum mean difference, and train the deep kernel of the measurement model with machine text features and human handwritten text features. The optimization goals of the training process include increasing the inter-class distance between machine text features and human handwritten text features, and reducing the intra-class distance of human handwritten text features, so as to obtain the trained MMD deep kernel.

[0039] Maximum Mean Discrepancy (MMD) is a metric used to measure the difference between two probability distributions P and Q, and is particularly used in parameter-free and kernel methods in machine learning and statistics. In this embodiment, MMD is used as a metric model to measure the difference between two distributions.

[0040] In one example, the above measurement model is:

[0041]

[0042] In the formula, are random samples from different distributions, represents a distribution of machine-generated text, represents a distribution of human handwritten text, is a Hilbert space with a reproducing kernel k, and f is a mapping of the distribution X or Y data to the real space. The function of is the set of functions f, and represents the expectation of two distributions sampled from machine text features and human handwritten text features, k(X,X′) represents the intra-class distance of machine text features, k(Y,Y′) represents the intra-class distance of human handwritten text features, and k(X,Y) represents the inter-class distance between machine text features and human handwritten text features.

[0043] In this embodiment, the kernel-based MMD method is used, and the optimization goal of the training process is:

[0044]

[0045] Among them are

[0046]

[0047] H ij :=k(x i ,x j )-k(x i ,y j )-k(y i ,x j )+k(y i ,y j ) (4);

[0048] In the formula, k ω represents the MMD kernel based on parameter set ω, express The coefficient of difference between the two distributions is It represents the test power of the optimization target and characterizes the certainty of the distribution difference. A higher test power indicates a greater certainty of the distribution difference.

[0049] When training the deep kernel of MMD-D, it tends to aggregate human handwritten text and all possible machine-generated text separately, that is, to reduce the intra-class distance, and at the same time push the two types of text away from each other, that is, to increase the inter-class distance. However, when the machine-generated text contains multiple different population samples, the optimization of this process becomes extremely difficult. Therefore, we removed the optimization of the intra-class distance of machine-generated samples during the optimization of the MMD kernel.

[0050] As another preferred embodiment of the present invention, in step 2, the intra-class distance of the machine text feature is ignored. With this as the goal, a multi-population perception optimization target with an MPP agent is established according to the optimization target. This embodiment defines it as a multi-population perception optimization target, abbreviated as MMD-MP, which includes:

[0051]

[0052] It should be noted that in order to ensure the non-negativity of the metric, we only removed the k w The optimization part of (Y,Y′) is used, while the original formula is still used during testing. The rationality of this optimization goal is further analyzed below.

[0053] Similar to MMD, we can still use U-statistics to approximate the proposed optimization objective proxy MPP, and it also has many ideal statistical properties to facilitate theoretical analysis. Below, we give an approximate estimate Progressiveness:

[0054]

[0055] in Indicates different Based on this, we can further conclude that test_power:

[0056]

[0057] in Φ is the standard normal cumulative distribution function, n is the number of samples, and the above formula shows that for any given constant r and and a sufficiently large n, the test_power of the MPP is determined by the first term Φ, and since The larger the test_power:

[0058]

[0059] Right now The higher the test_power, the better. So our training goal is to increase

[0060] The architecture design and training process of the MMD deep kernel are as follows:

[0061] Defining the MMD depth kernel For a neural network equipped with a feature extractor Use OpenAI's RoBERTabed-based GPT-2 detector model as the feature extractor f, and regard its last hidden state as the feature of the input text. It consists of a hidden layer transformer, a projector, and a multi-layer perceptron. During the training process, a fixed feature extractor is first used to extract features from human handwritten text and machine-generated text sentences, and then the deep core network is trained by maximizing the objective J in the equation Used for paragraph-level and sentence-level detection scenarios.

[0062] Step 3: For the detection task of the paragraph text to be tested, the MMD deep kernel is used to calculate the difference between the human handwritten text and the paragraph text to be tested, which is defined as the first difference value.

[0063] Step 4: randomly mix the paragraph text to be tested with the human handwritten text and split it into two mixed texts, calculate the difference between the two mixed texts using the MMD deep kernel, and define a second difference value.

[0064] Step 5, repeat step 4 according to a preset number of times to obtain multiple second difference values, calculate the differences between the multiple second difference values ​​and the first difference value, accumulate the number of differences greater than 0, calculate the ratio between the number and the preset number of times, if the ratio is greater than the preset probability threshold, it indicates that the paragraph text to be tested is generated by AI.

[0065] For paragraph-level detection, in one example, the detection process of the paragraph text to be tested is:

[0066]

[0067] In the formula, H ij :=k(x i ,x j )-k(x i ,y j )-k(y i ,x j )+k(y i ,y j ), is The Borel measure on , are independent and identically distributed observations, that is, the MMD is estimated by an unbiased U-statistic 2 Then estimate the MMD.

[0068] For the above formula (5), the calculated MMD value is first defined as "est", and then the mixed sentences are iteratively randomized and split, and the MMD value is calculated again, which is defined as "permi". By comparing "permi" with "est", if most of the "permi" values ​​are greater than "est", it means that the test paragraph is quite different from the human handwritten text, that is, the test paragraph is judged to be machine-generated text. On the contrary, if most of the "permi" values ​​are less than "est", the test paragraph is judged to be human handwritten text, that is, the ratio of the MMD after random scrambling to the MMD before scrambling is calculated, and this ratio can represent the probability that the text is generated by AI.

[0069] It also includes step 6, for the detection task of the single sentence text to be tested, using the MMD deep kernel to calculate the difference between the single sentence text to be tested and the human handwritten text, which is defined as the first difference value, and judging whether the single sentence text to be tested is generated by AI according to the first difference value.

[0070] The detection process of the single sentence text to be tested is:

[0071]

[0072] In the formula, represents a reference human handwritten text dataset, x i ,x j is a single sample in the dataset, To test a single sentence text.

[0073] The larger the MMD distance value, the greater the difference between the test sentence and the reference sample (human handwritten text). In order to better distinguish, a binary classification training model is added to convert the generated MMD distance value into 1 or 0, where 1 indicates that the test sentence is machine-generated text, and 0 indicates that the test sentence is human handwritten text.

[0074] The following experimental data proves the effectiveness of the method proposed by the present invention. Figure 2 and Figure 3 .

[0075] Figure 2The experimental comparison diagram of the method proposed in the present invention applied to multiple machine-generated text data sets in paragraph detection shows that this method (MMD-MP) is superior to the traditional MMD method (MMD-O, MMD-N) and the two-sample detection method (C2ST-S, C2ST-S) in the detection of machine-generated texts generated by multiple language models and mixed sets of multi-domain machine-generated texts;

[0076] Figure 3 The experimental comparison chart of the method proposed in the present invention applied to multiple machine-generated text datasets in single sentence detection shows that this method (MMD-MP) is superior to the existing metric-based methods (Likelihood, Rank, Log-Rank, Entropy, DetectGPT-d, DetectGPT-z) and model-based methods (OpenAI-D, ChatGPT-D) in the detection of machine-generated texts generated by multiple language models and mixed sets of multi-domain machine-generated texts.

[0077] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A machine-general text detection method based on maximum mean difference, characterized in that: The following steps are involved: Step 1: extract features from the text generated by the language model and the human handwritten text, which are defined as machine text features and human handwritten text features respectively; Step 2, establishing a measurement model based on maximum mean difference, training the deep kernel of the measurement model with the machine text features and the human handwritten text features, wherein the optimization goal of the training process includes increasing the inter-class distance between the machine text features and the human handwritten text features, and reducing the intra-class distance of the human handwritten text features, to obtain the trained MMD deep kernel; Step 3, for the detection task of the paragraph text to be tested, using the MMD deep kernel to calculate the difference between the human handwritten text and the paragraph text to be tested, which is defined as a first difference value; Step 4, randomly mixing the paragraph text to be tested with the human handwritten text, and splitting the mixed text into two sections, using the MMD deep kernel to calculate the difference between the two sections of the mixed text, and defining a second difference value; Step 5, repeat step 4 according to a preset number of times to obtain multiple second difference values, calculate the differences between the multiple second difference values ​​and the first difference value, accumulate the number of the difference values ​​greater than 0, calculate the ratio between the number and the preset number of times, if the ratio is greater than the preset probability threshold, it indicates that the paragraph text to be tested is generated by AI.

2. The machine-generalized text detection method based on maximum mean difference as claimed in claim 1, characterized in that: It also includes step 6, for the detection task of the single sentence text to be tested, using the MMD deep kernel to calculate the difference between the single sentence text to be tested and the human handwritten text, defined as a first difference value, and judging whether the single sentence text to be tested is generated by AI according to the first difference value.

3. The machine-generalized text detection method based on maximum mean difference as claimed in claim 1, characterized in that: The measurement model is: In the formula, are random samples from different distributions, represents a distribution of machine-generated text, represents a distribution of human handwritten text, is a Hilbert space with a reproducing kernel k, and f is a mapping of the distribution X or Y data to the real space. function, F is the set of functions f, and represents the expectation of two distributions sampled from machine text features and human handwritten text features, k(X,X′) represents the intra-class distance of machine text features, k(Y,Y′) represents the intra-class distance of human handwritten text features, and k(X,Y) represents the inter-class distance between machine text features and human handwritten text features.

4. The machine-generalized text detection method based on maximum mean difference as claimed in claim 3, characterized in that: The optimization goal is: Among them are H ij :=k(x i ,x j )-k(x i ,y j )-k(y i ,x j )+k(y i ,y j ) (4); In the formula, k ω represents the MMD kernel based on parameter set ω, express The coefficient of difference between the two distributions is It represents the test power of the optimization objective and characterizes the certainty of the distribution difference.

5. The machine-generalized text detection method based on maximum mean difference as claimed in claim 4, characterized in that: In step 2, the intra-class distance of the machine text feature is ignored.

6. The machine-generalized text detection method based on maximum mean difference as claimed in claim 5, characterized in that: A multi-population-aware optimization target with an MPP agent is established according to the optimization target, and the multi-population-aware optimization target includes:

7. The machine-generalized text detection method based on maximum mean difference as claimed in claim 6, characterized in that: The detection process of the paragraph text to be tested is: In the formula, H ij :=k(x i ,x j )-k(x i ,y j )-k(y i ,x j )+k(y i ,y j ), is The Borel measure on , are independent and identically distributed observations.

8. The machine-generalized text detection method based on maximum mean difference as claimed in claim 7, characterized in that: The detection process of the single sentence text to be tested is: In the formula, represents a reference human handwritten text dataset, x i ,x j is a single sample in the dataset, To test a single sentence text.

Images