Machine generated text detection method and system based on direct differential learning

Abstract

The invention provides a machine generated text detection method and system based on direct differential learning, and relates to the technical field of artificial intelligence safety. Comprising the following steps of: regenerating or polishing and rewriting a human text by using a large language model, and constructing machine-generated text data so as to obtain a data pair of the human written text and the machine-generated text; based on the constructed data pair, the scoring model is optimized by adopting direct difference learning, and the optimization target is to maximize the conditional probability difference of the machine-generated text and minimize the conditional probability difference of the human text at the same time; obtaining a to-be-detected text, and calculating a conditional probability difference of the to-be-detected text by using the optimized scoring model; and based on the conditional probability difference, estimating the probability that the to-be-detected text belongs to the machine-generated text by adopting a reference clustering method. According to the method, a task target of machine generation text detection is explicitly modeled through direct differential learning, and the method has remarkable advantages in the aspects of detection robustness, cross-domain generalization ability and calculation efficiency.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, and in particular relates to a machine-generated text detection method and system based on direct difference learning. Background Technology

[0002] Advanced generative large language models (LLMs) can easily generate text that is virtually indistinguishable from human handwriting. However, the misuse of such technologies could pose serious risks to society. For example, in academic research, the inability to distinguish between human-written papers and LLM-generated content could undermine the credibility of academic work. Similarly, if we cannot determine whether news reports are written by human journalists or fabricated by LLMs, the integrity of information and public trust in the media could be severely compromised. To address these concerns, the Machine Generated Text Detection (MGTD) task has emerged. MGTD is a binary classification task designed to distinguish whether a given text is entirely human-written or machine-generated (or modified). The MGTD task includes the detection of machine-generated text (MGT, i.e., entirely machine-generated) and machine-modified text (MRT, i.e., machine-edited or modified human-written text).

[0003] Most existing detection methods utilize pre-trained language models (called scoring models) to estimate the lexical probabilities of a given text and use these probabilities to calculate some manually designed classification metric. These detection methods can be further divided into zero-shot detection methods (detection methods that do not require additional training) and training-based detection methods (detection methods that use specific data to train or fine-tune the detector).

[0004] Zero-shot methods typically rely on the raw output distribution of the pre-trained scoring model, which limits their performance. This limitation arises because scoring models are often relatively small language models (e.g., GPT-Neo-2.7B) with limited knowledge and relatively simple output patterns. However, detection often occurs in text that deviates from its inherent distribution, making reliable detection difficult for these methods.

[0005] Training-based methods use supervised fine-tuning (SFT) or preference learning to align the output distribution of the scoring model with that of a specific model. While this approach can improve the detection performance of a particular model, it often leads to overfitting of the scoring model to the training data. Therefore, the detector's generalization ability and robustness are affected when detecting data outside the training distribution. Furthermore, despite recent advancements in research on machine-generated text detection methods, a comprehensive and integrated benchmark dataset remains lacking. Existing benchmarks suffer from several significant shortcomings: 1. Limited focus on machine-generated text: Most benchmark datasets, such as MGTBench, focus only on pure machine-generated text and ignore the detection of machine-modified text; 2. Narrow scope of source LLMs: Most benchmarks rely on small-scale open-source models, while real-world applications typically involve advanced closed-source LLMs such as GPT-4o and DeepSeek-R1; 3. Limited text domain coverage: Benchmarks such as HC3 typically sample text from only one or more domains, ignoring the domain sensitivity of machine-generated text.

[0006] These shortcomings highlight the significant gap between existing benchmark datasets and real-world applications, indicating that current benchmarks fail to comprehensively and objectively reflect real-world detection challenges. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, this invention provides a machine-generated text detection method and system based on direct difference learning. The proposed direct difference learning explicitly models the task objective of machine-generated text detection and has significant advantages in detection robustness, cross-domain generalization ability and computational efficiency.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a machine-generated text detection method based on direct difference learning.

[0009] A machine-generated text detection method based on direct difference learning includes the following steps: Sample human text, use a large language model to regenerate or polish and rewrite the human text, construct machine-generated text data, and then obtain pairs of human-written text and machine-generated text data. Based on the constructed data pairs, direct difference learning is used to optimize the scoring model. The optimization objective is to maximize the conditional probability difference of machine-generated text while minimizing the conditional probability difference of human text. Obtain the text to be detected and use the optimized scoring model to calculate the conditional probability difference of the text to be detected; Based on the conditional probability difference, the probability that the text to be detected belongs to machine-generated text is estimated using the reference clustering method.

[0010] A second aspect of the present invention provides a machine-generated text detection system based on direct difference learning.

[0011] Machine-generated text detection systems based on direct difference learning include: The data acquisition module is configured to: sample human text, use a large language model to regenerate, polish, or rewrite the human text, construct machine-generated text data, and thus obtain pairs of human-written text and machine-generated text data. The model optimization module is configured to optimize the scoring model using direct difference learning based on the constructed data pairs, with the optimization objective being to maximize the conditional probability difference of machine-generated text while minimizing the conditional probability difference of human text. The conditional probability difference calculation module is configured to: acquire the text to be detected and calculate the conditional probability difference of the text to be detected using the optimized scoring model; The classification module is configured to estimate the probability that the text to be detected belongs to machine-generated text based on the conditional probability difference using a reference clustering method.

[0012] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the machine-generated text detection method based on direct difference learning as described in the first aspect of the present invention.

[0013] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the machine-generated text detection method based on direct difference learning as described in the first aspect of the present invention.

[0014] The above one or more technical solutions have the following beneficial effects: This invention provides a machine-generated text detection method and system based on direct difference learning. The proposed direct difference learning explicitly models the task objective of machine-generated text detection, namely, maximizing the gap between the conditional probability difference value of machine-generated text and the conditional probability difference value of human-written text. This enables the scoring model to better learn the core semantics of the detection task during the optimization process, thereby enhancing the robustness and generalization ability of detection. It has significant advantages in detection robustness, cross-domain generalization ability and computational efficiency.

[0015] The scoring model proposed in this invention, which is optimized by direct difference learning, can support general detection across domains, tasks, and models with only a small number of simple samples for training. It is particularly suitable for automatic identification and screening of complex, multi-source, and large-scale text content.

[0016] The MIRAGE dataset created by this invention achieves breakthroughs in the comprehensiveness, realism, and diversity of evaluation, fills the gaps in existing benchmark tests, further promotes the construction of industry testing standards, and is expected to lead the industrialization of testing technologies.

[0017] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0019] Figure 1 This is a flowchart of the method in Example 1.

[0020] Figure 2 This is a diagram of the direct difference learning process in Example 1.

[0021] Figure 3 The graph shows the performance test results of the method in Example 1 compared with other methods.

[0022] Figure 4 This is a flowchart illustrating the sampling process of human text during the construction of the MIRAGE benchmark dataset in Example 1.

[0023] Figure 5 This is a schematic diagram of the data generation process in the construction of the MIRAGE benchmark dataset in Example 1.

[0024] Figure 6 Other detailed diagrams from the construction of the MIRAGE benchmark dataset in Example 1.

[0025] Figure 7 This is a diagram showing the integration effect of the MIRAGE benchmark dataset in Example 1. Detailed Implementation

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0028] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0029] Overall invention concept: Existing machine-generated text detection methods face two major technical challenges: First, zero-shot detection methods rely on the original probability distribution output of the scoring model, which is difficult to adapt to complex text outside the distribution, resulting in limited detection performance. Second, training-based detection methods align the output distribution of the scoring model with a specific model through supervised fine-tuning or preference learning, which can easily lead to overfitting of training data and lack of inherent awareness of the machine-generated text detection task itself, thus limiting its generalization ability in diverse scenarios.

[0030] To address the aforementioned issues, this invention proposes a novel optimization strategy—Direct Difference Learning (DDL). DDL explicitly incorporates prior knowledge of machine-generated text detection into the training process of the scoring model, enabling the model to directly learn the task-related semantic differences between human-written text (HWT) and machine-generated text (MGT), rather than merely fitting the lexical probability distribution of the training data.

[0031] Direct difference learning allows scoring models to abandon their original language model features and focus on the intrinsic features of the detection task, thereby significantly improving detection accuracy, cross-domain robustness, and generalization ability.

[0032] Overall, the machine-generated text detection method based on direct difference optimization provided by this invention includes: a. For a given text Use a reference model. The text is resampled and then scored using a scoring model. The text is modeled on a token-by-token basis, and the conditional probability discrepancy (a metric that measures the difference between a text and its resampled version) is calculated using the token probability distribution of the text estimated by the scoring model. b. Optimize the scoring model using Direct Discrepancy Learning (DDL) by maximizing the conditional probability difference of machine-generated text while minimizing the conditional probability difference of human text, enabling the scoring model to directly learn task-oriented discriminative features. c. Use reference clustering techniques to cluster and estimate the probability difference distribution between machine-generated text and human text, and then estimate the probability that the text to be tested belongs to machine-generated text.

[0033] To ensure the reliability and practical application value of the detection method, this invention constructs the MIRAGE benchmark dataset, which is the most comprehensive multi-task machine-generated text detection and evaluation framework to date. MIRAGE samples human-written text from 10 corpora across 5 common text domains and utilizes 17 advanced mainstream large language models (including 13 commercial closed-source models and 4 open-source models) for text generation or revision, creating over 93,000 HWT-MGT data pairs. Through strategies such as multi-domain sampling, realistic LLM usage, and language style enhancement, MIRAGE significantly improves the realism and diversity of the evaluation, establishing a more reliable evaluation standard that closely reflects real-world applications, and to some extent bridging the gap between research and real-world scenarios.

[0034] Example 1 The rapid development of large language models (LLMs) has blurred the lines between human-written text and machine-generated text, making machine-generated text detection (MGTD) a highly sought-after task. However, existing MGTD methods perform poorly in complex real-world scenarios: zero-shot detectors heavily rely on the output distribution of the scoring model, while training-based detectors are often limited by overfitting to the training data, thus restricting their generalization ability.

[0035] We found that the performance bottleneck of training-based detectors stems from the mismatch between the training objective and the task requirements: optimizing for the text lexical distribution rather than for the machine-generated text detection task itself. To address this issue, we propose Direct Difference Learning, a novel optimization strategy that leverages task-oriented knowledge to directly optimize the scoring model.

[0036] Our proposed method enables the scoring model to better learn the core semantics of the detection task during the optimization process, thereby enhancing the robustness and generalization ability of the detection.

[0037] Meanwhile, given the shortcomings of existing machine-generated text detection benchmark datasets, such as insufficient diversity of source models, insufficient diversity of text domains, or incomplete coverage of detection tasks, we constructed a large-scale and highly diverse multi-task machine-generated text detection benchmark dataset. Experiments on this dataset revealed the limitations of existing methods in complex environments, while the model trained using our proposed optimization strategy demonstrated robust and general detection performance, achieving state-of-the-art performance on all three detection tasks.

[0038] This embodiment discloses a machine-generated text detection method based on direct difference learning. The method described in this embodiment is an improvement on Fast-DetectGPT, determining whether text is machine-generated by measuring the conditional probability difference between the original text and its resampled version. Overall, as... Figure 1 As shown, the method in this embodiment achieves high efficiency, strong robustness, and excellent generalization ability through three steps: "sampling perturbation," "difference calculation," and "reference clustering." Specifically: A machine-generated text detection method based on direct difference learning includes the following steps: Sample human text, use a large language model to regenerate or polish and rewrite the human text, construct machine-generated text data, and then obtain pairs of human-written text and machine-generated text data. Based on the constructed data pairs, direct difference learning is used to optimize the scoring model. The optimization objective is to maximize the conditional probability difference of machine-generated text while minimizing the conditional probability difference of human text. Obtain the text to be detected and use the optimized scoring model to calculate the conditional probability difference of the text to be detected; Based on the conditional probability difference, the probability that the text to be detected belongs to machine-generated text is estimated using the reference clustering method.

[0039] Specific examples of large language models include GPT-4o, DeepSeek-R1, and Claude-3.7-sonnet.

[0040] This embodiment proposes a machine-generated text (MGT) detection method based on Direct Discrepancy Learning (DDL), and a large-scale evaluation benchmark dataset, MIRAGE, covering multi-task, multilingual models, multi-text domains, and multilingual styles. Compared with existing technologies, the proposed method has significant advantages in detection robustness, cross-domain generalization ability, and computational efficiency, while the MIRAGE dataset achieves breakthroughs in the comprehensiveness, realism, and diversity of evaluation, filling the gaps in existing benchmarks.

[0041] The method described in this embodiment will be explained in detail below.

[0042] (I) Basic Assumptions This embodiment shares a common assumption with Fast-DetectGPT and DetectGPT: machine-generated text tends to consist of high-probability tokens at each position, while human-written text exhibits greater variability. Although sampling strategies like top-k and top-p introduce some randomness, LLM typically still selects tokens with relatively high probabilities. Therefore, features in the token probability distribution can serve as useful clues to distinguish between machine-generated and human-written text.

[0043] (ii) Differences in conditional probabilities Given text and rating model When using another language model When a disturbance occurs, the probability difference (also known as probability curvature) can be expressed as:

[0044] in yes quilt The perturbated version.

[0045] When calculating this difference, It can be easily invoked, thus enabling efficient computation. .

[0046] However, since the logarithmic probability is calculated using a Markov chain, even a small perturbation requires recalculating the entire chain. Therefore, estimating... The expected value of the logarithmic probability is very complex. Fast-DetectGPT introduces conditional probability, which provides a biased but computationally efficient raw probability estimate:

[0047] By introducing such conditional probabilities, the probability differences mentioned above can be further rewritten as conditional probability differences:

[0048] in:

[0049]

[0050] Based on the assumption, machine-generated text It often has a high logarithmic probability, while its perturbation version This shows a lower log probability. In contrast, human-written text... It typically has a low logarithmic probability. When perturbed... The log probability tends to increase because the perturbation process replaces the model with a more probable substitute word. The words in the text. Therefore, we expect to achieve:

[0051] This inequality forms the basis of machine-generated text detection and also provides interpretability for our direct difference learning.

[0052] (iii) Direct Difference Learning As shown in Equations (3) and (6), the key to improving detector performance lies in increasing the difference in lexical probability distribution between machine-generated text and human-written text in the scoring model. Although ImBD has achieved significant performance improvements by combining Direct Preference Optimization (DPO) to optimize the scoring model, we believe that DPO is not the optimal optimization method for machine-generated text detection tasks.

[0053] This is because DPO implicitly aligns the scoring model with the preferences of another language model, rather than directly optimizing the scoring model to classify machine-generated text and human-written text. Furthermore, this alignment can mislead the scoring model, causing it to shift from learning knowledge for the machine-generated text detection task to learning the language style preferences of the training set data, thus impairing the detector's generalization performance.

[0054] When classifying text as either machine-generated or human-written, we rely on the difference in conditional probabilities output by the scoring model. We believe that ImBD's performance improvement using DPO stems from learning the distribution of the source-specific model's output. This implicitly moves towards the goal of reducing the conditional probability difference in human-written text and increasing the conditional probability difference in machine-generated text. However, this implicit optimization prevents the scoring model from directly learning task-oriented knowledge; that is, this optimization objective tells the scoring model that we need to learn a text distribution, but it doesn't tell the model that we need to distinguish between the two types of text.

[0055] Direct difference learning explicitly models the objective of machine-generated text detection: maximizing the gap between the conditional probability difference values ​​of machine-generated text and human-written text. We achieve this objective by maximizing the conditional probability difference of machine-generated text while minimizing the absolute value of the conditional probability difference of human-written text.

[0056] Specifically, the optimization objective of direct difference learning is as follows:

[0057] in Representing humanity in writing texts. This represents machine-generated text.

[0058] In practice, we set This allows the resampling step to be combined with the conditional probability difference calculation step into a single step, meaning that the word vectors corresponding to each word position predicted by the scoring model are directly used as resamples. These are manually set hyperparameters used to assign a target value to the conditional probability difference of machine-generated text. This prevents the direct difference from increasing indefinitely during gradient descent, which could lead to an underweighted proportion of the conditional probability difference of human-written text in the objective function, causing the model to ignore feature learning from human-written text. Our subsequent experiments demonstrate that direct difference learning is crucial for… The value of is not sensitive; in practice, we recommend setting . It is a sufficiently large positive number.

[0059] Direct difference learning incorporates prior knowledge relevant to machine-generated text detection tasks, namely, that the conditional probability difference for human-written text approaches zero, while the conditional probability difference for machine-generated text is positive. Because this prior knowledge is artificially introduced, direct difference learning helps scoring models learn the knowledge required for machine-generated text detection tasks more easily and directly.

[0060] like Figure 2 As shown, the scoring model receives pairs of human-written text and machine-generated text data and calculates the conditional probability difference for each pair. We optimize the scoring model by minimizing the conditional probability difference for human-written text while maximizing the conditional probability difference for machine-generated text.

[0061] (iv) Reference Clustering We use the reference clustering method to convert the conditional probability differences output by the scoring model into the probability that the model judges the text as machine-generated text in practical applications. The reference clustering method aims to estimate the probability that a given value belongs to a specific distribution, and includes two stages: data aggregation and probability estimation.

[0062] In the data aggregation phase, we first collect a certain amount of machine-generated text as a reference dataset. And collect approximately the same amount of human-written text as a reference dataset. Then, we use the scoring model that will be used for detection. Calculate separately and Difference in conditional probability between each text segment Therefore, we can obtain and Chinese text Conditional probability difference distribution under rating and .

[0063] In the probability estimation stage, we choose Mid-to-target value The closest Each value is used as a search window. :

[0064] in This is a hyperparameter that is determined by the size of the reference dataset. For a large reference dataset, The bigger the better, because it can provide higher performance. Precision.

[0065] Then, we count the number of MGT text and HWT text within the window:

[0066]

[0067] Finally, we use local statistical ratios to estimate the probability that text x belongs to the MGT. :

[0068] Due to the window The method is adaptively determined based on the data distribution, thus maintaining stability under different data densities and improving the robustness of real-world machine-generated text detection.

[0069] like Figure 3 As shown, our method (denoted as DetectAnyLLM) and previous state-of-the-art methods (including Fast-DetectGPT and ImBD) were tested on the created dataset MIRAGE-DIG. The performance improvement was calculated using the following formula: (new-old) / (1.0-old). The method described in this embodiment demonstrates a leading advantage in AUROC, ACC, MCC, and TPR@5% metrics.

[0070] (v) A diversified benchmark for machine-generated text detection and evaluation To facilitate further research, we have constructed a multi-task machine-generated text detection benchmark dataset that provides a rich variety of commercial LLM types and comprehensive text domains.

[0071] Previous benchmarks have shown significant limitations in terms of text domain diversity, source LLM coverage, and evaluation task diversity. These limitations hinder comprehensive and integrated evaluation and result in poor generalization ability of detection models outside of specific benchmark scenarios.

[0072] To provide more robust evaluations that reflect real-world applications, we developed the MIRAGE benchmark dataset. Overall, the MIRAGE benchmark dataset provided in this embodiment is a large-scale, multi-task machine-generated text detection evaluation dataset that covers diverse generative models, language styles, and text domains. Its features include: a. The dataset contains 10 original corpora sampled from five text domains: news, academic, commentary, email, and web pages; b. The text was generated or polished using 17 mainstream large language models (including 13 closed-source commercial models and 4 open-source models); c. The dataset covers three task types: Generate, Polish, and Rewrite, with each task including pairing machine-generated text with original human text; d. The texts are uniformly processed in terms of sampling, task allocation, length control, and language style diversity to ensure consistency and practical applicability in training and evaluation.

[0073] The aforementioned language styles include 16 styles such as formal, colloquial, academic, literary, critical, narrative, descriptive, lyrical, objective, subjective, original, casual, explanatory, argumentative, journalistic, and poetic. The language style is generated by controlling the prompt words.

[0074] Our invented MIRAGE benchmark is the most comprehensive multi-task machine-generated text detection and evaluation framework to date, integrating cross-domain generative and revised text and employing state-of-the-art LLMs, including 13 proprietary commercial LLMs and 4 advanced open-source LLMs.

[0075] (vi) Dataset Construction Our invented MIRAGE benchmark dataset takes into account the different performance of LLM in different text domains. It extracts human-written English text samples from 10 corpora, including five domains: news, email, comments, academic, and web pages.

[0076] Following the methods established in previous commonly used test sets, we designed three different machine-generated text detection tasks: generation (a paragraph entirely generated by machine), polishing, and rewriting (machines making minor improvements to human-written text). Specifically: 1. The generation task involves creating new text based on the first 30 morphemes of human-written text.

[0077] 2. The polishing task refines existing human-written texts while preserving their details and meaning.

[0078] 3. A rewriting task involves reconstructing a given human-written text without altering its meaning or essential details.

[0079] In real-world scenarios, people typically rely on powerful commercial closed-source LLMs to generate or modify text. However, most existing benchmarks, such as MAGE, RAID, and DetectRL, simply depend on open-source LLMs or less capable early commercial LLMs to create data, resulting in a gap between current benchmarks and real-world evaluations.

[0080] To address this issue, our MIRAGE benchmark integrates 13 mainstream commercial LLMs for text generation or revision. Furthermore, given the increasing deployment of high-performance open-source models in localized applications, we have integrated four advanced open-source LLMs to ensure comprehensive coverage of the contemporary LLM ecosystem.

[0081] We removed the "\n" character from each data entry to prevent the detector from recognizing machine-generated text based on the presence of the "\n" symbol. We then tokenized the text based on spaces and filtered the dataset for text containing 100-200 words to control for length-based detection bias.

[0082] We consider two different evaluation scenarios to reflect real-world applications: 1. Disjoint-Input Generation (DIG): This evaluates the detector's ability to distinguish between original human-written text and corresponding machine-generated or machine-revised text generated by a single LLM.

[0083] 2. Shared-Input Generation (SIG): This evaluates the detector's ability to simultaneously distinguish between multiple machine-generated texts or machine-revised texts (generated by different LLMs) and homologous human-written texts.

[0084] We designed each LLM to generate 2,000 samples for each machine-generated text task, evenly distributed between the DIG and SIG scenarios (1,000 samples per scenario). To ensure the balance of text fields, both DIG and SIG maintain the same domain distribution rate.

[0085] Sampling begins with constructing a domain-level HWT dataset by proportionally merging source datasets within each domain. This dataset blending strategy mitigates dataset bias by preventing oversampling from a single dataset. In implementation, the SIG is treated as a separate “model” and participates in the sampling process along with 17 independent LLMs. For each model (including the SIG), data is sampled sequentially from each domain dataset. After sampling is complete, data items are removed from the corresponding domain dataset to maintain the distinction between the DIG and SIG data. Within each text domain, data is continuously sampled until the sample size for that domain reaches the requirement specified in the supplementary materials. Once data sampling for one text domain is complete, the process moves on to the next text domain, repeating until all text domains have been sampled.

[0086] This method generates a DIG dataset and a comprehensive SIG dataset for each LLM, and then combines them to form a complete sample set for each LLM across all tasks.

[0087] For data augmentation, we consider language style to be a key distinguishing feature between human-written and machine-generated text, making text detection with human language style a more challenging task for detectors. Therefore, we introduce data augmentation based on the language style of LLM. Specifically, we augment the phrase "in a <style>style”融入到输入提示中。我们手动选择16种不同的语言风格，并在每次 LLM 推理过程中随机选择一种，以实现风格多样性。

[0088] 这种方法有助于评估检测器对不同语言风格内容的鲁棒性。此外，在实践中，使用LLM辅助写作时，通常会在提示中添加语言风格约束。因此，这进一步增强了我们所发明的基准测试模拟真实场景的能力。

[0089] 从上述人类书写文本数据生成机器生成或润色文本后，我们对生成的数据进行数据清理。首先，删除所有"\n”和"\r”的实例，以防止通过符号特征进行检测。接下来，我们删除少于90个单词或超过220个单词的文本，以防止文本长度变化对检测的影响。

[0090] 通过以上这些步骤，我们实现了对我们发明的MIRAGE基准数据集的构建。如图4、图5和图6 所示，为MIRAGE 基准数据集构建流程图。图4中展示了采样人类文本流程图，图5中展示了数据生成流程示意图，图6中展示了其他细节。MIRAGE包含超过93,000个人类书写文本—机器生成文本对，显著展示了文本域、源LLM和生成任务的多样性，同时使用写作风格控制作为数据增强。如图7所示，我们的 MIRAGE 基准测试强调跨领域、任务、评估场景和源LLM的多样性，从而实现了全面而稳健的评估。

[0091] （七）评估指标与先前的研究一致，我们采用受试者工作特征曲线下面积 (AUROC)作为主要评估指标。为了评估特定阈值下的性能，我们进一步结合了5%假阳性率下的TPR(TPR@5%)作为补充指标。此外，考虑到MIRAGE-SIG是一个类别不平衡的数据集，我们还报告了Matthews相关系数 (MCC)和平衡准确率，以提供更全面的评估。

[0092] 综合起来，这套多样化的指标可以全面评估检测器的性能，确保评估结果既能反映理论的完整性，又能反映实际应用。

[0093] 本发明提出的机器生成文本检测方法及其核心优化策略直接差异学习（DDL）具有显著的经济应用价值：在教育与科研诚信领域，该技术可广泛用于自动化检测学生作业、科研论文、新闻稿件等是否由大语言模型（LLM）生成，从而有效遏制学术不端行为，维护知识生产的真实性，降低人工审核成本，提升审查效率。在数字内容审核与安全监管场景中，本发明可为政府、媒体平台和企业提供高鲁棒性与强泛化能力的检测工具，帮助甄别虚假信息与伪造内容，提升内容合规审核效率，降低平台法律风险与信任成本。

[0094] 此外，使用我们提出的直接差异学习优化得到的评分模型只需少量简单的样本进行训练便可支持跨领域、跨任务、跨模型的通用检测，特别适用于对复杂、多源、大规模文本内容的自动甄别与筛查任务，在司法鉴定、法律证据评估、金融文本合规审计等领域也具备极高应用价值。

[0095] 本发明构建的MIRAGE基准数据集进一步推动了行业检测标准的构建，有望引领检测技术的产业化落地。

[0096] 综合而言，本发明显著提升了机器生成内容的可识别性与监管能力，为人工智能安全、数字版权保护、教育诚信和媒体真实性等关键领域提供了高效、可靠、可扩展的技术支持，具有广阔的市场前景和显著的经济社会效益。

[0097] 实施例二本实施例公开了基于直接差异学习的机器生成文本检测系统。

[0098] 基于直接差异学习的机器生成文本检测系统，包括：数据获取模块，被配置为：采样人类文本，使用大语言模型对人类文本进行重生成或润色、重写，构建机器生成文本数据，进而得到人类书写文本和机器生成文本数据对；模型优化模块，被配置为：基于构建的数据对，采用直接差异学习对评分模型进行优化，优化目标为最大化机器生成文本的条件概率差异，同时最小化人类文本的条件概率差异；条件概率差异计算模块，被配置为：获取待检测文本，使用优化后的评分模型计算待检测文本的条件概率差异；分类模块，被配置为：基于条件概率差异，采用参考聚类方法估算出待检测文本属于机器生成文本的概率。实施例三本实施例的目的是提供计算机可读存储介质。

[0099] 计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本公开实施例1所述的基于直接差异学习的机器生成文本检测方法中的步骤。

[0100] 实施例四本实施例的目的是提供电子设备。

[0101] 电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的程序，所述处理器执行所述程序时实现如本公开实施例1所述的基于直接差异学习的机器生成文本检测方法中的步骤。

[0102] 以上实施例二、三和四的装置中涉及的各步骤与方法实施例一相对应，具体实施方式可参见实施例一的相关说明部分。术语"计算机可读存储介质”应该理解为包括一个或多个指令集的单个介质或多个介质；还应当被理解为包括任何介质，所述任何介质能够存储、编码或承载用于由处理器执行的指令集并使处理器执行本发明中的任一方法。

[0103] 本领域技术人员应该明白，上述本发明的各模块或各步骤可以用通用的计算机装置来实现，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。本发明不限制于任何特定的硬件和软件的结合。

[0104] 上述虽然结合附图对本发明的具体实施方式进行了描述，但并非对本发明保护范围的限制，所属领域技术人员应该明白，在本发明的技术方案的基础上，本领域技术人员不需要付出创造性劳动即可做出的各种修改或变形仍在本发明的保护范围以内。< / style>

Claims

1. A machine-generated text detection method based on direct difference learning, characterized in that, Includes the following steps: Sample human text, use a large language model to regenerate or polish and rewrite the human text, construct machine-generated text data, and then obtain pairs of human-written text and machine-generated text data. Based on the constructed data pairs, direct difference learning is used to optimize the scoring model. The optimization objective is to maximize the conditional probability difference of machine-generated text while minimizing the conditional probability difference of human text. Obtain the text to be detected and use the optimized scoring model to calculate the conditional probability difference of the text to be detected; Based on the conditional probability difference, the probability that the text to be detected belongs to machine-generated text is estimated using the reference clustering method.

2. The machine-generated text detection method based on direct difference learning as described in claim 1, characterized in that, The conditional probability difference used is expressed as follows: ； ； ； in, The original text to be detected; For the scoring model; It is a language model used to generate resampled versions; yes through The version obtained by resampling; For conditional probability differences; This represents the expected value.

3. The machine-generated text detection method based on direct difference learning as described in claim 1, characterized in that, When optimizing the scoring model using direct difference learning: The reward for machine-generated text is the negative of the L1 distance between its conditional probability difference and the hyperparameter γ, which controls the desired conditional probability difference value of the machine-generated text. The reward for human text is the negative of the L1 norm of its conditional probability difference.

4. The machine-generated text detection method based on direct difference learning as described in claim 2, characterized in that, The optimization objective is expressed as: ； in, Representing humanity in writing texts. Represents machine-generated text; It's a hyperparameter.

5. The machine-generated text detection method based on direct difference learning as described in claim 1, characterized in that, The probability that the text to be detected belongs to machine-generated text is estimated using a reference clustering method, specifically including: Obtain the same number of machine-generated texts and human text ; Calculate using the scoring model respectively and The conditional probability differences of each text segment were obtained respectively. and Set of conditional probability difference distributions of Chinese text and ; The conditional probability difference value of the text to be detected is calculated using a scoring model, and this conditional probability difference value is used as a baseline value. and Select the k closest reference samples from the list; Using the reference sample as the neighborhood, within the neighborhood of the baseline value, the number of machine-generated texts and human texts is counted, and the probability that the original text to be detected is machine-generated is estimated based on the ratio.

6. The machine-generated text detection method based on direct difference learning as described in claim 5, characterized in that, The probability that the original text to be detected was generated by a machine is estimated based on the ratio, specifically: in, This indicates the probability that the text to be detected was generated by a machine; This indicates the number of machine-generated texts within the statistics window range; This indicates the number of human texts within the statistical window.

7. The machine-generated text detection method based on direct difference learning as described in claim 1, characterized in that, The machine-generated text data includes text generated by the machine and text modified by the machine.

8. A machine-generated text detection system based on direct difference learning, characterized in that, include: The data acquisition module is configured to: sample human text, use a large language model to regenerate, polish, or rewrite the human text, construct machine-generated text data, and thus obtain pairs of human-written text and machine-generated text data. The model optimization module is configured to optimize the scoring model using direct difference learning based on the constructed data pairs, with the optimization objective being to maximize the conditional probability difference of machine-generated text while minimizing the conditional probability difference of human text. The conditional probability difference calculation module is configured to: acquire the text to be detected and calculate the conditional probability difference of the text to be detected using the optimized scoring model; The classification module is configured to estimate the probability that the text to be detected belongs to machine-generated text based on the conditional probability difference using a reference clustering method.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the machine-generated text detection method based on direct difference learning as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the machine-generated text detection method based on direct difference learning as described in any one of claims 1-7.

Images