Large model generation text detection method based on multi-agent cooperation

Abstract

The invention discloses a large model generation text detection method based on multi-agent cooperation. The method comprises the following steps: 1) input text processing: carrying out differential disturbance processing on an input text through three heterogeneous agents; 2) semantic similarity evaluation: performing similarity scoring on an original text and a modified text generated by each agent by using a scoring method BARTScore based on a sequence-to-sequence pre-training model, and normalizing to an interval of [0, 1]; 3) threshold value judgment: determining the optimal threshold value of each agent through the Youden's J statistical magnitude; and 4) a voting decision mechanism: performing final classification judgment by adopting a majority voting mechanism according to judgment results of the three agents. The method does not need to depend on annotated data and field fine tuning, through multi-dimensional feature decoupling and a dynamic adaptive strategy, the detection accuracy and cross-field robustness in a zero sample scene are remarkably improved, and the method is suitable for practical scenes such as educational evaluation and content auditing.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a large model-based text detection method based on multi-agent collaboration. Background Art

[0002] In recent years, large language models (LLMs) have demonstrated remarkable capabilities in text generation, with the fluency and coherence of their generated text approaching human-level performance. However, this technological breakthrough has also posed significant challenges in areas such as content security, academic integrity, and copyright protection. It is difficult for ordinary readers to effectively distinguish text generated by large models from human-generated content based solely on subjective judgment. Therefore, building an efficient and reliable system for detecting text generated by large models is an urgent need.

[0003] Existing detection methods fall into two main categories: Within the supervised learning paradigm, classification is achieved by fine-tuning pre-trained models or building deep neural networks (such as TextCNN). However, these methods have two key limitations: they rely heavily on large-scale annotated data, resulting in significant performance degradation in cross-domain and cross-task scenarios; and they are insufficiently adaptable to different large models. When the detection target is changed from GPT-3.5 to Claude-3.5, the model accuracy performance is unstable. While the zero-shot prompt engineering method directly queries LLMs for generation probability by designing discriminant prompts, this reduces reliance on annotated data. However, its performance is highly dependent on manually designed prompt templates, resulting in poor stability in complex semantic scenarios. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a large-scale model-generated text detection method based on multi-agent collaboration. By constructing heterogeneous agents to perform differential perturbation processing on the text, combining semantic similarity evaluation and threshold judgment mechanism, the ultimate method is to combine integrated decision-making to achieve efficient and reliable detection of generated text in zero-sample scenarios.

[0005] The object of the present invention is achieved as follows: a large model generation text detection method based on multi-agent collaboration comprises the following steps:

[0006] 1) Input text processing: Three heterogeneous agents perform differential perturbation on the input text;

[0007] 2) Semantic similarity assessment: The original text and the modified text generated by each agent are scored using BARTScore, a scoring method based on a sequence-to-sequence pre-trained model, and the similarity is normalized to the range [0, 1].

[0008] 3) Threshold determination: Determine the optimal threshold for each agent using Youden's J statistic;

[0009] 4) Voting decision-making mechanism: Based on the judgment results of the three intelligent agents, a majority voting mechanism is used to make the final classification decision.

[0010] As a further limitation of the present invention, the three heterogeneous agents in step 1) specifically include:

[0011] The text revision agent directly modifies the input text and generates a new version with the same semantics but different expression. The optimization goal of the agent is formalized as:

[0012]

[0013] Among them, s represents the source text, s ′ Indicates that the agent revised the text, represents the candidate text space subject to dynamic constraints of the generated length, S sem (·) is used to measure semantic consistency, D lex (·) is the lexical distribution difference measure, and λ is used to measure the relative importance of semantics and lexical distribution;

[0014] The grammar correction agent performs grammatical correction on the input text and generates a grammatically standardized version. The correction process is modeled as follows:

[0015]

[0016] Where s represents the source text, s″ represents the revised text of the agent, and γ is used to measure the relative importance of grammatical compliance and syntactic structure changes. represents the candidate text space subject to dynamic constraints of the generated length, L syn (·) Calculate the grammatical compliance loss based on the dependency parser, C struct (·) Using tree edit distance to quantify syntactic structure changes;

[0017] The feature reconstruction agent learns the features of the input text and regenerates new text. Its reconstruction mechanism can be defined as:

[0018] A feature (s)=arg max[α·F core (s,s″″)-β·R surf (s,s″′)] (3)

[0019] Among them, s represents the source text, s″″ represents the revised text of the agent, α represents the semantic feature matching degree, β represents the reconstruction difference, and F core (·) is the matching degree of core semantic features, R surf (·) Reconstruct differences for surface forms.

[0020] As a further limitation of the present invention, the step 2) specifically includes:

[0021] Step 2.1) For source text x i With the revised text BARTScore is used as the core scoring mechanism to measure the semantic coverage and generation quality between the source and target texts.

[0022] Step 2.2) The BART model is parameterized by θ and is calculated by i After the given revision The log probability of x as the source text; the target text is tokenized into a series of tokens: i ={x i1 ,x i2 ,…,x ik}, where k is the sequence length;

[0023] Step 2.3) For each label x i The BART model calculates the token generation probability and takes the logarithm of all token generation probabilities to sum up to get the BARTScore. Based on the score, the similarity between the source text and the target text is judged. The calculation formula of BARTScore is:

[0024]

[0025] Among them, θ is the weight and bias of the BART model, t is the generation order of each word in the target text, and j represents the context dependency in the generation process;

[0026] Step 2.4) Normalize the scores to the interval [0, 1] and use the exit function to process the similarity scores:

[0027]

[0028] Among them, s expit is the final similarity score, σ is the expit function;

[0029] As a further limitation of the present invention, the step 3) specifically includes:

[0030] Step 3.1) Use Youden's J statistic to determine the optimal threshold τ for each agent * ; through the threshold τ * Achieve optimal performance in the classifier to distinguish between human-generated text and text generated by large models;

[0031] Step 3.2) Calculate the receiver operating characteristic curve of the scoring results. For each agent n∈{revise,correct,feature}, according to the scoring results generated by each single agent Receiver operating characteristic curves were calculated;

[0032]

[0033] Among them, TPR represents the true positive rate, that is, the classifier correctly classifies and is itself a positive example, FPR represents the false positive rate, the classifier incorrectly classifies itself as a negative example, TP represents the predicted positive and the actual positive, FN represents the predicted negative and the actual negative, FP represents the predicted positive and the actual negative, TN represents the predicted negative and the actual negative, and τ is the classification threshold, which controls the strictness of the model's judgment on positive examples.

[0034] Step 3.3) Calculate Youden's J statistic t, which is defined as:

[0035] J(τ)=TPR(τ)-FPR(τ) (8)

[0036] Step 3.4) Determine the optimal threshold, this optimal threshold τ * To maximize the value of Youden's J statistic:

[0037]

[0038] As a further limitation of the present invention, the step 4) specifically includes:

[0039] Step 4.1) For the i-th text sample, each agent The judgment result is a binary classification value Where 0 indicates that the text is judged to be human text, and 1 indicates that the text is judged to be generated by a large model;

[0040] Step 4.2) For each agent The judgment result Based on ratings and the optimal threshold τ * calculate:

[0041]

[0042] Step 4.3) The judgment results of the three agents are finally judged and the result is obtained. Calculated by the following formula:

[0043]

[0044] Here, n represents different categories of agents.

[0045] The present invention adopts the above technical solution, and compared with the existing technology, the beneficial effects are: by constructing a heterogeneous collaborative framework of text revision agents, grammar correction agents and feature reconstruction agents, the text is differentially disturbed from three dimensions: semantic consistency, grammatical standardization and logical coherence, breaking through the traditional method's dependence on labeled data; on the other hand, it innovatively combines BARTScore semantic similarity evaluation and adopts a majority voting strategy to integrate multi-agent judgment results, significantly improving detection robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The overall framework diagram of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 The method for generating text detection based on a large model of multi-agent collaboration is shown, comprising the following steps:

[0048] 1) Input text processing: Three heterogeneous agents perform differential perturbation on the input text;

[0049] The text revision agent directly modifies the input text and generates a new version with the same semantics but different expression form, simulating the adjustment of language expression by humans. The optimization goal of this agent can be formalized as:

[0050]

[0051] Among them, s represents the source text, s ′ Indicates that the agent revised the text, represents the candidate text space subject to dynamic constraints of the generated length, S sem (·) is used to measure semantic consistency, D lex (·) is the lexical distribution difference measure, and λ is used to measure the relative importance of semantics and lexical distribution;

[0052] The grammar correction agent performs grammatical correction on the input text and generates a grammatically standardized version. The correction process can be modeled as follows:

[0053]

[0054] Where s represents the source text, s″ represents the revised text of the agent, and γ is used to measure the relative importance of grammatical compliance and syntactic structure changes. represents the candidate text space subject to dynamic constraints of the generated length, L syn (·) Calculate the grammatical compliance loss based on the dependency parser, C struct (·) Using tree edit distance to quantify syntactic structure changes;

[0055] The feature reconstruction agent uses an abstract processing method to extract the key features of the input text, and then regenerates a new text that conforms to human expression habits based on the key features. Its reconstruction mechanism can be defined as:

[0056] A feature (s)=arg max[α·F core (s,s″″)-β·R surf (s,s″′)] (3)

[0057] Among them, s represents the source text, s″″ represents the revised text of the agent, α represents the semantic feature matching degree, β represents the reconstruction difference, and F core (·) is the matching degree of core semantic features, R surf (·) is the surface form reconstruction difference. The following is an example of generating human text in the reddit_dli5 dataset:

[0058]

[0059] 2) Semantic similarity evaluation: The original text and the modified text generated by each agent are scored using BARTScore, a scoring method based on the sequence-to-sequence pre-training model (BART model), and the similarity is normalized to the range [0, 1].

[0060] Step 2.1) For source text x i With the revised text BARTScore is used as the core scoring mechanism to measure the semantic coverage and generation quality between the source and target texts.

[0061] Step 2.2) The BART model is parameterized by θ and is calculated by i After the given revision The log probability of x as the source text; the target text is tokenized into a series of tokens: i ={x i1 ,x i2 ,…,x ik}, where k is the sequence length;

[0062] Step 2.3) For each label x i The BART model calculates the token generation probability and takes the logarithm of all token generation probabilities to sum up and obtain the BARTScore. Based on the score, the similarity between the source text and the target text is judged. The higher the score, the higher the similarity between the source text and the target text, indicating that the text is more likely to conform to the habits or logic of human writing. The calculation formula of BARTScore is:

[0063]

[0064] Among them, θ is the weight and bias of the BART model, t is the generation order of each word in the target text, and j is the context dependency relationship in the generation process;

[0065] Step 2.4) Normalize the scores to the interval [0, 1] and use the exit function to process the similarity scores:

[0066]

[0067] Among them, s expit is the final similarity score, and σ is the exit function.

[0068] 3) Threshold determination: Determine the optimal threshold for each agent using Youden's J statistic;

[0069] Step 3.1) Use Youden's J statistic to determine the optimal threshold τ for each agent * ; through the threshold τ * It enables the classifier to achieve optimal performance and distinguish between human-generated text and text generated by large models. It also sets a threshold to make the classification results as accurate as possible. If the threshold is too high, many positive samples may be missed, while if the threshold is too low, negative samples may be mistakenly classified as positive.

[0070] Step 3.2) Calculate the receiver operating characteristic curve (ROC curve) of the scoring results. For each agent n∈{revise,correct,feature}, the scoring results generated by each single agent are Calculate the receiver operating characteristic curve (ROC curve);

[0071]

[0072] Among them, TPR represents the true positive rate, that is, the classifier correctly classifies and is itself a positive example, FPR represents the false positive rate, the classifier incorrectly classifies itself as a negative example, TP represents the predicted positive, the actual positive, FN represents the predicted negative, the actual negative, FP represents the predicted positive, the actual negative, TN represents the predicted negative, the actual negative, τ is the classification threshold, which controls the strictness of the model's judgment on positive examples;

[0073] Step 3.3) Calculate Youden's J statistic, which is defined as:

[0074] J(τ)=TPR(τ)-FPR(τ) (8)

[0075] Step 3.4) Determine the optimal threshold, this optimal threshold τ * To maximize the value of Youden's J statistic:

[0076]

[0077] 4) Voting decision-making mechanism: Based on the judgment results of the three agents, a majority voting mechanism is used to make the final classification decision;

[0078] Step 4.1) For the i-th text sample, each agent The judgment result is a binary classification value Where 0 indicates that the text is judged to be human text, and 1 indicates that the text is judged to be generated by a large model;

[0079] Step 4.2) For each agent The judgment result Based on ratings and the optimal threshold τ * calculate:

[0080]

[0081] Step 4.3) The judgment results of the three agents are finally judged and the result is obtained. Calculated by the following formula:

[0082]

[0083] Here, n represents different categories of agents.

[0084] The sources of error for different agents may be different. Majority voting can offset the errors of individual agents by integrating the results of multiple agents, and the accuracy of the integration is usually better than the average performance of a single agent.

[0085] In order to test the effectiveness of the present invention, experiments were conducted on 11 datasets, including the HC3-English dataset (Finance, Open_qa, Medicine, wiki_csai, reddit_dli5), the XSum dataset, the PubMedQA dataset, etc. Among them, 1,000 texts generated by humans and large models were extracted from the Finance and reddit_dli5 fields, and all data were used in the remaining fields. In the main experiment, gpt-3.5-turbo was used as the core model, and the generated text was judged in combination with a multi-agent voting mechanism. The experimental results show (as shown in Table 1) that the method of the present invention performs well on most datasets, and its voting accuracy is significantly better than other baseline models (such as SDA, TextCNN, Bert, etc.). For example, on dataset A, the integrated prediction (voting) accuracy of the method of the present invention reached 91.50%, far exceeding baseline experiments such as SDA, TextCNN, and Bert.

[0086] In the Finance and reddit_dli5 fields of the HC3-English dataset, the accuracy of the proposed method (93.55% and 99.30%) is close to that of RoBERTa-MPU after fine-tuning on the HC3-English dataset (99.05% and 99.94% after fine-tuning), and has the following advantages: no need for fine-tuning for specific fields, reducing deployment costs; and maintaining high accuracy when labeled data is scarce.

[0087] Table 1 Accuracy of experimental results

[0088]

[0089] This paper proposes a large-scale model-generated text detection method based on multi-agent collaboration. By constructing a heterogeneous collaborative framework consisting of text revision agents, grammar correction agents, and feature reconstruction agents, this method performs differential perturbations on text along three dimensions: semantic consistency, grammatical standardization, and logical coherence. This method overcomes the traditional method's reliance on annotated data. Furthermore, it innovatively incorporates BARTScore semantic similarity assessment and employs a majority voting strategy to integrate multi-agent judgment results, significantly improving detection robustness.

[0090] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solutions disclosed in the present invention, those skilled in the art can make some substitutions and modifications to some of the technical features therein according to the disclosed technical content without creative labor, and these substitutions and modifications are all within the protection scope of the present invention.

Claims

1. A large model-generated text detection method based on multi-agent collaboration, characterized in that: The following steps are involved: 1) Input text processing: Differentiated perturbations are performed on the input text by three heterogeneous agents; 2) Semantic similarity assessment: The original text and the modified text generated by each agent are scored using BARTScore, a scoring method based on a sequence-to-sequence pre-trained model, and the similarity is normalized to the range [0, 1]. 3) Threshold determination: Determine the optimal threshold for each agent using Youden's J statistic; 4) Voting decision-making mechanism: Based on the judgment results of the three intelligent agents, a majority voting mechanism is used to make the final classification decision.

2. A large model generated text detection method based on multi-agent collaboration according to claim 1, characterized in that: The three heterogeneous agents described in step 1) specifically include: The text revision agent directly modifies the input text and generates a new version with the same semantics but different expression. The optimization goal of the agent is formalized as: Among them, s represents the source text, s ′ Indicates that the agent revised the text, represents the candidate text space subject to dynamic constraints of generation length, S sem (·) is used to measure semantic consistency, D lex (·) is the lexical distribution difference measure, and λ is used to measure the relative importance of semantics and lexical distribution; The grammar correction agent performs grammatical correction on the input text and generates a grammatically standardized version. The correction process is modeled as follows: Where s represents the source text, s″ represents the revised text of the agent, and γ is used to measure the relative importance of grammatical compliance and syntactic structure changes. represents the candidate text space subject to dynamic constraints of the generated length, L syn (·) Compute the grammatical compliance loss based on the dependency parser, C struct (·) Using tree edit distance to quantify syntactic structure changes; The feature reconstruction agent learns the features of the input text and regenerates new text. Its reconstruction mechanism can be defined as: A feature (s)=arg max[α·F core (s,s″″)-β·R surf (s,s″′)] (3) Among them, s represents the source text, s″″ represents the revised text of the agent, α represents the semantic feature matching degree, β represents the reconstruction difference, and F core (·) is the matching degree of core semantic features, R surf (·) Reconstruct the difference for surface form.

3. The method for detecting text generated by a large model based on multi-agent collaboration according to claim 1, characterized in that: The step 2) specifically includes: Step 2.1) For source text x i With the revised text BARTScore is used as the core scoring mechanism to measure the semantic coverage and generation quality between the source and target texts. Step 2.2) The BART model is parameterized by θ and is calculated by i After the given revision The log probability of x as the source text; the target text is tokenized into a series of tokens: i ={x i1 ,x i2 ,…,x ik }, where k is the sequence length; Step 2.3) For each label x i The BART model calculates the token generation probability and takes the logarithm of all token generation probabilities to sum up to get the BARTScore. Based on the score, the similarity between the source text and the target text is judged. The calculation formula of BARTScore is: Among them, θ is the weight and bias of the BART model, t is the generation order of each word in the target text, and j represents the context dependency in the generation process; Step 2.4) Normalize the scores to the interval [0, 1] and use the exit function to process the similarity scores: Among them, s expit is the final similarity score, and σ is the exit function.

4. The method for detecting text generated by a large model based on multi-agent collaboration according to claim 1, characterized in that: The step 3) specifically includes: Step 3.1) Use Youden's J statistic to determine the optimal threshold τ for each agent * ; through the threshold τ * Achieve optimal performance in the classifier to distinguish between human-generated text and text generated by large models; Step 3.2) Calculate the receiver operating characteristic curve of the scoring results. For each agent n∈{revise,correct,feature}, according to the scoring results generated by each single agent Receiver operating characteristic curves were calculated; Among them, TPR represents the true positive rate, that is, the classifier correctly classifies and is itself a positive example, FPR represents the false positive rate, the classifier incorrectly classifies itself as a negative example, TP represents the predicted positive, the actual positive, FN represents the predicted negative, the actual negative, FP represents the predicted positive, the actual negative, TN represents the predicted negative, the actual negative, τ is the classification threshold, which controls the strictness of the model's judgment on positive examples; Step 3.3) Calculate Youden's J statistic, which is defined as: J(τ)=TPR(τ)-FPR(τ) (8) Step 3.4) Determine the optimal threshold, this optimal threshold τ * To maximize the value of Youden's J statistic:

5. The method for detecting text generated by a large model based on multi-agent collaboration according to claim 1 is characterized in that: The step 4) specifically includes: Step 4.1) For the i-th text sample, each agent The judgment result is a binary classification value 0 indicates that the text is human-generated, and 1 indicates that the text is generated by a large model. Step 4.2) For each agent The judgment result Based on ratings and the optimal threshold τ * calculate: Step 4.3) The judgment results of the three agents are finally judged and the result is obtained. Calculated by the following formula: Here, n represents different categories of agents.

Images