An AI-generated text detection method, system, device and medium

Abstract

The application discloses an AI generated text detection method, system, device and medium, and relates to the technical field of text processing. The application converts a Chinese text sentence to be detected into a tone category sequence composed of tone categories. In this process, the categories of Chinese tones are used as a quantitative index of phonological structure, and text analysis is converted from a high-dimensional and high-cost semantic space to a low-dimensional and high-efficiency phonological feature space. The application is not dependent on training data of a specific model and is adaptable to text detection of multiple models. Then, N-gram analysis is performed on the generated tone category sequence, and the occurrence frequency of each N-gram combination is calculated to form a feature vector. Finally, an AI generated text is judged through a pre-trained classification model. In this process, the calculation type in processing is mainly string processing and frequency statistics, and the inconsistency problem of the model in multiple model invocations is avoided, so that the AI generated text can be accurately detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text processing, and in particular to an AI-generated text detection method, system, device and medium. BACKGROUND

[0002] With the rapid development of large language models (LLMs), the quality and efficiency of artificially intelligent generated text have been significantly improved, but the risk of its abuse has also become increasingly prominent, such as potential harm in scenarios of false information dissemination, academic misconduct, etc. Therefore, AI-generated content detection technology has become a hot topic of research and application. Currently, the detection technology for AI-generated content is mainly divided into training classifiers and zero-shot classifiers.

[0003] The training classifier is trained based on text data generated by a specific AI model (such as ChatGPT), and a classification model is constructed by analyzing the semantics, vocabulary and statistical characteristics of the text. However, this method highly depends on the source of training data, resulting in insufficient generalization ability of the model. When detecting text generated by other models, the performance significantly decreases, making it difficult to adapt to the actual environment where multiple models coexist. The zero-shot classifier does not require specific training data, but relies on the source language model for real-time probability calculation and perturbation analysis. This approach requires multiple calls to large language model APIs, but there are differences in model structures when calling large language models multiple times, resulting in unsatisfactory detection effect of AI-generated text.

[0004] Therefore, the current training classifier has poor performance when detecting text generated by other models, making it difficult to adapt to the actual environment where multiple models coexist. The zero-shot classifier needs to call large language model APIs multiple times, resulting in poor detection effect of AI-generated text. Therefore, the current detection method is difficult to accurately detect AI-generated text. SUMMARY

[0005] The present application provides an AI-generated text detection method, system, device and medium, which can solve the problems in the prior art.

[0006] The present application provides an AI-generated text detection method, comprising the following steps:

[0007] Obtaining a Chinese text sentence to be detected, and a Chinese text sentence of a preset standard and an AI-generated Chinese text sentence;

[0008] For Chinese text sentences to be detected, based on preset standards and AI-generated text sentences, the pinyin and tone markings of each Chinese character in the Chinese text sentence are obtained. Based on preset tone category mapping rules, the tone markings of each Chinese character are mapped to a tone category, so as to transform the Chinese text sentence into a tone category sequence composed of tone categories.

[0009] Based on tone category sequences, an N-gram model covering orders 1 to N is constructed, and all N-gram combinations within the tone category sequences are obtained. The occurrence frequency of each N-gram combination in the tone category sequences is counted, and the relative frequency of each N-gram combination is calculated. The frequency values ​​of all N-gram combinations are arranged to obtain feature vectors representing the phonological rhythm patterns of Chinese text sentences to be detected, preset standards, and AI-generated sentences.

[0010] Based on the feature vectors representing the phonetic rhythm patterns of the Chinese text sentences to be detected, the preset standards, and the AI-generated Chinese text sentences, the probability values ​​of the detected Chinese text sentences are obtained, and the probability values ​​are compared with the preset probability thresholds to determine whether the Chinese text sentences to be detected are AI-generated text.

[0011] Preferably, obtaining the tone category sequence includes:

[0012] Use the G2PM library, a pinyin conversion tool, to obtain the pinyin and tone markings for each Chinese character in a Chinese text sentence;

[0013] According to the preset tone category mapping rules, the tone mark of each Chinese character is mapped to a tone category, so as to transform the entire Chinese text sentence into a tone category sequence composed of tone categories;

[0014] Among them, the tone category mapping rule is defined based on the pitch variation pattern of Chinese tones;

[0015] Set the first and second tones to category P, the third and fourth tones to category Z, and the neutral tone to category W.

[0016] Preferably, obtaining the feature vector includes:

[0017] Based on tone category sequences, an N-gram model covering orders from 1 to N is constructed, where N is 7, and all N-gram combinations within the tone category sequence are generated;

[0018] Count the occurrences of each N-gram combination in the sequence and calculate its frequency value; for N-gram combinations that appear in all N-gram combinations but not in the sequence, fill their frequency values ​​with zero;

[0019] The frequency values ​​of all N-gram combinations are arranged in a predetermined order to form a fixed-dimensional feature vector representing the phonological rhythmic statistical pattern of the sentence.

[0020] Preferably, determining whether the Chinese text sentence to be detected is AI-generated text includes:

[0021] The feature vectors of the phonetic rhythm patterns of Chinese text sentences generated by preset standards and AI are converted into a structured dataset. The optimized distributed gradient boosting library XGBoost model is trained using the structured dataset to obtain the trained XGBoost model.

[0022] The feature vector representing the phonological rhythm pattern of the Chinese text sentence to be detected is input into the trained XGBoost model, which outputs a probability value and compares the probability value with a preset probability threshold.

[0023] If the probability value is higher than or equal to the preset probability threshold, the Chinese text sentence to be detected is determined to be AI-generated text; if the probability value is lower than the preset probability threshold, the Chinese text sentence to be detected is determined to be human-generated text.

[0024] This invention also provides an AI-generated text detection system, comprising:

[0025] The text module is used to obtain the Chinese text sentence to be detected, as well as the Chinese text sentence with preset standards and the Chinese text sentence generated by AI;

[0026] The tone sequence extraction module is used to obtain the pinyin and tone annotation of each Chinese character in the Chinese text sentence to be detected, based on preset standards and AI-generated Chinese text sentences. Based on preset tone category mapping rules, the tone annotation of each Chinese character is mapped to a tone category, so as to transform the Chinese text sentence into a tone category sequence composed of tone categories.

[0027] The feature vector generation module is used to construct an N-gram model covering orders 1 to N based on the tone category sequence, and obtain all N-gram combinations within the tone category sequence; count the occurrence frequency of each N-gram combination in the tone category sequence, calculate the relative frequency of each N-gram combination, and arrange the frequency values ​​of all N-gram combinations to obtain feature vectors representing the phonological rhythm patterns of Chinese text sentences to be detected, based on preset standards, and generated by AI.

[0028] The classification and judgment module is used to obtain the probability value of the detected Chinese text sentence based on the feature vector representing the phonetic rhythm pattern of the Chinese text sentence to be detected, the preset standard, and the AI-generated Chinese text sentence, and compare the probability value with the preset probability threshold to determine whether the Chinese text sentence to be detected is AI-generated text.

[0029] This invention also provides an electronic device, including a memory and a processor;

[0030] The memory is used to store computer programs;

[0031] When the processor executes the computer program stored in the memory, it implements the steps of the AI-generated text detection method described above.

[0032] This invention also provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of an AI-generated text detection method as described above.

[0033] This invention provides an AI-generated text detection method, system, device, and medium, which have the following advantages compared with the prior art:

[0034] This invention maps the tone of each Chinese character to a tone category using a pre-defined tone category mapping rule. This transforms the Chinese text sentence to be detected into a tone category sequence composed of tone categories. This process uses the categories of Chinese tones as a quantitative indicator of phonological structure, transforming text analysis from a high-dimensional, high-cost semantic space to a low-dimensional, high-efficiency phonological feature space. The extracted tone sequence features are inherent attributes of the Chinese language itself, rather than fingerprints generated by a specific AI model. It does not rely on training data of a specific model and can be adapted to text detection using multiple models. Then, N-gram analysis is performed on the generated tone category sequence, and the frequency of occurrence of each N-gram combination is calculated to form a feature vector. Finally, a pre-trained classification model is used to judge the AI-generated text. This process transforms the continuous and fuzzy phonological "rhythm" into discrete and quantifiable statistical features. The subsequent processing mainly involves string processing and frequency statistics, avoiding inconsistencies in models when called multiple times, thus enabling accurate detection of AI-generated text. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the feature extraction process of an AI-generated text detection method provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0037] At present, with the rapid development of large language models (LMs), the quality and efficiency of AI-generated text have been significantly improved, but the risk of its abuse is also becoming increasingly prominent, such as the potential harm in scenarios such as the spread of false information and academic misconduct. Therefore, AI-generated content detection technology has become a hot topic in research and application. At present, the existing technologies in this field are mainly divided into three categories: trained classifiers, zero-shot classifiers, and watermark detection methods.

[0038] Trainable classifiers are trained on text data generated by specific AI models (such as ChatGPT) and build classification models by analyzing the semantic, lexical and statistical features of the text. However, such methods are highly dependent on the source of the training data, resulting in insufficient generalization ability of the model. For example, detectors trained on ChatGPT show a significant performance drop when faced with text generated by other models (such as Tongyi Qianwen), making it difficult to adapt to real-world environments where multiple models coexist.

[0039] Zero-shot classifiers (such as FastDetectGPT) do not require specific training data, but rely on the source language model for real-time probability calculation and perturbation analysis. Although such methods have a certain degree of generalization, they are computationally expensive, require multiple calls to large language model APIs, resulting in huge response latency and resource consumption, and their detection performance for short texts is not ideal. In addition, the method completely fails when the source model is unavailable, and the deployment threshold is high.

[0040] Watermark detection technology achieves detection by actively embedding hidden markers during the content generation stage, but its application scope is limited. It cannot effectively detect massive amounts of historical text without watermarks, and it requires the cooperation of a generation model, which limits its practical application scenarios.

[0041] Traditionally, human language generation, whether oral or written, is influenced by long-established cognitive habits and physiological rhythms; this influence causes the text to naturally exhibit certain statistical preferences in the rising, falling, turning, and closing of rhymes and the cadence of rhythms; while the essence of AI large language models is to learn the co-occurrence relationship between words and semantics by mathematically modeling large-scale text data, and their generation process lacks this internal constraint that is deeply bound to human language physiology and cognition; based on this, in view of the problems existing in the current technology, by quantitatively analyzing this rhyme and rhythm pattern, reliable features for distinguishing human and AI texts can be constructed; specifically, the present invention uses the categories of Chinese tones as a quantitative index of the rhyme structure, converts text analysis from a high-dimensional and high-cost semantic space to a low-dimensional and high-efficiency rhyme feature space, and makes classification judgments based on the statistical pattern of this feature sequence.

[0042] Specifically:

[0043] Step 1: Data processing and feature extraction, the overall process is as Figure 1 shown.

[0044] (1) Data preparation: Prepare data for model training, and the data row format includes answers generated by humans and answers generated by AI (such as: <questions, human_answers, chatgpt_answers>). Among them, the Chinese text sentences generated by humans are the preset standard Chinese text sentences.

[0045] (2) Sentence segmentation: Use a program package (such as Pandas) to read the data, extract the human text column and the AI text column; perform sentence-level segmentation on the content of the two columns through end-of-sentence punctuation marks (such as ".", "!", "?").

[0046] (3) Generation of tone category sequences:

[0047] Traverse each segmented sentence and filter out sentences that are too short (for example, with a length less than 7 characters) to ensure the effectiveness of the features.

[0048] Use a pinyin conversion tool (such as the G2PM library) to obtain the pinyin and tone annotation of each Chinese character (for example, "你好" is processed as [ni3, hao3]).

[0049] According to the preset mapping rules, map each tone annotation to a tone category to form a tone category sequence.

[0050] In one specific embodiment, the first and second tones (high level / rising tone type) can be denoted as category "P", the third and fourth tones (falling tone / turning tone type) can be denoted as category "Z", and other cases such as neutral tone can be denoted as category "W"; thus, the sequence obtained by "[ni3, hao3]" is [Z, Z].

[0051] (4) Construction of N-gram frequency features:

[0052] An N-gram model is constructed for the generated tone category sequence (e.g., [P, Z, P, W]). This invention uses 1 to 7-gram (n-gram, n=1 to 7).

[0053] Count the occurrences of all N-gram combinations in each sentence and calculate their frequencies to form a feature vector representing the phonological rhythm pattern of the sentence; for N-gram combinations that do not appear, their frequencies are filled with 0.

[0054] (5) Constructing the dataset: Label the processed human text feature vectors as label=0 and the AI ​​text feature vectors as label=1, and save them as a structured dataset.

[0055] Step 2: Model training and validation.

[0056] (1) Model selection: Since the features generated by this invention are structured frequency data, a machine learning model with high efficiency and high performance in processing such data is selected, such as the XGBoost gradient boosting tree model. Compared with large language models, this model has the advantages of fewer parameters, faster training and lower hardware requirements, and is more suitable for industrial application scenarios.

[0057] (2) Training and validation: K-fold cross-validation (e.g., 5-fold) is used to train and evaluate the performance of the model. Before training, feature selection can be performed, such as selecting the most important (Top-10) N-gram frequency features for classification.

[0058] (3) Performance Evaluation: The model performance was evaluated using metrics such as precision, recall, F1-score, accuracy, and AUROC curve. The results of 5-fold cross-validation on a mainstream public dataset showed that the accuracy of this scheme was close to 100% (>99%), and the model demonstrated excellent performance in terms of precision, recall, and F1 score. The above results are based on the HC3-Chinese dataset (which is a publicly available standard test set, mainly consisting of text generated by the ChatGPT and Tongyi Qianwen models).

[0059] Step 3: Model application and detection.

[0060] The trained XGBoost model is deployed, and the Chinese text sentences to be detected are processed as follows:

[0061] (1) Sentence segmentation, tone category sequence extraction, and N-gram feature vector generation are performed on the text to be detected.

[0062] (2) Input the generated feature vector into the trained XGBoost model and output the probability value of the sentence being AI-generated text.

[0063] (3) Compare the probability value with a preset threshold (e.g., 0.5). If it is greater than or equal to the threshold, it is determined to be generated by AI; if it is less than the threshold, it is determined to be generated by humans.

[0064] The detection method of this invention is mainly applicable to detecting general written texts with a certain syntactic structure and length, such as:

[0065] News reports and commentary articles: These texts usually follow standard grammar and writing rhythm, and have relatively stable phonological patterns.

[0066] Question-and-answer text: such as online Q&A, product introductions, etc., with relatively complete sentence structures, similar to the features of the training dataset of this solution (such as HC3-Chinese).

[0067] Medium to long social media posts and blogs: When the text length exceeds a certain threshold (such as more than 50 characters), it can provide a sufficient sample for N-gram statistics.

[0068] This invention improves the generalization ability of detection: the tone sequence features extracted by this invention are inherent properties of the Chinese language itself, rather than fingerprints generated by a specific AI model; therefore, it does not rely on the training data of a specific model, and can still maintain high detection performance when facing texts generated by different language models (such as ChatGPT and Tongyi Qianwen), solving the technical pain point of "single model adaptation" in existing training detectors; even if future large language models become closer to humans in terms of semantics and lexicality, their statistical bias in phonology and rhythm will still be difficult to completely eliminate.

[0069] This invention reduces computational and deployment costs: the computational types of this invention are mainly string processing, dictionary lookup (such as any tool that supports Chinese character pinyin and tone conversion, such as G2PM), and frequency statistics. The inference overhead of the classifier (XGBoost) is extremely small and can run efficiently on a general-purpose CPU. In contrast, zero-shot detectors, represented by FastDetectGPT, rely on multiple API calls to large language models (such as GPT-Turbo) to obtain word probabilities, resulting in high computational costs, network latency, and strong dependence on third-party services.

[0070] This invention opens up a detection path based on phonological structure, which complements the current mainstream detection methods based on semantics and vocabulary statistics. When AI models become more sophisticated in semantic imitation in the future, this non-semantic structured feature may become a more robust distinguishing criterion, providing a new technical approach for the field of AI-generated content detection.

[0071] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An AI-generated text detection method, characterized by, The method comprises the following steps: obtaining a Chinese text sentence to be detected, and a Chinese text sentence of a preset standard and an AI-generated Chinese text sentence; obtaining the pinyin and tone mark of each Chinese character in the Chinese text sentence to be detected, the Chinese text sentence of the preset standard and the AI-generated Chinese text sentence, mapping the tone mark of each Chinese character to a tone category based on a preset tone category mapping rule, and converting the Chinese text sentence into a tone category sequence composed of tone categories; based on the tone category sequence, constructing an N-gram model covering 1 to N orders, and obtaining all N-gram combinations in the tone category sequence; counting the number of occurrences of each N-gram combination in the tone category sequence, and calculating the relative frequency of each N-gram combination; arranging the frequency values of all N-gram combinations to obtain a feature vector representing the phonological rhythm and cadence mode of the Chinese text sentence to be detected, the Chinese text sentence of the preset standard and the AI-generated Chinese text sentence; According to the feature vector representing the phonological rhythm and cadence mode of the Chinese text sentence to be detected, the Chinese text sentence of the preset standard and the AI-generated Chinese text sentence, obtaining the probability value of the detected Chinese text sentence, and comparing the probability value with the preset probability threshold, to determine whether the Chinese text sentence to be detected is an AI-generated text; The acquisition of the tone category sequence comprises: obtaining the pinyin and tone mark of each Chinese character in the Chinese text sentence using the G2PM library of the pinyin conversion tool; According to the preset tone category mapping rule, the tone mark of each Chinese character is mapped to a tone category, and the entire Chinese text sentence is converted into a tone category sequence composed of tone categories; The tone category mapping rule is defined based on the pitch variation mode of Chinese tones; tones 1 and 2 are set as category P, tones 3 and 4 are set as category Z, and light tone is set as category W; The judgment of whether the Chinese text sentence to be detected is an AI-generated text comprises: convert the feature vector of the phonological rhythm and cadence mode of the Chinese text sentence of the preset standard and the AI-generated Chinese text sentence into a structured data set, train an optimized distributed gradient boosting library XGBoost model using the structured data set, and obtain the trained XGBoost model; input the feature vector representing the phonological rhythm and cadence mode of the Chinese text sentence to be detected into the trained XGBoost model, output a probability value, and compare the probability value with the preset probability threshold; If the probability value is higher than or equal to the preset probability threshold, the Chinese text sentence to be detected is determined to be an AI-generated text; if the probability value is lower than the preset probability threshold, the Chinese text sentence to be detected is determined to be a human-generated text.

2. The AI-generated text detection method of claim 1, wherein, The acquisition of the feature vector comprises: based on the tone category sequence, constructing an N-gram model covering 1 to N orders, wherein N is 7, and generating all N-gram combinations in the tone category sequence; counting the number of occurrences of each N-gram combination in the sequence, and calculating its frequency value; for the N-gram combination that appears in all N-gram combinations but does not appear in the sequence, the frequency value is filled as zero; The frequency values of all N-gram combinations are arranged in a predetermined order to form a fixed-dimensional feature vector representing the statistical pattern of the prosodic rhythm of the sentence.

3. An AI-generated text detection system, comprising: The method comprises the following steps: The text module is configured to obtain a Chinese text sentence to be detected, and a Chinese text sentence of a preset standard and an AI-generated Chinese text sentence. The tone sequence extraction module is configured to obtain the pinyin and tone label of each Chinese character in the Chinese text sentence to be detected, the Chinese text sentence of the preset standard, and the AI-generated Chinese text sentence, and map the tone label of each Chinese character to a tone category based on a preset tone category mapping rule, so as to convert the Chinese text sentence into a tone category sequence composed of tone categories. The feature vector generation module is configured to construct an N-gram model covering 1 to N orders based on the tone category sequence, and obtain all N-gram combinations in the tone category sequence; count the number of occurrences of each N-gram combination in the tone category sequence, and calculate the relative frequency of each N-gram combination; arrange the frequency values of all N-gram combinations to obtain a feature vector representing the prosodic rhythm pattern of the Chinese text sentence to be detected, the Chinese text sentence of the preset standard, and the AI-generated Chinese text sentence. The classification and judgment module is configured to obtain a probability value of the detected Chinese text sentence based on the feature vector representing the prosodic rhythm pattern of the Chinese text sentence to be detected, the Chinese text sentence of the preset standard, and the AI-generated Chinese text sentence, and compare the probability value with a preset probability threshold to determine whether the Chinese text sentence to be detected is an AI-generated text. The obtaining of the tone category sequence comprises: The pinyin and tone label of each Chinese character in the Chinese text sentence are obtained using the G2PM library of the pinyin conversion tool. The tone label of each Chinese character is mapped to a tone category based on a preset tone category mapping rule, so as to convert the entire Chinese text sentence into a tone category sequence composed of tone categories. The tone category mapping rule is defined based on the pitch variation pattern of Chinese tones. Tones 1 and 2 are set as category P, tones 3 and 4 are set as category Z, and light tone is set as category W. The determination of whether the Chinese text sentence to be detected is an AI-generated text comprises: The feature vector representing the prosodic rhythm pattern of the Chinese text sentence of the preset standard and the AI-generated Chinese text sentence is converted into a structured data set, an optimized distributed gradient boosting library XGBoost model is trained using the structured data set, and a trained XGBoost model is obtained; The feature vector representing the prosodic rhythm pattern of the Chinese text sentence to be detected is input into the trained XGBoost model, and a probability value is output, which is compared with a preset probability threshold; If the probability value is higher than or equal to the preset probability threshold, the Chinese text sentence to be detected is determined to be an AI-generated text; if the probability value is lower than the preset probability threshold, the Chinese text sentence to be detected is determined to be a human-generated text.

4. An electronic device, comprising: The method comprises the following steps: The memory is configured to store a computer program. The processor is configured to execute the computer program stored in the memory to implement the steps of the AI-generated text detection method according to any one of claims 1-2. ​ 5. A computer readable storage medium, characterized in that, A computer program product for storing a computer program which, when executed by a processor, implements the steps of a method of AI-generated text detection according to any one of claims 1-2.

Images