Text detection method and device and medium

By performing perturbation processing and syntax tree analysis on the text to be tested, combined with the graph convolutional neural network, the problem of degradation of text detection accuracy in the existing technology is solved, and high-precision detection and robustness detection of the generated text after perturbation is achieved .

CN119068503APending Publication Date: 2024-12-03INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411015967.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing large-scale model generated text detection method has decreased detection accuracy when text field changes or word replacement, and has poor detection effect on text with the same distribution of training samples, which cannot effectively solve the text perturbation detection problem in actual application scenarios.

Method used

By perturbing the text to be tested, perturbing text is generated, and the nodes and adjacency matrix of the graph neural network are constructed using syntax tree analysis, vectorized the perturbing text, combined with high-dimensional vector representation and syntax tree information, and through graph convolutional neural network, the graph convolution is carried out The network is trained to achieve high-precision detection of the generated text after perturbation.

Benefits of technology

It realizes effective detection of various text perturbations, improves the robustness and practicality of the detection method, and can more accurately identify the categories of text generated after perturbation, making up for the existing methods in terms of detection accuracy and robustness. flaws.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068503A_ABST
    Figure CN119068503A_ABST
Patent Text Reader

Abstract

The invention provides a text detection method and device and a medium, and the method comprises the steps: carrying out the disturbance processing of a to-be-detected text, and obtaining a disturbed text; performing syntax tree analysis on the disturbance text, identifying a plurality of words of the text and syntax structures among the words, and generating a syntax tree; traversing the syntax tree, constructing each word in the syntax tree into a node required by graph neural network training, and coding a syntax structure between the words into an adjacent matrix; performing vectorization processing on the disturbance text to obtain high-dimensional vector representation of the text; and fusing the high-dimensional vector representation and the adjacent matrix, and training through a graph convolutional neural network to predict a category label of the to-be-tested text. According to the method, high-precision detection on the text generated after disturbance is realized, and meanwhile, the detection method has very high robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of large model generated text detection, and specifically relates to a text detection method, device and medium. Background Art

[0002] The detection of text generated by a large model refers to predicting whether a piece of text is derived from text generated by a large model through methods such as machine learning.

[0003] At present, the detection method of large model generated text is mainly achieved through classification, that is, by building a classifier, a piece of text is judged as generated by a large model or generated by humans. There are three common methods: (1) Feature-based text detection. This method uses the features of the text itself, such as the probability of calculating the next word in the text, ranking distribution, entropy, perplexity and other features, and then uses these features to train the machine learning model for classification. It performs well on some simple texts, but its disadvantage is that it has poor generalization. When the training text and the test text are in different fields, the generation model is different, and the text style is different, the effect will be significantly reduced; (2) Text detection based on fine-tuning pre-trained models, such as fine-tuning Bert, Roberta, Transformer and other models. This method has good performance in many natural language processing tasks, but its effect depends on the amount of fine-tuned text and the quality of generation; (3) Text detection based on zero-shot mainly uses the differences between human text and machine text for classification, such as using the self-consistency of machine text, masking the text and then predicting, comparing the similarity before and after, or comparing the changes in the probability curve after perturbing the text. Machine text usually changes greatly, so the probability difference between machine text and human text is used to achieve text detection. However, the disadvantage is that some zero-shot methods such as DetectGPT are white-box detection, which is more difficult to apply in practice.

[0004] Although the above-mentioned existing technologies can detect human and machine texts, they have high requirements on the quality of the text itself. Test texts with the same distribution as training samples can be detected, but if the test text is changed to another field or some words in it are replaced, the detection accuracy will drop rapidly and may even be impossible to train, unless the classifier is retrained with new text. However, this cannot fundamentally solve the problem and is not suitable for actual application scenarios. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a text detection method, device, and medium, which achieve high-precision detection of text generated after disturbance, while making the detection method highly robust.

[0006] In order to achieve the above object, the present invention provides a text detection method, comprising:

[0007] Performing disturbance processing on the text to be tested to obtain a disturbed text;

[0008] Performing syntax tree analysis on the disturbed text, identifying several words in the text and the grammatical structure between the words, and generating a syntax tree;

[0009] Traverse the syntax tree, construct each word in the syntax tree as a node required for graph neural network training, and encode the grammatical structure between words into an adjacency matrix;

[0010] Vectorize the perturbed text to obtain a high-dimensional vector representation of the text;

[0011] The high-dimensional vector representation and the adjacency matrix are fused and trained through a graph convolutional neural network to predict the category label of the text to be tested.

[0012] In some embodiments, the perturbation process includes at least one of deleting, replacing, inserting and repeating words.

[0013] In some embodiments, the replacing words includes random replacement, synonym replacement, and phrase replacement.

[0014] In some embodiments, an adjacency matrix is ​​constructed based on the parent-child node relationship in the syntax tree, where the adjacency matrix is ​​a two-dimensional matrix, and the rows and columns represent words of the text;

[0015] If the syntax tree has a unidirectional edge from a parent node to a child node, it is marked in the adjacency matrix.

[0016] In some embodiments, the perturbed text is vectorized using a pre-trained language model to learn semantic representations of words in the text;

[0017] Convert the semantic representation of words into high-dimensional vector representation.

[0018] In some embodiments, the high-dimensional vector representation is used as a node feature input of a graph convolutional neural network, and the adjacency matrix is ​​used as a graph structure input;

[0019] Train a multi-layer graph convolution neural network, and perform high-dimensional vector aggregation and dimensionality reduction processing on the unidirectional edges in the syntax tree in each layer of the network, so that the model can learn the syntax tree features for classifying the text to be tested.

[0020] In some embodiments, the category labels include large model generated text and human generated text.

[0021] Another aspect of the present invention also discloses a text detection device, which adopts the above-mentioned text detection method and at least comprises:

[0022] A disturbance processing module is used to perform disturbance processing on the text to be tested to obtain a disturbed text;

[0023] A syntax tree analysis module, used to perform syntax tree analysis on the disturbance text, identify several words in the text and the grammatical structure between the words, and generate a syntax tree; and

[0024] Traverse the syntax tree, construct each word in the syntax tree as a node required for graph neural network training, and encode the grammatical structure between words into an adjacency matrix;

[0025] A vectorization processing module is used to perform vectorization processing on the disturbed text to obtain a high-dimensional vector representation of the text;

[0026] The training module is used to fuse the high-dimensional vector representation and the adjacency matrix, and train the graph convolutional neural network to predict the category label of the text to be tested.

[0027] In some embodiments, the perturbation processing module performs at least one of deleting, replacing, inserting and repeating words.

[0028] Another aspect of the present invention further discloses a computer-readable storage medium, wherein at least one program code is stored in the computer-readable storage medium, and the at least one program code is loaded and executed by a processor to implement the text detection method as described above.

[0029] It can be seen from the above scheme that the advantages of the present invention are:

[0030] The text detection method disclosed by the present invention, on the one hand, performs disturbance processing on the text to be tested by fitting the disturbance operation of the real scene, thereby ensuring the effective detection of various text disturbance situations, and being able to more realistically simulate the repeated editing problem faced by the generated text detection method in practical applications; on the other hand, an adjacency matrix required by the graph neural network is constructed through a syntax tree, and combined with the text vectorization operation in the pre-trained model, by fusing high-dimensional vector information and syntax tree information, the defect that the existing method cannot accurately detect the generated text after the disturbance is achieved, and high-precision detection of the generated text after the disturbance is achieved, while making the detection method highly robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the overall process of a text detection method provided by an embodiment of the present invention is shown;

[0032] Figure 2 A schematic diagram showing the text detection method being applied to a specific example;

[0033] Figure 3 A framework diagram of a text detection device disclosed in another embodiment of the present invention is shown.

[0034] in:

[0035] 300: text detection device;

[0036] 301: disturbance processing module;

[0037] 302: syntax tree analysis module;

[0038] 303: vector processing module;

[0039] 304: Training module. DETAILED DESCRIPTION

[0040] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0041] See also Figure 1-Figure 3 As shown, Figure 1 FIG. 1 shows a schematic diagram of the overall process of a text detection method provided by an embodiment of the present invention. Figure 2 A schematic diagram showing the application of the text detection method to a specific example is disclosed, Figure 3 A framework diagram of a text detection device disclosed in another embodiment of the present invention is shown.

[0042] See also Figure 1 As shown, in one embodiment of the present invention, a text detection method is also provided, comprising the following steps:

[0043] Step S1, performing disturbance processing on the text to be tested to obtain a disturbed text.

[0044] In practical applications, machine-generated texts are often artificially disturbed to a certain extent, such as deleting, replacing, inserting or repeating certain words. These disturbances may destroy certain features of the original text, making it difficult to accurately identify whether the modified text is generated by the machine using the text detection method without disturbance directly. Therefore, in this embodiment, the input text is disturbed to make the detection method closer to the actual application scenario and improve the robustness and practicality of the detection method. By disturbing the input text, various text editing situations that may occur in practical applications can be simulated, thereby evaluating the performance of the detection method in these situations. The text after disturbance processing can be used as training data or test data to train or evaluate the detection model, and improve the recognition ability of the model to the disturbed text.

[0045] In some embodiments, the disturbance processing includes at least one of deletion, replacement, insertion and repetition of words. Among them, the deletion processing refers to randomly deleting some words from the text, simulating the deletion operation that may occur when the user edits. The replacement processing refers to replacing some words in the text with other words, including random replacement, synonym replacement and phrase replacement. Among them, synonym replacement can use WordNet to obtain all synonyms of a word and select one for replacement; phrase replacement can count the high-frequency phrase combinations in Wikipedia for replacement. Insertion processing refers to randomly inserting a certain proportion of words in the text, simulating the insertion operation that may occur when the user edits. Repeating operation refers to repeating some words or sentences in the text, simulating the repeated operation that may occur when the user edits.

[0046] Among the above perturbation operations, deletion, random replacement, insertion, and word duplication are operations that destroy text coherence, while synonym replacement and phrase replacement can maintain the semantic coherence of the text. By adopting multiple perturbation methods, more realistic and comprehensive user editing behaviors are covered, ensuring effective detection of various text perturbations, so that everything from synonym replacement to maintain coherence to simple language structure destruction can be effectively identified.

[0047] Step S2: Performing syntax tree analysis on the disturbed text, identifying several words in the text and the grammatical structures between the words, and generating a syntax tree.

[0048] The disturbance operation often destroys the grammatical structure of the text, which requires that when detecting text, it is necessary not only to capture the semantic features of the text, but also to learn the grammatical structure features of the text. However, the existing technologies are not robust to texts with damaged language structures. To this end, in this embodiment, a syntax tree analysis is performed on the disturbed text to convert the disturbed text into a structured representation form - a syntax tree. A syntax tree is a tree diagram used to represent the grammatical structure of a sentence. It clearly shows the part of speech of each word in the sentence (such as noun, verb, adjective, etc.) and the grammatical relationship between them (such as subject-predicate relationship, verb-object relationship, etc.). Syntax tree analysis tools (such as Spacy, NLTK, etc.) are usually used to parse the disturbed text, identify several words in the text and the grammatical structure between words, and store this information in the form of a syntax tree. The constructed syntax tree is as follows: Figure 2 as shown in .

[0049] Step S3: traverse the syntax tree, construct each word in the syntax tree as a node required for graph neural network training, and encode the grammatical structure between words into an adjacency matrix.

[0050] After obtaining the syntax tree, we need to convert each word in the syntax tree into the nodes required for graph neural network (GNN) training. This is because graph neural networks can effectively process graph structure data, and syntax trees are a special kind of graph structure. By traversing the syntax tree, each word in the syntax tree is regarded as a node, and the attributes of the node can include information such as the word's part of speech and word frequency.

[0051] In addition, the syntax tree not only contains information about the words themselves, but also contains the grammatical relationship between words. In order to introduce this relationship information into the subsequent graph neural network training, it is necessary to further encode the grammatical structure between words into an adjacency matrix. The adjacency matrix is ​​a matrix that represents the connection relationship between nodes in a graph. If there is an edge between node i and node j, the element at the corresponding position in the matrix is ​​1 (or other values ​​indicating the strength of the connection), otherwise it is 0. Specifically, an adjacency matrix can be constructed based on the parent-child node relationship in the syntax tree. The adjacency matrix is ​​a two-dimensional matrix, where rows and columns represent words in the text; if the syntax tree has a unidirectional edge from a parent node to a child node, it is marked in the adjacency matrix, such as marking the element at the corresponding position in the matrix as 1. The generated adjacency matrix is ​​as follows: Figure 2 as shown in .

[0052] Through the operations of the above steps S2 and S3, the perturbed text is analyzed through the syntax tree, the nodes required for graph neural network training are encoded, and the grammatical structure of the words is encoded as an adjacency matrix.

[0053] Step S4: vectorize the disturbed text to obtain a high-dimensional vector representation of the text.

[0054] In natural language processing, text usually needs to be converted into a numerical form that can be processed by computers, that is, vectorized processing. By converting the semantic information of the text into a vector representation in a high-dimensional space, it is convenient for subsequent training of machine learning or deep learning models.

[0055] In some embodiments, a pre-trained language model (such as the Roberta model) is used to vectorize the perturbed text, that is, to segment words and construct word vectors to learn the semantic representation of words in the text; and convert the semantic representation of words into a high-dimensional vector representation.

[0056] In this embodiment, through the operations of the above steps S2-S4, the structural information of the text is effectively converted into a computer-processable form by performing syntax tree analysis and vectorization processing on the disturbed text, providing a data basis for subsequent detection.

[0057] Step S5: Fusing the high-dimensional vector representation and the adjacency matrix, and training through a graph convolutional neural network to predict the category label of the text to be tested.

[0058] In this embodiment, after obtaining the high-dimensional vector representation of the text and the adjacency matrix of the syntax tree, it is necessary to fuse these two parts of information and train them through a graph convolutional neural network. The graph convolutional neural network can process graph structure data and learn the node features in the graph and the connection relationships between them through convolution operations. Specifically, the high-dimensional vector representation is used as the node feature input of the graph convolutional neural network, and the adjacency matrix is ​​used as the graph structure input to train the multi-layer graph convolutional neural network. In each layer of the network, the high-dimensional vector representation is aggregated and reduced in dimension for the unidirectional edges in the syntax tree, so that the model learns the syntax tree features for classifying the text to be tested. Through the training of the graph convolutional neural network, the model can learn the semantic features of words in the text and their grammatical structure features, thereby realizing the prediction of text categories. The prediction results usually include two categories: large model generated text and human generated text, that is, the category labels include large model generated text and human generated text. As Figure 2 As shown in , for the text to be tested, after the text vectorization operation and the construction of the adjacency matrix required by the graph neural network through the syntax tree, it is input into the graph convolutional neural network for training, and finally the test results include the classification ratio of the text generated by the large model and the text generated by humans.

[0059] In summary, compared with the prior art, the text detection method disclosed in the present invention, on the one hand, performs perturbation processing on the text to be tested by fitting the perturbation operation of the real scene, thereby ensuring the effective detection of various text perturbations, and can more realistically simulate the repeated editing problems faced by the generated text detection method in practical applications; on the other hand, the adjacency matrix required by the graph neural network is constructed through the syntax tree, and combined with the text vectorization operation in the pre-trained model, by fusing high-dimensional vector information and syntax tree information, the defect that the existing method cannot accurately detect the generated text after the perturbation is achieved, and high-precision detection of the generated text after the perturbation is achieved, while making the detection method highly robust.

[0060] In addition, it should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the process of constructing the adjacency matrix required for the graph neural network through the syntax tree recorded in the present invention and the text vectorization operation process, these two processes can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, the present invention is not limited here.

[0061] In addition, corresponding to the above-mentioned text detection method embodiment, the present invention also provides an embodiment of a text detection device. Figure 3, a text detection device 300 provided by an embodiment of the present invention at least comprises:

[0062] The disturbance processing module 301 is used to perform disturbance processing on the text to be tested to obtain a disturbed text.

[0063] A syntax tree analysis module 302 is used to perform syntax tree analysis on the disturbance text, identify a number of words in the text and the grammatical structure between the words, and generate a syntax tree; and

[0064] Traverse the syntax tree, construct each word in the syntax tree as a node required for graph neural network training, and encode the grammatical structure between words into an adjacency matrix;

[0065] The vectorization processing module 303 is used to perform vectorization processing on the disturbed text to obtain a high-dimensional vector representation of the text.

[0066] The training module 304 is used to fuse the high-dimensional vector representation and the adjacency matrix, and perform training through a graph convolutional neural network to predict the category label of the text to be tested.

[0067] In addition, the implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0068] As for the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only illustrative, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0069] Corresponding to the embodiment of the aforementioned text detection method, the embodiment of the present application further provides a computer device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the following Figure 1 In addition, in addition to the processor, memory, DMA controller, disk, and non-volatile memory, the computer device may also include other hardware, usually based on the actual function of the device, which will not be described in detail.

[0070] Corresponding to the embodiments of the aforementioned text detection method, the embodiments of the present invention further provide a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the text detection method in the aforementioned embodiment is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capability in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be any device with data processing capability, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (FlashCard), etc. equipped on the device. Further, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and can also be used to temporarily store data that has been output or is to be output.

[0071] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text detection method, characterized in that: Include: Performing disturbance processing on the text to be tested to obtain a disturbed text; Performing syntax tree analysis on the disturbed text, identifying several words in the text and the grammatical structure between the words, and generating a syntax tree; Traverse the syntax tree, construct each word in the syntax tree as a node required for graph neural network training, and encode the grammatical structure between words into an adjacency matrix; Vectorize the perturbed text to obtain a high-dimensional vector representation of the text; The high-dimensional vector representation and the adjacency matrix are fused and trained through a graph convolutional neural network to predict the category label of the text to be tested.

2. The method according to claim 1, characterized in that The perturbation process includes at least one of deleting, replacing, inserting and repeating words.

3. The method according to claim 2, characterized in that The replacement words include random replacement, synonym replacement and phrase replacement.

4. The method according to claim 1, characterized in that: According to the parent-child node relationship in the syntax tree, an adjacency matrix is ​​constructed. The adjacency matrix is ​​a two-dimensional matrix, and the rows and columns represent the words of the text; If the syntax tree has a unidirectional edge from a parent node to a child node, it is marked in the adjacency matrix.

5. The method according to claim 1, characterized in that Use the pre-trained language model to vectorize the perturbed text and learn the semantic representation of the words in the text; Convert the semantic representation of words into high-dimensional vector representation.

6. The method according to claim 1, characterized in that The high-dimensional vector representation is used as the node feature input of the graph convolutional neural network, and the adjacency matrix is ​​used as the graph structure input; Train a multi-layer graph convolution neural network, and perform high-dimensional vector aggregation and dimensionality reduction processing on the unidirectional edges in the syntax tree in each layer of the network, so that the model can learn the syntax tree features for classifying the text to be tested.

7. The method according to claim 1, characterized in that The category labels include large model generated text and human generated text.

8. A text detection device, characterized in that: Include: A disturbance processing module is used to perform disturbance processing on the text to be tested to obtain a disturbed text; A syntax tree analysis module is used to perform syntax tree analysis on the disturbance text, identify several words in the text and the grammatical structure between the words, and generate a syntax tree; as well as Traverse the syntax tree, construct each word in the syntax tree as a node required for graph neural network training, and encode the grammatical structure between words into an adjacency matrix; A vectorization processing module is used to perform vectorization processing on the disturbed text to obtain a high-dimensional vector representation of the text; The training module is used to fuse the high-dimensional vector representation and the adjacency matrix, and train the graph convolutional neural network to predict the category label of the text to be tested.

9. The device according to claim 8, characterized in that The disturbance processing module performs at least one operation of deleting, replacing, inserting and repeating words.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the text detection method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Text detection method, electronic device and computer program product

    CN122470751A