Generated text detection method and related equipment

Abstract

One or more embodiments of the invention provide a generated text detection method and related equipment. The method comprises the following steps: acquiring a to-be-detected text; according to the to-be-detected text, lexical unit coherence of the to-be-detected text is obtained, and the lexical unit coherence represents the semantic closeness degree between words in the to-be-detected text; according to the to-be-detected text, obtaining an original predicted value through a basic detection model; and obtaining a final detection result according to the original predicted value and the lexical coherence. Through the method, the generated text detection effect can be improved.

Description

Technical Field

[0001] One or more embodiments of this application relate to the field of natural language processing technology, and in particular to a method for generating text detection and related equipment. Background Art

[0002] Large Language Models (LLMs) are machine learning models with a large number of parameters and complex computational structures. These models are typically built from deep neural networks, possessing billions or even hundreds of billions of parameters. They learn complex models and features by training on massive amounts of data, exhibiting stronger generalization capabilities and the ability to make accurate predictions on unseen data. The purpose of large models is to improve the expressive power and predictive performance of models, enabling them to handle more complex tasks and data.

[0003] With the development of machine learning technology and its application in various fields, the ability and efficiency of large language models (LLMs) in the field of natural language processing are remarkable. Among these capabilities, large language models can generate text that resembles human writing.

[0004] This enhanced capability has also raised concerns about its potential misuse. Therefore, effective detection of text generated by large language models has become an important principle to ensure the responsible use of large language models.

[0005] However, the detection schemes in related technologies still suffer from problems such as single feature extraction and unsatisfactory detection results. Summary of the Invention

[0006] In view of this, the purpose of one or more embodiments of this application is to provide a text generation detection method and related equipment to solve the problems raised in the background art.

[0007] To achieve the above objectives, one or more embodiments of this application provide a method for generating text detection. This includes:

[0008] Obtain the text to be detected;

[0009] Based on the text to be detected, the lexical cohesion of the text to be detected is obtained, and the lexical cohesion represents the semantic closeness between the words in the text to be detected;

[0010] Based on the text to be detected, the original predicted value is obtained through the basic detection model;

[0011] The final detection result is obtained based on the original predicted value and the word cohesion.

[0012] Optionally, obtaining the lexical cohesion of the text to be detected based on the text to be detected includes:

[0013] Based on the text to be detected, multiple copies of the text to be detected are created, each copy of the text to be detected including a portion of the text to be detected.

[0014] Based on the text to be detected and its copy, the lexical cohesion of the text to be detected is obtained through a lexical cohesion calculation model.

[0015] Optionally, the lexical cohesiveness computation model includes:

[0016] Where x represents the text to be detected. This refers to a copy of the text to be detected. n represents the number of copies of the text to be detected.

[0017] Optionally, the basic detection model is trained based on a zero-shot detector.

[0018] Optionally, obtaining the final detection result based on the original predicted value and the lexical cohesion includes:

[0019] Based on the original predicted value and the word cohesion, the final predicted value is obtained using the following formula;

[0020] w(x) = e u(x) ×v(x);

[0021] Where u(x) represents the lexical cohesion of the text to be detected, and v(x) represents the original predicted value of the text to be detected;

[0022] Based on the final predicted value, it is determined whether the text to be detected is generated text.

[0023] Optionally, determining whether the text to be detected is generated text based on the final predicted value includes:

[0024] If the final predicted value is greater than a preset threshold, the text to be detected is determined to be generated text;

[0025] In response to the final predicted value being less than or equal to a preset threshold, the text to be detected is determined to be non-generated text.

[0026] Optionally, obtaining the lexical cohesion of the text to be detected through the lexical cohesion calculation model includes:

[0027] The word cohesive computing model is invoked by performing a preset operation, so that the word cohesive computing model provides computing services in a preset manner.

[0028] Based on the same inventive concept, one or more embodiments of this application also provide a text generation detection device, including:

[0029] The acquisition module is configured to acquire the text to be detected.

[0030] The first calculation module is configured to obtain the lexical cohesion of the text to be detected based on the text to be detected, wherein the lexical cohesion represents the semantic closeness between words in the text to be detected;

[0031] The second calculation module is configured to obtain the original predicted value based on the text to be detected using a basic detection model.

[0032] The third calculation module is configured to obtain the final detection result based on the original predicted value and the word cohesion.

[0033] Based on the same inventive concept, one or more embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the text generation detection method as described in any of the above.

[0034] Based on the same inventive concept, one or more embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute any of the above-described text generation detection methods.

[0035] As can be seen from the above, the text detection method provided in one or more embodiments of this application involves: acquiring a text to be detected; obtaining the lexical cohesion of the text to be detected, wherein the lexical cohesion represents the semantic closeness between words in the text to be detected; obtaining an original prediction value through a basic detection model based on the text to be detected; and obtaining a final detection result based on the original prediction value and the lexical cohesion.

[0036] The technical solution of this application combines the lexical cohesive features of the text to be detected with the basic detection results to detect the text, thereby improving the detection effect of the generated text.

[0037] The text generation detection device, electronic device, and computer-readable storage medium provided in this application can all implement the steps of the above-described text generation detection method, and therefore also have the beneficial effects of the above-described text generation detection method. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in one or more embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating one or more embodiments of the text detection method of this application;

[0040] Figure 2 This is a schematic diagram of the structure of a generated text detection model according to one or more embodiments of this application;

[0041] Figure 3 This is a schematic diagram of the experimental results of lexical cohesion in one or more embodiments of this application;

[0042] Figure 4 This is a pseudocode diagram illustrating the text generation and judgment process of one or more embodiments of this application;

[0043] Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to one or more embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0045] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in one or more embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0046] As described in the background section, the detection of generated text can be regarded as a binary classification task, that is, classifying the text to be detected into generated text generated by a large language model or text written by humans.

[0047] In related technologies, the above tasks are mainly performed through supervised learning detectors or zero-shot detectors.

[0048] In the first type of approach, supervised learning detectors are typically trained on manually labeled data, utilizing the statistical features or neural representations of the data to distinguish between text generated by large language models and human-written text. One popular trend is to directly fine-tune the pre-trained language model, i.e., by adding a classification unit to the pre-trained model to enable it to classify text, such as BERT or RoBERTa, for text classification tasks. However, supervised learning detectors are trained by learning the differences between text generated by large language models and human-written text in labeled data. Therefore, they are prone to overfitting to their specific training domain, may have poor generalization ability, and require periodic retraining.

[0049] In the second category of related techniques, zero-shot detectors do not require training and typically exhibit better generalization ability. The core idea is to extract various statistical features, such as log-likelihood, perplexity, normalized logarithmic ranking perturbation, probability, and conditional probability curvature, and threshold these features to distinguish between text generated by large language models and human-written text. These features can be collected from the source large language model itself (white-box setting) or a surrogate model (black-box setting). Furthermore, some black-box zero-shot detectors based on large language model rewriting have emerged in related techniques. The core idea of ​​these methods is to rewrite text paragraphs using another large language model (usually ChatGPT) and then evaluate the similarity or overlap between the original and rewritten text. Paragraphs with higher similarity or overlap are considered to have been generated by the large language model.

[0050] Zero-shot detectors extract diverse features that distinguish between the two types of text and directly threshold these features without training, making them less susceptible to domain-specific factors and generally exhibiting better generalization ability. However, traditional zero-shot detectors often extract only a single statistical feature. On the one hand, Transformer-based language models consistently use a "causal self-attention mechanism" when generating text, ensuring that the generation of each word is closely related to the words preceding it. This results in higher lexical cohesion in the text generated by the language model, and this meaningful difference for text detection cannot be captured using traditional zero-shot methods, leading to information loss.

[0051] In developing this disclosure, the applicant discovered that samples generated by language models typically exhibit higher word coherence than manually written text. Combining this word coherence feature can effectively improve the validity of the detection results.

[0052] Therefore, this application proposes a text generation detection method to solve the above problems.

[0053] refer to Figure 1 The text detection method generated according to one or more embodiments of this application includes the following steps:

[0054] Step S101: Obtain the text to be detected.

[0055] The above text represents sentences, paragraphs, or chapters composed of word units.

[0056] Step S102: Based on the above text to be detected, obtain the lexical cohesion of the above text to be detected. The lexical cohesion represents the semantic closeness between the words in the above text to be detected.

[0057] In developing this application, the applicant discovered that for text generated by a language model, each word is generated based on all its preceding words, which naturally promotes close relationships between words, leading to higher cohesion. However, for manually written text, there is no such explicit constraint on word generation, which may result in looser relationships between words and thus lower cohesion. Therefore, the applicant proposes the hypothesis that "samples generated by language models typically exhibit higher lexical cohesion than manually written text."

[0058] Based on this assumption, the applicant conducted the following verification work:

[0059] Let P LLM P represents the distribution of text generated by the language model. Human This represents the distribution of manually written text. Based on the above assumptions, the lexical cohesion u(x) is in the range x~P LLM The values ​​are often higher in samples with x ~ P Human The values ​​are often lower in the sample.

[0060] The applicant empirically verified this hypothesis using an automated method. Specifically, the applicant randomly selected 500 news articles as human-written data and used four different language models to generate "language model-generated text" based on the first 30 words of each human-written article. For each original text, 10 copies were created, and 1.5% of the words in each copy were randomly removed. The negative BARTScore was then used as the semantic difference measure DIFF(·,·), and the expected value E(·) was approximated by the average of the 10 copies. Figure 3 The experimental results show that the lexical cohesion distribution of text generated by the language model is significantly different from that of manually written text. The samples generated by the language model do indeed tend to have higher lexical cohesion.

[0061] Therefore, this application proposes a new statistical feature for zero-shot text detection—lexical cohesion—and combines it with traditional zero-shot detectors, enabling traditional detectors to additionally consider semantic dimension features for more precise detection of text generated by language models.

[0062] In some embodiments, a semantic similarity measurement model, such as BARTScore, can be used to calculate the lexical cohesion of candidate paragraphs. This model is relatively small and can be used directly without any fine-tuning.

[0063] In some embodiments, lexical cohesion can be defined as: given a candidate text x, let Let x and x represent candidate text copies created by randomly removing a small subset of words from x. The expected value of the semantic difference between the two is the lexical cohesion of candidate text x. Lexical cohesion essentially measures the semantic tightness between words in a paragraph. The tighter the semantic relationship between words, the higher the lexical cohesion.

[0064] Thus, in some embodiments of this application, it can be achieved through formulas Calculate the lexical cohesion of the text to be detected. Where x represents the text to be detected. This refers to a copy of the text to be detected. n represents the number of copies of the text to be detected.

[0065] In some embodiments, the lexical cohesion described above is obtained through a lexical cohesion calculation model. This lexical cohesion calculation model can be used as a pluggable module to improve the base detection model. For example, LRR and Fast-DetectGPT can be selected as the base detection models, and the lexical cohesion calculation model can be applied to the detector as a pluggable module.

[0066] Step S103: Based on the above text to be detected, obtain the original predicted value through the basic detection model.

[0067] In some embodiments, the basic detection model described above can be trained based on a zero-shot detector.

[0068] In some embodiments, the text to be detected is first input into a first channel, which creates a copy of the text to be detected and calculates lexical cohesion based on the text to be detected; then the text to be detected is input into a second channel, which obtains the original prediction value through a basic detection model.

[0069] Step S104: Obtain the final detection result based on the above original predicted values ​​and the above word cohesion.

[0070] In some embodiments, the calculation results obtained from the above two steps can be used to calculate the result using the formula w(x) = e u(x) The final predicted value is obtained by multiplying u(x) by v(x), where u(x) represents the lexical cohesion of the text to be detected and v(x) represents the original predicted value of the text to be detected. Then, based on the final predicted value, it is determined whether the text to be detected is generated text.

[0071] In some embodiments, in response to a final predicted value greater than a preset threshold, the text to be detected is determined to be generated text; in response to a final predicted value less than or equal to a preset threshold, the text to be detected is determined to be non-generated text.

[0072] In some embodiments, the pseudocode for the above determination process is as follows: Figure 4 As shown.

[0073] Based on the above, this application proposes the concept of lexical cohesion and reveals that it can serve as a new standard for distinguishing between text generated by language models and human-written text. Based on this new discovery, this application designs a detection device with a structure of "basic detection model + lexical cohesion calculation model". This device is a general dual-channel detection paradigm that uses lexical cohesion as a pluggable module to improve existing zero-shot detectors without changing the original detector's structure or requiring additional training. It only requires a small additional time and space cost to improve the detection accuracy of the original detector.

[0074] It is understandable that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0075] It should be noted that the methods of one or more embodiments of this application can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this application, and the multiple devices will interact with each other to complete the method described.

[0076] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a text generation detection device.

[0078] like Figure 2 As shown, the above-mentioned text generation detection device includes:

[0079] Module 11 is configured to acquire the text to be detected;

[0080] The first calculation module 12 is configured to obtain the lexical cohesion of the text to be detected based on the text to be detected, wherein the lexical cohesion represents the semantic closeness between words in the text to be detected.

[0081] The second calculation module 13 is configured to obtain the original predicted value based on the above-mentioned text to be detected through the basic detection model.

[0082] The third calculation module 14 is configured to obtain the final detection result based on the above-mentioned original predicted value and the above-mentioned word cohesion.

[0083] Optionally, the lexical cohesion of the text to be detected, obtained from the text to be detected, includes:

[0084] Based on the above text to be detected, multiple copies of the text to be detected are created, each copy including a portion of the above text to be detected.

[0085] Based on the text to be detected and its copy, the lexical cohesion of the text to be detected is obtained through a lexical cohesion calculation model.

[0086] Optionally, the above-mentioned lexical cohesion calculation model includes:

[0087] Where x represents the text to be detected. This refers to a copy of the text to be detected. n represents the number of copies of the text to be detected.

[0088] Optionally, the above-mentioned basic detection model is trained based on a zero-shot detector.

[0089] Optionally, the final detection result obtained based on the above-mentioned original predicted values ​​and the above-mentioned lexical cohesion includes:

[0090] Based on the above original predicted values ​​and the above word cohesion, the final predicted value is obtained through the following formula;

[0091] w(x) = e u(x) ×v(x);

[0092] Where u(x) represents the lexical cohesion of the text to be detected, and v(x) represents the original predicted value of the text to be detected.

[0093] Based on the final predicted value, determine whether the text to be detected is generated text.

[0094] Optionally, the determination of whether the text to be detected is generated text based on the final predicted value includes:

[0095] If the final predicted value is greater than a preset threshold, the text to be detected is determined to be generated text.

[0096] If the final predicted value is less than or equal to a preset threshold, the text to be detected is determined to be non-production text.

[0097] Optionally, the lexical cohesion of the text to be detected obtained through the lexical cohesion calculation model includes:

[0098] By executing a preset operation, the above-mentioned word cohesive computing model is invoked so that the above-mentioned word cohesive computing model provides computing services in a preset manner.

[0099] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, when implementing one or more embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0100] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0101] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0102] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0103] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0104] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0105] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0106] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0107] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this application, and not necessarily all the components shown in the figures.

[0108] The electronic devices described above are used to implement the corresponding methods in the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0109] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0110] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0111] Additionally, to simplify the description and discussion, and to avoid obscuring one or more embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be illustrated in block diagram form to avoid obscuring one or more embodiments of this application, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which one or more embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that one or more embodiments of this application may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0112] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0113] One or more embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of this disclosure.

Claims

1. A generated text detection method, characterized in that: include: Get the text to be detected; According to the text to be detected, the word unit cohesion of the text to be detected is obtained, and the word unit cohesion represents the semantic closeness between each word in the text to be detected; According to the text to be detected, an original prediction value is obtained through a basic detection model; A final detection result is obtained according to the original prediction value and the word unit cohesion.

2. The method according to claim 1, characterized in that The step of obtaining the word unit cohesion of the text to be detected according to the text to be detected includes: According to the text to be detected, creating multiple copies of the text to be detected, wherein the copies of the text to be detected include part of the content of the text to be detected; According to the text to be detected and the copy of the text to be detected, the word unit cohesion of the text to be detected is obtained through a word unit cohesion calculation model.

3. The method according to claim 1, characterized in that The word unit cohesion calculation model includes: Wherein, x represents the text to be detected, represents the copy of the text to be detected, n represents the number of the text copies to be detected.

4. The method according to claim 3, characterized in that The basic detection model is trained based on a zero-shot detector.

5. The method according to claim 3, characterized in that: The obtaining of a final detection result according to the original prediction value and the word unit cohesion includes: According to the original prediction value and the word unit cohesion, the final prediction value is obtained by the following formula: w(x)=e u(x) ×v(x); Wherein, u(x) represents the word unit cohesion of the text to be detected, and v(x) represents the original prediction value of the text to be detected; According to the final prediction value, it is determined whether the text to be detected is a generated text.

6. The method according to claim 5, characterized in that The step of judging whether the to-be-detected text is a generated text according to the final prediction value includes: In response to the final prediction value being greater than a preset threshold, determining that the text to be detected is a generated text; In response to the final prediction value being less than or equal to a preset threshold, it is determined that the text to be detected is not a generated text.

7. The method according to claim 2, characterized in that The step of obtaining the word unit cohesion of the text to be detected by using the word unit cohesion calculation model includes: The word-unit cohesion calculation model is called by executing a preset operation, so that the word-unit cohesion calculation model provides a calculation service in a preset manner.

8. A generated text detection device, characterized in that: include: An acquisition module is configured to acquire the text to be detected; A first calculation module is configured to obtain the word-unit cohesion of the text to be detected according to the text to be detected, wherein the word-unit cohesion represents the semantic closeness between the words in the text to be detected; A second calculation module is configured to obtain an original prediction value through a basic detection model according to the text to be detected; The third calculation module is configured to obtain a final detection result according to the original prediction value and the word unit cohesion.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute any one of claims 1 to 7.

Images