Generative artificial intelligence authorship detection

Abstract

A method of detecting whether a submitted text was generated by an artificial intelligence computer program includes: feeding a textual prompt, in response to which the submitted text had been written, into a plurality of artificial intelligence composition programs; generating comparison texts with the artificial intelligence composition programs that are responsive to the textual prompts; comparing the comparison texts to the submitted text; and based on the results of the comparing step, determining a probability whether the submitted text is derived from one or more of the comparison texts.

Description

GENERATIVE ARTIFICIAL INTELLIGENCE AUTHORSHIP DETECTIONField of the Invention

[0001] The present Application relates to the field of forensic analysis of text, and more specifically, but not exclusively, to methods and systems for determining whether a text was produced by a human or by an artificial intelligence, through generation of Al-derived comparison texts.Background of the Invention

[0002] As the capabilities of artificial intelligence (Al) increase, artificial intelligence computer programs are becoming increasingly adept at generating text. One well-known computer program is ChatGPT. Such computer programs are based on large language models that include, but are not limited to, Bard, ChatGPT and GPT 4.

[0003] In various contexts, it is valuable to determine whether a text was written by a human as opposed to by an artificial intelligence. These contexts include, without limitation: education (ensuring that a student personally completed an assignment) and search engine optimization (ensuring that the search engine considers only examples generated by a human as opposed to those generated by a machine).

[0004] Various "passive" metrics exist for evaluating whether a text was generated by a human or by an Al. "Passive" means that the analysis is performed solely on the text itself, without considering any information about how the text was generated. Passive Al checkers operate by evaluating the text with models that are trained on human-written and Al-generated text, applying a "black-box" stylealgorithm to the text, and outputting a determination. Artificial intelligence checkers based on passive metrics are able to predict the presence of text generated by an artificial intelligence with a high degree of accuracy. For example, Copyleaks®, the assignee of the present invention, has developed an artificial intelligence checker with over 99% accuracy.Summary of the Invention

[0005] While passive Al detectors have made great progress, they are still not foolproof. On those days, according to internal testing, up to 1% of texts may be misclassified - in either direction. Some human-written texts may be misclassified as written by Al, and some Al-generated texts may be misclassified as being written by humans. Whatever the purpose of the Al detection, it is desirable to achieve greater accuracy. For example, so long as approximately 1% of all Al-generated text may go undetected, a student (for example) may plausibly deny that the text that he submitted was in fact generated by Al.

[0006] In addition, as the ability of Al to approach human-like text continues to increase, the use of "passive" techniques for verifying human authorship is likely to become futile. Simply put, the Al is expected to get "smarter" and learn how to match its writing style even more closely to that of humans. Accordingly, additional techniques are needed in order to verify, with a high probability, that a given text was produced by a human.

[0007] The present disclosure introduces systems and methods for increasing the ability to detect Al-generated text. In particular, the present disclosure introduces the use of "generative" artificial intelligence authorship detection. Theterm "generative" means that, instead of analyzing the submitted text using passive techniques, the system generates comparison texts, from the prompt for the assignment, using known Al text generation programs. The assignment text is then compared to the comparison texts. This comparison may be performed using methods similar to those utilized by plagiarism checkers. Optionally, semantic variants of the prompt are also fed to the Al text generation programs, and the resulting texts are added to the collection of comparison texts. The generative Al detection thus not only enables an estimation of the probability of Al-generated text, but also enables demonstration of the process by which the Al-generated text was made, thereby providing incontrovertible proof of the use of Al.

[0008] According to the first aspect, a method of detecting whether a submitted text was generated by an artificial intelligence computer program is disclosed. The method includes: feeding a textual prompt, in response to which the submitted text had been written, into a plurality of artificial intelligence composition programs; generating comparison texts with the artificial intelligence composition programs that are responsive to the textual prompts; comparing the comparison texts to the submitted text; and based on the results of the comparing step, determining a probability whether the submitted text is derived from one or more of the comparison texts.

[0009] In another implementation according to the first aspect, the method further includes: generating semantically similar variants to the textual prompt; feeding the semantically similar variants into the plurality of artificial intelligencecomputer programs, to generate additional comparison texts; and performing the comparing and determining steps with reference to the additional comparison texts.

[0010] In another implementation according to the first aspect, the method further includes: feeding the textual prompt into the plurality of artificial intelligence composition programs using a plurality of distinct user accounts to thereby generate different comparison texts corresponding to pre-registered preferences for each of the distinct user accounts.

[0011] In another implementation according to the first aspect, the method further includes comparing the submitted text with a one or more additional texts submitted contemporaneously, and, based on the comparison, determining a probability that the submitted text and the one or more additional texts were all generated by an artificial intelligence computer program.

[0012] In another implementation according to the first aspect, the method further includes comparing the submitted text with one or more additional texts that were previously submitted and stored in a database, and, based on the comparison, determining a probability that the submitted text was derived from the one or more additional texts.

[0013] According to a second aspect, a system for detecting whether a submitted text was generated by an artificial intelligence computer program is disclosed. The system includes a program database comprising a list of artificial intelligence composition programs; a first software module configured to generate comparison texts by feeding a textual prompt, in response to which the submitted text had been generated, to one or more of the artificial intelligence compositionprograms of the database; and a second software module configured to compare the submitted text to one or more comparison texts, and to determine a probability that the submitted text is derived from one or more of the comparison texts.

[0014] In another implementation according to the second aspect, the second software module is further configured to generate semantically similar variants to the textual prompt; feed the semantically similar variants into the plurality of artificial intelligence computer programs, generate additional comparison texts; and perform the comparing and determining steps with reference to the additional comparison texts.

[0015] In another implementation according to the second aspect, the first software module is configured to feed the textual prompt into the plurality of artificial intelligence composition programs using a plurality of distinct user accounts to thereby generate different comparison texts corresponding to pre-registered preferences for each of the distinct user accounts.

[0016] In another implementation according to the second aspect, the system further includes a text database containing a plurality of previously submitted texts, and the second software module is further configured to compare the submitted text with previously submitted texts and with other contemporaneously submitted comparison texts.Brief Description of the Drawings

[0017] FIG. 1A illustrates output of a prior art plagiarism detector on plagiarized text;

[0018] FIG. IB illustrates output of a prior art plagiarism detector on Al- generated text;

[0019] FIG. 1C illustrates output of a prior art Al detector on Al-generated text;

[0020] FIG. 2 illustrates steps in a method for determining whether a text was generated by Al, using comparisons to Al-generated text and to database texts, according to embodiments of the present disclosure;

[0021] FIG. 3A illustrates an original textual prompt and a modified textual prompt, according to embodiments of the present disclosure;

[0022] FIG. 3B illustrates Al-generated responses to the prompts of FIG. 3A, according to embodiments of the present disclosure; and

[0023] FIG. 4 illustrates the collection and comparison of texts for inclusion in a database, according to embodiments of the present disclosure.Detailed Description of the Invention

[0024] The present Application relates to the field of forensic analysis of text, and more specifically, but not exclusively, to methods and systems for determining whether a text was produced by a human or by an artificial intelligence, through generation of Al-derived comparison texts.

[0025] Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or theExamples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0026] For purposes of exemplification, the present disclosure uses examples of writing assignments given to high school students. This is a common scenario in which there is an incentive and opportunity to pass off Al-generated text as human- written. Of course, the principles of the present disclosure apply equally well to any context in which text generated by artificial intelligence may be passed off as written by humans. These contexts may include, but are not limited to, journalism, scholarship, advertising, copywriting, and speechwriting.

[0027] One exemplary assignment described herein asks for comment about the interweaving of fictional characters with historical figures in Tolstoy's War and Peace. A second exemplary assignment asks for a listing and description of the turning points in World War II. Each student in the class submits an essay in response, and the teacher attempts to determine whether the submitted essay was written by an artificial intelligence or not.

[0028] In such circumstances, it is especially tempting for the student to submit the prompt to an artificial intelligence text generator. FIGS. 1A-1C exemplify why this is the case. In FIG. 1A, a student has submitted an essay response that is copied, verbatim, from an article published on the subject by the Sparknotes® website. The plagiarism checker detects that all 184 words of the submitted paragraph are identical to a selection from Sparknotes, and assigns a 100% plagiarism score. In FIG. IB, the student, cognizant of the teacher's skill with plagiarism checkers, has decided instead to generate a new text using an artificialintelligence text generator. The plagiarism checker was not able to find any matches, appropriately, since the text does not necessarily correspond to any published texts or websites. In FIG. 1C, the same text is fed to a passive Al detector. The passive Al detector, relying only on analysis of the text as it appears, determines a 92% probability that the text was generated by Al. While this risk of being caught may well discourage the student from submitting this response, the student does have plausible deniability regarding whether the text was, in fact, generated by Al. The student may seek to exploit this deniability to avoid punishment.

[0029] FIG. 2 illustrates steps in a method 100 for determining whether a text was generated by artificial intelligence. The method may proceed in parallel through generation of Al-generated texts and consultation with a proprietary database of Al- generated texts.

[0030] The method is implemented by a system, e.g., a computer including a processor and a memory. The memory is a non-transitory medium storing instructions that, when executed by the processor, cause the performance of steps in the method 100. The system is described herein as a "generative Al detector." The system is further described herein as having various "software modules" which are configured to perform various functions, and various "databases" which are used to store and provide access to lists of generative Al programs and documents.

[0031] At step 101, a "first software module" of the generative Al detector feeds a textual prompt into an Al text generator. The textual prompt is, in the case of an assignment to a student, the prompt that was given to the student. For example,as illustrated in FIG. 3A, on the left side, in the context of a literature assignment, the textual prompt may be as follows:War and Peace is a historical novel. Tolstoy made great efforts to ensure the accuracy of his facts and dates, and the characters of Tsar Alexander I, Napoleon, Speranski, and other dignitaries generally respect historical factuality. Yet almost all the important and interesting characters in the novel are fictional. Why does Tolstoy merge fact and fiction in this fashion?This textual prompt may be supplied to various chatbot sites and proprietary programs for Al-based text generation. A list of these sites and programs is stored, and continually updated, on a database of the system. For example, without limitation, the programs for Al-based text generation may include ChatGPT, ChatSonic, Jasper, YouChat, Google Bard, Socratic by Google, and Microsoft Bing Al.

[0032] Optionally, at step 102, the program may modify the textual prompt using semantically similar terms. The program submits the modified prompt to the programs for Al-based text generation. This step is advantageous because even a change of a few words may induce the Al-text generators to produce different texts with regard to content, order, and structure.

[0033] For example, at the right side of FIG. 3A, the user has modified the assigned prompt to read:War and Peace addresses historical events. Tolstoy cared to ensure the accuracy of his facts and dates, and the characters of Tsar Alexander I, Napoleon, Speranski, and other dignitaries are portrayed with historicalaccuracy. Yet many interesting characters in the novel are made up. Why does Tolstoy combine fact and fiction?This prompt is not very different from the original prompt, but it resulted in a different answer, as shown on the right side of FIG. 3B. While this answer is also written in a style that is characteristic of Al-generated text, and which may be identified as such using the passive metrics described above, it is sufficiently different from the answer on the left side of FIG. 3B that the teacher, possessing only the Al-generated text on the left side of FIG. 3B, may not have the extra level of conclusive proof that the submitted text on the right side of FIG. 3B was Al- generated.

[0034] In addition, the program may submit the same prompt to the same text generation program, but using different user accounts. Such duplication is useful because artificial intelligence text generation programs routinely give different responses to the same prompt. The different responses may respond to various considerations, including differences in previous interactions between the same user and the Al text generator. Thus, an artificial intelligence text generator may give a different response based on whether its interlocutor typically uses more formal or informal language, has previously expressed interest in one topic vs. another topic, or has expressed preferences for longer or shorter answers, etc.

[0035] At step 103, the generative Al detector generates and collects the comparison texts. The comparison texts are saved and made available for analysis.

[0036] At step 104, the program, in a second software module, compares the submitted text to the comparison texts. The technical process of comparison issimilar to that of a plagiarism checker. In such analyses, the following types of similarities may be indicative of copying: similar order of topics, similar sequences of words, similar length of answers, and / or similarity of meaning. The comparison may proceed on a basis of words, lines, groups of lines, as well as on the level of the document as a whole. The comparison results in identification text that is exactly copied from the Al-generated text, and identification of text that is semantically similar to the Al-generated text.

[0037] At step 105, the program determines a probability of whether the submitted text was derived from one or more of the comparison texts. In this context, the program may, at step 109, generate a report. This report may include the percentage of copied words, and the degree of semantic similarity between the submitted texts and the Al-generated text. The program may further report a prediction, with a degree of confidence, that the student used a particular artificial intelligence program in order to generate the submitted response.

[0038] Steps 106-108 illustrate a second aspect of checking for Al-generated text. Whereas, in the first aspect, the comparison texts are generated as a predicate for the comparison, in the second aspect, at steps 106-108, the comparison texts are considered from other student-generated texts and / or from texts stored in a text database.

[0039] The student-generated texts are texts that are submitted contemporaneously by other students to the same assignments. Often, two students may independently use artificial intelligence text generators in order to answer thesame assignments. These text generators may generate texts that are different from each other, but are similar in style and scope.

[0040] To exemplify this point, a second example is considered. In this example, a student is asked to describe the turning points of World War II. This is a multifaceted question, which necessarily is open to many answers. Indeed, when the same question is asked three consecutive times to an Al text generator, the text generator produces three different lists. These lists are illustrated in FIG. 4. The lists differ both with respect to which events were selected, how the events are named (for events that appear on all lists), and how the events are ordered. Although, for brevity, the detailed answers are not provided, it is apparent that there would be minor variations in the text of each paragraph about the events as well.

[0041] Thus, it is conceivable that two students generated answers derived from artificial intelligence text generators but which are facially different from each other. Thus, at step 107, two or more student-submitted texts are compared to each other. When (1) both a conventional artificial intelligence checker determines that each text was likely written by artificial intelligence (e.g., through syntax and sentence structure), and both student-submitted texts contain similarly written paragraphs and similar wording, as described above, it is possible to predict with a higher degree of confidence, at step 108, that the texts were generated from artificial intelligence. In other words, while a single student may retain some vestige of plausible deniability as to whether his text was written by an artificial intelligence, when two or more papers exhibit the same phenomena, and are also deemed to besimilar to each other, the chances of the determination being mistaken are reduced to virtually nil.

[0042] Still referring to steps 107 and 108, the same logic may apply equally well to using other texts in a database to check for copying or plagiarism of Al- derived texts. A long-term database of Al-derived texts may be stored, and may be used for as a basis for comparison with respect to the submitted text. The database may be built through any suitable method, including but not limited to feeding prompts to Al-based text generators and storing the outcomes, as well as storing texts that have otherwise been checked and determined to be Al-generated. Comparison to the Al-derived texts in the database enables detection of student plagiarizing of Al-derived texts as well as of Al models plagiarizing themselves.

[0043] For example, in the context of education, students may collectively submit a large group of texts that are responsive to the same question. A portion of such texts may be deemed to be generated by artificial intelligence, with extremely high probability, through the methods described above. This group of texts may be introduced into the long-term database and further used to evaluate whether the submitted text was written by a human or by an artificial intelligence. The maintenance of a long-term database may be particularly relevant in order to detect secondary copying from an artificial intelligence. In addition, the long-term text database may be used in order to evaluate whether an Al-based text generator copied or plagiarized from itself, i.e., from a text that it itself had previously generated.

[0044] The long-term text database is especially useful for detecting copying from Al-derived texts which may be difficult to reproduce directly. In the same way that an Al-based text generator may generate different answers when a prompt is given to it multiple times, Al-generated answers to the same question may exhibit slight variation over time. This is because, among other factors, the inputs that are considered by Al-based text generators in providing its responses are constantly updated as new information is published and disseminated. The Al-based response is a snapshot of the information available to the Al-based text generator at any given moment. Thus, suppose a student copied an answer from another student, who had submitted an answer during the prior year from an Al-based text generator. If the previous student had copied a full year ago, it may be impossible to reproduce the exact text that served as the source for this year's student. However, using the database, the source of the text may be preserved.

[0045] The text database may also serve as a proprietary database for the purposes of checking for plagiarism. For example, many of the submitted texts may not be publicly available. If the submitted text is very similar to a text in the database, even if neither text is flagged as being Al-generated, the text may still be flagged as having a high likelihood of being plagiarized.

[0046] Referring to step 109, the comparison to database texts may likewise be concluded with a report regarding the predicted source of the submitted text. This report may include, for example, information about the database text, such as the provenance of the database text and when the text was entered into the database.

[0047] In the foregoing description, the generative Al detector was described as a standalone program. In addition, the generative Al detector may be combined with other types of Al detection software. This includes Al detection software based on keystroke and input logging. In addition, the generative Al detector may be a plugin that may be incorporated into other software programs. These software programs may be, for example, word processing editing programs. In addition, the generative Al detector may be integrated into a suitable program that utilizes the output of the checker. For example, a content moderation may determine whether a text should be published on an online platform on the basis of whether the text was generated by a human or Al. The term "publish" is used in a broad sense and may refer to any form of publicizing or calling attention to a text. Likewise, a search engine optimization tool, or a social media platform, may choose whether to prioritize certain content on the basis of whether the text was generated by a human or by an Al.

Claims

What is claimed is:

1. A method of detecting whether a submitted text was generated by an artificial intelligence computer program, comprising: feeding a textual prompt, in response to which the submitted text had been written, into a plurality of artificial intelligence composition programs; generating comparison texts with the artificial intelligence composition programs that are responsive to the textual prompts; and comparing the comparison texts to the submitted text; and based on the results of the comparing step, determining a probability whether the submitted text is derived from one or more of the comparison texts.

2. The method of claim 1, further comprising: generating semantically similar variants to the textual prompt; feeding the semantically similar variants into the plurality of artificial intelligence computer programs, to generate additional comparison texts; and performing the comparing and determining steps with reference to the additional comparison texts.

3. The method of claim 1, further comprising feeding the textual prompt into the plurality of artificial intelligence composition programs using a plurality of distinct user accounts to thereby generate different comparison texts corresponding to preregistered preferences for each of the distinct user accounts.

4. The method of claim 1, further comprising comparing the submitted text with a one or more additional texts submitted contemporaneously, and, based on the comparison, determining a probability that the submitted text and the one or more additional texts were all generated by an artificial intelligence computer program.

5. The method of claim 1, further comprising comparing the submitted text with one or more additional texts that were previously submitted and stored in a database, and, based on the comparison, determining a probability that the submitted text was derived from the one or more additional texts.

6. A system for detecting whether a submitted text was generated by an artificial intelligence computer program, comprising: a program database comprising a list of artificial intelligence composition programs; a first software module configured to generate comparison texts by feeding a textual prompt, in response to which the submitted text had been generated, to one or more of the artificial intelligence composition programs of the database; and a second software module configured to compare the submitted text to one or more comparison texts, and to determine a probability that the submitted text is derived from one or more of the comparison texts.

7. The system of claim 6, wherein the second software module is further configured to generate semantically similar variants to the textual prompt; feed the semantically similar variants into the plurality of artificial intelligence computer programs, generate additional comparison texts; and perform the comparing and determining steps with reference to the additional comparison texts.

8. The system of claim 6, wherein the first software module is configured to feed the textual prompt into the plurality of artificial intelligence composition programs using a plurality of distinct user accounts to thereby generate different comparison texts corresponding to pre-registered preferences for each of the distinct user accounts.

9. The system of claim 6, further comprising a text database containing a plurality of previously submitted texts, and wherein the second software module is further configured to compare the submitted text with previously submitted texts and with other contemporaneously submitted comparison texts.

Images