Methods for generating tests to differentiate between humans and computers

Abstract

A method is provided for generating tests to distinguish between humans and computers. The computer may include a machine learning model. A reference sample is selected from a first category. A first plurality of samples are selected from the first category. The samples may be images. Adversarial examples are created for one or more of the first plurality of samples. Each of the adversarial examples is created for one or more other categories different from the first category. A second plurality of samples are selected from the one or more other categories different from the first category. Adversarial examples are created for the first category from one or more of the second plurality of samples. The reference sample and the first and second plurality of samples modified into adversarial examples are presented to a user for testing to determine whether the user is a human or a computer.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to protecting online platforms, and more specifically to a method of generating a test for distinguishing between humans and computers. BACKGROUND

[0002] Some online platforms require users to register on the platform before accessing the platform. For example, forums, online marketplaces, email services, online games, etc. can only be accessible by registered users. Typically, the services are intended to be used by humans. However, it is possible to create computer programs that register themselves as humans. These computer programs are sometimes referred to as "bots", which is an abbreviation of "robot". The use of bots can provide an opportunity for an attacker to abuse the online services.

[0003] Some online services use tests designed to help distinguish between human users and bots. One test used for this purpose is known as the Turing test. A form of the reverse Turing test commonly used is known as "Completely Automated Public Turing test to tell Computers and Humans Apart", more commonly known as CAPTCHA. Bots created for the purpose of accessing online services use machine learning (ML) models that are trained to classify, for example, images. To thwart bots, text CAPTCHAs can use distorted images of some text, while the bots can have a text recognition module that will attempt to guess the text in the image. As bot availability becomes stronger, more difficult CAPTCHA images are being used. Unfortunately, the more difficult CAPTCHAs can be too difficult for humans to consistently crack. As a result, a large proportion of CAPTCHA tests provide false positives, while also frustrating many human users.

[0004] Accordingly, there is a need for a test for distinguishing between humans and bots that addresses the above problems. SUMMARY

[0005] According to a first aspect of the application, there is provided a method of generating a test for distinguishing between humans and computers, the method comprising:

[0006] selecting a first category;

[0007] selecting a reference sample in the selected first category;

[0008] selecting a first plurality of samples, the first plurality of samples being selected from the first category;

[0009] creating adversarial examples for one or more of the first plurality of samples to create a modified first plurality of samples, each of the adversarial examples being for one or more other categories different from the first category;

[0010] selecting a second plurality of samples, the second plurality of samples selected from one or more other categories different from the first category;

[0011] creating adversarial examples of one or more of the second plurality of samples for the first category to create a modified second plurality of samples; and

[0012] presenting the reference sample and the first and second plurality of samples modified with the adversarial examples for a user to test to determine whether the user is human or a computer.

[0013] In one or more embodiments, the method further comprises creating adversarial examples of the reference sample, the adversarial examples for a second category different from the first category.

[0014] In one or more embodiments, the reference sample and the first and second plurality of samples are images.

[0015] In one or more embodiments, the reference sample and the first and second plurality of samples are audio files.

[0016] In one or more embodiments, the adversarial examples are created by adding noise to a sample.

[0017] In one or more embodiments, creating adversarial examples for one or more of the first and second plurality of samples further comprises creating adversarial examples for all of the first and second plurality of samples.

[0018] In one or more embodiments, creating adversarial examples for one or more of the first plurality of samples further comprises creating adversarial examples for one or more of the first plurality of samples, the adversarial examples for one or more of the same categories as the second plurality of samples.

[0019] In one or more embodiments, the presenting for a user to test to determine whether the user is human or a computer further comprises testing using a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA).

[0020] In one or more embodiments, creating adversarial examples for the one or more of the first and second plurality of samples further comprises creating each of the adversarial examples to be for a different machine learning algorithm.

[0021] According to a second aspect of the application, there is provided a method of generating a test for distinguishing between humans and computers, the method comprising:

[0022] selecting a first category;

[0023] selecting a reference sample in a selected one of the first categories;

[0024] creating adversarial examples of the reference sample for a second category different from the first category;

[0025] selecting a first plurality of samples selected from the first category;

[0026] creating adversarial examples for one or more of the first plurality of samples to create a modified first plurality of samples, each of the adversarial examples being for one or more other categories different from the first category;

[0027] selecting a second plurality of samples selected from the one or more other categories;

[0028] creating adversarial examples of one or more of the second plurality of samples for the first category to create a modified second plurality of samples; and

[0029] presenting the reference sample and the modified first and second pluralities of samples for a user to test to determine whether the user is human or computer.

[0030] In one or more embodiments, the reference sample and the first and second pluralities of samples are images.

[0031] In one or more embodiments, creating adversarial examples for one or more of the first and second pluralities of samples further comprises creating adversarial examples for all of the first and second pluralities of samples.

[0032] In one or more embodiments, creating adversarial examples for one or more of the first plurality of samples further comprises creating adversarial examples for one or more of the first plurality of samples, the adversarial examples being for the same category as one or more of the second plurality of samples.

[0033] In one or more embodiments, the for a user to test to determine whether the user is human or computer further comprises using a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) to conduct the testing.

[0034] In one or more embodiments, creating adversarial examples for the one or more of the first and second pluralities of samples further comprises creating each of the adversarial examples to be for a different machine learning algorithm.

[0035] According to a third aspect of the present invention, a non-transitory machine-readable storage medium is provided, the non-transitory machine-readable storage medium comprising computer-readable instructions executable by a microprocessor to perform the following operations:

[0036] Select the first category;

[0037] Select a reference sample from the first selected category;

[0038] Select a first plurality of samples, wherein the first plurality of samples are selected from the first category;

[0039] Adversarial examples are created for one or more of the first plurality of samples to create modified first plurality of samples, each of the adversarial examples being used for one or more other categories different from the first category;

[0040] Select a second plurality of samples, wherein the second plurality of samples are selected from one or more other categories different from the first category;

[0041] Adversarial examples are created for one or more samples from the second plurality of samples against the first category to create modified second plurality of samples; and

[0042] The reference sample and the modified first and second samples are presented for user testing to determine whether the user is a human or a computer.

[0043] In one or more embodiments, the non-transitory machine-readable storage medium further includes instructions for creating adversarial examples of the reference sample for a second category different from the first category.

[0044] In one or more embodiments, the instructions for creating adversarial examples for one or more of the first and second plurality of samples further include instructions for creating adversarial examples for all of the first and second plurality of samples.

[0045] In one or more embodiments, the instructions for creating adversarial examples for one or more of the first plurality of samples further include instructions for creating adversarial examples for one or more of the first plurality of samples, the adversarial examples being of the same category as the second plurality of samples.

[0046] In one or more embodiments, the instructions for creating adversarial examples for one or more of the first and second plurality of samples further include instructions for creating each of the adversarial examples for a different machine learning algorithm.

[0047] These and other aspects of the present application will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0048] The present application is illustrated by way of example and not limitation in the figures of which like references indicate similar elements. Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.

[0049] Figure 1 A method for distinguishing between humans and computers according to an embodiment is shown.

[0050] Figure 2 A sample test set for distinguishing between humans and computers in a first part of a method according to an embodiment is shown.

[0051] Figure 3 A modified sample test set from the test set of Figure 2 according to a second part of the method is shown.

[0052] Figure 4 A data processing system usable to implement the method of Figure 1 is shown. DETAILED DESCRIPTION

[0053] In general, a method of generating a test for distinguishing between humans and computers is provided. The method comprises generating a CAPTCHA comprising adversarial examples. According to an embodiment, the CAPTCHA comprises a plurality of samples from a reference class. The samples can be images. The reference samples are selected from the plurality of samples in the reference class. Some of the plurality of samples are in the reference class and some are in a different, non-reference class. Some or all of the plurality of samples are modified into adversarial examples. The adversarial examples are arranged in the CAPTCHA and the CAPTCHA is presented to a user when accessing access to, for example, an online service. The purpose of the test is to trick an ML model of an automated program to identify the adversarial examples as being in a different and incorrect class, while a human will be able to correctly classify them relatively easily.

[0054] In this patent application, an adversarial example is a data sample comprising intentional perturbations of features that cause an ML model to incorrectly classify the adversarial example. The modification or perturbation can be a small pixel change to a feature of, for example, an image that is not perceptible to the human eye, such that a human would not incorrectly describe the image, but a machine learning model of an attacking computer would likely incorrectly classify the image.

[0055] The method provides a defense against automated program attacks that use ML models for input sample classification. The described method makes images more difficult for automated programs to recognize while not making the images more difficult for humans to classify.

[0056] According to embodiments, a method of generating tests for distinguishing between humans and computers is provided, the method comprising: selecting a first category; selecting a reference sample in the selected first category; selecting a first plurality of samples, the first plurality of samples selected from the first category; creating adversarial examples for one or more of the first plurality of samples to create a modified first plurality of samples, each of the adversarial examples for one or more other categories different from the first category; selecting a second plurality of samples, the second plurality of samples selected from the one or more other categories different from the first category; creating adversarial examples for one or more of the second plurality of samples for the first category to create a modified second plurality of samples; and presenting the reference sample and the first and second plurality of samples modified with the adversarial examples for a user test to determine whether the user is a human or a computer. The method can further comprise creating adversarial examples for the reference sample for a second category different from the first category. The reference sample and the first and second plurality of samples can be images. The reference sample and the first and second plurality of samples can be audio files. The adversarial examples can be created by adding noise to the samples. Creating adversarial examples for one or more of both the first and second plurality of samples can further comprise creating adversarial examples for all of the first and second plurality of samples. Creating adversarial examples for one or more of the first plurality of samples can further comprise creating adversarial examples for one or more of the first plurality of samples, the adversarial examples for one or more of the same categories as the second plurality of samples. The user test to determine whether the user is a human or a computer can further comprise testing using a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA). Creating adversarial examples for one or more of the first and second plurality of samples can further comprise creating each of the adversarial examples to be for a different machine learning algorithm.

[0057] In another embodiment, a method of generating a test for distinguishing between a human and a computer is provided, the method comprising: selecting a first category; selecting a reference sample in the selected first category; creating adversarial examples of the reference sample for a second category different from the first category; selecting a first plurality of samples, the first plurality of samples selected from the first category; creating adversarial examples for one or more of the first plurality of samples to create a modified first plurality of samples, each of the adversarial examples for one or more other categories different from the first category; selecting a second plurality of samples, the second plurality of samples selected from the one or more other categories; creating adversarial examples for one or more of the second plurality of samples for the first category to create a modified second plurality of samples; and presenting the reference sample and the modified first and second pluralities of samples for a user test to determine whether the user is a human or a computer. The reference sample and the first and second pluralities of samples can be images. Creating adversarial examples for one or more of both the first and second pluralities of samples can further comprise creating adversarial examples for all of the first and second pluralities of samples. Creating adversarial examples for one or more of the first plurality of samples can further comprise creating adversarial examples for one or more of the first plurality of samples, the adversarial examples for the same category as one or more of the second plurality of samples. Presenting for a user test to determine whether the user is a human or a computer can further comprise using a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) to conduct the test. Creating adversarial examples for one or more of the first and second pluralities of samples can further comprise creating each of the adversarial examples to be for a different machine learning algorithm.

[0058] In yet another embodiment, a non-transitory machine-readable storage medium is provided that includes computer-readable instructions executable by a microprocessor to select a first category, select a reference sample in the selected first category, select a first plurality of samples selected from the first category, create adversarial examples for one or more of the first plurality of samples to create a modified first plurality of samples, each of the adversarial examples being for one or more other categories different from the first category, select a second plurality of samples selected from the one or more other categories different from the first category, create adversarial examples for one or more of the second plurality of samples for the first category to create a modified second plurality of samples, and present the reference sample and the modified first and second pluralities of samples for a user to test to determine whether the user is human or a computer. The non-transitory machine-readable storage medium can further include instructions to create adversarial examples for a reference sample for a second category different from the first category. The instructions to create adversarial examples for one or more of the first and second pluralities of samples can further include instructions to create adversarial examples for all of the first and second pluralities of samples. The instructions to create adversarial examples for one or more of the first plurality of samples can further include instructions to create adversarial examples for one or more of the first plurality of samples for the same category as the second plurality of samples. The instructions to create adversarial examples for one or more of the first and second pluralities of samples can further include instructions to create each of the adversarial examples for a different machine learning algorithm.

[0059] Figure 1 A method 10 of generating a test for distinguishing between humans and computers according to an embodiment is shown. Reference will be made to Figures 1-3 The method 10 is described. In the described embodiment, the samples are images. The images in the described embodiment can be any kind of image, such as a photograph, a video, a drawing, a diagram, text, etc. In another embodiment, the method 10 can be applied to other types of samples. For example, the method 10 can be applied to audio samples for people with visual impairments. The audio samples can include speech, music, or various other sounds such as traffic, animals, construction, etc. The method begins by constructing a CAPTCHA as shown in Figure 2 There are various ways to construct and arrange the images of the CAPTCHA. Figure 2CAPTCHA is just one example. In general, in a CAPTCHA, a reference image is presented, and the user is required to select one or more images from a set of images in the same category as the reference image. Successful identification of the image category will allow the user to access the resource protected by the CAPTCHA. The method 10 will be described in detail below after the description of Figure 2 and Figure 3 The method 10 will be described in detail below after the description of

[0060] Figure 2 A test set 30 of samples used to distinguish between human responses and computer responses in the first part of the method 10 is shown. Figure 2 includes a reference sample 32 and a plurality of samples 34, 36, 38, 40, 42, 44, 46, 48, and 50. While Figure 2 nine samples are shown in the CAPTCHA, any number can be used. Each of the samples is selected from a category of images. The actual images selected for the CAPTCHA are not important to the description of the invention. The images can be any images that a human would be relatively easy to identify as being in a particular category. The reference sample 32 is an image selected from a category X, where X represents the category. Example categories can include photographs or other images of cars, people, animals, fish, road signs, houses, etc. Figure 2 Each of the plurality of samples in the CAPTCHA is selected from the indicated category. For example, samples 34, 40, 42, 44, and 48 are each different images that can be visually identified by a human as being in the same category X as the reference sample 32. If a user correctly labels samples 34, 40, 42, 44, and 48 as being in the same category as the reference sample 32, then the CAPTCHA will indicate that the user is a human. While there are five correct images in Figure 2 the CAPTCHA, any number of correct images can be present in other embodiments. The remaining images of the CAPTCHA are filled with images that a human would be relatively easy to identify as being different from category X. For example, image 36 is from category B, image 38 is from category D, image 46 is from category E, and image 50 is from category Y. It does not matter what the specific categories are if the images selected from those categories can be identified by a human as being from a different category than category X. The samples are selected so that most people will be able to identify samples 34, 40, 42, 44, and 48 as belonging to the same category as the reference sample 32. For example, category X can be a photograph of a turkey, and selecting those images will determine that the user is a human, and then the user will be allowed to access the protected resource. However, machine learning models are becoming more and more available and can successfully identify many images. It should be noted that there can be any number of total samples in the CAPTCHA, where any number of the total samples are in the same category as the category of the reference image, and where the remaining other images are from one or more categories different from the reference category.

[0061] Figure 3 modified test set 30' of samples of Figure 2 According to a second part of method 10, the Figure 3 creation of adversarial examples is shown in Figure 3 for example X→Y, where X represents the original image class, and →Y indicates that an adversarial example is created from a class X image for class Y. This means that a human would still recognize the image as belonging to class X as previously described, but a computer with the ML model would be misled and would recognize the image as belonging to class Y, where class Y is different from class X. As Figure 2 indicated in

[0062] Referring back to Figure 1 , method 10 begins at step 12. At step 12, a first class is selected for Figure 2 reference samples 32. At step 14, reference samples 32 are selected from the first class (class X). In one embodiment, the class of reference samples is selected according to method 10 by preparing a software application for creating CAPTCHAs. At step 16, adversarial examples are created from reference samples 32. Step 16 is shown in Figure 3 where reference samples 32 are modified to adversarial examples 32' (X→Y).

[0063] Adversarial examples can be created by adding noise to the samples. There are various ways to add noise to the samples. In one embodiment, a small amount of noise can be added to disturb a few pixels of the added image. In another embodiment, a large amount of noise is added to a small area of the image. In most adversarial examples, the amount of noise is almost imperceptible to a human, but will cause the ML model to misclassify the image. There are various known methods to create adversarial examples, so how to create adversarial examples will not be discussed further.

[0064] In method 10, at step 18, a first plurality of samples is selected from the first class or as Figure 2The reference class X is shown in FIG. 1. As an example, five samples are selected as the first plurality of samples, namely, samples 34, 40, 42, 44, and 48. A different number can be selected for another embodiment. At step 20, one or more adversarial examples of the first plurality of samples are created for one or more other classes different from the first class to create a modified first plurality of samples. As discussed above, modifying an image into an adversarial example is intended to cause the ML model to misclassify the image without interfering with human recognition of the image. For example, an adversarial example 40' is created from sample 40, where the adversarial example 40' is for class T. Class T is a different class than class X. Likewise, an adversarial example 44' is created from sample 44, and an adversarial example 48' is created from sample 48. Both adversarial examples 44' and 48' are for class T. In Figure 3 In FIG. 1, no adversarial examples are created from samples 34 and 42, but in a different embodiment, adversarial examples can be created from both samples 34 and 42. It should be noted that generally, just one incorrect answer will cause a user to fail the CAPTCHA test, so even if the ML model is not fooled by all adversarial examples, the ML model can be fooled by at least one of the adversarial examples. To make the probability that at least one adversarial example will cause a misclassification greater, different techniques can be used to create adversarial examples to cover more than one different ML algorithm. In another embodiment, all of the first plurality of samples can be modified into adversarial examples.

[0065] At step 22, a second plurality of samples is selected from one or more other classes different from the reference class. The number of samples depends on the number of the first plurality of samples and the total number of samples in the CAPTCHA. For example, in the illustrated embodiment, there are nine samples in total, with the first plurality of samples being five, so the second plurality of samples is four. In other embodiments, the total number of samples and the first and second pluralities can be different. At step 24, adversarial examples are created from one or more of the second plurality of samples to create a modified second plurality of samples. The adversarial examples can be for the first class, as Figure 2 in FIG. 1, class X. As Figure 3 As shown in FIG. 1, samples 36 and 46 are modified into adversarial examples 36' and 46', respectively, which are both for class X. The reason class X is used for adversarial examples 36' and 46' is so that the ML model of the attacking computer will not provide the correct class classification as a possible result classification but with a lower confidence. Samples 38 and sample 50 are not modified, but can also be modified into adversarial examples. At step 26, the CAPTCHA test 30' can then be presented to a potential user of the protected resource. The CAPTCHA test can be presented in various ways. For example, the CAPTCHA test 30' is presented as Figure 3A set of possible response images that are positioned in proximity to the reference sample 32' shown in the middle. In another embodiment, the CAPTCHA can be presented in different ways.

[0066] The method provides defense against computers or automated programs that attack the use of ML models for sample classification. The described method makes it more difficult for computers with ML models to pass a CAPTCHA test while also not making it more difficult for humans to pass the test.

[0067] Figure 4 A data processing system 60 for implementing the method 10 of Figure 1 The data processing system 60 can be implemented on one or more integrated circuits and can be used to implement the described embodiments. The data processing system 60 includes a bus 62. A processor 64, a memory 66, a user interface 68, an instruction memory 70, and a network interface 72 are connected to the bus 62. The processor 64 can be any hardware device capable of executing instructions stored in the memory 66 or the instruction memory 70. For example, the processor 64 can execute instructions that implement ML algorithms that can be used to perform the method 10. The processor 64 can have multiple processing cores. The processor 64 can be, for example, a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or similar device. The processor 64 can be implemented in a secure hardware element and can be tamper-resistant.

[0068] The memory 66 can be any kind of memory, such as an LI, L2, or L3 cache or a system memory. The memory 66 can include volatile memory, such as static random access memory (SRAM) or dynamic RAM (DRAM), or can include non-volatile memory, such as flash memory, read-only memory (ROM), or other volatile or non-volatile memory. Also, the memory 66 can be implemented in a secure hardware element. Alternatively, the memory 66 can be a hard disk drive implemented external to the data processing system 60. In one embodiment, the memory 66 is used to store weight matrices of ML models or some images used to create a CAPTCHA.

[0069] The user interface 68 can be connected to one or more devices for enabling communication with a user, such as an administrator. For example, the user interface 68 can be enabled for coupling to a display, a mouse, a keyboard, or other input / output devices. The network interface 72 can include one or more devices for enabling communication with other hardware devices. For example, the network interface 72 can include or be coupled to a network interface card (NIC) configured to communicate according to an Ethernet protocol. Also, the network interface 72 can implement a TCP / IP stack for communicating according to a TCP / IP protocol. Data samples for classification can be input via the network interface 72 or similar interface. Various other hardware or configurations for communication can be available.

[0070] Instruction memory 70 can include one or more machine-readable storage media for storing instructions executed by processor 64. In other embodiments, both memories 66 and 70 can store data operable by processor 64. Memories 66 and 70 can also store, for example, encryption, decryption, and verification applications. Memories 66 and 70 can be implemented in secure hardware elements and be tamper-resistant.

[0071] While the application is described herein with reference to specific embodiments thereof, various modifications and changes can be made therein without departing from the scope of the present application as set forth in the claims that follow. The specification and drawings should therefore be construed in an illustrative rather than a restrictive sense, and all such modifications and variations should be deemed to be within the ambit of the present application. It should be understood that any benefits, advantages, or solutions to problems that are described herein as associated with any elements of specific embodiments mentioned herein can also be achieved by means of any other embodiment of the application. For example, benefits, advantages, or solutions to problems described herein that can be associated with one embodiment can also be achieved by means of another embodiment without departing from the scope of the present application.

[0072] Various embodiments or portions thereof can be implemented in hardware or as instructions on a non-transitory machine-readable storage medium, including any mechanism for storing information in a form readable by a machine, such as a personal computer, notebook computer, file server, smart phone, or other computing device. The non-transitory machine-readable storage medium can include volatile and nonvolatile memory, such as read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash memory, and the like. The non-transitory machine-readable storage medium does not include a transitory signal.

[0073] Furthermore, as used herein, the term "a" is defined as one or more than one. Also, the use of the leading article "the" in a claim to a phrase such as "at least one" and "one or more" should not be interpreted as implying that any particular claim element introduced by the indefinite article "a" will contain only one such element, even if that is the number of such element recited in that claim. The same applies to the use of the definite article "the" in a claim.

[0074] Unless otherwise stated, the terms such as "first" and "second" are used to arbitrarily distinguish one element from another element. Therefore, these terms are not necessarily intended to indicate a temporal priority or other priority.

Claims

1. A method of generating a test for distinguishing between a human and a computer, characterized by, The method comprises: selecting a first category; selecting a reference sample in the selected first category; selecting a first plurality of samples, the first plurality of samples selected from the first category; creating adversarial examples for one or more of the first plurality of samples to create a modified first plurality of samples, each of the adversarial examples for one or more other categories different from the first category; selecting a second plurality of samples, the second plurality of samples selected from the one or more other categories different from the first category; creating adversarial examples for one or more of the second plurality of samples for the first category to create a modified second plurality of samples; and presenting the reference sample and the first and second plurality of samples modified with the adversarial examples for a user to test to determine whether the user is human or computer, wherein the method further comprises creating adversarial examples for the reference sample, the adversarial examples for the reference sample for a second category different from the first category, and wherein creating adversarial examples for one or more of both the first and second plurality of samples further comprises creating adversarial examples for all of the first and second plurality of samples, wherein the reference sample and the first and second plurality of samples are images or audio files, and wherein the adversarial examples are created by adding noise to a sample.

2. The method of claim 1, wherein, The presenting for a user to test to determine whether the user is human or computer further comprises testing using a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA).

3. The method of claim 1, wherein, Creating adversarial examples for the one or more of the first and second plurality of samples further comprises creating each of the adversarial examples to be for a different machine learning algorithm.

Images