Image-based captcha challenge

Abstract

The present disclosure relates to image-based CAPTCHA challenges. Systems and methods of determining image features are provided. More specifically, a first image having unknown features can be obtained. The first image can be provided to a plurality of user devices in a verification challenge. The verification challenge can include one or more instructions to be presented to a user of each user device. The instructions are determined based at least in part on the first image. User responses can be received, and the unknown features of the first image can be determined based at least in part on the received responses. After the unknown features of the first image are determined, one or more machine learning models can be trained based at least in part on the determined features.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201680053314.5, filed on December 1, 2016. Technical Field

[0003] The present disclosure relates generally to CAPTCHA challenges, and more particularly to determining image features using image-based CAPTCHA challenges. Background Art

[0004] Trust is an asset in web-based interactions. For example, to feel comfortable interacting with an entity, a user must trust that entity to provide adequate mechanisms to confirm and protect their identity or other confidential information. Furthermore, entities providing web resources must be able to prevent automated attacks attempting to obtain web resources for malicious purposes. Therefore, sophisticated authentication mechanisms that can distinguish resource requests originating from humans from those generated by automated machines are essential tools for securing web resources from automated attacks and fostering the necessary trust between resource providers and users.

[0005] CAPTCHA systems (Completely Automated Public Turing test to tell Computers and Humans Apart) can provide this authentication mechanism. One goal of CAPTCHA systems is to exploit situations where humans are known to perform tasks better than automated machines. Therefore, as part of the verification process, CAPTCHA systems can provide a challenge that is solvable by humans but generally unsolvable by machines. Summary of the Invention

[0006] Solutions and advantages of the embodiments of the present disclosure will be given in part in the following description, or may be understood from the description, or may be understood through practice of the embodiments.

[0007] An example embodiment of the present disclosure relates to a computer-implemented method for determining image features. The method includes obtaining, by one or more computing devices, a first image having unknown features. The method also includes providing, by one or more computing devices, the first image to a plurality of user devices in a verification challenge. The verification challenge includes one or more instructions to be presented to a user of each user device. The instructions are determined based at least in part on the first image. The method also includes receiving, by one or more computing devices, responses to the verification challenge from at least a subset of the plurality of user devices. The method also includes determining, by one or more computing devices, the unknown features of the first image based at least in part on the received responses. The method also includes, after determining, by one or more computing devices, the unknown features of the first image, training, by one or more computing devices, one or more machine learning models based at least in part on the determined features.

[0008] The method may also include the following optional features. The method may also include receiving, via one or more computing devices, a request from a user computing device to participate in a verification process, and obtaining one or more first images in response to receiving the request. The method may also include obtaining, via the one or more computing devices, a plurality of second images having known features; and providing, via the one or more computing devices, at least a subset of the plurality of second images along with the one or more first images in the verification challenge. The unknown features of the one or more first images may be determined as unknown based at least in part on one or more machine learning models, the one or more machine learning models configured to receive one or more images as input and determine one or more features of the one or more images as output. The known features of the plurality of second images may be determined based at least in part on the one or more machine learning models. The one or more features of the one or more images may be related to one or more items or objects depicted in the one or more images. The one or more machine learning models may also be configured to provide a certainty probability indicating a degree of certainty that the output of the model is correct. When the certainty probability associated with the one or more first images is below a certainty threshold, the unknown features of the one or more first images may be determined as unknown. One or more instructions associated with the verification challenge may prompt the user to interact with one or more of the first or second images that meet the criteria. The one or more instructions associated with the verification challenge may prompt a user to interact with at least a portion of the one or more first images that meet the criteria. The one or more instructions associated with the verification challenge may prompt the user to select a portion of at least one of the one or more first images. The verification challenge may be determined based at least in part on an unknown characteristic of the one or more first images. The method may further include providing, by one or more computing devices, a third image in the verification challenge, the third image depicting an item or object, and wherein the one or more instructions associated with the verification challenge prompt the user to select one or more images from at least a subset of the one or more first images or the second images that depict items or objects of the same classification as the item or object depicted in the third image. The method may further include determining, by the one or more computing devices, whether to verify each of the plurality of user devices based at least in part on responses to the verification challenge. Determining, by the one or more computing devices, the unknown characteristic of the one or more first images may include inferring, by the one or more computing devices, the unknown characteristic of the one or more first images when a threshold number of user responses indicate a characteristic of the one or more first images.

[0009] In accordance with the exemplary aspects of providing the above-described methods, other exemplary aspects of the present disclosure are directed to systems, apparatuses, tangible, non-transitory computer-readable media, user interfaces, memory devices, and electronic devices for determining image features using image-based verification challenges.

[0010] The above solutions seek to solve problems associated with how to train machine learning models for image classification and image feature determination.

[0011] These and other features, aspects and advantages of various embodiments will become better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The description provides a detailed discussion of embodiments for those skilled in the art, and the description refers to the accompanying drawings, in which:

[0013] Figure 1 An overview of an example system for determining image features using an image-based CAPTCHA challenge according to an example embodiment of the present disclosure is shown.

[0014] Figure 2-Figure 4 An example CAPTCHA challenge is shown according to an example embodiment of the present disclosure.

[0015] Figure 5 A flowchart illustrating an example method for determining image features based on CAPTCHA challenge results according to an example embodiment of the present disclosure is shown.

[0016] Figure 6 A flowchart illustrating an example method of authenticating a user according to an example embodiment of the present disclosure.

[0017] Figure 7 An example system according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Reference will be made below in detail to the embodiments, one or more examples of which are illustrated in the accompanying drawings. Each example is provided by way of explanation of the embodiments, not by way of limitation of the present disclosure. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments without departing from the scope or spirit of the present disclosure. For example, a feature shown or described as part of one embodiment may be used in conjunction with another embodiment, thereby providing yet another embodiment. Therefore, it is intended that multiple aspects of the present disclosure encompass these modifications and variations.

[0019] Example schemes of the present disclosure are directed to determining image classification based, at least in part, on an image-based verification challenge. For example, an image having one or more unknown features may be obtained and provided to a plurality of user devices as part of a verification challenge, such as a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) challenge. Specifically, the challenge may include one or more instructions prompting a user to interact with a first image in a specified manner. A response to the challenge may be obtained from each of the plurality of user devices. At least one of the one or more unknown features of the first image may then be determined based, at least in part, on the responses from the plurality of user devices. Once the at least one unknown feature is determined, the first image may be used to train one or more machine learning models.

[0020] More specifically, a user may attempt to access an online service or resource through a user device. As used herein, a user device may be a laptop computer, a desktop computer, a smartphone, a tablet computer, a wearable computing device, or any other suitable user computing device. In order to access an online service, the user may first be required to interact with a computerized CAPTCHA system to verify the user's human identity, thereby preventing "bots" from disrupting the online service. For example, the CAPTCHA system may present a verification challenge to the user, and if the user "passes" the challenge, the user may gain access to the online service. In some embodiments, the verification challenge may be an image-based challenge, including instructions prompting the user to solve the challenge by interacting with one or more images.

[0021] For example, as indicated, one or more first images having one or more unknown features can be provided to the user in a verification challenge. For example, the unknown feature can be an uncertain classification associated with the image. In this manner, an image having one or more unknown features can be an image for which the classification of the content depicted in the image cannot be determined within a reasonable degree of certainty. As another example, the unknown feature can be associated with the location of an object or item in the image. In some embodiments, one or more machine learning models can be utilized to determine whether the features of an image are known or unknown. For example, one or more machine learning models (e.g., image classification, object detection, or other models) can be configured to receive an image as input and identify, classify, categorize, or otherwise detect one or more objects or items depicted in the image as output. The model can be further configured to determine a certainty probability corresponding to a degree of certainty that the output of the model is correct. In some embodiments, if the image has an associated certainty probability below a threshold, then the image feature may be unknown. For example, the first image having one or more unknown features can be an image for which the certainty probability is below a threshold.

[0022] In some embodiments, multiple second images may also be obtained, and at least a subset of the multiple second images may be provided to multiple user devices for use in the verification challenge. The multiple second images may be images having known characteristics. For example, the multiple second images may be images with an associated certainty probability above a threshold. As another example, the second images may be images with associated descriptive information, such as metadata or other information describing the image content. In this manner, at least a subset of the multiple second images may be provided along with the first image in the verification challenge.

[0023] The verification challenge can be an image-based challenge. In some embodiments, the verification challenge can be associated with an unknown feature of the first image. For example, the verification challenge can include instructions prompting the user to interact with at least a portion of the first image in a specified manner. Specifically, the instructions can prompt the user to interact with at least a portion of the first image that meets criteria. For example, the instructions associated with the verification challenge can prompt the user to locate and interact with a portion of the first image that depicts a specific object. As another example, the verification challenge can prompt the user to interact with at least one of the first or second images. For example, the verification challenge can prompt the user to select all images that depict a specific object or item or are associated with a specific category or classification. In this manner, the verification challenge can be tailored to determine or infer unknown features using responses to the verification challenge. As another example, in some embodiments, a reference image can be provided to the user along with the first and second images in the verification challenge. In such embodiments, the instructions associated with the verification challenge can prompt the user to select each of the first and second images that depict an object or item from the same category or classification as the object or item depicted in the reference image. As another example, the verification challenge can include one or more images and instructions prompting the user to select one or more categories from a list of categories describing the image.

[0024] A verification challenge may be presented to one or more user devices attempting to gain access to one or more online services. A response to the verification challenge may then be received from the user device. For example, if the verification challenge prompts a user to select one or more images that meet specified criteria, the response from the user may include data indicating the images selected by the user. The response from the user may be used to determine whether the user device should be granted access to the online service. For example, if the response satisfactorily completes or resolves the selected challenge, the computerized CAPTCHA system may authenticate the user computing device. As an example, once the computerized CAPTCHA system has been sufficiently satisfied that the user device is operated by a human (e.g., based on the response to the verification challenge), the computerized CAPTCHA system may provide a verification token or certificate to the user computing device. The user computing device may then provide the verification token to a resource provider associated with the online service. In turn, the resource provider may confirm the validity of the verification token with the computerized CAPTCHA system. After confirming the token's validity, the resource provider may provide resources to the user device. In some embodiments, the user's response to the verification challenge may not be decisive in determining whether to authenticate the user device. For example, even if the user correctly resolves the verification challenge, the user may still be denied (e.g., unauthenticated). As another example, a user may be authenticated even if the user does not correctly resolve the authentication challenge. In such an embodiment, one or more additional signals associated with the user and / or the user's computing device (e.g., in conjunction with the user's response) may be used to determine whether to authenticate the user. For example, a user who has nearly resolved the authentication challenge may be authenticated based at least in part on the one or more additional signals.

[0025] Responses to the verification challenge can further be used to determine or infer unknown features of the first image(s). Specifically, when a threshold number of user responses indicate features of the first image(s), the unknown features of the first image(s) can be determined or inferred. For example, a user's interaction with the first image(s) in response to the verification challenge can indicate an object or item depicted in the image(s). As an example, if an image-based verification challenge includes instructions prompting a user to select all images depicting tigers, then whether the user selects the first image in response to the verification challenge can indicate whether the first image depicts a tiger. In this manner, if enough users (e.g., a number exceeding a threshold) select the first image in response to receiving the verification challenge, it can be inferred that the image depicts a tiger. If the number of users who select the first image is below a threshold, it can be inferred that the first image does not depict a tiger. In some embodiments, this inference can be made based at least in part on the percentage of users who select the image. As another example, in embodiments where the verification challenge includes instructions prompting a user to locate and interact with a portion of the first image depicting a specific object, the user selection can be used to determine the location of the object in the first image.

[0026] In some embodiments, a user's response to a verification challenge can be used to validate a known image. For example, a user's response to a verification challenge can be used to verify that a known feature of a known image is correct. In this manner, if sufficient user responses associated with a known image indicate that the known feature is incorrect, the image can be removed from the set of known images. As another example, the known feature can be adjusted or refined based at least in part on the user responses.

[0027] Once the unknown features are determined, the first image(s) may be used to train one or more machine learning models. Specifically, data indicating the first image(s) may be added to a corpus of training data used to train the model. For example, the machine learning model may be used in image recognition applications, object detection applications, and / or other suitable applications. Specifically, a set of training data may be used to train the machine learning model. Such training data may include data (e.g., vector data or other data) indicating one or more images having known features. For example, the training data may include data indicating one or more images, the content of which is known. In this manner, the training data may include multiple input / output pairs, where the input is vector data indicating an image and the output is data indicating the content of the image (e.g., one or more items depicted in the image). The machine learning model may be trained based at least in part on the input / output pairs of the training data. For example, vector data associated with the image may be input into the machine learning model(s), and the output of the model may be compared to the output of the training data. The model(s) may be adjusted or improved based at least in part on the comparison results.

[0028] The following is a more detailed discussion of exemplary embodiments of the present disclosure with reference to the accompanying drawings. Figure 1 An overview of an example system 100 for providing image-based authentication challenges is shown. Specifically, the system 100 may include a computerized CAPTCHA system 102, a machine learning network 104, and a resource provider 106. In some implementations, the CAPTCHA system 102, the machine learning network 104, and / or the resource provider 106 may communicate with each other via a network.

[0029] The CAPTCHA system 102 may be implemented using one or more computing devices (e.g., one or more servers). The CAPTCHA system 102 may be configured to determine a verification CAPTCHA challenge and provide such a challenge to the user device in response to a verification request from the user device. In some embodiments, the verification challenge may be an image-based CAPTCHA challenge. For example, the CAPTCHA challenge may include providing one or more known and / or unknown images to the user. Such an image-based challenge may have associated instructions prompting the user to resolve the challenge by interacting with one or more images. For example, as indicated above, the challenge may include instructions prompting the user to select all presented images that match or belong to a particular category or classification. As another example, the challenge may include instructions prompting the user to select or otherwise interact with a portion of an image depicting a particular item, object, animal, or the like.

[0030] As shown, the CAPTCHA system 102 can access image data from an image database 108. The image database 108 may include multiple images to be used in one or more verification challenges. In an example embodiment, the image database 108 may be a single database located in a single location, or may include multiple databases distributed across multiple locations. Specifically, the image database 108 may include known images and unknown images. The known images may include images having one or more known features. For example, the known images may be images that have been classified or categorized. In some embodiments, the one or more known features may be determined based at least in part on metadata or other descriptive information associated with the image. In some embodiments, the one or more known features may be determined based at least in part on a machine learning network 104. For example, the machine learning network 104 may be used to classify, categorize, recognize, or otherwise detect one or more objects depicted in an image.

[0031] Image database 108 may also include one or more unknown images having one or more unknown features. Similar to known images, the unknown feature(s) may be determined to be unknown based at least in part on a lack of metadata or other descriptive information associated with the unknown feature(s). As another example, the unknown feature(s) may be determined to be unknown based at least in part on the machine learning network 104.

[0032] In some embodiments, known and / or unknown images can be determined relative to the verification challenge. For example, in embodiments where the instructions associated with the verification challenge prompt the user to select each image depicting a tiger, the known images can include one or more images from the image database 108 that are known to depict tigers and / or known not to depict tigers. In this manner, the unknown image(s) can include images where the content depicted in the image is unknown. In some embodiments, the verification challenge can then be used to determine or infer the content depicted in the unknown image(s).

[0033] The machine learning network 104 may include one or more machine learning models, such as one or more neural networks. For example, one or more models may be configured to classify an image and / or detect an object depicted in an image. In this manner, the machine learning network 104 may be configured to receive an image as input and provide an image classification as output. In some embodiments, the machine learning network 104 may also be configured to provide a deterministic probability or other metric associated with the output with a specified confidence level for output. In some embodiments, the machine learning network 104 may be configured to communicate with the image database 108 to classify one or more images associated with the image database 108. In this manner, in some embodiments, an unknown image may be an image with an associated deterministic probability below a threshold, while a known image may be an image with an associated deterministic probability above a threshold.

[0034] The machine learning network 104 can be further configured to train one or more machine learning models. For example, various suitable algorithms can be used to improve or adjust the machine learning models based at least in part on a set of training data. In this manner, the training data can be used to improve the machine learning models. It should be understood that although the machine learning network 104 is shown as a separate and distinct system from the CAPTCHA system 102, in some embodiments, the machine learning network 104 can be implemented within the CAPTCHA system 102.

[0035] System 100 also includes a resource provider 106. Resource provider 106 may be associated with an online or other service. Resource provider 106 may receive a request from a user device to access the online service. In response to the request, resource provider 106 may coordinate with CAPTCHA system 102 to provide a verification challenge to the user device. In this manner, CAPTCHA system 102 may determine the verification challenge and provide it to the user. In some embodiments, CAPTCHA system 102 may determine the verification challenge based at least in part on the unknown image. For example, CAPTCHA system 102 may adjust the verification challenge so that the user's response to the verification challenge can be used as evidence for the classification or categorization of the unknown image. In some embodiments, the verification challenge may be determined based at least in part on the confidence level that the request was provided by a human rather than a robot. For example, the difficulty of the verification challenge may be adjusted based at least in part on the confidence level.

[0036] The user's response to the verification challenge can be used to determine the unknown feature(s) of the unknown image(s). For example, the user's response can be used as evidence related to the classification of the unknown image. For example, in an embodiment where the verification challenge includes an instruction to select all images depicting tigers, whether the user selects an unknown image can indicate whether the unknown image depicts a tiger. In this way, unknown images can be presented to multiple users in one or more verification challenges, and the responses from the users to the one or more verification challenges can be used to infer the unknown features of the images. For example, if enough users (e.g., a sufficiently high percentage of users and / or a number of users exceeding a threshold) select an unknown image in response to a verification challenge that instructs the user to select an image containing a tiger, then it can be inferred that the unknown image depicts a tiger.

[0037] Once the unknown features can be inferred with reasonable certainty, the image can be annotated to indicate the image's classification. The image can then be included in the training dataset used to train the machine learning model(s) of the machine learning network 104. In this way, the image can be used to improve the model's classification capabilities. In some embodiments, the annotated image can also be used in future verification challenges determined by the CAPTCHA system 102. For example, the annotated image can then be used as a known image in one or more verification challenges.

[0038] Figure 2Depicted is a CAPTCHA challenge 110 based on an example image according to an example embodiment of the present disclosure. As shown, the CAPTCHA challenge 110 includes instructions 112 and an associated reference image 114 that is known to depict a dog. The instructions 112 prompt the user to select all images that "look like" the reference image 114. The CAPTCHA challenge 110 also includes known images 116 and negative known images 118. The known images 116 are images that are known to depict dogs, and the negative known images 118 are images that are known not to depict dogs. Although each of the negative known images 118 depicts a cat, it should be understood that the negative known images may depict one or more other objects, items, plants, animals, humans, and the like. The CAPTCHA challenge 110 also depicts an unknown image 120. The unknown image 120 may be an image whose image content is unknown. As Figure 2 As shown, unknown image 120 depicts a lizard.

[0039] As described above, upon receiving the CAPTCHA challenge 110, the user can follow instructions and select one or more images that, in the user's opinion, "look like" the reference image 114. The user's selection can determine whether the user is allowed to access the resources or services provided by the resource provider. For example, if the user "passes" the challenge, the user can be granted access to the resource or service. For example, if the user selects the correct image combination, or a combination of images that is close to the correct image combination, the user can pass the challenge. In the CAPTCHA challenge 110, if the user selects every known image 116 and there are no negative known images 118, the user will pass. In some embodiments, if the user does not "pass", a different CAPTCHA challenge can be presented to the user.

[0040] Because unknown image 120 does not depict a dog, it is unlikely that most human users who would respond to the verification challenge would select unknown image 120 in response to CAPTCHA challenge 110. In this way, it can be inferred that unknown image 120 does not depict a dog. Similarly, if the unknown image does depict a dog, it is likely that most human users would select the unknown image in response to the CAPTCHA challenge. In this way, it can be inferred that the unknown image does depict a dog.

[0041] Once it is inferred that the unknown image 120 does not depict a dog, it can be used in subsequent verification challenges. For example, it can be used as an unknown image in a verification challenge with a reference image that does not depict a dog. In some embodiments, the unknown image 120 can then be used as a negative known image in a verification challenge with a reference image that depicts a dog.

[0042] Figure 3 and Figure 4An additional example image-based CAPTCHA challenge according to an example embodiment of the present disclosure is shown. For example, Figure 3 Depicted is a CAPTCHA challenge 122 with a reference image 124 depicting a beverage. The instructions prompt the user to select one or more appropriate categories associated with the reference image 124. One or more inferences and / or confirmations can be made about the reference image 124 based at least in part on the user selections. Figure 4 A CAPTCHA challenge 126 is shown with a reference image 128 depicting a scene. As shown, the reference image 128 is divided into a plurality of regions (e.g., blocks). The CAPTCHA challenge 126 also includes an instruction prompting the user to "select all blocks with a business name." In this manner, one or more inferences can be made regarding the location of the business name in the reference image 128 based on the user's response. For example, most human users would likely select image blocks 130 and 132 in response to the instruction. In this manner, it can then be inferred that the business name is located in the lower left portion of the reference image 128 in image blocks 130 and 132.

[0043] Figure 5 A flow chart illustrating an example method (200) for determining one or more image features using a computerized CAPTCHA system according to an example embodiment of the present disclosure. The method (200) may be performed by one or more computing devices (e.g., Figure 7 In addition, Figure 5 The steps are shown to be performed in a specific order for the purpose of illustration and discussion. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the steps of any method discussed herein may be adjusted, reconfigured, expanded, omitted, or modified in various ways without departing from the scope of the present disclosure.

[0044] At (202), the method (200) may include receiving a request to participate in a verification process. For example, the request may be provided by a user device operated by a user seeking to obtain access to an online service or resource. At (204), the method (200) may include obtaining a first image having an unknown feature. In some embodiments, one or more first images having an unknown feature may be obtained. As described above, the unknown feature may be associated with content depicted in the image. For example, the unknown feature may be associated with a classification or category of the image content. As another example, the unknown feature may be associated with detection of an object depicted in the image.

[0045] In some embodiments, an unknown feature can be determined to be unknown based at least in part on one or more machine learning models. For example, an image can be identified as an unknown image if the machine learning model cannot classify or categorize the image with reasonable certainty. Specifically, an image can be identified as an unknown image with an unknown feature if the model provides a certainty probability for the image that is below a threshold. As another example, an unknown feature can be determined to be unknown if there is insufficient metadata or other descriptive data associated with the feature.

[0046] At (206), the method (200) may include obtaining a plurality of second images having known characteristics. Similar to the first image, the known characteristics of the second image may be determined based at least in part on a machine learning model and / or metadata associated with the image. In some embodiments, the second image may be selected based at least in part on the first image and / or the unknown characteristics of the first image(s).

[0047] At (208), the method (200) may include generating a CAPTCHA challenge based at least in part on the first image and the second image. For example, the CAPTCHA challenge may include the first image and / or at least a subset of the plurality of second images, along with instructions prompting a user to interact with the image(s). As described above, in some embodiments, the challenge may be selected and / or generated based at least in part on unknown features of the first image.

[0048] At (210), the method (200) may include sending a CAPTCHA challenge to a plurality of user devices.For example, a CAPTCHA challenge may be sent to a plurality of user devices seeking to gain access to one or more online services or resources associated with one or more resource providers.

[0049] At (212), the method (200) may include receiving one or more responses to the CAPTCHA challenge from at least a subset of the plurality of user devices. The responses may include data indicating that the user interacted with the CAPTCHA challenge. In some embodiments, the responses may be utilized to determine whether to grant the user access to the online service or resource.

[0050] At (214), the method (200) may include determining an unknown characteristic of the first image based at least in part on the user responses. For example, the user's selection and / or interaction with the image provided in the CAPTCHA challenge may be used as evidence of one or more characteristics associated with the first image. For example, the user responses may be analyzed according to the instructions provided in the CAPTCHA challenge to determine one or more characteristics of the first image. In this manner, if sufficient user responses indicate a characteristic of the first image, then the characteristic may be inferred. For example, if sufficient user responses instruct the user to select the first image in a CAPTCHA challenge that depicts every image of a tiger, then it may be inferred that the first image depicts a tiger.

[0051] At (216), the method (200) may include training one or more machine learning models based at least in part on the determined features of the first image. For example, training the one or more machine learning models may include providing data indicative of the image and the determined features to a set of training data used to train the model, thereby adding the data indicative of the image and the determined features to the training data set. In this manner, the data indicative of the first image may be input into the model(s), and the output of the model(s) may be compared to the determined features. The model(s) may then be adjusted or improved based at least in part on the comparison results.

[0052] In some embodiments, the first image can then be used in subsequent verification challenges. For example, the first image can be used as a known image in a subsequent CAPTCHA challenge. In this way, the first image can be added to a set of known images that can be used in CAPTCHA challenges and removed from a set of unknown images.

[0053] Figure 6 A flowchart of an example method (300) for authenticating a user device according to an example embodiment of the present disclosure is shown. The method (300) may be performed by one or more computing devices (e.g., Figure 7 In addition, Figure 6 The steps are shown to be performed in a specific order for the purpose of illustration and discussion. Using the disclosure provided herein, one of ordinary skill in the art will appreciate that the steps of any method discussed herein may be adjusted, reconfigured, expanded, omitted, or modified in various ways without departing from the scope of the present disclosure.

[0054] At (302), a user device may request a resource or access to an online service from a resource provider. At (304), the resource provider may receive a request from the user device.

[0055] At (306), the resource provider may instruct the user device to directly engage in the verification process through the computerized CAPTCHA system. At (308), the user device may receive instructions from the resource provider to directly engage with the computerized CAPTCHA system. As an example, the resource provider may instruct the user device at (306) to directly engage with the computerized CAPTCHA system and begin the verification process by redirecting a browser of the user device.

[0056] As another example, at (306), the resource provider may provide a client-side script to the user device that, when executed by the user device, enables the user device to interface directly with the computerized CAPTCHA system. For example, the client-side script may be included in an iframe, embedded object, portlet, or other embedded application or widget. In some embodiments, the client-side script may be included in a plug-in provided to the resource provider from the computerized CAPTCHA system. Furthermore, in some embodiments, at (306), the resource provider may provide a public key to the user device that identifies the resource provider to the computerized CAPTCHA system.

[0057] At (310), the user device may send a request to participate in the verification process directly to the computerized CAPTCHA system.At (312), the computerized CAPTCHA system may receive the request from the user device.

[0058] As an example, the request sent at 310 can be formatted according to an application programming interface associated with a computerized CAPTCHA system. For example, the request can be sent as a result of executing a client-side script provided to the user device at 306. Furthermore, in some embodiments, the request sent at 310 can include a public key associated with the resource provider.

[0059] At (314), the computerized CAPTCHA system may obtain one or more first images having one or more unknown image features. For example, the first image may be an image that cannot or has not been classified or categorized within a reasonable degree of certainty. As another example, the first image may be an image in which the location of an item, object, plant, animal, human, etc. cannot be determined within a reasonable degree of certainty. As described above, the unknown image may be determined to be unknown based at least in part on metadata or other descriptive information associated with the image and / or one or more machine learning models.

[0060] At (316), based at least in part on the first image and / or unknown features of the first image(s), the computerized CAPTCHA system may select a CAPTCHA challenge and provide the selected challenge to the user device. For example, the computerized CAPTCHA system may include a database of CAPTCHA challenges having different formats, difficulties, shapes, sizes, file formats, programming languages, or other variable parameters. For example, in some embodiments, the CAPTCHA challenge may be image-based. In some embodiments, selecting the CAPTCHA challenge may include obtaining one or more second images having known image features. As described above, the second image may be selected based at least in part on instructions associated with the CAPTCHA challenge and / or the first image(s).

[0061] As an example, for a situation where the content depicted in the first image obtained at (314) is unknown, a CAPTCHA challenge may be selected to determine the classification of the image. For example, the CAPTCHA challenge may include multiple known images, and the instructions may prompt the user to select each image that shares a particular common feature. As described above, the CAPTCHA challenge may be adjusted so that the user's response to the CAPTCHA challenge can be used as evidence to determine and / or narrow down the features of the first image(s). As another example, if the unknown features of the first image are associated with an unknown location of an object in the image, the CAPTCHA challenge may prompt the user to interact (e.g., select, highlight, annotate, or otherwise interact) with the portion(s) of the image depicting the object.

[0062] In some embodiments, at (316), the challenge format, challenge type, or other attributes of the challenge may be selected based on various other suitable factors. For example, where the user device is a mobile device (e.g., a smartphone or tablet), at (316), the computerized CAPTCHA system may select a CAPTCHA challenge designed for mobile devices.

[0063] At (318), the user device may receive a CAPTCHA challenge from the computerized CAPTCHA system and present it to the user (e.g., on a display of the user device). At (320), the user device may receive one or more responses from the user and send the response(s) to the computerized CAPTCHA system. At (322), the computerized CAPTCHA system may receive the responses from the user device.

[0064] At (324), if one or more verification conditions are met, the computerized CAPTCHA system may generate a verification token and provide it to the user device. At (326), the user device may receive the verification token from the computerized CAPTCHA system.

[0065] As an example, if the response received at (322) correctly solves or otherwise complies with the challenge provided at (316), the computerized CAPTCHA system may generate a verification token at (324) and provide it to the user device. The verification token may be an authentication certificate or other security or authentication device or mechanism. For example, in some embodiments, the verification token may include a hash of a user device identifier or other information, or may contain a public key of a resource provider.

[0066] It should be understood that in some embodiments, steps (310)-(326) may be considered a verification process. Furthermore, in some embodiments, steps (310)-(326) may occur via an iframe, embedded object, portlet, or other embedded widget or application included in the resource provider's website.

[0067] At (328), the user device may provide the verification token to the resource provider. At (330), the resource provider may receive the verification token from the user device.

[0068] At (332), the resource provider may send the verification token to the computerized CAPTCHA system. In some embodiments, at (332), the resource provider may also send its private key along with the verification token to the computerized CAPTCHA system.

[0069] If the verification token is valid, then the computerized CAPTCHA system may provide confirmation of the verification token to the resource provider at (334).If the verification token is invalid or has been tampered with, then the computerized CAPTCHA system may notify the resource provider that the verification token is invalid.

[0070] At (336), the resource provider may receive confirmation of the verification token from the computerized CAPTCHA system. In response to receiving the confirmation at (336), at (338), the resource provider may provide access to the resource to the user device. At (340), the user device may access the resource.

[0071] Figure 7 An example computing system 400 is shown that can be used to implement methods and systems according to example aspects of the present disclosure. System 400 can be implemented using a client-server architecture that includes a server 410 communicating with one or more client devices 430 via a network 440. System 400 can be implemented using other suitable architectures, such as a single computing device.

[0072] System 400 includes a server 410, such as a web server. Server 410 can host a computerized CAPTCHA system configured to grant or deny access to online services or resources. Server 410 can be implemented using any suitable computing device(s). Server 410 can have one or more processors 412 and one or more memory devices 414. Server 410 can also include a network interface for communicating with one or more client devices 430 via a network 440. The network interface can include any suitable components for interfacing with one or more networks, such as transmitters, receivers, ports, controllers, antennas, or other suitable components.

[0073] The one or more processors 412 may include any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, logic device, or other suitable processing device. The one or more memory devices 414 may include one or more computer-readable media, including, but not limited to, non-transitory computer-readable media, RAM, ROM, hard drive, flash drive, or other memory devices. The one or more memory devices 414 may store information accessible by the one or more processors 412, including computer-readable instructions 416 executable by the one or more processors 412. The instructions 416 may be any set of instructions that, when executed by the one or more processors 412, cause the one or more processors 412 to perform operations. For example, the instructions 416 may be executed by the one or more processors 412 to implement, for example, the challenge generator 420, the user verifier 422, and the image classifier 424. For example, the challenge generator 420 may be configured to obtain one or more unknown images and / or one or more known images and generate a CAPTCHA challenge based at least in part on the images. The user verifier 422 may be configured to receive a user response to the CAPTCHA challenge and determine whether to grant access to the user based at least in part on the response. The image classifier 424 may be configured to determine one or more features of the unknown image based at least in part on the user response, and to provide the image and the determined features to the machine learning network 450 .

[0074] like Figure 7 As shown, one or more memory devices 414 may also store data 418, which may be retrieved, manipulated, created, or stored by one or more processors 412. Data 418 may include, for example, image data, challenge data, and other data. Data 418 may be stored in one or more databases. One or more databases may be connected to server 410 via a high-bandwidth LAN or WAN, or may be connected to server 410 via network 440. One or more databases may be split up and located at multiple locations.

[0075] The server 410 may exchange data with one or more client devices 430 via a network 440. Figure 7 Only one client device 430 is shown, but any number of client devices 430 may be connected to server 410 via network 440. Each client device 430 may be any suitable type of computing device, such as a general purpose computer, a special purpose computer, a laptop computer, a desktop computer, a mobile device, a navigation system, a smartphone, a tablet computer, a wearable computing device, a display with one or more processors, or other suitable computing device.

[0076] Similar to the server 410, the client device 430 may include one or more processors 432 and memory 434. The one or more processors 432 may include one or more central processing units (CPUs), graphics processing units (GPUs) dedicated to efficiently rendering images or performing other specialized calculations, and / or other processing devices. The memory 434 may include one or more computer-readable media and may store information accessible by the one or more processors 432, including instructions 436 and data 438 that can be executed by the one or more processors 432. For example, the memory 434 may store instructions 436 for implementing a user interface module for displaying a CAPTCHA challenge according to example aspects of the present disclosure.

[0077] Figure 7 The client device 430 may include various input / output devices for providing and receiving information to and from the user, such as a touch screen, a touchpad, data input keys, a speaker, and / or a microphone suitable for speech recognition. For example, according to an exemplary aspect of the present disclosure, the client device 430 may have a display device 435 for presenting a user interface displaying a semantic place name.

[0078] Client device 430 may also include a network interface for communicating with one or more remote computing devices (e.g., server 410) over network 440. The network interface may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, receiver, port, controller, antenna, or other suitable components.

[0079] The network 440 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), a cellular network, or some combination thereof. The network 440 can also include a direct connection between the client device 430 and the server 410. Generally, the communication between the server 410 and the client device 430 can be carried out via a network interface using any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0080] Server 410 may also exchange data with one or more machine learning networks (e.g., machine learning network 450). Similar to server 410 and client device 430, the machine learning network may include one or more processors (multiple) 452 and memory 454. The one or more processors (multiple) 452 may include one or more central processing units (CPUs) and / or other processing devices. Memory 454 may include one or more computer-readable media and may store information accessible by the one or more processors 452, including instructions 456 and data 458 that can be executed by the one or more processors 452. For example, memory 454 may store instructions 456 for implementing a model trainer configured to train one or more machine learning models. Data 458 may include one or more models and training data. Machine learning network 450 may also include a network interface for communicating with one or more remote computing devices (e.g., server 410) via network 440.

[0081] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information transferred to and from these systems. Those skilled in the art will appreciate that the inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the server processes discussed herein can be implemented using a single server or multiple servers working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0082] Although the subject matter has been described in detail with respect to specific example embodiments of the invention, it will be appreciated that modifications, variations, and equivalents of these embodiments may readily occur to those skilled in the art upon gaining an understanding of the foregoing. Accordingly, the scope of the present disclosure is given by way of example and not limitation, and the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter that would be apparent to one of ordinary skill in the art.

Claims

1. A computer-implemented method of providing access to a resource, the method comprising: Obtaining, by one or more computing devices, one or more first images having unknown characteristics and one or more second images having known characteristics; providing, by the one or more computing devices, the one or more first images and the one or more second images to a user device in a verification challenge, the verification challenge comprising one or more instructions to be presented to a user of the user device, the instructions determined at least in part based on the unknown features of the one or more first images and the known features of the one or more second images; receiving, by the one or more computing devices, a response to the authentication challenge from the user device; determining, by the one or more computing devices, the unknown characteristic of the one or more first images based at least in part on the received responses; providing, by the one or more computing devices, a verification token to the user device in response to determining the unknown characteristic of the one or more first images; receiving, by one or more computing devices, the verification token from a resource provider in response to providing the verification token to a user device; validating, by the one or more computing devices, the verification token from the resource provider; as well as Data indicative of the confirmation is provided to the resource provider by the one or more computing devices to provide the user device with access to resources controlled by the resource provider.

2. The computer-implemented method of claim 1 , further comprising receiving, by one or more computing devices, a request from a user computing device to participate in a verification process, and wherein the one or more first images are obtained in response to receiving the request.

3. The computer-implemented method of claim 1 , further comprising: obtaining, by the one or more computing devices, a plurality of second images having known characteristics; as well as At least a subset of the plurality of second images is provided along with the one or more first images in the verification challenge by the one or more computing devices.

4. The computer-implemented method of claim 1 , wherein: The unknown features of the one or more first images are determined to be unknown based at least in part on one or more machine learning models configured to receive one or more images as input and determine one or more features of the one or more images as output.

5. The computer-implemented method of claim 4, wherein: The one or more features of the one or more images relate to one or more items or objects depicted in the one or more images.

6. The computer-implemented method of claim 4, wherein: The one or more machine learning models are further configured to provide a certainty probability indicating a degree of certainty that an output of the model is correct.

7. The computer-implemented method of claim 6, wherein: The unknown feature of the one or more first images is determined to be unknown when the certainty probability associated with the one or more first images is below a certainty threshold.

8. The computer-implemented method of claim 4, wherein the known features of the plurality of second images are determined at least in part based on the one or more machine learning models.

9. The computer-implemented method of claim 3, wherein: The one or more instructions associated with the verification challenge prompt the user to interact with one or more of the first image or the second image that meets criteria.

10. The computer-implemented method of claim 1 , wherein: The one or more instructions associated with the verification challenge prompt the user to interact with at least a portion of the one or more first images that meets criteria.

11. The computer-implemented method of claim 10, wherein: The one or more instructions associated with the verification challenge prompt the user to select a portion of at least one of the one or more first images.

12. The computer-implemented method of claim 11, wherein: The authentication challenge is determined based at least in part on the unknown feature of the one or more first images.

13. The computer-implemented method according to claim 3 further includes providing a third image in the verification challenge via the one or more computing devices, the third image depicting an item or object, and wherein one or more instructions associated with the verification challenge prompt the user to select one or more images from the at least one subset of the one or more first images or second images that depict items or objects of the same category as the item or object depicted in the third image.

14. The computer-implemented method of claim 1, further comprising determining, by the one or more computing devices, whether to authenticate the user device based at least in part on a response to the authentication challenge.

15. The computer-implemented method of claim 1 , wherein determining, by the one or more computing devices, the unknown feature of the one or more first images comprises: The unknown feature of the one or more first images is inferred by the one or more computing devices when a threshold number of user responses are indicative of a feature of the one or more first images.

16. A computing system comprising: one or more processors; as well as one or more memory devices storing computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: obtaining a first image having unknown features and a second image having known features; providing the first image and the second image to a user device in a verification challenge, the verification challenge providing one or more instructions to a user of the user device based at least in part on the unknown feature of the first image and the known feature of the second image; receiving a response to the authentication challenge from the user device; determining the unknown characteristic of the first image based at least in part on the received response; providing a verification token to the user device in response to determining the unknown feature of the first image; receiving the verification token from a resource provider in response to providing the verification token to a user device; validating the verification token received from the resource provider; and Data indicative of the confirmation is provided to the resource provider to provide the user device with access to resources controlled by the resource provider.

17. The computing system of claim 16, further comprising: obtaining a plurality of second images having known characteristics; as well as At least a subset of the plurality of second images is provided along with the first image in the verification challenge.

18. The computing system of claim 17, wherein determining the unknown feature of the first image comprises: The unknown feature of the first image is inferred when a threshold number of user responses are indicative of a feature of the first image.

19. One or more tangible, non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: obtaining a first image having unknown features and a second image having known features; providing the first image and the second image to a user device in a verification challenge, the verification challenge providing one or more instructions to a user of the user device based at least in part on the first image; receiving a response to the authentication challenge from the user device; determining the unknown characteristic of the first image based at least in part on the received response; providing a verification token to the user device in response to determining the unknown feature of the first image; receiving the verification token from a resource provider in response to providing the verification token to a user device; validating the verification token received from the resource provider; as well as Data indicative of the confirmation is provided to the resource provider to provide the user device with access to resources controlled by the resource provider.

20. The one or more tangible, non-transitory computer-readable media of claim 19, wherein determining the unknown feature of the first image comprises: The unknown feature of the first image is inferred when a threshold number of user responses are indicative of a feature of the first image.

Images