Secure management of consumption of computing resources

Abstract

In some embodiments, a method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device. An access management computing system receives an incoming request from the application for a resource-managed application programming interface computing system (RM API computing system). The incoming request includes credentials associated with the developer and the application for access to the access management computing system. The access management computing system determines a level of throttling to apply to an outgoing request based on a remaining resource allocation in the record, wherein the outgoing request is based on the incoming request. Based on the level of throttling, the access management computing system selectively transmits the outgoing request to the RM API computing system identified in the incoming request; or blocks the outgoing request.

Description

SECURE MANAGEMENT OF CONSUMPTION OF COMPUTING RESOURCESCROSS-REFERENCE(S) TO RELATED APPLICATION(S)

[0001] This application claims the benefit of Provisional Application No. 63 / 582742, filed September 14, 2023, the entire disclosure of which is hereby incorporated by reference herein for all purposes.BACKGROUND

[0002] Many advanced resources, including but not limited to artificial intelligence (Al) services, are made available to applications via application programming interfaces (APIs). However, the development of advanced applications leveraging innovative technology7accessible through APIs presents significant challenges, at least in part due to the difficultyin scaling usage. Typically, access to Al APIs is restricted on a user-by-user basis, with individual users each being provided a limited amount of access. This makes it difficult to scale shared services, such as applications that call Al APIs provided by third parties, since the individual access afforded to the app developer is quickly exhausted.

[0003] Previous attempts to redistribute Al API access have experienced problems, including but not limited to a lack of security. In these attempts, end users shared access keys that control their limited amount of access to each of the Al APIs with the application developer. This is very insecure for the end user since they must trust that the application developer will not misuse the access keys. Further, even if the application developer uses the Al API access keys as authorized, this technique does not allow the end user to control how much of their own resource allocation the application developer will consume on their behalf.SUMMARY

[0004] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary7is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In some embodiments, a method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device. An access management computing system stores a record of a resource allocation for the end user. The access management computing system receives an incoming request from the application for aresource-managed application programming interface computing system (RM API computing system). The incoming request includes credentials associated with the developer and the application for access to the access management computing system. The access management computing system determines a level of throttling to apply to an outgoing request based on a remaining resource allocation in the record, wherein the outgoing request is based on the incoming request. Based on the level of throttling, the access management computing system selectively transmits the outgoing request to the RM API computing system identified in the incoming request; or blocks the outgoing request.

[0006] In some embodiments, a method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device is provided. An access management computing system stores a record of a resource allocation for the end user. The access management computing system receives an incoming request from the application for a resource-managed application programming interface computing system (RM API computing system). The incoming request includes credentials associated with the developer and the application for access to the access management computing system. The access management computing system transmits an outgoing request to the RM API computing system identified in the incoming request, wherein the outgoing request is based on the incoming request. The access management computing system receives an incoming response to the outgoing request from the RM API computing system, and transmits an outgoing response to the application based on the incoming response. The access management computing system determines an amount of computing resources consumed by the outgoing request, and adjusts the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request.

[0007] In some embodiments, a non-transitory computer-readable medium having computer-executable instructions stored thereon is provided. The instructions, in response to execution by one or more processors of an access management computing system, cause the access management computing system to perform a method as described above.

[0008] In some embodiments, a system for secure management of consumption of computing resources used by an application is provided. The system comprises an access management computing system configured to perform a method as described above.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to thefollowing detailed description, when taken in conjunction with the accompanying drawings, wherein:

[0010] FIG. 1 is a schematic illustration of previous efforts to incorporate functionality exposed through application programming interfaces into an application.

[0011] FIG. 2 is a schematic illustration of an improved system for securely managing consumption of computing resources, according to various aspects of the present disclosure.

[0012] FIG. 3 is a block diagram that illustrates aspects of a non-limiting example embodiment of an access management computing system, according to various aspects of the present disclosure.

[0013] FIG. 4A - FIG. 4C are a flowchart that illustrates a non-limiting example embodiment of a method of secure management of computing resources, according to various aspects of the present disclosure.

[0014] FIG. 5 is a sequence diagram that illustrates a non-limiting example of a subroutine for obtaining authorization for an application to use a resource allocation of an end user, according to various aspects of the present disclosure.

[0015] FIG. 6A -FIG. 6C and FIG. 7A - FIG. 7E are schematic illustrations of non-limiting example embodiments of resource allocation adjustments based on a resource cost, according to various aspects of the present disclosure.DETAILED DESCRIPTION

[0016] FIG. 1 is a schematic illustration of previous efforts to incorporate functionality exposed through application programming interfaces into an application. A developer, using a developer computing system 106, creates an application 104, which is deployed to one or more end user computing devices 102. In some embodiments, the application 104 may be downloadable, executable software that is installed on the end user computing device 102. For ease of description, the majority of the present disclosure describes this embodiment wherein the application 104 is installed on the end user computing device 102. However, in some embodiments, the application 104 may be fully or partially implemented as web-based software that is accessed via the end user computing device 102 but that is executed at least partially on the developer computing system 106 on behalf of the end user.

[0017] In order to provide complex functionality (including but not limited to Al functionality), the application 104 may transmit requests to one or more resource-managed application programming interface computing systems (RM API computing systems 108). RM API computing systems 108 may include systems that provide artificial intelligencefunctionality, which may include, but is not limited to. text, audio, code, and / or image generation; content summary; speech-to-text conversion; image description; and / or content search. Some non-limiting examples of such RM API computing systems 108 are gpt3.5- turbo (offering text generation), ElevenLabs (offering text-to-speech and voice generation), Replicate (offering cloud-based model execution), Bard (offering text generation and chatbot functionality), and PaLM (offering responses to reasoning tasks including code, math, classification, question answering, translation, and natural language generation). While the discussion herein is mainly of such artificial intelligence API (Al API) computing systems, one will recognize that RM API computing systems 108 may provide other types of functionality instead of or in addition to artificial intelligence functionality.

[0018] The APIs accessed in the system 100 are referred to as “resource managed” because the computing resources consumed are tracked and limited. Typically, in order to access an RM API computing system 108, an account is established with the RM API computing system 108, and credentials for the account are associated with each submitted request. An allocation of computing resources is assigned to the account, and when computing resources are consumed by requests submitted for the account, the consumed computing resources are removed from the allocation.

[0019] In simple cases, the developer may create an account with each RM API computing system 108 used to support the application 104. and the credentials and resource allocation of the developer may be used by the application 104. Unfortunately, this strategy is only- practical at very small scales - once many end users and / or end user computing devices 102 are using the application 104, and once the application 104 incorporates functionality from multiple RM API computing systems 108. the resource allocation of the developer will quickly be exhausted. Thus, development of a successful application 104 is a double-edged sword, as it may be impractical for the developer to obtain a resource allocation that is adequate to support execution of the application 104 at scale.

[0020] One technique used in the past by developers to overcome this problem is to have end users establish accounts with the RM API computing systems 108, and to use the credentials of the end user (and thereby the resource allocation of the end user) within the application 104 when submitting requests to RM API computing systems 108. While this does pass the responsibility for managing the resource allocation from the developer to the end user, it has significant technical drawbacks. For example, the complexity of requiring the end user to establish accounts with each of the RM API computing systems 108 is undesirable, particularly when a large number of RM API computing systems 108 are being used. More significantly, the end user is required to share their otherwise confidentialcredentials with the developer, thereby undermining the security and privacy of the confidential information of the end user. In addition, by providing the confidential credentials to the developer, the end user loses control over the amount of the resource allocation consumed by the developer and the application 104. This puts the end user at risk of overconsumption of the resource allocation, either by an unscrupulous developer who uses the confidential credentials and resource allocation of the end user for purposes other than providing the application 104 to the end user, or by an otherwise trustworthy developer whose application 104 includes behavior that causes unexpected amounts of the resource allocation to be consumed. What is desired are techniques that allow a developer to use computing resources allocated to an end user for requests to RM API computing systems 108 instead of solely using computing resources allocated to the developer, while also providing the end user with adequate security7for their confidential information and control over the use of their resource allocation.

[0021] FIG. 2 is a schematic illustration of an improved system for securely managing consumption of computing resources, according to various aspects of the present disclosure. In the system 200. a developer (via developer computing system 212) again creates an application 208 and causes it to be deployed to one or more end user computing devices 210. The end user computing devices 210 may be any type of computing devices, including but not limited to desktop computing devices, server computing devices, laptop computing devices, mobile computing devices (e.g., smartphone computing devices, tablet computing devices, etc.), or any other types of computing device capable of executing one or more applications. As in the system 100 illustrated in FIG. 1, the application 208 may be implemented fully or partially as a downloadable executable application installed on the end user computing device 210, fully or partially implemented as web-based software that is accessed via the end user computing device 210 but that is executed at least partially on the developer computing system 212 on behalf of the end user, or in any other suitable architecture. Though a single developer computing system 212 and application 208 are illustrated for the sake of clarity, in some embodiments the access management computing system 202 may concurrently support multiple different developers / developer computing systems 212, as well as multiple applications 208.

[0022] Unlike the system 100 of FIG. 1 in which the application 104 communicated directly with the RM API computing systems 108, the application 208 of the system 200 instead communicates with an access management computing system 202. Each end user may establish an account with the access management computing system 202 (e g., using browser 206 to connect to an interface provided by the access management computingsystem 202). and the access management computing system 202 may establish a resource allocation for each end user. Once an account is established by the end user at the access management computing system 202, the end user may authorize use of their resource allocation by the application 208. In some embodiments, the access management computing system 202 may use an identity provider computing system 204 to store account information, credentials, authorization information, and other relevant information.

[0023] To access an RM API computing system 108, the application 208 submits a request to the access management computing system 202 instead of directly to the RM API computing system 108. The access management computing system 202 determines whether the application 208 has been authorized to use the computing resources of the end user and if so, provides the request to the appropriate RM API computing system 108 using credentials of the developer for connecting to the RM API computing system 108. The access management computing system 202 determines an amount of computing resources consumed by the request, deducts the used resources from the resource allocation of the end user, and provides the response to the request to the application 208. In some embodiments, the access management computing system 202 may determine whether there are adequate computing resources remaining in the resource allocation of the end user to cover the computing resources likely to be consumed by the request, and may take steps to throttle or block the request if the resource allocation remaining to the end user is too small.

[0024] The use of an access management computing system 202 as disclosed herein can provide many various improvements. For example, some embodiments of the access management computing system 202 provided for simplified integration between an application 208 and various RM API computing systems 108. By providing a common interface to multiple different RM API computing systems 108 and a shared resource allocation for an end user for multiple different RM API computing systems 108, the developer is relieved from the burden of creating separate interfaces and managing separate resource allocations for each of the RM API computing systems 108. As another example, by using the resource allocation of each end user instead of a resource allocation of the developer, the computing resource burden is spread amongst the end users, thus allowing support for greater numbers of installed applications 208 and end users without overwhelming the computing resources allocated to the developer. As still another example, the access management computing system 202 provides security, transparency, and control for end users that was not possible when the developer managed communication between the application 104 and the RM API computing systems 108. The access management computing system 202 securely controls credentials and authorization for access to theresource allocations of the end users that would be unavailable if end users shared credentials or other access information with the developer. Further details of how these (and other) technical improvements are provided by the access management computing system 202 are described in the remainder of the present disclosure.

[0025] FIG. 3 is a block diagram that illustrates aspects of a non-limiting example embodiment of an access management computing system according to various aspects of the present disclosure. The illustrated access management computing system 202 may be implemented by any computing device or collection of computing devices, including but not limited to a desktop computing device, a laptop computing device, a mobile computing device, a server computing device, a computing device of a cloud computing system, and / or combinations of one or more of each.

[0026] As shown, the access management computing system 202 includes one or more processors 302, one or more communication interfaces 304, an account data store 308, a resource allocation data store 310, and a computer-readable medium 306.

[0027] In some embodiments, the processors 302 may include any suitable type of general- purpose computer processor. In some embodiments, the processors 302 may include one or more special-purpose computer processors or Al accelerators optimized for specific computing tasks, including but not limited to graphical processing units (GPUs), vision processing units (VPUs), and tensor processing units (TPUs).

[0028] In some embodiments, the communication interfaces 304 include one or more hardware and or software interfaces suitable for providing communication links between components. The communication interfaces 304 may support one or more wired communication technologies (including but not limited to Ethernet, FireWire, and USB), one or more wireless communication technologies (including but not limited to Wi-Fi, WiMAX, Bluetooth, 2G, 3G, 4G, 5G, and LTE), and / or combinations thereof.

[0029] As shown, the computer-readable medium 306 has stored thereon logic that, in response to execution by the one or more processors 302, cause the access management computing system 202 to provide a user identification engine 312. a request handling engine 314, and an allocation engine 316.

[0030] As used herein, "computer-readable medium" refers to a removable or nonremovable device that implements any technology capable of storing information in a volatile or non-volatile manner to be read by a processor of a computing device, including but not limited to: a hard drive; a flash memory; a solid state drive; random-access memory(RAM); read-only memory (ROM); a CD-ROM, a DVD. or other disk storage; a magnetic cassette; a magnetic tape; and a magnetic disk storage.

[0031] In some embodiments, the user identification engine 312 is configured to manage accounts and credentials for end users and developers, may manage account records in the account data store 308, and may communicate with an identity provider computing system 204 for improved security of confidential information. In some embodiments, the request handling engine 314 is configured to receive requests from applications 208, ensure that the requests are authorized and are likely to be covered by the resource allocation available to the associated end user, transmit the requests to the appropriate RM API computing systems 108, and return responses back to the applications 208. In some embodiments, the allocation engine 316 is configured to determine the amount of computing resources consumed by requests and to make adjustments to resource allocations stored within the resource allocation data store 310 accordingly.

[0032] Further description of the configuration of each of these components is provided below.

[0033] As used herein, "engine" refers to logic embodied in hardware or software instructions, which can be written in one or more programming languages, including but not limited to C. C++. C#, COBOL, JAVA™. PHP. Perl. HTML, CSS, JavaScript, VBScript, ASPX, Go, and Python. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be callable from other engines or from themselves. Generally, the engines described herein refer to logical modules that can be merged with other engines, or can be divided into sub-engines. The engines can be implemented by logic stored in any type of computer-readable medium or computer storage device and be stored on and executed by one or more general purpose computers, thus creating a special purpose computer configured to provide the engine or the functionality thereof. The engines can be implemented by logic programmed into an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA). or another hardware device.

[0034] As used herein, "data store" refers to any suitable device configured to store data for access by a computing device. One example of a data store is a highly reliable, highspeed relational database management system (DBMS) executing on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key -value store. However, any other suitable storage technique and / or device capable of quickly and reliably providing the stored data in response to queries may beused, and the computing device may be accessible locally instead of over a network, or may be provided as a cloud-based service. A data store may also include data stored in an organized manner on a computer-readable storage medium, such as a hard disk drive, a flash memory, RAM, ROM, or any other type of computer-readable storage medium. One of ordinary skill in the art will recognize that separate data stores described herein may be combined into a single data store, and / or a single data store described herein may be separated into multiple data stores, without departing from the scope of the present disclosure.

[0035] FIG. 4A - FIG. 4C are a flowchart that illustrates a non-limiting example embodiment of a method of secure management of computing resources, according to various aspects of the present disclosure. One will note that, while the method 400 is illustrated as a sequence of blocks, in some embodiments, the actions described in the method 400 may be performed at least partially concurrently for multiple end users, applications 208, and / or requests, except where explicitly described as occurring serially.

[0036] From a start block, the method 400 proceeds to block 402, where a user identification engine 312 of an access management computing system 202 creates an account for a developer. In some embodiments, the account for the developer is created as a record in the account data store 308, and may include information such as an identity of the developer (e.g., one or more of a name, physical address, e-mail address, organization, etc.), one or more application identifiers associated with one or more applications 208, and / or an API key associated with each of the application identifiers for each of the applications 208 to use to access the access management computing system 202. The combination of an application identifier and an API key, potentially along with other information, may be used as the developer credentials as described below. Creating the account for the developer may also include the allocation engine 316 establishing a resource allocation usable by the developer, and storing a record of the resource allocation in the resource allocation data store 310. In some embodiments, a service such as Azure API Management, provided by Microsoft Corporation, may be used by the user identification engine 312 for registration of the developer, as such services provide useful functionality such as API key generation, registration, rate limit-based throttling, and other reusable functionality.

[0037] At block 404, the user identification engine 312 creates an account for an end user. In some embodiments, the user identification engine 312 may generate a web-based interface, and the end user may access the web-based interface using the browser 206 of the end user computing device 210. In some embodiments, the end user may be prompted or directed to create an account at the access management computing system 202 by theapplication 208 itself, while in other embodiments, the end user may initiate the creation of the account with the access management computing system 202 prior to interacting with the application 208. To create the account, the user identification engine 312 may create a record for the account in the account data store 308 of the access management computing system 202, and / or may communicate with an identity provider computing system 204 to generate and / or store credentials associated with the account.

[0038] At block 406, an allocation engine 316 of the access management computing system 202 stores a resource allocation for the end user in a resource allocation data store 310 of the access management computing system 202. In some embodiments, the resource allocation is a record that indicates an amount of computing resources allocated to the end user, and the amount of computing resources provided by the resource allocation may be determined in any suitable way. In some embodiments, the resource allocation may be a preconfigured amount of computing resources. In some embodiments, the resource allocation may automatically generate periodically, such that the resource allocation serves to prevent any particular user from consuming an unfair amount of computing resources. In some embodiments, the amount of the resource allocation may be based on a subscription or other model. In some embodiments, the amount of the resource allocation may be determined by the developer.

[0039] At block 408, the end user installs an application 208 from the developer on an end user computing device 210. Any suitable technique for installing the application 208 on the end user computing device 21 may be used, including but not limited to downloading and executing an installer, obtaining the application 208 from an app marketplace (e.g., the App Store from Apple Corporation; the Google Play store from Google LLC, etc.), navigating to a web site that provides at least a portion of the functionality of the application 208, or any other suitable technique.

[0040] The method 400 then proceeds to subroutine block 410, where a subroutine is performed wherein the end user computing device 210 communicates with the access management computing system 202 to obtain authorization for the application 208 to use the resource allocation of the end user. Any suitable subroutine that maintains the security of the end user credentials and ensures that only authorized applications 208 are allowed to use the resource allocation of the end user may be used, including but not limited to the subroutine illustrated in the sequence diagram of FIG. 5 and described in further detail below.

[0041] The method 400 then proceeds through a continuation terminal ("terminal A") to block 412, where the application 208 transmits a request to the access management computing system 202 to access an RM API computing system 108. In some embodiments, the request includes credentials associated with the developer. In some embodiments, the developer credentials included in the request may include one or more of credentials associated with the developer for accessing the access management computing system 202 (e.g., an API key for the access management computing system 202, etc.), credentials that authorize the developer to use the resources of the end user (e.g., the access token, a sub value, etc.), or credentials associated with the developer for accessing the RM API computing system 108. In some embodiments, instead of including an API key for the access management computing system 202 in the developer credentials, the access management computing system 202 may authenticate and uniquely identify the developer and / or the application 208 using a different technique (e.g., a passkey, etc.).

[0042] The request to the access management computing system 202 replaces the direct request from the application 208 to the RM API computing system 108, and in some embodiments, may be highly similar to a direct request from the application 208 to the RM API computing system 108, but for the addition of developer credentials associated with the access management computing system 202 and routing information indicating the RM API computing system 108 being accessed. Further, in some embodiments, the endpoint may be provided as a REST API if the RM API is provided by the RM API computing system 108 as a REST API. In this way, minimal changes may be applied to switch between a direct request to the RM API computing system 108 and a request through the access management computing system 202, easing testing of and integration with the access management computing system 202.

[0043] As a non-limiting example, a direct request to a DALLE-2 RM API computing system 108 may be as follows:{"prompt" : "A cute baby sea otter" ,"n" : 2 ,"size" : " 1024x1024"}11SUBSTITUTE SHEET (RULE 26)

[0044] This direct request can then be altered by adding the information relevant to the access management computing system 202 (ACMP) as follows:{"prompt" : "A cute baby sea otter" ,"n" : 2 ,"size" : "1024x1024" ,"ACMP" : {"provider" : "openai" ,"provider-api " : "dalle-2" ,"provider_keY" • "<developers_openai_key_value>" ,"app_id" : "<developers_platform_app_id_value>" , "access_token" :"<users_access_token_value_returned_by_authO>" ,"sub" : "<users_sub_value_returned_by_authO>"}}

[0045] As shown, the DALLE-2 request has been updated with the developer credentials (e.g., the provider key value, the app id value, the access token value, and the sub value) and routing information (e.g., the provider value and the provider-api value).

[0046] Since the endpoint provided by the access management computing system 202 for receiving the request is likely to receive significant traffic, in some embodiments, the access management computing system 202 may use rate limiting techniques to avoid overloading of the access management computing system 202. For example, a system such as Azure API Management may be used for providing the endpoint, and functionality of Azure API Management such as rate limiting, API key management, and logging may be leveraged to improve reliability and functionality of the endpoint.

[0047] It should be noted that systems such as Azure API Management do not provide a way to determine the identity of the application 208 based on the API key, and instead will allow access to the endpoint if any valid API key is used. Though systems such as Azure API Management can provide significant benefits related to rate limiting and other scaling issues, allowing any valid API key to be used can lead to insecure and inefficient behavior within the access management computing system 202. For example, if a developer had multiple applications 208 (or otherwise had multiple app ids), a first application 208 could12SUBSTITUTE SHEET (RULE 26)use an API key associated with a second application 208, and the request handling engine 314 would have to look up the app id associated with the API key being used for every request. This would be computationally expensive for the access management computing system 202, and would limit the ability of the access management computing system 202 to scale.

[0048] To address these issues, some embodiments of the present disclosure (such as in the example request provided in the text above) have the application 208 include the app id as well as the API key. The service such as Azure API Management can be configured with a policy that will append the app id associated with the API key being used to the headers of the request automatically. Accordingly, the request handling engine 314 may verify that the correct API key is being used for the correct application 208 by comparing the app id sent in the request with the app id added to the header of the request to ensure that they match. By offloading this lookup to Azure API Management, computing resources are conserved at the access management computing system 202, and the access management computing system 202 may support a larger volume of requests.

[0049] A non-limiting example of such a policy for Azure API Management follows:<policies><inbound><base / ><set-header name="X-Subscription-Id" exists- ac ti on= "over r ide "><value>@ (context . Subscription . Id) < / value>< / set-header ><set-header name="X-Subscription-Name" exists- ac ti on= "over r ide "><value>@ (context . Subscription . Name) < / value>< / set-header >< / inbound><backend><base / >< / backend><outbound><base / >13SUBSTITUTE SHEET (RULE 26)

[0050] At block 414, the user identification engine 312 ensures that the developer and application 208 associated with the incoming request are authorized to access the resource allocation of the end user. In some embodiments, the user identification engine 312 may ensure the authorization by checking that the access token associated with the end user and the application 208 is valid (e.g., was correctly issued, has not expired, etc.), in addition to checking that the other developer credentials (such as the API key) are also valid for the request.

[0051] Assuming the user identification engine 312 was able to successfully ensure that the developer and application 208 are authorized, the method 400 proceeds to block 416, where a request handling engine 314 of the access management computing system 202 determines a predicted resource usage for the incoming request. Any suitable technique may be used to determine the predicted resource usage. In some embodiments, a maximum per-request resource usage may be pre-defined for each of the RM API computing systems 108 accessible by the access management computing system 202, and the appropriate maximum per-request resource usage for the requested RM API computing system 108 may be used as the predicted resource usage for the incoming request. In some embodiments, the request handling engine 314 may generate an estimated resource usage based on characteristics of the incoming request.

[0052] For example, the request handling engine 314 may use a cost function associated with the RM API computing system 108 targeted by the incoming request that uses characteristics of the incoming request to calculate the predicted resource usage. As a nonlimiting example of such a cost function, some RM API computing systems 108, such as some text-processing RM API computing systems 108 offered by OpenAI, measure resource usage based on a number of tokens associated with the request. Such a cost function may multiply the total number of tokens in the input and the output by a per-token resource usage, or may multiply the number of input tokens by an input token resource usage and the number of output tokens by an output token resource usage. Since the number of tokens in the output may be variable, the request handling engine 314 may use a configurable maximum number of output tokens that will be provided to the RM API14SUBSTITUTE SHEET (RULE 26)computing system 108 as the number of output tokens, and the actual number of tokens in the request as the number of input tokens. As another non-limiting example of such a cost function, some RM API computing systems 108, such as some image-processing RM API computing systems 108 offered by OpenAI, measure resource usage based on a requested size of the output images (e.g.. requested output image resolution) multiplied by the number of requested output images.

[0053] In some embodiments, if an express cost function is not available the request handling engine 314 may provide features of the incoming request to a predictive model trained based on previous requests in order to generate the estimated resource usage, and may use the estimated resource usage as the predicted resource usage for the incoming request. In some embodiments, instead of attempting to calculate or estimate the predicted resource usage, the request handling engine 314 may use a function offered by an RM API computing system 108 that will provide an estimated resource usage upon request, prior to consuming the resources, and this estimated resource usage provided by the RM API computing system 108 may be used as the predicted resource usage for the incoming request.

[0054] In some embodiments, multiple different predicted resource usages may be determined by the request handling engine 314. For example, the request handling engine 314 may determine both a worst-case predicted resource usage (e.g., if a maximum-sized output is generated, for variable-sized output) and an average- or best-case predicted resource usage (e.g., if an average-sized or minimum-sized output is generated), and both of the predicted resource usages may be used later in the method 400. In some embodiments, the method 400 may not perform throttling based on a predicted resource usage, and in such embodiments, the actions of block 416 may be skipped.

[0055] The method 400 then proceeds to a continuation terminal ("terminal B"). From terminal B (FIG. 4B), the method 400 proceeds to block 418, where the request handling engine 314 determines a level of throttling, if any, to be applied to the incoming request. In some embodiments, the method 400 may not be configured to perform throttling, and in such embodiments, the method 400 may skip directly from terminal B to terminal C.

[0056] As illustrated, the method 400 supports three different types of throttling, and the level of throttling to be applied may be determined based on a comparison of the predicted resource usage to the remaining resource allocation for the end user as indicated in the resource allocation data store 310. Throttling may be applied to ensure that no more computing resources are used than are allocated to the end user and / or expected by thedeveloper, even when the cost of each request is variable and when multiple requests may generally be processed concurrently. In some embodiments, a different number of levels of throttling may be available. For example, in some embodiments, the hard block and the soft block processing may be available, but the rate limiting processing may be missing. As another example, in some embodiments, the hard block and the rate limiting processing may be available, but the soft block processing may be missing.

[0057] At decision block 420. a decision is made based on whether the determined level of throttling was a hard block. A hard block may be applied if the predicted resource usage is greater than the remaining resource allocation for the end user (in other words, in response to determining that there are not enough resources allocated to the end user to cover the predicted resource usage), and the developer has configured the system to apply the hard block to avoid expending more than the allocated resources.

[0058] If it was determined that a hard block should occur, then the result of decision block 420 is YES, and the method 400 proceeds to block 422, where the request handling engine 314 denies the incoming request. In some embodiments, the access management computing system 202 may generate a response to the request that indicates the reason for the hard block, and the developer may configure the application 208 to take an appropriate action. For example, in some embodiments, the application 208 may prompt the end user to contact the access management computing system 202 to obtain an additional allocation of resources before attempting a subsequent request. As another example, the application 208 may hide the fact that the request failed from the end user, and may instead provide alternative content or functionality to replace the content or functionality expected in response to the request. The method 400 then proceeds to a continuation terminal ("terminal D").

[0059] Returning to decision block 420, if it was determined that a hard block should not occur, then the result of decision block 420 is NO, and the method 400 proceeds to decision block 424. At decision block 424, a decision is made based on whether the determined level of throttling was a soft block. A soft block may be applied if the predicted resource usage is greater than the remaining resource allocation for the end user, and the developer has configured the system to apply the soft block instead of the hard block in order to balance an improved end user experience with the avoidance of consuming more than the allocated resources. A soft block may also be applied if the predicted resource usage is within a threshold of the remaining resource allocation for the end user, and the developer has configured the threshold to trigger a soft block.

[0060] If it was determined that a soft block should occur, then the result of decision block 424 is YES, and the method 400 proceeds to block 426, where the request handling engine 314 performs an action regarding blocking future incoming requests from the end user. In some embodiments, the action may include setting a flag to apply a hard block to the next request if the resource allocation is not replenished, but allowing the present incoming request to be processed. In some embodiments, the action may include allowing the request to be processed, but causing a notification to be sent to the developer and / or the end user via a push notification, email, an update to the interface of the application 208, or via other appropriate means to indicate that the resource allocation of the end user has been depleted. The method 400 then proceeds to decision block 428.

[0061] Returning to decision block 424, if it was determined that a soft block should not occur, then the result of decision block 424 is NO, and the method 400 proceeds from decision block 424 directly to decision block 428. At decision block 428, a decision is made based on whether it was determined that the incoming request may be processed in parallel. Typically, the method 400 may process two or more requests associated with the end user in parallel, in order to provide increased performance. However, this leads to a race condition where two or more incoming requests may pass through decision block 424 without being blocked because the incoming requests individually do not exhaust the remaining resource allocation, but where the two or more incoming requests collectively consume more than the remaining resource allocation.

[0062] Accordingly, incoming requests may be processed serially when there is a risk of multiple concurrent incoming requests collectively consuming more computing resources than the remaining resource allocation. In some embodiments, a maximum per-request resource usage is multiplied by a maximum number of concurrent incoming requests, and the product is compared to the remaining resource allocation. If the remaining resource allocation is less than the product, then it may be decided to process the incoming request (and future incoming requests for the end user) serially.

[0063] If it w as determined that the incoming request should not be executed in parallel with other incoming requests for the end user, then the result of decision block 428 is NO, and the method 400 proceeds to block 430, where the request handling engine 314 adds the incoming request to a sequential request queue. Requests in the sequential request queue will be processed serially instead of in parallel, thus ensuring that the resource allocation is checked accurately for each request. In some embodiments, sequential processing may continue to be applied until the sequential request queue is empty, or until the resourceallocation is replenished to be greater than the product. The method 400 then proceeds to a continuation terminal ("terminal C").

[0064] Returning to decision block 428, if it was determined that the incoming request may be processed in parallel, then the result of decision block 428 is YES, and the method 400 proceeds directly from decision block 428 to terminal C. If the result of decision block 428 was YES, then the remaining actions of the method 400 may occur in parallel for multiple incoming requests.

[0065] From terminal C (FIG. 4C), the method 400 proceeds to block 432, where the request handling engine 314 transmits an outgoing request to the RM API computing system 108 based on the incoming request. In some embodiments, the outgoing request may include credentials associated with the developer for accessing the RM API computing system 108. The request handling engine 314 may obtain the credentials associated with the developer (e.g., an API key associated with the developer and the RM API computing system 108) from the identity provider computing system 204, or the credentials associated with the developer may be provided in the incoming request, and may simply be copied over to the outgoing request. In some embodiments, the outgoing request may include credentials associated with the access management computing system 202 instead of the developer for accessing the RM API computing system 108, in which case a resource allocation of the access management computing system 202 may be used instead of as illustrated in FIG. 6A - FIG. 6C and FIG. 7A - FIG. 7E where a resource allocation of the developer is used.

[0066] In some embodiments, the outgoing request may be created by copying the entirety of the incoming request other than the portions specifically directed to the access management computing system 202 (e.g., the identifier of the application 208 and the access token associated with the end user). In this way, the access management computing system 202 may support any RM API computing system 108 for which a developer can craft the incoming request without requiring updates to the request handling engine 314.

[0067] At block 434, the request handling engine 314 receives a response from the RM API computing system 108, and at block 436, an allocation engine 316 of the access management computing system 202 determines a resource cost expended by the outgoing request based on the response. In some embodiments, the resource cost may be included within the response received from the RM API computing system 108, and the allocation engine 316 may extract the resource cost from the response. In some embodiments, the allocation engine 316 may calculate the resource cost using a cost function as described atblock 416. For some cost functions (e.g., cost functions that include both a number of input tokens in the request and a number of output tokens in the response), the calculation at block 416 was an estimate because the size of the response was not known, but once the response is received at block 434, the size of the response is known, and so the allocation engine 316 can use the cost function to accurately calculate the amount of computing resources expended.

[0068] The method 400 then proceeds to subroutine block 438, where a subroutine is performed wherein the allocation engine 316 determines one or more resource allocation adjustments based on the resource cost and updates the resource allocation data store 310 based on the one or more resource allocation adjustments. Any suitable technique may be used to determine the one or more resource allocation adjustments, including but not limited to the technique illustrated in FIG. 6 A - FIG. 6C, or the technique illustrated in FIG. 7 A - FIG. 7E, each of which is described in further detail below. As illustrated, the method 400 assumes that at least one resource allocation adjustment will be appropriate (e.g., the response indicates a successful query7and the resource cost is non-zero). In some embodiments, if this assumption is not valid (e.g., the response indicates a failed query, or the response indicates a successful query but the resource cost is nevertheless zero), the method 400 may skip subroutine block 438, and / or may otherwise not make any resource allocation adjustments.

[0069] At block 440, the request handling engine 314 transmits the result to the application 208. Prior to transmitting the result to the application 208, the request handling engine 314 may add optional augmented information to the result. For example, the request handling engine 314 may add an indication of the remaining resource allocation for the end user, an indication of a level of throttling applied, or any other type of information that may be useful to the application 208. The application 208 may then take any appropriate action to process the results, including but not limited to generating content based on the results to be presented to the end user.

[0070] The method 400 then proceeds to terminal D, and then to decision block 442, where a decision is made regarding whether the method 400 is finished executing. Typically, the method 400 may execute as long as the application 208 is executing on the end user computing device 210, and / or as long as the access management computing system 202 is operating and available for processing requests.

[0071] If it is decided that the method 400 is not finished executing, then the result of decision block 442 is NO, and the method 400 returns to terminal A to process the nextrequest from the application 208. Otherwise, if it is decided that the method 400 is finished executing, then the result of decision block 442 is YES, and the method 400 proceeds to an end block and terminates.

[0072] FIG. 5 is a sequence diagram that illustrates a non-limiting example of a subroutine for obtaining authorization for an application to use a resource allocation of an end user, according to various aspects of the present disclosure. This subroutine is a non-limiting example of a subroutine suitable for use at subroutine block 410 of the method 400 described above. In FIG. 5, components of the system 200 are illustrated as boxes with vertical arrows extending therefrom. Time extends from the top of the figure to the bottom of the figure, and horizontal arrows illustrate communication between the components of the system 200. The circled numbers are references to points within the sequence, and are provided for the sake of illustration. One will recognize that additional communication may occur between the components (e.g., interface elements of a web site may be transmitted, an illustrated communication may take place over the course of two or more transmissions, etc.), but that these additional communications are not illustrated for the sake of brevity7and so as to not obscure the inventive portions of the subroutine.

[0073] The subroutine begins at point 502, where the application 208 transmits a request to the request handling engine 314 (or another element of the access management computing system 202) for a one-time use code. The request includes developer credentials (e.g.. an API key) used by the developer for connecting to the access management computing system 202. The request handling engine 314 generates a random, highly unique, one-time use code, which may be referred to as a Manual Callback Flow (MCF) Auth Code, or MAC. When the MAC is generated, the request handling engine 314 stores an association between the MAC and the developer credentials. The MAC is configured to expire either when used or after a threshold period of time appropriate for completing the subroutine, such as 15 minutes. Once expired, the stored association between the MAC and the developer credentials may be retained for various purposes.

[0074] At point 504, the request handling engine 314 transmits a response to the application 208 that includes the MAC and a URI to an endpoint provided by the access management computing system 202 (e.g., by the user identification engine 312) at which functionality for authorizing the application 208 may be accessed. In some embodiments, the URI may include the MAC as a query parameter (e.g., if the MAC is 1234. the URI may be “https: / / example.com / auth?mac=1234”) in order to provide all of the information in a single value for the sake of simplicity7.

[0075] At point 506. the application 208 provides the URI to the browser 206. In some embodiments, the application 208 itself may access the URI, but by providing the URI to the browser 206, the developer can utilize the capabilities of the browser 206 and the web interface provided by the user identification engine 312 instead of having to build an interface for the authorization processing into the application 208.

[0076] The browser 206 then uses the URI to conduct an authorization workflow with the user identification engine 312 and the identity provider computing system 204. One will recognize that the authorization workflow is similar to an OAuth workflow, and that the identity provider computing system 204 may be operated by a provider such as AuthO. Accordingly, at point 508, the browser 206 presents an interface requesting the end user to input their login credentials for the access management computing system 202 (e.g., a username and password), and the login credentials are transmitted to the user identification engine 312.

[0077] At point 510, the provides the end user login credentials to the identity provider computing system 204. The identity provider computing system 204 verifies the end user login credentials, and at point 512, the identity provider computing system 204 transmits authentication details related to the end user to the user identification engine 312. The user identification engine 312 stores the authentication details in the account data store 308, and associates the end user's authentication details with the developer credentials.

[0078] The authentication details received from the identity provider computing system 204 may include one or more of an access token (the credential usable by the application 208 to access the resource allocation of the end user), an ID token that includes end user profile attributes, a token expiration timestamp that indicates when the access token and ID token expire, a token lifetime value that indicates a lifetime of the access token and ID token in seconds, a refresh token that can be used to obtain a renewed access token without the end user re-authenticating, a scope value that indicates a level of access given to the application 208. or a token type value (typically "Bearer"). The authentication details may also include a user info object that includes further information relating to the end user, including but not limited to one or more of an aud value indicating an audience for which the token is intended, an email address of the end user, a value indicating whether the email address is verified, a timestamp value indicating when the token expires, a timestamp value indicating when the token was issued, a value identifying the issuer of the token, a name of the end user, a nickname of the end user, a nonce value usable to mitigate token replay attacks, a URL of a profile picture of the end user, a session ID for the authenticated session, a sub value issued by the identity provider computing system 204 that uniquelyidentifies the end user, and a timestamp indicating when the information for the end user was last updated. Because the authentication details may include sensitive information and other information that is not needed by the application 208, the authentication details may be stored at the access management computing system 202 by the user identification engine 312, but only a subset of the authentication details may be provided to the application 208.

[0079] The actions from point 508 to point 12 assume that the end user had previously created an account with the access management computing system 202. If the end user did not previously have an account, the user identification engine 312 and the identity provider computing system 204 may manage the process of establishing a new account for the end user prior to providing the authentication details at point 512.

[0080] Once the login process has completed and the authentication details have been received by the user identification engine 312, at point 514, the user identification engine 312 transmits a request to the browser 206 for the end user to configure the application 208 for use with the access management computing system 202. Since the URI provided at point 506 and used by the browser 206 to contact the access management computing system 202 at point 508 includes the MAC, and since the MAC is associated with the application identifier of the application 208, the configuration request may include any details of the application 208 available to the access management computing system 202, including but not limited to the name of the application 208, the identity of the developer, typical resource usage from other end users, descriptions of the application 208, a logo of the application 208, and / or other information.

[0081] At point 516. the end user provides application configuration information to the user identification engine 312. The configuration information may include at least whether the end user authorizes the application 208 to use their resource allocation. In some embodiments, the end user may be provided with finer-grained control, such as being able to set a limit or quota on the amount of their resource allocation that the application 208 is allowed to use, limits on RM API computing systems 108 that the application 208 is authorized to use the resource allocation for, and so on.

[0082] The user identification engine 312 stores the configuration information in the account data store 308. In some embodiments, the user identification engine 312 stores the configuration information along with one or more of the authentication details of the end user received at point 512, the credentials of the developer, and the MAC. Even though the MAC has expired because it was used to complete the end user login, it may still be used by the developer to retrieve the end user authentication information in following steps.

[0083] Once the configuration is complete, at point 518 the user identification engine 312 transmits a notification to the browser 206 that the configuration has been completed, and at point 520, the end user closes the browser 206 and returns to the application 208. In some embodiments, the interface presented by the browser 206 may prompt the end user to return to the application 208. In other embodiments, the interface may automatically bring the application 208 back to the foreground. Upon returning to the application 208, at point 522 the application 208 transmits a request for user authentication information that includes the developer credentials and the MAC. In some embodiments, the application 208 may automatically detect that it has been reactivated and may advance to point 522 automatically. In some embodiments, the application 208 may present an interface element (e.g., a button) asking the end user to confirm that the configuration has been completed, and may advance to point 522 in response to actuation of the interface element.

[0084] Even though the MAC is expired, it can be used by the request handling engine 314 to find the information in the account data store 308 indicating that the end user authorized the application 208. Assuming that the end user authorized the application 208, at point 524, the request handling engine 314 returns user authentication information to the application 208. In some embodiments, the user authentication information is a subset of the user authentication details provided at point 512, and may include one or more of an access token, a timestamp at which the access token will expire, a lifetime in seconds of the access token, a refresh token usable to acquire a new access token upon expiration, a sub value (i.e.. a unique, stable identifier of the end user established by the identity provider computing system 204), contact information (e.g., an email address) of the end user, a name of the end user, and / or a profile picture of the end user.

[0085] Once the application 208 receives the user authentication information at point 524, it may use the information (typically, the access token and the sub value) to prove to the access management computing system 202 that it has been authorized to use the resource allocation of the end user.

[0086] FIG. 6A -FIG. 6C provide schematic illustrations of a non-limiting example embodiment of resource allocation adjustments made by the allocation engine based on a resource cost, according to various aspects of the present disclosure. The illustrated resource allocation adjustments are a non-limiting example of a suitable subroutine for use at subroutine block 438. In FIG. 6A. a resource cost 602 determined by the allocation engine 316 at block 436 of the method 400 is illustrated. The resource cost 602 is represented by a consumed resource balance 604, which may be stored in the resource allocation data store 310, or may be temporarily tracked in memory by the allocation engine316 during the subroutine. As shown, the allocation engine 316 determined that five units of computing resources were consumed by the processed request, and so the consumed resource balance 604 starts in FIG. 6A as negative five units.

[0087] FIG. 6A also shows two tracked resource allocations: a developer allocation 606 and an end user allocation 608. The tracked resource allocations are typically persistently stored in the resource allocation data store 310. The end user allocation 608 was discussed in detail above, and represents an amount of computing resources allocated to the end user. Since all consumption of computing resources by the RM API computing system 108 is to be accounted for, the developer allocation 606 is provided to track an amount of computing resources that have been consumed by the developer (e.g., by the application 208) without having been attributed to an end user.

[0088] FIG. 6B illustrates a first resource allocation adjustment made by the allocation engine 316. The developer allocation 606 is adjusted to transfer five units of allocated computing resources from the developer allocation 606 in order to satisfy the consumed resource balance 604. Essentially, this is the situation that the developer would be in within the old system 100 illustrated in FIG. 1, wherein the developer was individually responsible for all computing resources consumed by the application 208.

[0089] FIG. 6C illustrates an improvement to this situation that is provided by embodiments of the present disclosure, as the allocation engine 316 adjusts the end user allocation 608 and the developer allocation 606 to transfer five units of allocated computing resources from the end user allocation 608 to the developer allocation 606. As discussed above, this allows the developer to support a larger volume of end users, as the developer allocation 606 is not solely responsible for covering the resource cost 602, but may instead use computing resources allocated to the end users.

[0090] FIG. 7A - FIG. 7E provide schematic illustrations of a non-limiting example embodiment of more complex resource allocation adjustments made by the allocation engine based on a resource cost, according to various aspects of the present disclosure. As with FIG. 6A - FIG. 6C, the illustrated resource allocation adjustments in FIG. 7A - FIG. 7E are non-limiting examples of a suitable subroutine for use at subroutine block 438.

[0091] In FIG. 7A, a resource cost 702, a developer allocation 708, and an end user allocation 710 similar to those illustrated in FIG. 6A are shown. How ever, in FIG. 7 A, the resource cost 702 includes both a consumed resource balance 704 and a markup balance 706. The consumed resource balance 704 represents the amount of computing resources consumed by the RM API computing system 108 in processing the request, as determined atblock 436. The markup balance 706 represents additional adjustments to be made by the allocation engine 316, as discussed in further detail below.

[0092] FIG. 7B is similar to FIG. 6B, and shows a first resource allocation adjustment, wherein the allocation engine 316 adjusts the developer allocation 708 to transfer five units of allocated computing resources from the developer allocation 708 in order to satisfy the consumed resource balance 704. As a result, the consumed resource balance 704 is covered, and the developer allocation 708 is reduced by five units.

[0093] FIG. 7C illustrates subsequent resource allocation adjustments made by the allocation engine 316. The allocation engine 316 adjusts the end user allocation 710, but instead of adjusting the end user allocation 710 by an amount that matches the initial consumed resource balance 704, the allocation engine 316 adjusts the end user allocation 710 by an amount equal to the consumed resource balance 704 (five units) plus a markup value (one unit). The developer allocation 708 is adjusted by the amount of the initial consumed resource balance 704 (five units), and the markup value is added to the markup balance 706.

[0094] Further adjustments to resource allocations may then be made to distribute the markup balance 706. FIG. 7D and FIG. 7E illustrate two non-limiting examples of adjustments made to distribute the markup balance 706 shown in FIG. 7C. In FIG. 7D, the allocation engine 316 adjusts the developer allocation 708 to add the entire markup balance 706 to the developer allocation 708. This can be beneficial to the developer and incentivize the developer to provide additional functionality (e.g., services to end users who have not yet been granted resource allocations or who have exhausted their resource allocations, transferring the extra allocated resources to other parties for their use, etc.).

[0095] In FIG. 7E, a more complicated distribution of the markup balance 706 is shown. In addition to the developer allocation 708 and the end user allocation 710, a system allocation 712 and a third-party allocation 714 are also tracked. The system allocation 712 may be associated with the access management computing system 202 itself, and may represent computing resources usable by the access management computing system 202 to either provide the services described herein or other services. The third-party allocation 714 may be associated with another party that provides services used by the access management computing system 202, including but not limited to a cloud service provider, a network provider, or other types of service providers. As shown, the allocation engine 316 splits the one unit of the markup balance 706, and adjusts the developer allocation 708 by0.5 units, adjusts the system allocation 712 by 0.17 units, and adjusts the third-party allocation 714 by 0.03 units.

[0096] Though the markup balance 706 is shown in FIG. 7E as being split between three allocations, this should not be seen as limiting, and in other embodiments, more, fewer, or different allocations may be adjusted. In some embodiments, the initial amount of the markup balance 706 and the split amounts for distributing the markup balance 706 may be configurable by the developer, and may be configured by the developer differently for each application 208.

[0097] In addition to configuring the amount and distribution of the markup balance 706, in some embodiments, the developer may be able to configure how the markup balance 706 is treated for purposes of throttling. For example, if an end user allocation 710 is adequate to cover a predicted resource usage for an incoming request, but is not enough to cover the predicted resource usage plus the markup, this may be taken into consideration when choosing a level of throttling for the incoming request. While the developer may choose to hard block an incoming request for which the remaining resource allocation is smaller than the predicted resource usage, the developer may choose to merely soft block an incoming request for which the remaining resource allocation is greater than the predicted resource usage but smaller than the predicted resource usage plus the markup. Alternatively, the developer may choose to include both the remaining resource allocation and the markup as a combined total to be used in all of the throttling decisions.

[0098] Throughout FIG. 6A - FIG. 6C and FIG. 7A - FIG. 7E, the numbers of units should not be seen as limiting, but are instead provided as non-limiting example amounts to illustrate the types of adjustments made by the allocation engine 316. Further, while these figures refer to generic '‘units,’’ one will recognize that any suitable type unit that may be used to represent the consumption of computing resources by an RM API computing system 108 may be used, including but not limited to CPU time, tokens processed, images generated, memory used, data transferred, or any other suitable unit.

[0099] Some embodiments of the system 200 as described above may be used to implement the following non-limiting example experience for an end user.

[0100] Brooke, a non-tech savvy iOS user hears about a new app called "Storyteller." Story teller (an application 208 configured for use with the access management computing system 202) is blowing up on social media because of its ability to tell highly engaging and interactive stories to children. Brooke thinks that her 5-year-old daughter, Elie, would absolutely love something like this, so she downloads it to try it out for herself.

[0101] Brooke opens the Storyteller app on her iPhone (a non-limiting example of an end user computing device 210) and is asked to create an account. She logs in with her Facebook ID (utilizing Facebook as an identity provider computing system 204) and finishes the sign-up process. Storyteller then displays a playful, animated avatar of a children’s fairy. It asks Brooke to tell the fairy what kind of story she’d like to hear. Brooke says aloud, “tell me a story about rainbows and princes," knowing that Elie loves both of those things. After a brief moment, the experience works magically, and Brooke is treated to a full, narrated story about rainbows and princes, with accompanying music and relevant imagery. Brooke is blown away at how her simple suggestion turned into such an experience and knows that Elie will love this. She tries out the app a couple more times for herself and is blown away each time.

[0102] After the third story is finished. Storyteller indicates that Brooke has used up her trial stories. Unbeknownst to Brooke, the app may have been using computing resources allocated to the developer of Storyteller, and Brooke's share of these computing resources has now been exhausted. The app tells Brooke that if she would like to continue, she needs to login with her account on the access management computing system 202. Brooke has never heard of the access management computing system 202 before, but loving the experience provided by the app, she clicks on the link.

[0103] Brooke is taken to a landing page that explains what the access management computing system 202 provides: a unique service designed to make using innovative, AI- powered apps like Storyteller easy, accessible, and safe by automatically preventing unexpected resource consumption. With the access management computing system 202, she will be able to enjoy as many stories as she wants, though each of the stories will consume computing resources allocated to her. She can also set up a quota for the maximum amount of her resources she would like Storyteller to consume and the access management computing system 202 will ensure that she never exceeds it, and that the developer does not use her resources for other purposes. The landing page features a simple, user-friendly- interface with bright, appealing visuals and easy-to-understand language.

[0104] Feeling reassured and eager to use the app again, Brooke clicks on the "Sign Up" button and proceeds to create her account on the access management computing system 202. She is guided through the setup process step by step, linking her accounts and setting a comfortable quota for Storyteller.

[0105] Once Brooke finishes setting up her account, she is returned to the Story teller app. This time, she invites Elie to join in. Together, they listen to an enchanting story about aprince who found a magical rainbow. Brooke is relieved that she doesn't have to worn’ that the Storyteller app will unexpectedly consume more resources than she had been allocated. With the peace of mind offered by the access management computing system 202, Brooke and Elie enjoy countless magical storytelling sessions, turning their bedtime routine into a creative and imaginative experience.

[0106] Some embodiments of the system 200 as described above may be used to implement the following non-limiting example experience for a developer.

[0107] David is a developer interested in releasing apps that use the latest Al innovations to create novel experiences. After coming up with the concept for the Story teller app, he crunches the numbers and realizes that the Al API resources consumed by generating each story are too high to be sustainable.

[0108] Luckily, David has heard of the access management computing system 202, which is going viral on developer networks. He’s happy to learn that his code requires virtually no modifications to use the access management computing system 202 instead of using RM API computing systems 108 such as OpenAI directly, since the exact same APIs are available, with no discrepancy in supported features or languages. Further, since the access management computing system 202 will manage all of the resource consumption associated with the Al API usage and will associate it with end users causing the consumption of the resources, David can create the Storyteller app without worrying about scaling Al API usage as his app becomes more popular.

[0109] He follows the instructions on the access management computing system 202 website to create a developer account and registers the ’‘Storyteller” app as an application 208 with the access management computing system 202. After generating an API key for accessing the access management computing system 202, he walks through a quick tutorial on how to integrate the login experience of the access management computing system 202 into his app. and then call some OpenAI APIs using both his API key for the access management computing system 202 and his OpenAI API key. He learns that the access management computing system 202 will manage all of the resource management associated with using OpenAI and sees several other supported API providers listed as well. He also learns that the end user gets to decide how many resources they are willing to consume, and how to handle cases where the end user has reached their quota.

[0110] David finds that the small amount of extra code to integrate with the access management computing system 202 is more than worth the peace of mind knowing that as his app gets more popular, it will scale properly. David releases his Storyteller app, and itimmediately goes viral. Users are experiencing wonderful stories with engaging technology that was never before possible. As his user count reaches the millions, the Platform ensures that his personal Al API resource consumption remains minimal.

[0111] While illustrative embodiments have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the invention.EXAMPLES

[0112] The following numbered examples describe non-limiting example embodiments of the disclosed subject matter.

[0113] Example 1. A method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device, the method comprising: storing, by an access management computing system, a record of a resource allocation for the end user; receiving, by the access management computing system, an incoming request from the application for a resource-managed application programming interface computing system (RM API computing system), wherein the incoming request includes credentials associated with the developer and the application for access to the access management computing system; transmitting, by the access management computing system, an outgoing request to the RM API computing system identified in the incoming request, wherein the outgoing request is based on the incoming request; receiving, by the access management computing system, an incoming response to the outgoing request from the RM API computing system; transmitting, by the access management computing system, an outgoing response to the application based on the incoming response; determining, by the access management computing system, an amount of computing resources consumed by the outgoing request; and adjusting, by the access management computing system, the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request.

[0114] Example 2. The method of Example 1, wherein the credentials associated with the developer and the application for access to the access management computing system include an access token associated with the end user.

[0115] Example 3. The method of Example 2, further comprising: verifying, by the access management computing system, the access token to authorize the application to use computing resources from the resource allocation for the end user.

[0116] Example 4. The method of Example 3, further comprising obtaining, by the access management computing system, the access token from an identity provider.

[0117] Example 5. The method of Example 4, wherein the identity7provider is AuthO.

[0118] Example 6. The method of any one of Examples 1-5, wherein the outgoing request includes credentials associated with the developer for access to the RM API computing system.

[0119] Example 7. The method of any one of Examples 1-6, wherein the incoming request includes an identification of the RM API computing system.

[0120] Example 8. The method of any one of Examples 1-7, wherein the record of the resource allocation for the end user includes a quota identifying a maximum amount of computing resources allowed to be consumed by the application on behalf of the end user.

[0121] Example 9. The method of any one of Examples 1-8, further comprising: receiving, by the access management computing system, a second incoming request from the application for the RM API computing system; and determining, by the access management computing system, a predicted resource usage for a second outgoing request based on the second incoming request.

[0122] Example 10. The method of Example 9, wherein determining the predicted resource usage for the second outgoing request based on the second incoming request includes at least one of: determining a count of tokens used by the second incoming request; or determining a size of a result requested by the second incoming request and a number of results requested by the second incoming request.

[0123] Example 11. The method of any one of Examples 9-10, further comprising: determining, by the access management computing system, a level of throttling to apply based on a remaining resource allocation in the record.

[0124] Example 12. The method of Example 11. wherein determining the level of throttling to apply based on the remaining resource allocation in the record includes: comparing, by the access management computing system, the predicted resource usage for the second outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the second outgoing request is greater than the remaining resource allocation in the record, blocking, by the access management computing system, the second outgoing request.

[0125] Example 13. The method of any one of Examples 11-12, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the predicted resource usage for the second outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the second outgoing request is within a threshold amount of the remaining resource allocation, causing, by the access management computing system a future outgoing request for the end user to be blocked while allowing the second outgoing request to proceed.

[0126] Example 14. The method of any one of Examples 11-13, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the remaining resource allocation to a product of a maximum per-request resource usage and a maximum number of concurrent incoming requests; and in response to determining that the remaining resource allocation is within a threshold amount of the product, serializing, by the access management computing system, the outgoing request and one or more other outgoing requests associated with the end user to the RM API computing system.

[0127] Example 15. The method of any one of Examples 1-14, wherein adjusting the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request includes adjusting the record of the resource allocation for the end user by the amount of computing resources consumed by the outgoing request plus a markup.

[0128] Example 16. The method of Example 15, further comprising: adjusting one or more resource allocations for parties other than the end user based on the markup.

[0129] Example 17. A method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device, the method comprising: storing, by an access management computing system, a record of a resource allocation for the end user; receiving, by the access management computing system, an incoming request from the application for a resource-managed application programming interface computing system (RM API computing system), wherein the incoming request includes credentials associated with the developer and the application for access to the access management computing system; determining, by the access management computing system, a level of throttling to apply to an outgoing request based on a remaining resource allocation in the record, wherein the outgoing request is based on the incoming request; and based on the level of throttling: selectively transmitting, by the access management computing system, the outgoing request to the RM API computing system identified in the incoming request; or blocking the outgoing request.

[0130] Example 18. The method of Example 17. further comprising: determining, by the access management computing system, a predicted resource usage for an outgoing request based on the incoming request.

[0131] Example 19. The method of Example 18. wherein determining the predicted resource usage for the second outgoing request based on the second incoming requestincludes at least one of: determining a count of tokens used by the second incoming request; or determining a size of a result requested by the second incoming request and a number of results requested by the second incoming request.

[0132] Example 20. The method of any one of Examples 18-19, wherein determining the level of throttling to apply based on the remaining resource allocation in the record includes: comparing, by the access management computing system, the predicted resource usage for the outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the outgoing request is greater than the remaining resource allocation in the record, blocking, by the access management computing system, the outgoing request.

[0133] Example 21. The method of any one of Examples 18-20, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the predicted resource usage for the outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the outgoing request is within a threshold amount of the remaining resource allocation, causing, by the access management computing system a future outgoing request for the end user to be blocked while allowing the outgoing request to proceed.

[0134] Example 22. The method of any one of Examples 17-21, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the remaining resource allocation to a product of a maximum per-request resource usage and a maximum number of concurrent incoming requests; and in response to determining that the remaining resource allocation is within a threshold amount of the product, serializing, by the access management computing system, the outgoing request and one or more other outgoing requests associated with the end user to the RM API computing system.

[0135] Example 23. The method of any one of Examples 17-23, wherein the credentials associated with the developer and the application for access to the access management computing system include an access token associated with the end user.

[0136] Example 24. The method of Example 23, further comprising: verifying, by the access management computing system, the access token to authorize the application to use computing resources from the resource allocation for the end user.

[0137] Example 25. The method of any one of any one of Examples 23-24, further comprising obtaining, by the access management computing system, the access token from an identity provider.

[0138] Example 26. The method of Example 25. wherein the identity provider is AuthO.

[0139] Example 27. The method of any one of Examples 17-26, wherein the outgoing request includes credentials associated with the developer for access to the RM API computing system.

[0140] Example 28. The method of any one of Examples 17-27, wherein the incoming request includes an identification of the RM API computing system.

[0141] Example 29. The method of any one of Examples 17-28, wherein the record of the resource allocation for the end user includes a quota identifying a maximum amount of computing resources allowed to be consumed by the application on behalf of the end user.

[0142] Example 30. The method of any one of Examples 17-29, further comprising: receiving, by the access management computing system, an incoming response to the outgoing request from the RM API computing system; transmitting, by the access management computing system, an outgoing response to the application based on the incoming response; determining, by the access management computing system, an amount of computing resources consumed by the outgoing request; and adjusting, by the access management computing system, the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request.

[0143] Example 31. The method of Example 30, wherein adjusting the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request includes adjusting the record of the resource allocation for the end user by the amount of computing resources consumed by the outgoing request plus a markup.

[0144] Example 32. The method of Example 31. further comprising: adjusting one or more resource allocations for parties other than the end user based on the markup.

[0145] Example 33. A non-transitory computer-readable medium having computerexecutable instructions stored thereon that, in response to execution by one or more processors of an access management computing system, cause the access management computing system to perform a method as recited in any one of Examples 1-32.

[0146] Example 34. A system for secure management of consumption of computing resources used by an application, the system comprising: an access management computing system configured to perform a method as recited in any one of Examples 1-32.

[0147] Example 35. The system of Example 34. further comprising: at least one artificial intelligence computing system, wherein the at least one artificial intelligence computing system includes one or more computing devices configured to provide access to one or more artificial intelligence services by providing an resource-managed application programming interface.

[0148] Example 36. The system of Example 35, wherein the at least one artificial intelligence computing sy stem is configured to provide at least one of: a gpt-3.5 API; a gpt- 4 API; an embeddings API: or a DALL-E API.

[0149] Example 37. The system of any one of Examples 34-36, further comprising: at least one end user computing device, wherein the end user computing device is associated with an end user, and wherein the end user computing device is configured to access the application provided by the developer.

[0150] Example 38. The system of Example 37, wherein accessing the application includes executing the application.

[0151] Example 39. The system of any one of Examples 37-38, wherein accessing the application includes receiving information generated by the application while the application is executing on a separate computing device.

[0152] Example 40. The system of Example 39, wherein the separate computing device includes a computing device of a cloud computing system.

[0153] Example 41. The system of any one of Examples 34-40, further comprising: an identity provider configured to authenticate end users and provide access tokens to the access management computing system.

Claims

CLAIMSThe embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device, the method comprising: storing, by an access management computing system, a record of a resource allocation for the end user; receiving, by the access management computing system, an incoming request from the application for a resource-managed application programming interface computing system (RM API computing system), wherein the incoming request includes credentials associated with the developer and the application for access to the access management computing system; determining, by the access management computing system, a level of throttling to apply to an outgoing request based on a remaining resource allocation in the record, wherein the outgoing request is based on the incoming request; and based on the level of throttling: selectively transmitting, by the access management computing system, the outgoing request to the RM API computing system identified in the incoming request; or blocking the outgoing request.

2. The method of claim 1, further comprising: determining, by the access management computing system, a predicted resource usage for an outgoing request based on the incoming request.

3. The method of claim 2, wherein determining the predicted resource usage for the second outgoing request based on the second incoming request includes at least one of: determining a count of tokens used by the second incoming request; or determining a size of a result requested by the second incoming request and a number of results requested by the second incoming request.

4. The method of claim 2, wherein determining the level of throttling to apply based on the remaining resource allocation in the record includes:comparing, by the access management computing system, the predicted resource usage for the outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the outgoing request is greater than the remaining resource allocation in the record, blocking, by the access management computing system, the outgoing request.

5. The method of claim 2, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the predicted resource usage for the outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the outgoing request is within a threshold amount of the remaining resource allocation, causing, by the access management computing system a future outgoing request for the end user to be blocked while allowing the outgoing request to proceed.

6. The method of claim 1, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the remaining resource allocation to a product of a maximum per-request resource usage and a maximum number of concurrent incoming requests; and in response to determining that the remaining resource allocation is within a threshold amount of the product, serializing, by the access management computing system, the outgoing request and one or more other outgoing requests associated with the end user to the RM API computing system.

7. The method of claim 1, wherein the credentials associated with the developer and the application for access to the access management computing system include an access token associated with the end user.

8. The method of claim 7, further comprising: verifying, by the access management computing system, the access token to authorize the application to use computing resources from the resource allocation for the end user.

9. The method of claim 8, further comprising obtaining, by the access management computing system, the access token from an identity provider.

10. The method of claim 9, wherein the identity provider is AuthO.

11. The method of claim 1. wherein the outgoing request includes credentials associated with the developer for access to the RM API computing system.

12. The method of claim 1, wherein the incoming request includes an identification of the RM API computing system.

13. The method of claim 1, wherein the record of the resource allocation for the end user includes a quota identifying a maximum amount of computing resources allowed to be consumed by the application on behalf of the end user.

14. The method of claim 1, further comprising: receiving, by the access management computing system, an incoming response to the outgoing request from the RM API computing system; transmitting, by the access management computing system, an outgoing response to the application based on the incoming response; determining, by the access management computing system, an amount of computing resources consumed by the outgoing request; and adjusting, by the access management computing system, the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request.

15. The method of claim 14, wherein adjusting the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request includes adjusting the record of the resource allocation for the end user by the amount of computing resources consumed by the outgoing request plus a markup.

16. The method of claim 15, further comprising: adjusting one or more resource allocations for parties other than the end user based on the markup.

17. A method for secure management of consumption of computing resources on behalf of an end user by an application provided by a developer and accessed by the end user via an end user computing device, the method comprising: storing, by an access management computing system, a record of a resource allocation for the end user; receiving, by the access management computing system, an incoming request from the application for a resource-managed application programming interface computing system (RM API computing system), wherein the incoming request includes credentialsassociated with the developer and the application for access to the access management computing system; transmitting, by the access management computing system, an outgoing request to the RM API computing system identified in the incoming request, wherein the outgoing request is based on the incoming request; receiving, by the access management computing system, an incoming response to the outgoing request from the RM API computing system; transmitting, by the access management computing system, an outgoing response to the application based on the incoming response; determining, by the access management computing system, an amount of computing resources consumed by the outgoing request; and adjusting, by the access management computing system, the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request.

18. The method of claim 17, wherein the credentials associated with the developer and the application for access to the access management computing system include an access token associated with the end user.

19. The method of claim 18, further comprising: verifying, by the access management computing system, the access token to authorize the application to use computing resources from the resource allocation for the end user.

20. The method of claim 19, further comprising obtaining, by the access management computing system, the access token from an identity provider.

21. The method of claim 20, wherein the identify provider is AuthO.

22. The method of claim 17, wherein the outgoing request includes credentials associated with the developer for access to the RM API computing system.

23. The method of claim 17, wherein the incoming request includes an identification of the RM API computing system.

24. The method of claim 17, wherein the record of the resource allocation for the end user includes a quota identify ing a maximum amount of computing resources allowed to be consumed by the application on behalf of the end user.

25. The method of claim 17, further comprising: receiving, by the access management computing system, a second incoming request from the application for the RM API computing system; and determining, by the access management computing system, a predicted resource usage for a second outgoing request based on the second incoming request.

26. The method of claim 25, wherein determining the predicted resource usage for the second outgoing request based on the second incoming request includes at least one of: determining a count of tokens used by the second incoming request; or determining a size of a result requested by the second incoming request and a number of results requested by the second incoming request.

27. The method of claim 25, further comprising: determining, by the access management computing system, a level of throttling to apply based on a remaining resource allocation in the record.

28. The method of claim 27, wherein determining the level of throttling to apply based on the remaining resource allocation in the record includes: comparing, by the access management computing system, the predicted resource usage for the second outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the second outgoing request is greater than the remaining resource allocation in the record, blocking, by the access management computing system, the second outgoing request.

29. The method of claim 27, wherein determining the level of throttling to apply based on the remaining resource allocation includes: comparing, by the access management computing system, the predicted resource usage for the second outgoing request to the remaining resource allocation in the record; and in response to determining that the predicted resource usage for the second outgoing request is within a threshold amount of the remaining resource allocation, causing, by the access management computing system a future outgoing request for the end user to be blocked while allowing the second outgoing request to proceed.

30. The method of claim 27, wherein determining the level of throttling to apply based on the remaining resource allocation includes:comparing, by the access management computing system, the remaining resource allocation to a product of a maximum per-request resource usage and a maximum number of concurrent incoming requests; and in response to determining that the remaining resource allocation is within a threshold amount of the product, serializing, by the access management computing system, the outgoing request and one or more other outgoing requests associated with the end user to the RM API computing system.

31. The method of claim 17, wherein adjusting the record of the resource allocation for the end user based on the amount of computing resources consumed by the outgoing request includes adjusting the record of the resource allocation for the end user by the amount of computing resources consumed by the outgoing request plus a markup.

32. The method of claim 31, further comprising: adjusting one or more resource allocations for parties other than the end user based on the markup.

33. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by one or more processors of an access management computing system, cause the access management computing system to perform a method as recited in any one of claim 1 to claim 32.

34. A system for secure management of consumption of computing resources used by an application, the system comprising: an access management computing system configured to perform a method as recited in any one of claim 1 to claim 32.

35. The system of claim 34, further comprising: at least one artificial intelligence computing system, wherein the at least one artificial intelligence computing system includes one or more computing devices configured to provide access to one or more artificial intelligence services by providing an resourcemanaged application programming interface.

36. The system of claim 35, wherein the at least one artificial intelligence computing system is configured to provide at least one of: a gpt-3.5 API; a gpt-4 API; an embeddings API; ora DALL-E API.

37. The system of claim 34, further comprising: at least one end user computing device, wherein the end user computing device is associated with an end user, and wherein the end user computing device is configured to access the application provided by the developer.

38. The system of claim 37, wherein accessing the application includes executing the application.

39. The system of claim 37, wherein accessing the application includes receiving information generated by the application while the application is executing on a separate computing device.

40. The system of claim 39, wherein the separate computing device includes a computing device of a cloud computing system.

41. The system of claim 34, further comprising: an identity provider configured to authenticate end users and provide access tokens to the access management computing system.

Images