System and method for providing reasonable allocation of shared capacity in multi-tenant resource environment

Abstract

Systems and methods for providing reasonable allocation of resources in a multi-tenant environment are disclosed herein. Systems and methods are configured for identifying a plurality of tenants participating in a multi-tenant environment. For each tenant of the plurality of tenants, the system determines a tenant state as a donating tenant, a reasonable borrowing tenant, or an unreasonable borrowing tenant, and applies a different borrowing algorithm to each tenant of the plurality of tenants based on the corresponding tenant state determined for each tenant. The different borrowing algorithms are configured to determine borrowing restrictions of the different resources from the common pool of resources for each tenant.

Description

Background Technology

[0001] Large Language Models (LLMs) are trained on massive amounts of text data and are able to generate coherent and context-sensitive sentences by predicting the probability of words based on previously used words in the text. LLMs have a wide range of applications, including but not limited to text generation, translation, summarization, and question answering.

[0002] LLMs require significant computing resources to run, particularly graphics processing units (GPUs), which are well-suited for the parallel processing tasks involved in training and running these models. The cost of these resources, both in terms of acquisition and operation, can be very high. Therefore, maximizing the utilization of these resources is a priority for organizations operating LLMs.

[0003] One way to maximize resource utilization is through multi-tenancy, where multiple users or tenants share access to the same hardware resources. In such a multi-tenant environment, each tenant is typically allocated a certain amount of total available hardware capacity. However, at any given time, some tenants may not use their full allocation, resulting in unused capacity.

[0004] Resource allocation in a multi-tenant environment is a complex task. It involves ensuring that each tenant has access to the resources allocated to them, while also allowing efficient use of any unused capacity. This requires a system capable of tracking active resource usage, managing resource allocation and deallocation, and ensuring proper access to resources among tenants.

[0005] Various strategies and algorithms have been proposed for resource allocation in multi-tenant environments. These include queue-based rational sharing algorithms, time-slice allocation algorithms, and max-min rational algorithms. However, these algorithms have limitations and do not always provide the desired level of rationality or resource utilization efficiency, especially for controversial resources in the context of LLM.

[0006] In view of the above, there is a need to provide improved methods and systems for the allocation of shared resources in a multi-tenant environment.

[0007] The subject matter claimed herein is not limited to embodiments that address any shortcomings or operate only in environments such as those described above. Rather, this background art is provided merely to illustrate an exemplary technical field in which some of the embodiments described herein can be practiced. Summary of the Invention

[0008] The disclosed embodiments relate to systems and methods for managing shared resources in a multi-tenant environment. Typically, the system is configured to identify multiple tenants participating in the multi-tenant environment. For each of the multiple tenants, the system determines the tenant status as a donating tenant, a legitimate borrowing tenant, or an unreasonable borrowing tenant, and applies different borrowing algorithms to each of the multiple tenants based on the corresponding tenant status determined for each tenant.

[0009] Some implementations include guaranteeing requests from tenants determined to be legitimate borrowers and rejecting requests from tenants determined to be illegitimate borrowers. Different borrowing algorithms are configured to determine different resource borrowing limits from the shared resource pool for each tenant, depending on whether the tenant is determined to be legitimate or illegitimate borrower.

[0010] More specifically, some embodiments also relate to systems and methods for providing equitable resource allocation in a multi-tenant environment. For example, for each of multiple tenants, the system first determines a fixed limit on the amount of resource capacity guaranteed to be available to each tenant. The system also periodically evaluates each tenant to identify unused resources and allocates the unused resources to a common pool.

[0011] The system then calculates the reasonable borrowing limit for resources allocated to each tenant from the public pool. This reasonable borrowing limit is based, at least in part, on a proportional analysis of the percentage of resources currently borrowed by each tenant from the public pool, relative to the sum of all fixed limits for resources corresponding to multiple tenants, compared to a fixed limit guaranteeing the available resource capacity for each tenant. Based on this proportional analysis, the system determines which tenants are reasonable borrowers (i.e., tenants reasonably borrowing resources from the public pool) and which are unreasonable borrowers (i.e., tenants unreasonably borrowing resources from the public pool).

[0012] The system then generates a new, calculated borrowing limit for the resources allocated to the unreasonable borrowing tenant from the common pool. This new calculated borrowing limit is lower than the previously calculated reasonable borrowing limit for the resources. Finally, in some instances, if it is determined that a request from the unreasonable borrowing tenant to borrow additional resources from the common pool would cause the resources allocated to the unreasonable borrowing tenant from the common pool to exceed the new calculated borrowing limit, the system will reject the request from the unreasonable borrowing tenant.

[0013] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter.

[0014] Additional features and advantages will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the teachings herein. The features and advantages of the invention can be realized and obtained by the instruments and combinations particularly pointed out in the appended claims. The features of the invention will become more fully apparent from the following description and the appended claims, or may be learned by practicing the invention as set forth below. Attached Figure Description

[0015] To describe how the above and other advantages and features can be obtained, a more specific description of the subject matter briefly described above will be presented by referring to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments and are therefore not intended to limit their scope; the embodiments will be described and explained with additional features and details using the drawings, in which:

[0016] Figure 1 The illustration shows an embodiment with a flowchart of multiple actions associated with a method for providing shared allocation of resources in a multi-tenant resource environment.

[0017] Figure 2 Another example embodiment is illustrated with a flowchart of multiple actions associated with a method for providing shared allocation of resources in a multi-tenant resource environment.

[0018] Figure 3 The illustration shows an example embodiment of shared resources in a multi-tenant environment.

[0019] Figure 4 The illustration shows an example implementation of fixed restrictions per tenant in a multi-tenant resource environment.

[0020] Figure 5 The illustration shows an example implementation of evaluating used and unused capacity for donating resources to a public pool.

[0021] Figures 6 to 7 The illustration shows an example implementation of fulfilling a request to borrow resources from a public pool.

[0022] Figure 8 An example embodiment for determining tenant status is illustrated.

[0023] Figure 9 The illustration shows an example implementation of determining resource allocation limits based on different tenant states.

[0024] Figure 10 Another example embodiment for determining tenant status is illustrated.

[0025] Figure 11 An example computing environment is illustrated, in which a computing system is combined with and / or used to perform the disclosed aspects of the disclosed embodiments. Detailed Implementation

[0026] The disclosed embodiments can be used to improve resource allocation in multi-tenant resource environments. Specifically, this document includes systems and methods for applying different rational algorithms for resource allocation among different tenants. As mentioned above, one way to maximize resource utilization is through multi-tenancy, where multiple users (e.g., client systems) or tenants share access to the same hardware resources. In such a multi-tenant environment, each tenant is typically allocated a certain amount of total available hardware capacity.

[0027] As described herein, a tenant can include, for example, a single, independent computer system belonging to a user. In other cases, a tenant is a distributed network of computer systems, such as an enterprise system comprising multiple distinct, independent computer systems linked together and associated with a tenant entity that includes an enterprise.

[0028] At any given time, some tenants may not use their full allocation, resulting in unused capacity. Therefore, the disclosed embodiments facilitate improved resource utilization across all tenants participating in a shared resource environment. It should be understood that in some cases, a tenant refers to a human user, while in others, a tenant is any computer-based user (e.g., a robot or software application) utilizing the resources. In some cases, a multi-tenant environment includes both human-based and computer-based tenants. Technological benefits

[0029] As will be described in more detail in this paper, there are many technical problems in conventional systems for allocating shared resources. Some of these technical problems are related to conventional systems. For example, in time-sliced ​​allocation systems, CPU time is divided into equal time slices and allocated sequentially to each process. A process runs its allocated time slice and then pauses until a new time slice is allocated to it. However, this approach relies on interruptible processes. When processing requests using LLM, requests cannot be interrupted (i.e., tenants must be able to maintain the required amount of resources for the entire duration the system processes the LLM request). Some conventional systems manage shared resources by allocating network bandwidth based on reasonable capacity slices for different tasks. However, when rating requests on LLM, a minimum amount of capacity must be reserved for the entire duration of the task, and typically, the duration is unknown when allocating resource capacity. This problem is addressed by allocating resources in a way that is not limited by time slices.

[0030] Some conventional resource management solutions attempt to address the problem of rational resource allocation by maintaining more resources than are needed. However, this can lead to resource waste and underutilization. In view of the aforementioned technical problems, the disclosed embodiments relate to technical solutions that can be used to help solve the problem of rational resource allocation while also providing maximum utilization, as will be described in more detail below.

[0031] For example, the disclosed embodiments relate to systems and methods that utilize near real-time approaches to provide a reasonable allocation of shared resource capacity across multiple tenants for an LLM, ensuring that tenants have guaranteed access to their fixed limits and reasonable access to additional resources beyond their fixed limits. This is achieved by: tracking tenant capacity active usage through leases, periodically assessing tenants' unused capacity, maintaining a public pool of unused resources from which tenants can reclaim or borrow resources, and finally, forcing tenants to return borrowed capacity to the public pool when tenants below their fixed limits require more resources.

[0032] Any tenant, including those who borrow unreasonably, can borrow from the common pool, thus promoting maximum utilization of all available resources in a multi-tenant environment. However, while any tenant can borrow from the common pool, reasonable access to resources is achieved by reserving resources in the common pool for donating or reasonably borrowing tenants (i.e., unreasonable borrowing tenants will leave sufficient reserved capacity in the common pool for other tenants).

[0033] In summary, the disclosed embodiments provide numerous technical benefits associated with improved systems and methods for managing resources in a multi-tenant environment, such as providing minimum guaranteed access to resources for each tenant, providing maximum reasonable access to additional resources for each tenant based on the current utilization status, and maximizing the utilization of all resources across all tenants in a multi-tenant resource environment. The disclosed embodiments can also reduce request processing response time when processing requests from different tenants by ensuring that a minimum amount of resources is always available to different requesting tenants, while also preventing unreasonable borrowing by tenants from exhausting the resources required by the system to process requests from different requesting tenants. introduction

[0034] Now we will focus on Figure 1 The diagram illustrates a flowchart or method 100, including various actions (actions 110, 120, and 130) associated with an exemplary method for providing reasonable resource allocation in a multi-tenant environment, which can be implemented by computing system 1110.

[0035] The first illustrated action is provided to identify multiple tenants participating in a multi-tenant environment (Action 110). For each of the multiple tenants, the system determines the tenant's status as either a donating tenant who is donating some or all of a fixed limit of resources guaranteed for that particular tenant, a reasonable borrowing tenant who is borrowing resources at a level equal to or lower than a previously determined borrowing limit, or an unreasonable borrowing tenant who is borrowing resources at a level higher than a previously determined borrowing limit (Action 120). By identifying whether a tenant is a donating tenant, a reasonable borrowing tenant, or an unreasonable borrowing tenant, the system is able to better determine the allocation or reallocation of resources available to each tenant, thereby resulting in (i) improved reasonable allocation among all tenants, and (ii) system optimization to achieve maximum utilization of all available resources in the multi-tenant environment.

[0036] Additionally, tenant status provides the system with an efficient way to establish a baseline by which new resource allocations are determined. According to some disclosed embodiments, it would be unreasonable to allow tenants who were initially allocated only a small amount of resources (i.e., the fixed limit guaranteed to the tenant's resources) but are already borrowing a higher amount of resources than other tenants to borrow even more resources from the common pool, thereby preventing tenants initially allocated less than their fixed resource limit from being allocated at least their fixed resource limit or borrowing additional resources. Therefore, the system can use tenant status to "quickly view" the resource distribution across tenants in a multi-tenant environment at any given time and take into account the different tenants' specifications when determining how resources should be allocated.

[0037] For example, the system is configured to apply different borrowing algorithms (Action 130) to each of multiple tenants based on the corresponding tenant state determined for each tenant. These different borrowing algorithms, or rational algorithms, are configured to determine new borrowing limits from the shared resource pool for each tenant. By applying different borrowing algorithms to tenants based on their tenant state, the system can promote improved maximization of available resources across the entire system (by allocating unused resources as they become available), while still maintaining a rational allocation of resources among different tenants using varying amounts of resources.

[0038] Generally, to ensure the fair allocation of resources, resource borrowing restrictions associated with reclaiming or borrowing from the common pool are least restrictive for donating tenants and most restrictive for unreasonable borrowing tenants. It is worth noting that resource borrowing restrictions for reasonable borrowing tenants are more restrictive than those for donating tenants, but less restrictive than those for unreasonable borrowing tenants.

[0039] A donating tenant is a tenant using less than their initial allocation (i.e., fixed limit) of resources within the resource system, and donating the unused resources to the public resource pool. A reasonable borrowing tenant is a tenant using all of their initial allocation (i.e., fixed limit) of resources within the resource system, and reasonably borrowing additional resources from the public pool. An unreasonable borrowing tenant is a tenant using all of their initial allocation (i.e., fixed limit) of resources within the resource system, but unreasonablely borrowing additional resources from the public pool.

[0040] The public pool can maintain a minimum resource threshold that cannot be borrowed or can be reclaimed by tenants whose resource usage is below its fixed limit. The system may also include reserving additional resources in the public pool for tenants who are borrowing less than their reasonable share of resources.

[0041] Several different methods (described in more detail below) are used to determine whether a tenant is a legitimate or illegitimate borrower. The system is also configured to satisfy or reject reclamation requests and additional borrowing requests based on tenant status and available resources in the common pool. In some instances, the system is configured with different backpressures or triggers to force resource reallocation by different borrowers in order to satisfy resource requests from other borrowers with less stringent borrowing restrictions associated with the common resource pool.

[0042] In some instances, shared resources are represented by tags. For example, when a tenant wants to make a request to a machine learning model (e.g., LLM), the system allocates a number of tags equal to the maximum capacity the system might consume in that request. The allocated tags represent the amount of resources the tenant is expected to consume in processing the LLM request. If the request requires more tags than allowed by the new borrowing limits determined for the tenant, the system will reject the request. If the request requires fewer tags than or equal to the number of tags allowed by the new borrowing limits, the system will accept the request and allocate any additional tags from the public pool to the tenant to process the request.

[0043] However, the label is merely an exemplary representation of the various types of resources that can be allocated using the disclosed technology. In other instances, resources allocated by the disclosed system using the disclosed technology include predetermined amounts of memory or storage, processing cycles, bandwidth, session duration, virtual machines, services, and / or applications. The referenced resources may also include other resources not listed above, but which may be quantified and allocated by the disclosed system to the referenced tenant. Example Method

[0044] Now we will focus on Figure 2The diagram illustrates a flowchart or method 200, including various actions (actions 210, 220, and 230) associated with an exemplary method for providing a reasonable allocation of resources in a multi-tenant environment, which can be implemented by computing system 1110. It is worth noting that... Figure 2 The various actions and system components included will be referenced in this article. Figures 3 to 10 It was described further.

[0045] The first illustrated action is (Action 210) provided to determine a fixed limit (act 210) on the resource capacity guaranteed to be available for each of the multiple tenants associated with a multi-tenant resource environment. Determining a fixed limit on the resource capacity guaranteed for each tenant is beneficial because it allows the system to determine the amount of resources that can be initially allocated to each tenant from the total shared resources available in the multi-tenant environment. It also provides a way to ensure that each tenant is guaranteed a minimum amount of resources at a given time or during a given time frame. This means that each tenant is able to access its initial, reasonable share of resources from the system. For example, Figure 3 A multi-tenant resource environment is illustrated, in which multiple tenants (e.g., tenant A, tenant B, tenant C, etc.) access resource 304 through resource management 302. It should be understood that although resource 304 is shown as a resource comprising thirteen resources or discrete capacities, resource 304 may include any number of resources or resource capacities available to any number of tenants. Resource capacity measurement

[0046] It is worth noting that the resources referred to herein can include any type of resource, including hardware and software resources, or even services. In some instances, resources can be measured in terms of tokens, where each token represents a unit of resource capacity required or allocated to a tenant by the system to process requests for the Large Language Model (LLM), such as requests for model processing prompts and outputs such as responses corresponding to those requests.

[0047] While a request is being processed, the resources being used remain active throughout the request's duration. Resources used to process a request can be released back to the tenant's availability limits or common pool upon request completion. Again, the terms resource and resource capacity should be interpreted broadly to include any desired allocation of computing resources, whether a fixed amount required to process requests and responses or an alternatively flexible amount. Corresponding resources may include power, time, applications, computing power, memory, storage, processors and other hardware components, network bandwidth, and / or any other type of resource that the computing system can utilize while processing a request, or any of the above-mentioned proxy measurements or metrics.

[0048] When a tenant wants to make a request to a machine learning model (e.g., an LLM), the system allocates a number of tags equal to the maximum capacity the system might consume in that request. The allocated tags represent the amount of resources the tenant is expected to consume while processing the LLM request. For example, an LLM request that scores 100 tags with a maximum request response length of 200 tags requests the allocation of resource capacity required to process at least 300 tags. The tenant keeps the allocated resources active for the duration of the request; during this time, the allocated tags are considered part of the tenant's active resource usage. Once the request is completed and the LLM returns a result, the allocated tags are released, thus reducing the tenant's active resource usage.

[0049] It should be understood that tags are only one capacity metric applicable to LLMs, as they are related to the memory utilization of the GPUs running these models, making them a good proxy for estimating tenant memory usage. However, the capacity metrics described herein (e.g., “resources” or “tags”) can be generalized to other types of GPU / hardware, software, and machine learning models. As previously stated, the resources referred to can be any predetermined amount of memory or storage, processing cycles, bandwidth, session duration, virtual machines, services, and / or applications.

[0050] In some instances, the referenced resource includes resource capacity, which refers to units of memory storage, processing cycles, and / or bandwidth required to process a request. In some instances, resource capacity refers to units of computer processing power (e.g., clock speed), such as the minimum processing speed required to process a particular request. In some instances, resource capacity refers to one or more instantiations of a machine learning model that will be used to complete the request. In some instances, the unit of measurement used to describe resource capacity in a multi-tenant shared resource environment will refer to a combination of the aforementioned resources. Fixed limitations of resources

[0051] like Figure 4 As shown, a fixed limit on resources is determined for each tenant. This fixed limit guarantees the amount of resource capacity available to a particular tenant. In some instances, this resource guarantee is associated with a predetermined time slack. In some instances, the predetermined time slack is relative to a resource request made by a tenant (e.g., for two minutes after tenant A makes a request, five marked resources will always be available to tenant A). Figure 4 As shown, the fixed limit 308 for tenant A consists of five tags from resource 304, the fixed limit 312 for tenant B consists of two tags from resource 304, and the fixed limit 316 for tenant C consists of six tags from resource 304. Therefore, each tenant can be initially allocated one or more resources from resource 304 until its corresponding resource fixed limit.

[0052] like Figure 5As shown, tenant A is assigned (i.e., resource allocation 306) all five tags associated with its fixed limit 308, tenant B is assigned (i.e., resource allocation 310) all two tags associated with its fixed limit 312, and tenant C is initially assigned (i.e., resource allocation 314) all six tags associated with its fixed limit 316. Unused resources

[0053] Reference Figure 2 The system also periodically evaluates each tenant to identify unused resources and allocates them to a common pool (Action 220). By periodically (or continuously) monitoring unused resources, the system can maximize resource utilization in a multi-tenant environment by allocating unused resources from one tenant to another that is requesting more resources. In some instances, the system continuously tracks resource usage in real time in response to changes in the use of available resources in a multi-tenant environment and can dynamically adjust resource allocation and borrowing limits. By periodically tracking unused resources, the system can adjust for different time periods, where shorter periods allow for more continuous monitoring but consume more computing power, memory, and processing than longer periods that still allow monitoring but use less computing power, memory, and processing. In some instances, the system collects information about the amount of resources being used by each tenant, such as processing power being consumed, memory being used, storage being occupied, instantiation of machine learning models, status of requests made to machine learning models, or network bandwidth being utilized. The system can then aggregate this data to determine the total active use of resources for each tenant.

[0054] By managing resource allocation based on each tenant's actual usage (rather than on their potential usage as in conventional systems), the system is better able to maximize the utilization of all available resources in a multi-tenant environment. This process ensures that resources are not idle in the system when they can be used by other tenants, thus improving the overall efficiency of the system. Furthermore, by allocating unused resources to a shared pool, the system provides different tenants with a mechanism to access additional resources beyond their predetermined limits when they request additional resources, subject to the availability of resources in the pool and reasonableness standards implemented by the system.

[0055] For example, such as Figure 5As shown, tenant A is using all five tags in its tags (i.e., used tags are indicated by dashed lines in the entire tag box, while unused tags are shown as empty boxes). Tenant B is using all two tags in its tags. Tenant C is not using any tags allocated to it from its fixed limit 316. Here, the system is able to determine that tenant C is not using some or all of the resources initially allocated or guaranteed to it based on fixed limit 316. The system then allocates any unused resources to the common pool (such as...). Figure 6 (As shown). In some instances, tenant C voluntarily donates (e.g., donation 318) all of its unused tokens associated with fixed limit 316 to public pool 320. Initial borrowing from the public pool

[0056] In some instances, a request to borrow resources from the common pool triggers the reallocation of unused resources from one or more tenants to the common pool 320 within a certain amount of time requested. Alternatively, unused resources may be returned after a predetermined amount of time since they were not used.

[0057] like Figure 6 As shown, tenant A issues a request (e.g., request 322) to borrow additional resources (e.g., a tag) from public pool 320. Tenant B also issues a request (e.g., request 324) to borrow additional resources (e.g., a tag) from public pool 320. Although Figure 6 The example shows a request being made for one tag, but it should be understood that a tenant can make a request to borrow any number of tags from the public pool.

[0058] After the additional borrowing request was satisfied, Figure 7 Tenant A is now shown using a total of six tokens, including five within its fixed limit 308 and one borrowed token from the public pool 320 (e.g., borrowed token 326). Tenant A uses a total of three tokens, including two from its fixed limit 312 and one borrowed token (e.g., borrowed token 328). Tenant C has not used any tokens (i.e., because they previously donated all resources corresponding to their fixed limit 316). The public pool 320 now has only four tokens available for borrowing or reclaiming by different tenants. The system will then determine whether each tenant has reasonably or unreasonably borrowed resources from the public pool. Reasonable Algorithm / Tenant Status

[0059] For example, refer to the return Figure 2The system applies a reasonable algorithm to calculate the reasonable borrowing limit for resources allocated to each tenant from the common pool (Action 230). By applying a reasonable algorithm or calculation method to calculate the reasonable borrowing limit for resources for each tenant, the system can strike a balance between maximizing utilization (by monitoring and allocating unused resources) and providing resource availability to different tenants without sacrificing reasonableness. The reasonable borrowing limit for resources is based at least in part on a proportional analysis of the percentage of resources currently borrowed by each tenant from the common pool, relative to the sum of all fixed limits for resources corresponding to multiple tenants, compared to a fixed limit guaranteeing the resource capacity available to each tenant.

[0060] For example, such as Figure 8 As shown, the system keeps track of each tenant's fixed limit 802, tokens being donated (i.e., donation 804), tokens being borrowed (borrowing 806), and the borrowing ratio based on the tokens being borrowed relative to the available tokens in the public pool. The system also determines the borrowing limit 810 for each tenant based on the fixed limit 802 relative to the total available tokens in resource 304.

[0061] Reference Figure 2 The system uses proportional analysis to determine which tenants are reasonably borrowing resources from the common pool (reasonable borrowing tenants) and which are unreasonably borrowing resources from the common pool (Action 240). As described above, by identifying whether a tenant is a donating tenant, a reasonably borrowing tenant, or an unreasonably borrowing tenant, the system can better determine the allocation or reallocation of resources available to each tenant, thereby leading to (i) improved reasonable allocation among all tenants and (ii) system optimization to achieve maximum utilization of all available resources in a multi-tenant environment. Tenant status is useful in providing the system with a “quick look” at the distribution of resources among tenants across the multi-tenant environment at any given time.

[0062] Therefore, as Figure 8 As shown, the system determines tenant status 814 based on a reasonable algorithm (e.g., algorithm 812). In some instances, as described above, the reasonable algorithm is based on comparing the borrowing ratio 808 with the borrowing limit 810. If the borrowing ratio 808 is less than or equal to the borrowing limit 810, the tenant is considered a reasonable borrowing tenant. If the borrowing ratio 808 is greater than the borrowing limit 810, the tenant is considered an unreasonable borrowing tenant. If a tenant does not use any of their fixed limits or uses less than their fixed limits, the tenant is considered a donating tenant.

[0063] Specifically, tenant A's borrowing rate 808 is less than tenant A's borrowing limit (e.g., 17% < 38%), so tenant A's status is reasonable. Tenant B's borrowing rate 808 is greater than tenant B's borrowing limit (e.g., 17% > 15%), so tenant B's status is unreasonable. Tenant C is a donating tenant because they are using less than their fixed limit (in this case, not using their fixed limit).

[0064] After determining the tenant status, the system is configured to determine a new resource borrowing limit, which corresponds to the amount of additional resources that each different tenant can request to borrow (or reclaim if they are a donating tenant) from the common pool 320 based on their respective tenant status. For example, refer to... Figure 2 The system generates a newly calculated borrowing limit for resources borrowed from the common pool by tenants who are borrowing unreasonably (or who are borrowing reasonably or who are donating), and this new calculated borrowing limit is lower than the previously calculated reasonable borrowing limit for the resources (Action 250). By applying different borrowing algorithms to tenants based on their tenant status, the system can promote the improvement of the maximum utilization of available resources throughout the system (by allocating unused resources as they become available), while still maintaining a reasonable allocation of resources among different tenants using different amounts of resources.

[0065] For example, because tenant C is a donating tenant, the system determines that tenant C can reclaim up to its fixed resource limit of 316. Figure 9 As illustrated, all four tokens currently available in the public pool (e.g., public pool 320C) are available for reclamation by tenant C.

[0066] If tenant C requests the reclamation of more tags than currently available in the public pool, the system may force the reallocation of unused resources from different tenants or force the borrowing tenant to return any borrowed tags. In some instances where a forced return is required to satisfy the donation tenant's request, the system will first return the borrowed tags from unreasonable borrowing tenants, and then return the borrowed tags from reasonable borrowing tenants as needed. In some instances, a forced return occurs immediately upon receiving a request from the donation tenant. Alternatively, a forced return occurs within a predetermined time margin associated with the donation tenant's fixed limit 316 (or guaranteed resource capacity). Alternatively, a forced return occurs after the borrowing tenant has generated results using the borrowed tags (i.e., after using the tags to generate outputs from the machine learning model). If a forced return is required, tenant C can begin using any tags already in the public pool to provide the requested total number of tags up to its fixed limit before the forced return occurs.

[0067] It should be understood that many techniques can be implemented for forcibly returning resources. Some of these techniques are resource-dependent. For example, when resources include storage, the system can assign leases to tenants for reserved or allocated storage. These leases can establish control over read and / or write locks on that storage for the tenant. The system can forcibly return the resource by revoking the lease on that storage, thus removing the tenant's control over read and write locks on that storage.

[0068] Additionally or alternatively, the system may rename the namespace of the allocated storage and may provide the new name for the namespace of that storage resource to another tenant of the storage that was allocated, revoked, and returned. This may also include overwriting data or otherwise cleaning up the referenced storage before reassigning it to a new tenant.

[0069] In another embodiment, when the resource includes bandwidth, the system can force the return of the resource by limiting the bandwidth allocated to a tenant whose allocated resource has been revoked. This essentially forces the bandwidth capacity back to the system for reallocation to new tenants. The system may include, for example, a network gateway and / or communicate with a third-party network gateway that manages and meters network flows to different tenants. Tables used by the network gateway to track bandwidth allocations can be updated to reflect changes in bandwidth allocations metered to different tenants, such as when resources are initially allocated and when the forced return of resources is implemented.

[0070] In yet another example, when resources include application or service instances assigned to a tenant, the system can issue a usage license (with the necessary authentication and / or verification credentials) to enable the tenant to utilize the application and service they were assigned. Then, when the resource is forcibly returned, the system can revoke the enabling license necessary for the tenant to utilize the application and service.

[0071] By enabling the system to force resource return in some instances and allowing donating tenants to reclaim resources, the system ensures that resources are not hoarded by tenants who do not currently require fixed-limit resources or who borrow beyond their fixed or borrowing limits. This also allows the system to dynamically expand the available resource pool, making these resources available to other tenants who may request additional resources, such as donating or legitimately borrowing tenants. Forced return also puts back pressure on any tenant using resources beyond their fixed or borrowing limits by ensuring all tenants have legitimate access to resources available in a multi-tenant environment.

[0072] The system also determines new borrowing limits for tenant A (who is a legitimate borrowing tenant). In some instances, the system reserves a portion of the available resources in the public pool (e.g., public pool 320A) for donating tenants (e.g., at least one token, in some instances, such as...). Figure 9 (As shown). Therefore, tenant A is able to borrow three additional tokens from the public pool.

[0073] Borrowing restrictions for tenant B are more restrictive than those for tenant A because they are unreasonable borrowers regarding the common pool. When considering requests from unreasonable borrowers, the system reserves a first portion of the common pool resources for any donating tenant and a second portion for any reasonable borrower (i.e., at least one token is reserved for tenant C as a donating tenant and at least one token for tenant A as a reasonable borrower). In some instances, the system reserves at least one token for each donating or reasonable borrower in a multi-tenant environment. Therefore, as... Figure 9 As shown, tenant B is able to borrow two additional tokens from the four tokens available in public pool 320B.

[0074] By reserving additional resources for tenants who borrow less than their reasonable share, this system configuration helps maintain equitable resource allocation in a multi-tenant environment. It ensures that tenants have access to their reasonable share of resources, preventing tenants with larger pre-defined limits from monopolizing resources, and ensuring that tenants with smaller pre-defined limits have access to the resources they request. This contributes to the overall efficiency and effectiveness of the system in managing shared resources within a multi-tenant environment.

[0075] In some instances, if tenant B requests more than two available tokens, the system will either reject the borrowing request or only satisfy requests up to the new borrowing limit (i.e., two tokens). For example, see reference back. Figure 2 When it is determined that a request from an unreasonable borrowing tenant to borrow additional resources from the common pool would cause the resources allocated from the common pool to the unreasonable borrowing tenant to exceed the newly calculated borrowing limit, the system will reject the request from that unreasonable borrowing tenant (260). By rejecting requests that would exceed the newly calculated borrowing limit, the system ensures that reasonable borrowing tenants and donating tenants will still have access to reclaim or borrow resources, and prevents any single tenant from borrowing all resources.

[0076] By determining the legitimacy of borrowing resources in this way, the system ensures that resources are used effectively and allocated reasonably among tenants. It prevents tenants with large pre-defined restrictions from monopolizing resources, while also ensuring that tenants with smaller pre-defined restrictions have access to the resources they request. This contributes to the overall efficiency and effectiveness of the system in managing shared resources in a multi-tenant environment. Improved and reasonable algorithm

[0077] Some embodiments disclosed herein involve utilizing improved rational algorithms to determine new borrowing limits based on fixed limits corresponding to borrowing tenants (rather than all tenants). For example, in such embodiments, the system applies a rational algorithm to calculate a rational borrowing limit for resources allocated to each tenant for borrowing from the common pool. The rational borrowing limit for resources is based at least in part on a proportional analysis of the percentage of resources currently borrowed by each tenant from the common pool, relative to the sum of all fixed limits for resources corresponding to one or more borrowing tenants, compared to a fixed limit guaranteeing the availability of resource capacity for each tenant.

[0078] For example, such as Figure 10 As shown, the system keeps track of each tenant's fixed limit 1002, the tokens being donated (i.e., donation 1004), the tokens being borrowed (borrowing 1006), and the borrowing ratio based on the borrowing tokens relative to the tokens available in the public pool. The borrowing ratio for tenant A is 1 borrowed token relative to the initial six tokens in public pool 320 (i.e., borrowing ratio = 17%). The borrowing ratio for tenant B is the same as for tenant A because tenant B also borrows 1 / 6 of a token from public pool 320. The borrowing ratio for tenant C is 0% because tenant C has not borrowed any tokens. The system also determines the borrowing limit 1010 for each tenant based on the fixed limit 1002 relative to the total fixed limits of resources corresponding to borrowing tenants. The borrowing limit for tenant A is 29% (e.g., five tokens associated with tenant A's fixed limit relative to seven tokens corresponding to the total fixed limits for borrowing tenants). The total fixed restrictions for borrowing tenants include five tokens from tenant A's fixed restrictions and two tokens from tenant B's fixed restrictions.

[0079] Then, based on the ratio analysis of borrowing ratio and borrowing limit, the system determines which tenants are "reasonably borrowing" resources from the public pool and which are "unreasonably borrowing" resources from the public pool. For example... Figure 10 As shown, based on an improved rational algorithm (e.g., algorithm 1012), the system determines tenant status 1014. In some instances, as described above, the rational algorithm is based on comparing the borrowing ratio 1008 with the borrowing limit 1010. If the borrowing ratio 1008 is less than or equal to the borrowing limit 1010, the tenant is considered a rational borrowing tenant. If the borrowing ratio 1008 is greater than the borrowing limit 1010, the tenant is considered an unreasonable borrowing tenant. If the tenant does not use any of its fixed limits or uses less than its fixed limits, the tenant is still considered a donating tenant.

[0080] According to Algorithm 1012, tenant A's status is justified because its borrowing ratio 1008 is less than its borrowing limit (e.g., 17% < 38%). Tenant B's status is also justified because its borrowing ratio 808 is also less than its new borrowing limit (e.g., 17% > 29%). Tenant C is a donating tenant because it uses less than its fixed limit (in this case, not using its fixed limit).

[0081] Each tenant can then borrow additional resources from the public pool based on a new borrowing limit determined by applying a reasonable algorithm to each tenant for different tenant states. Therefore, according to Algorithm 1012, since both tenant A and tenant B are reasonable borrowing tenants, either tenant A or tenant B can borrow three additional tokens from the four tokens available in public pool 320. Some resources (e.g., at least one token in some instances) will still be reserved for tenant C as a donating tenant. Example computing system

[0082] Now we will focus on Figure 11 The illustration depicts a computing system 1110 as part of a computing environment 1100, which includes multiple client systems 1120 and multiple third-party systems 1130 communicating with the computing system 1110 (via network 1140). As illustrated, the computing system 1110 is a server computing system configured to compile, modify, and implement one or more machine learning models, which are configured to provide appropriate resource allocation in a multi-tenant resource environment.

[0083] For example, computing system 1110 includes one or more processors (such as one or more hardware processors) and one or more hardware storage devices storing computer-readable instructions. One or more of the hardware storage devices are capable of accommodating any number of data types and / or any number of computer-executable instructions. When the computer-executable instructions are executed by the one or more hardware processors, computing system 1110 is configured to implement one or more aspects of the disclosed embodiments. The hardware storage devices are configured to include and / or access different resources that can be shared and allocated among different tenants for satisfying model requests using machine learning models (e.g., LLM).

[0084] The computing system 1110 is also shown to include (multiple) user interfaces and input / output (I / O) devices (such as audio inputs like microphones and other audio input devices, and audio outputs like speakers and other audio output devices).

[0085] like Figure 11As shown, the (multiple) hardware storage devices are represented as a single storage unit. However, it should be understood that the (multiple) hardware storage devices can be distributed storage devices, distributed across several separate and sometimes remote systems and / or (multiple) third-party systems. The computing system 1110 may also include distributed systems, wherein one or more components of the computing system 1110 are maintained / operated by different discrete systems that are remote from each other and each perform different tasks. In some instances, multiple distributed systems perform similar and / or shared tasks to implement the disclosed functionality, such as in a distributed cloud environment.

[0086] The computing system communicates with (multiple) client systems 1120, which include one or more processors, one or more user interfaces, one or more I / O devices, one or more computer-executable instruction sets, and one or more hardware storage devices.

[0087] The computing system also communicates with (multiple) third-party systems. In some instances, it is anticipated that (multiple) third-party systems 1130 will also include databases containing additional resources, such as resources not included in local storage. Additionally or alternatively, (multiple) third-party systems 1130 may include machine learning systems external to the computing system 1110.

[0088] Embodiments of the present invention may include or utilize a dedicated or general-purpose computer (e.g., computing system 1110), which includes computer hardware, as discussed in more detail below. Embodiments within the scope of the invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or dedicated computer system. Computer-readable media storing computer-executable / computer-readable instructions (e.g., Figure 11 The (multiple) hardware storage devices are physical hardware storage media / devices excluding transmission media. A computer-readable medium carrying computer-executable or computer-readable instructions in one or more carrier waves or signals is a transmission medium. Therefore, by way of example and not limitation, embodiments of the invention may include at least two distinctly different types of computer-readable media: physical computer-readable storage media / devices and transmission computer-readable media.

[0089] A physical computer-readable storage medium / device is hardware, including RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices (such as CDs, DVDs, etc.), magnetic disk storage devices or other magnetic storage devices, or any other hardware that can be used to store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer.

[0090] "Network" (e.g., Figure 11 A network (1140) is defined as one or more data links that enable the transmission of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer via a network or another communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately considers that connection as a transmission medium. The transmission medium may include a network and / or data link, which may be used to carry or be intended program code in the form of computer-executable instructions or data structures, and which may be accessible by a general-purpose or special-purpose computer. Combinations of the above are also included within the scope of computer-readable media.

[0091] Furthermore, upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from a transmission computer-readable medium to a physical computer-readable storage medium (and vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to the computer system RAM and / or to a less volatile computer-readable physical storage medium at the computer system. Therefore, computer-readable physical storage media can be included in computer system components that also (or even primarily) utilize the transmission medium.

[0092] Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or group of functions. Computer-executable instructions can be, for example, binary, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features and actions are disclosed as exemplary forms of implementing the claims.

[0093] Those skilled in the art will understand that this invention can be practiced in networked computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, pagers, routers, switches, etc. This invention can also be practiced in distributed system environments, where both local and remote computer systems linked via a network (via hardwired data links, wireless data links, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can reside on both local and remote memory storage devices.

[0094] Additionally or alternatively, the functions described herein may be performed at least in part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. Aspects of the disclosed embodiments

[0095] This invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments should be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than the foregoing description. All variations falling within the meaning and scope of equivalents of the claims should be included within their scope.

[0096] Various aspects of this subject are set forth below to review and / or supplement the embodiments described to date, with emphasis placed here on the interrelationships and interchangeability of the following embodiments in relation to the systems or methods disclosed herein. In other words, it is emphasized that each feature in the embodiments may be combined with each of the other features unless explicitly stated or logically impractical.

[0097] In some aspects, the technology described herein relates to a method for managing shared resources in a multi-tenant environment, the method comprising: for each of a plurality of tenants, determining a fixed limit on the resource capacity guaranteed to be available to each tenant; periodically evaluating each tenant to identify unused resources and allocating the unused resources to a common pool; calculating a reasonable borrowing limit on the resources allocated to each tenant for borrowing from the common pool, wherein the reasonable borrowing limit on the resources is based at least in part on: a proportional analysis of the percentage of resources currently borrowed by each tenant from the common pool relative to the sum of all fixed limits on resources corresponding to the plurality of tenants, compared to the fixed limit on the resource capacity guaranteed to be available to each tenant; and based on this proportional analysis, determining the percentage of resources currently borrowed by each tenant from the common pool among the plurality of tenants. The system identifies which tenants are legitimate borrowers and which of the multiple tenants are illegitimate borrowers. Legitimate borrowers are legitimately borrowing resources from the pool at a rate equal to or lower than the calculated legitimate borrowing limit, while illegitimate borrowers are illegitimately borrowing resources from the pool at a rate higher than the calculated legitimate borrowing limit. A new calculated borrowing limit is generated for the resources allocated to illegitimate borrowers from the pool, which is lower than the previously calculated legitimate borrowing limit. The system also rejects requests from illegitimate borrowers to borrow additional resources from the pool if it is determined that such a request would cause the allocation of resources from the pool to the illegitimate borrower to exceed the new calculated borrowing limit.

[0098] In some respects, the technology described in this article relates to a method that also includes retaining a first minimum number of donating tenants.

[0099] In some respects, the techniques described herein involve a method that also includes reserving a second minimum amount of resources in a public pool for reasonable borrowing tenants to borrow.

[0100] In some respects, the technique described herein relates to a method in which a reasonable algorithm is applied to calculate a reasonable borrowing limit for resources allocated to each tenant from a common pool, based at least in part on: a different percentage analysis of the percentage of resources currently borrowed by each tenant from the common pool relative to the sum of fixed limits for resources corresponding to one or more borrowing tenants among a plurality of tenants, compared to a different percentage of fixed limits guaranteeing the availability of resource capacity for each tenant.

[0101] In some respects, the technology described herein relates to a method that further includes: receiving a request from a donating tenant to reclaim resources from a public pool; determining that the public pool does not have sufficient resources to satisfy the request from the donating tenant; when it is determined that the public pool does not have sufficient resources to satisfy the request from the donating tenant, forcing the return of resources from one or more borrowing tenants to the public pool; and satisfying the request from the donating tenant by allocating resources from the public pool to the donating tenant.

[0102] In some respects, the technology described herein relates to a method in which a donating tenant can request the reclamation of resources from a public pool up to a fixed limit corresponding to the donating tenant, in order to trigger the forced return of resources from one or more borrowing tenants.

[0103] In some aspects, the technology described herein relates to a computing system for managing shared resources in a multi-tenant environment, the computing system comprising: a processor; and a hardware storage device storing computer-executable instructions, which can be executed by the processor to enable the computing system to: determine, for each of a plurality of tenants, a fixed limit guaranteeing the availability of resource capacity for each tenant; periodically evaluate each tenant to identify unused resources and allocate the unused resources to a common pool; and calculate a reasonable borrowing limit for resources allocated to each tenant from the common pool, wherein the reasonable borrowing limit for resources is based at least in part on: the percentage of the fixed limit guaranteeing the availability of resource capacity for each tenant relative to the sum of all fixed limits for resources corresponding to the plurality of tenants, calculated by each tenant currently borrowing from the common pool. A percentage analysis of the borrowed resources is performed. Based on this percentage analysis, it is determined which tenants among multiple tenants are reasonable borrowers and which are unreasonable borrowers. Reasonable borrowers are reasonably borrowing resources from the public pool at a rate equal to or lower than the calculated reasonable borrowing limit, while unreasonable borrowers are unreasonably borrowing resources from the public pool at a rate higher than the calculated reasonable borrowing limit. A new calculated borrowing limit is generated for the resources allocated to unreasonable borrowers from the public pool, which is lower than the previously calculated reasonable borrowing limit. And when it is determined that a request from an unreasonable borrower to borrow additional resources from the public pool would cause the resources allocated to the unreasonable borrower from the public pool to the unreasonable borrower to exceed the new calculated borrowing limit, the request from the unreasonable borrower is rejected.

[0104] In some respects, the technology described herein relates to a computing system in which the computing system is also configured to retain a first minimum number of donated tenants.

[0105] In some respects, the technology described herein relates to a computing system in which the computing system is also configured to retain a second minimum amount of resources in a public pool for reasonable borrowing tenants to borrow.

[0106] In some respects, the technique described herein relates to a computational system in which a reasonable algorithm is applied to calculate a reasonable borrowing limit for resources allocated to each tenant from a common pool, based at least in part on: a different percentage analysis of the percentage of resources currently borrowed by each tenant from the common pool relative to the sum of fixed resource limits corresponding to one or more borrowing tenants among a plurality of tenants, compared to a different percentage of fixed limits guaranteeing the availability of resource capacity for each tenant.

[0107] In some respects, the technology described herein relates to a computing system, wherein the computing system is further configured to: receive a request from a legitimate borrowing tenant to borrow additional resources from a public pool; determine that the public pool does not have sufficient resources to satisfy the request from the legitimate borrowing tenant; force the return of resources from an illegitimate borrowing tenant to the public pool; and satisfy the request from the legitimate borrowing tenant to borrow additional resources from the public pool.

[0108] In some respects, the techniques described herein relate to a computing system, wherein the computing system is further configured to: identify a result returned by a machine learning model based on the use of a set of resources being used by an unreasonable borrowing tenant associated with the machine learning model; and, after identifying the returned result, force a portion of the unused set of resources allocated to the unreasonable borrowing tenant to be returned to a public pool, such that the tenant who had been unreasonable in borrowing no longer uses resources exceeding the reasonable borrowing limits for that tenant.

[0109] In some respects, the technology described herein relates to a method for managing shared resources in a multi-tenant environment, the method comprising: identifying multiple tenants participating in the multi-tenant environment; determining, for each of the multiple tenants, a tenant status as a donating tenant, a legitimate borrowing tenant, or an unreasonable borrowing tenant; and applying different borrowing algorithms to each of the multiple tenants based on the corresponding tenant status determined for each tenant, wherein the different borrowing algorithms are configured to determine different resource borrowing limits from a common resource pool for each tenant.

[0110] In some respects, the techniques described herein relate to a method that also includes determining fixed limits on resources guaranteed to be available to each of the multiple tenants before determining the tenant's status.

[0111] In some respects, the technology described herein relates to a method in which a tenant's status is determined to be a donating tenant when a particular tenant among a plurality of tenants is donating some or all of the resources guaranteed to be available to that particular tenant in a public pool.

[0112] In some respects, the techniques described herein relate to a method in which a tenant's status is determined at least in part based on the percentage of resources currently borrowed by a particular tenant from a common pool, relative to the sum of all fixed limits on resources corresponding to multiple tenants, compared to a fixed limit guaranteeing the availability of resources for that particular tenant.

[0113] In some respects, the technology described herein relates to a method in which a tenant's status is determined at least in part based on the percentage of resources currently borrowed by the particular tenant from the common pool relative to the sum of all fixed limits on resources corresponding to one or more borrowing tenants, compared to a fixed limit guaranteeing the availability of resources for that particular tenant.

[0114] In some respects, the technology described herein relates to a method in which a first resource borrowing limit corresponding to an unreasonable borrowing tenant is less than a second resource borrowing limit corresponding to a reasonable borrowing tenant.

[0115] In some respects, the technology described herein relates to a method in which the third resource borrowing limit corresponding to the donating tenant is greater than the second resource borrowing limit.

[0116] In some respects, the technology described herein relates to a method that further includes rejecting a request from an unreasonable borrowing tenant to borrow additional resources from a common pool when it is determined that such a request would cause the amount of resources available in the common pool to fall below a minimum threshold.

[0117] It should be noted that all features, elements, components, functions, and steps described with respect to any embodiments provided herein are intended to be freely combined and substituted with those embodiments in any other embodiment. If a feature, element, function, or step is described with respect to only one embodiment, it should be understood that each feature, element, function, or step can be used in conjunction with any other embodiment described herein.

Claims

1. A method for managing shared resources in a multi-tenant environment, the method comprising: For each of the multiple tenants, determine a fixed limit on the resource capacity that is guaranteed to be available to each tenant; Each tenant is periodically evaluated to identify unused resources, and these unused resources are allocated to the public pool. Calculate a reasonable borrowing limit for resources allocated to each tenant from the public pool, wherein the reasonable borrowing limit for resources is based at least in part on: a percentage analysis of the percentage of resources currently borrowed by each tenant from the public pool relative to the sum of all fixed limits for resources corresponding to the plurality of tenants, compared to the percentage of the fixed limits that guarantee the resource capacity available to each tenant; Based on the ratio analysis, it is determined which tenants among the multiple tenants are reasonable borrowing tenants and which tenants among the multiple tenants are unreasonable borrowing tenants. The reasonable borrowing tenants are reasonably borrowing resources from the public pool at a level equal to or lower than the calculated reasonable borrowing limit of resources, and the unreasonable borrowing tenants are unreasonably borrowing resources from the public pool at a level higher than the calculated reasonable borrowing limit of resources. Generate a new calculated borrowing limit for resources allocated to tenants who are improperly borrowing from the public pool, the new calculated borrowing limit being lower than the previously calculated reasonable borrowing limit for the resources; and If it is determined that a request from an unreasonable borrowing tenant to borrow additional resources from the public pool would cause the resources allocated from the public pool to the unreasonable borrowing tenant to exceed the newly calculated borrowing limit, the request from the unreasonable borrowing tenant is rejected.

2. The method according to claim 1, further comprising: A first minimum amount of resources to be reclaimed by donating tenants is reserved in the public pool.

3. The method according to claim 1, further comprising: A second minimum amount of resources is reserved in the public pool for borrowing by a reasonable borrowing tenant.

4. The method of claim 1, wherein the reasonable borrowing limit for resources allocated to each tenant from the public pool is calculated at least in part based on: a different percentage analysis of the percentage of resources currently borrowed by each tenant from the public pool relative to the sum of fixed limits for resources corresponding to one or more borrowing tenants among the plurality of tenants, compared to different percentages of the fixed limits that guarantee the resource capacity available to each tenant.

5. The method according to claim 1, further comprising: Receive requests from donating tenants to reclaim resources from the public pool; It was determined that the public pool did not have sufficient resources to satisfy the request from the donating tenant; If it is determined that the public pool does not have sufficient resources to satisfy the request from the donating tenant, resources shall be forcibly returned to the public pool from one or more borrowing tenants; as well as The request from the donating tenant is fulfilled by allocating resources from the public pool to the donating tenant.

6. The method of claim 5, wherein the donating tenant requests to reclaim resources from the public pool up to a fixed limit corresponding to the donating tenant, in order to trigger a forced return of resources from the one or more borrowing tenants.

7. A computing system for managing shared resources in a multi-tenant environment, the computing system comprising: processor; as well as Hardware storage device, the hardware storage device storing computer-executable instructions, the computer-executable instructions being executed by the processor to enable the computing system to: For each of the multiple tenants, determine a fixed limit on the resource capacity that is guaranteed to be available to each tenant; Each tenant is periodically evaluated to identify unused resources, and these unused resources are allocated to the public pool. Calculate a reasonable borrowing limit for resources allocated to each tenant from the public pool, wherein the reasonable borrowing limit for resources is based at least in part on: a percentage analysis of the percentage of resources currently borrowed by each tenant from the public pool relative to the sum of all fixed limits for resources corresponding to the plurality of tenants, compared to the percentage of the fixed limits that guarantee the resource capacity available to each tenant; Based on the ratio analysis, it is determined which tenants among the multiple tenants are reasonable borrowing tenants and which tenants among the multiple tenants are unreasonable borrowing tenants. The reasonable borrowing tenants are reasonably borrowing resources from the public pool at a rate equal to or lower than the calculated reasonable borrowing limit, and the unreasonable borrowing tenants are unreasonable borrowing resources from the public pool at a rate higher than the calculated reasonable borrowing limit. Generate a new, calculated borrowing limit for resources allocated to tenants who are improperly borrowing from the public pool, the new calculated borrowing limit being lower than the previously calculated reasonable borrowing limit; and If it is determined that a request from an unreasonable borrowing tenant to borrow additional resources from the public pool would cause the resources allocated from the public pool to the unreasonable borrowing tenant to exceed the newly calculated borrowing limit, the request from the unreasonable borrowing tenant is rejected.

8. The computing system of claim 7, further comprising: A first minimum amount of resources to be reclaimed by donating tenants is reserved in the public pool.

9. The computing system of claim 7, further comprising: A second minimum amount of resources is reserved in the public pool for borrowing by a reasonable borrowing tenant.

10. The computing system of claim 7, wherein the reasonable borrowing limit for resources allocated to each tenant from the public pool is calculated at least in part based on: a different percentage analysis of the percentage of resources currently borrowed by each tenant from the public pool relative to the sum of fixed limits for resources corresponding to one or more borrowing tenants among the plurality of tenants, compared to different percentages of the fixed limits that guarantee the resource capacity available to each tenant.

11. The computing system of claim 7, wherein the computing system further comprises: Receive requests from legitimate borrowing tenants to borrow additional resources from the public pool; It was determined that the public pool did not have sufficient resources to satisfy the request from the legitimate borrowing tenant; Force the return of resources from unreasonably borrowed tenants to the public pool; and The request from the legitimate borrowing tenant to borrow additional resources from the public pool is satisfied.

12. The computing system of claim 7, further comprising: The identifier is the result returned by the machine learning model based on the use of a set of resources allocated to unreasonable borrowing tenants associated with the machine learning model; and After the result is returned with the identifier, a portion of the resource set allocated to the unreasonably borrowing tenant is forcibly returned to the public pool, so that the tenant who was previously unreasonably borrowed no longer uses resources that exceed the reasonable borrowing limit for the tenant's resources.

13. A method for managing shared resources in a multi-tenant environment, the method comprising: Identify the multiple tenants participating in the multi-tenant environment; For each of the plurality of tenants, the tenant status is determined as follows: a donating tenant who is donating some or all of the fixed limit of resources guaranteed to be available to the particular tenant; a reasonable borrowing tenant who is borrowing resources at a level equal to or lower than the previously calculated reasonable borrowing limit; or an unreasonable borrowing tenant who is borrowing resources at a level higher than the previously calculated reasonable borrowing limit. as well as Based on the corresponding tenant state determined for each tenant, a different borrowing algorithm is applied to each of the multiple tenants, wherein the different borrowing algorithms are configured to determine new borrowing restrictions associated with borrowing shared resources from the common pool in the multi-tenant environment.

14. The method of claim 13, wherein the shared resource in the multi-tenant environment is represented by a tag.

15. The method of claim 14, further comprising: Receive requests from specific tenants that need to be processed by the large language model; Determine the maximum capacity of the system associated with the multi-tenant environment that will be needed to process the request; The allocation flag used to satisfy the maximum capacity required to process the request is determined not to exceed the new borrowing limit determined for the specific tenant; as well as Multiple tokens are assigned to the specific tenant for processing the request, wherein the assigned tokens represent the amount of resources from the shared resource that the specific tenant is expected to consume in processing the request.

Images