Load testing and performance benchmarking for large language models using cloud computing platforms

Abstract

The techniques disclosed herein enable a system to perform repeatable and iterative load testing and performance benchmarking for artificial intelligence models deployed in a cloud computing environment. This is accomplished by utilizing load profiles and representative workloads generated based on the load profiles to evaluate artificial intelligence models in various workload contexts. The representative workload is then executed by the artificial intelligence model using the available computational infrastructure. Performance metrics are extracted from the execution and analyzed to provide insight into various performance dynamics, such as relationships between latency and data throughput. In addition, dynamically adjusting load profiles and input data sets to evaluate different scenarios and cases enables the system to automatically test artificial intelligence models across different applications. Furthermore, by comparing various iterations of the artificial intelligence model, a quality gate can be constructed to enhance a consistent and high-quality user experience.

Description

Background Art

[0001] Recent years have seen a rapid growth in the power and complexity of artificial intelligence (AI) software applications. Therefore, the computing requirements for implementing advanced AI applications have also grown significantly. For example, transformer-based AI models have seen widespread adoption due to their different processing capabilities in vision, speech, language, and decision making. Unlike other AI models such as recursive neural networks and long short-term memory (LSTM) models, transformer-based AI models use local self-attention mechanisms to identify fuzzy contexts and even synthesize new content (e.g., images, music). In some instances, some transformer-based models (also referred to as large language models) can include millions (if not billions) of individual parameters. In order to meet this demand, many organizations that provide large-scale computing infrastructure such as cloud computing provide AI platforms that are customized to enable users to train and deploy large language models of the prior art.

[0002] Within the AI ​​platform, users can dynamically scale computing resources to suit their needs. For example, users can scale resources vertically by selecting more or less powerful computing instances (e.g., virtual machines). Similarly, users can scale resources horizontally by selecting the number of computing instances to deploy. From the user's perspective, there are unlimited computing resources available for them to use to meet any needs. However, in reality, the set of available computing resources is limited and shared among multiple users.

[0003] In order to provide a consistent and high-quality user experience, AI platform providers must have accurate measurements of the performance and capabilities of the computing infrastructure applicable to the different workload contexts of the model environment. In addition, service providers must also gain detailed visibility into performance-to-cost dynamics in order to operate the AI ​​platform in a cost-effective manner. Without such insights, users may experience unacceptable performance degradation, which may result in both technical and financial negative impacts on the computing infrastructure as the provider spends time and engineering resources reacting to workload conditions. Summary of the invention

[0004] The technology described in this article provides a platform system for enhancing cloud computing platforms by introducing load testing and performance benchmarking for artificial intelligence models. This is achieved by utilizing load profiles representing various workload contexts to generate representative workloads. In various examples, in response to increasing computing requirements, representative workloads evaluate various performance characteristics of artificial intelligence models for cloud computing platforms, such as latency and data throughput. In addition, representative workload execution can be configured to test different workload parameters, including vector output versus marked output, streaming output versus non-streaming output, cache size, and the like.

[0005] As described above, models based on large language transformers have seen widespread adoption in different application spaces such as summarization, text completion, classification, content synthesis, etc. As such, large language models can be deployed in any number of configurations, depending on the intended use case. For example, large language models can be variably configured for throughput, number of inputs and / or outputs (e.g., tokens), and / or workload scenarios. Therefore, accurately benchmarking every possible configuration of a large language model is overly complex. Furthermore, large language models are often deployed as active resources. That is, the amount of available computing resources can grow or shrink rapidly based on demand.

[0006] To this end, the present disclosure addresses several technical challenges associated with using cloud computing platforms to deploy large language models. For example, utilizing load profiles to generate representative workloads enables the disclosed system to perform consistent and repeatable artificial intelligence model testing. In contrast, many existing solutions lack such a framework for evaluating the performance of artificial intelligence models. In addition, collecting clear insights into the performance impact of different factors such as computing requirements, hardware infrastructure, and model configuration enables service providers to guarantee user-specific performance levels. In addition, performance testing enables service providers to gain visibility into cost and performance dynamics, allowing service providers to operate cloud computing platforms in a cost-effective manner.

[0007] In another example of the technical benefits of the present disclosure, the performance testing system as described herein also enables the cloud computing platform to construct a quality gate for comparing performance measurements across various iterations of the artificial intelligence model. In this way, the quality gate enables the system to reveal performance trends across iterations and anticipate the performance impact of future updates and workloads. In addition, the quality gate enables the system to use regression metrics to establish deployment standards to ensure an ideal operating window for the cloud computing platform. Therefore, the quality gate prevents interruptions to the user experience and degradation of performance over time.

[0008] Features and technical benefits other than those explicitly described above will be apparent from a reading of the following specific embodiments and a review of the associated drawings. This summary is provided to introduce a selection of concepts in a simplified form, which will be further described in the following specific embodiments. This summary is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. For example, the term "technology" may refer to (multiple) systems, (multiple) methods, computer-readable instructions, (multiple) modules, algorithms, hardware logic and / or (multiple) operations as allowed by the above context and throughout the entire document. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items. References to individual items of a plurality of items may use reference numbers with letters in the alphabetical sequence to refer to each individual item. General references to items may use specific reference numbers without the alphabetical sequence.

[0010] Figure 1 is a block diagram of a system for performing load testing and performance benchmarking for artificial intelligence models using a cloud computing platform.

[0011] Figure 2A A first example of a representative workload generated from a load profile representing a first context is illustrated.

[0012] Figure 2B The diagram shows Figure 2A The computational demands induced by the first example of a representative workload captured in .

[0013] Figure 3A A second example of a representative workload generated from a load profile representing a second context is illustrated.

[0014] Figure 3B The diagram shows Figure 3A The computational demands induced by the second example of a representative workload captured in .

[0015] Figure 4A A first example of performance measurements resulting from the execution of a representative workload is illustrated.

[0016] Figure 4B Graphical representation of AI model performance versus operating costs.

[0017] Figure 4C Illustration showing a comparison of performance measures between various iterations of an AI model.

[0018] Figure 5 is a flow chart illustrating aspects of a routine for performing load testing and performance benchmarking against artificial intelligence models using a cloud computing platform.

[0019] Figure 6 is a computer architecture diagram illustrating an illustrative computer hardware and software architecture for a computing system capable of implementing aspects of the techniques and methods presented herein.

[0020] Figure 7 is a drawing that illustrates a distributed computing environment in which aspects of the techniques and methods presented herein can be implemented. DETAILED DESCRIPTION

[0021] The technology described herein enhances the operation of a cloud computing platform for deploying artificial intelligence models. Although the examples discussed below are particularly relevant to artificial intelligence models based on large language transformers (also referred to herein as large language models), it should be understood that the disclosed system can be used in conjunction with any type of implementation of artificial intelligence (AI) such as neural networks. As described above, as the capabilities and complexity of these models grow, service providers must deal with similar complex dynamics around performance to ensure a high quality and consistent user experience. For example, modern large language models typically include millions and even billions of parameters. In this way, deploying and utilizing large language models at cloud computing platforms can represent a major investment in computing resources. Therefore, it is important to understand the performance impact of large language models.

[0022] However, due to the number of possible permutations, load testing and performance benchmarking of large language models using traditional methods is time consuming and impractical. Therefore, a system is needed to automatically evaluate the different ranges of workload contexts that cloud computing platforms may encounter. This is achieved by using load profiles that define parameters representing various workload contexts. In various examples, the load profile is selected from a predetermined set of load profiles that are part of the setup layer. The setup layer is responsible for initializing the necessary components for testing artificial intelligence models, such as input data, model configuration, etc. Alternatively, load profiles can be generated in a customized manner to test specific workload contexts.

[0023] The selected load profile is then provided to the load generation layer, which generates a representative workload according to the parameters defined by the load profile. In one example, the load profile defines the number of containers that include the representative workload and the number of threads for each container. In this way, the representative workload evaluates the ability of the cloud computing platform to accommodate the load. In addition, the load generation layer can utilize the computing infrastructure of the cloud computing platform to generate the load. For example, the representative workload can be generated by a Kubernetes cluster.

[0024] The representative workload is then executed by the artificial intelligence model at the model deployment layer. In various examples, the model deployment layer is the computing hardware of the cloud computing platform including such as a graphics processing unit (GPU), a central processing unit (CPU), memory, etc. The model deployment layer also includes various infrastructures for supporting workload execution such as a load balancer, and user access points also referred to as endpoints for receiving user input.

[0025] As the workload is executed by the artificial intelligence model, the analysis layer extracts various performance metrics from the model deployment layer. Alternatively, the analysis layer extracts the performance metrics after the artificial intelligence model completes executing the representative workload (e.g., via a log file). In one example, in response to the computational demands imposed by the representative workload, the analysis layer is configured to track the latency of the artificial intelligence model. In another instance, in response to the incremental demand, the analysis layer is configured to monitor data throughput. In addition, the analysis layer can track multiple performance metrics (e.g., latency and data throughput) simultaneously.

[0026] The analysis layer then synthesizes performance measurements for the artificial intelligence model based on the performance metrics. As discussed below, the performance measurements can be represented as a curve graph (e.g., latency versus demand). When multiple test runs are completed, the resulting performance measurements can be used to compare various iterations of the artificial intelligence model to provide insights into performance trends over time. For example, a service provider may deploy a new iteration of the artificial intelligence model every month. To prevent performance loss, the provider performs the load testing and performance benchmarking discussed herein and collects performance measurements for the new iteration of the artificial intelligence model. In this way, the new iteration can be objectively compared to the previous iteration of the artificial intelligence model.

[0027] In addition, the performance measurements are provided to the feedback layer for updating in preparation for another test run of the artificial intelligence model. Thus, the feedback layer modifies input parameters or otherwise perturbs the selected load profile based on the performance measurements. In this way, the feedback layer generates an updated load profile using the initially selected load profile and the resulting performance measurements.

[0028] In one example, an initially selected load profile is configured to test the maximum computational demand that the artificial intelligence model can accommodate. After executing the representative workload, performance measurements show that the artificial intelligence model did not experience a significant increase in latency, indicating that the load profile did not adequately stress the artificial intelligence model. In response, the feedback layer modifies the load profile to generate an updated representative workload. The updated representative workload places incremental computational demands on the artificial intelligence model to find an upper limit on the level of latency that the artificial intelligence model experiences. In this way, the feedback layer enables the system to iteratively test different workload contexts by selecting load profiles and different permutations of a given workload context.

[0029] Below about Figures 1 to 7 Describes various examples, scenarios, and aspects of implementing load testing and performance benchmarking of AI models.

[0030] Figure 1A system 100 for implementing load testing and / or performance benchmarking for artificial intelligence models in a cloud computing platform is illustrated. Figure 1 As shown, the setup layer 102 is responsible for initializing the load testing environment by selecting a model configuration 104, an input data set 106, and a load profile 108. In various examples, the model configuration 104 defines various parameters of the artificial intelligence model to be deployed. For example, the model configuration 104 defines the output type, cache size, amount of available computing resources, etc. In addition, the model configuration 104 can also define the geographic area in which the artificial intelligence model is deployed, because the location of the user relative to the cloud computing center can affect the latency.

[0031] The input data set 106 is the data that the artificial intelligence model will process as part of a load test and / or performance benchmark. In various examples, the input data set 106 includes a synthetic benchmark data set, which is a large, publicly available data set that implements consistent and repeatable testing. In this way, the synthetic benchmark data set allows the system 100 to fairly compare performance between different iterations of the artificial intelligence model and regression testing. The input data set 106 may also include data from a live production environment pointing to the test environment. Including real-time data enables the system 100 to evaluate the artificial intelligence model under realistic conditions (e.g., external users). In addition, portions of the live production data can be specially prepared for the test environment. In other words, the input data set 106 can be generated by configuring a portion of the live production data that exhibits a specific traffic pattern with a time series component. This enables the system 100 to evaluate different iterations of the artificial intelligence model using a consistent data set representing actual user activity.

[0032] Then, the setup layer 102 selects a load profile 108. As described above, the load profile 108 defines various parameters of a representative workload including a set of containers, which includes a set of threads. In addition, the load profile 108 also defines the concurrency level of the representative workload. In other words, the load profile 108 defines the number of containers and threads that are active at a given time, which translates into computing demand. Therefore, different load profiles can be selected to represent different workload contexts. The characteristics of the workload context are various factors, such as workload type (e.g., text completion, classification, content generation), traffic patterns, peak computing demand, etc. For example, a first load profile defines a significant surge in computing demand for a context in which a large number of users submit workloads simultaneously. In contrast, a second load profile defines a smooth increase and subsequent decrease in computing demand for a context in which the number of users accessing the cloud computing platform naturally grows and shrinks over the course of a day.

[0033] After the initialization process of the setup layer 102, the load generation layer 110 utilizes the set of load generation agents 112 to generate a representative workload 114 according to the parameters defined by the load profile 108. In various examples, the generation of the representative workload 114 begins in response to a load trigger 116. The load trigger 116 can be a set of conditions that must be met before load generation can begin. For example, the load generation agents 112 can be instances of virtual machines that are each assigned a set of computing resources. Therefore, the load trigger 116 is configured to wait until a threshold amount of computing resources are available before starting to generate the representative workload 114.

[0034] As described in further detail below, the representative workload 114 includes a set of containers to simulate computing demands from external users (e.g., customers) arriving at the cloud computing platform. A container is a dedicated software package containing application code, libraries, and configuration components for enabling an application to execute in an independent environment. In addition, each container itself may include several software threads. The number of containers and threads active at a given time is defined by a load profile. In this way, concurrency (i.e., computing demands) is well defined and enables the system 100 to perform consistent and repeatable load testing and / or performance benchmarking.

[0035] Once the representative workload 114 is generated by the load generation agent 112, the representative workload 114 is dispatched to the model deployment layer 118 for execution. As shown, the representative workload 114 is received by the model endpoint 120. In various examples, the model endpoint 120 is a component that enables a user to interact with the cloud computing platform. For example, the representative workload 114 is formatted as a hypertext transfer protocol (HTTP) request, which is sent to the model endpoint 120 over the network. The request is then analyzed and verified before being routed to the load balancer 122 within the model deployment layer 118 for dispatching to the computing resource 124. In addition, the model endpoint 120 can be configured to scale in response to the amount of incoming requests. In addition, the model endpoint 120 also handles the communication between the model deployment layer 118 and the request source (e.g., user).

[0036] The computing resources 124 are any computing hardware that facilitates the execution of the representative workloads 114 via a graphics processing unit (GPU), a central processing unit (CPU), memory, storage, etc. As described above, the model endpoint 120 is composed of an artificial intelligence model 126. In various examples, the artificial intelligence model 126 can be a large language transformer-based model deployed via the model configuration 104. This utilizes the computing resources 124 to process the representative workloads 114. In other words, some or all of the representative workloads 114 are scheduled to the computing resources 124 for execution by the artificial intelligence model 126 available at the model endpoint 120.

[0037] During execution of the representative workload 114 by the artificial intelligence model 126 deployed in the model deployment layer 118 and / or after execution of the representative workload 114 by the computing resources 124, various performance metrics 128 are extracted from the artificial intelligence model 126 and / or the computing resources 124. The performance metrics 128 include service logs, outputs from the artificial intelligence model 126, and any other data derived from the representative workload 114. Aggregation and other analysis can be performed on the performance metrics 128 to gain insight into the behavior and / or performance of the artificial intelligence model 126 within the context embodied by the representative workload 114.

[0038] This is the responsibility of the analysis layer 130, which extracts the performance metrics 128 from the model deployment layer 118. The analysis layer 130 utilizes the performance analyzer 132 to aggregate the raw data from the performance metrics 128. In addition, the performance analyzer 132 can be configured with a specified granularity to enable trend analysis, data aggregation, and other performance analysis techniques. For example, the performance metrics 128 may include data throughput, latency, and other performance indicators. Therefore, the performance analyzer 132 can generate performance measurements 134 based on the performance metrics 128 to capture the behavior of the artificial intelligence model 126 in response to the representative workload 114. The performance measurement 134 is any data that is formatted to represent and quantify the performance of the current configuration of the artificial intelligence model 126 and / or computing resources 124. In one example, the performance measurement 134 is a graph representing the relationship between latency and data throughput.

[0039] In addition to the performance measurements 134, the analysis layer 130 can also be configured to generate quality gates 136. As discussed below, the quality gates 136 are generated based on a comparison of the performance measurements 134 for multiple iterations of the artificial intelligence model 126. In the illustrative example, the cloud service provider develops and releases updates for the artificial intelligence model 126 once a month. However, before actually releasing the update, the service provider uses the Figure 1The process shown and discussed above can be used to evaluate updates. In addition, updates can be evaluated together with previous iterations of the artificial intelligence model 126 as regression tests. In this way, multiple iterations of the artificial intelligence model 126 can be deployed simultaneously at the computing resources 124 for regression testing.

[0040] Based on the regression testing, the analysis layer 130 generates a quality gate 136 that defines an operating window including one or more performance thresholds that must be met before an iteration of the artificial intelligence model 126 can be deployed. In one example, the quality gate 136 defines a maximum acceptable latency for a given data throughput level. In addition, the quality gate can be used to select an iteration of the artificial intelligence model 126 selected from the model configuration 104 to deploy from a plurality of candidate iterations. It should be understood that an iteration can also be referred to as a build or version of software such as the artificial intelligence model 126.

[0041] To complete the iteration cycle, the feedback layer 138 reviews the results of the analysis layer 130, such as the performance measurements 134. Thus, the feedback layer 138 generates a modified set of load profile parameters 140 based on the performance measurements 134 resulting in an updated load profile 142. In addition, the feedback layer 138 can generate a modified input data set 144. With the updated load profile 142 and the modified input data set 144, the feedback layer 138 enables the system 100 to run the updated representative workload 114 to evaluate the performance of the artificial intelligence model 126 under different scenarios and / or contexts.

[0042] As described above, large language models have seen adoption for a wide variety of uses, such as summarization, text completion, document retrieval and search, classification, translation, prompt-based content generation, and the like. Each of these uses involves a general workflow that includes receiving input and / or context tags, to which the artificial intelligence model 126 applies a series of feed-forward pass calculations to derive a set of output tags. It is well understood that the size of the input and / or context tags, the tasks anticipated, and the model configuration 104 affect the size of the output tags and the time required to derive the output tags. In addition, the number of requests per second (i.e., demand) determines the data throughput and affects the latency for a given workload. Thus, when testing the artificial intelligence model 126, evaluating as many different contexts as possible is an important consideration.

[0043] By generating the modified load profile 142 and the modified input data set 144, the feedback layer 138 achieves several uses and technical benefits. In one example, the system 100 can investigate the impact of workload size on the performance of the artificial intelligence model 126. In another example, the feedback layer 138 identifies workload types that interact well and can therefore be processed by the same artificial intelligence model 126 and / or computing resources 124. For example, an artificial intelligence model 126 that primarily processes summary workloads can handle text completion workloads well. In contrast, the same artificial intelligence model 126 may suffer from performance degradation in image generation workloads. In this way, the system 100 can determine the optimal workload scheduling for a given set of computing resources 124. In addition, the system 100 can perform customized adjustments of the artificial intelligence model 126 to maximize efficiency for available computing power.

[0044] In yet another example, the feedback layer 138 can identify a load threshold at which a given workload must be directed to a new and / or additional computing resource 124 for appropriate processing, which is generally referred to as failover. In a typical scenario, failover is a highly serious problem that requires a lot of effort to solve. By identifying the specific load threshold at which failover may occur, the system 100 can handle failover gracefully. For example, by automatically managing workloads and / or by issuing advance warnings to system administrators or engineers. In yet another example, by iteratively modifying and testing input data sets 106 and load profiles 108, the system 100 can identify various levels of customer experience to provide to users. For example, users who pay a premium subscription to a cloud computing platform can receive lower latency with higher data throughput than free users.

[0045] Now go to Figure 2A , an example of a load profile 108 is shown and described. Specifically, Figure 2A An example of an ascending load profile 202 is illustrated. The ascending load profile 202 generates load through containers 204A to 204D that are active simultaneously, each container including a thread set 206. The ascending load profile 202 follows an arithmetic sequence of increasing computational demands over time. In many examples, the total demand imposed by the load profile 108 can be expressed using the following equation (1):

[0046]

[0047] In equation (1), TC t is the total demand, T is the time, n is the number of threads, and C is the number of containers.

[0048] The rising load profile 202 is useful for determining the maximum level of computing demand that the computing resource set 124 can handle using the artificial intelligence model 126 before encountering excessive latency degradation. Figure 2A As shown, the ramp-up load profile 202 defines a representative workload 114 that includes four containers 204. Therefore, each container 204 increases the number of its threads 206 by a predetermined increment 208 at each time step T. For example, at time T1, each container 204 contains one thread 206 and increases by an increment 208 of one for each time step. As a result, each container 204A to container 204D contains five threads 206 at time T5.

[0049] like Figure 2B As shown by Figure 2A The computing demand 210 imposed by the ramp-up load profile 202 quickly reaches a peak demand 212 imposed on the computing resources 124 and the artificial intelligence model 126, which returns to zero instantaneously when the representative workload 114 is completed. By utilizing the feedback layer 138 discussed above, the peak demand 212 can be adjusted iteratively.

[0050] Go to Figure 3A , another example of a load profile 108 is shown and discussed. In this example, a bell curve load profile 302 is illustrated. Similar to the ramp-up load profile 202 above, the bell curve load profile 302 defines a number of containers 304 and a number of threads 306 for each container 304. However, unlike the ramp-up load profile 202, the containers 304 are staggered with respect to time T. That is, the time T at which the first container 304A reaches the model endpoint 120 and begins to increase demand is different than the time T at which the second container 304B does. In addition, the increment 308 for each container varies as some containers 304 complete execution while other containers 304 are still stepping up their corresponding number of threads 306.

[0051] like Figure 3B As shown, as discussed, the staggered containers 304 cause the computing demand 310 to behave differently from the ramp-up load profile 202 by first going through a phase of increasing demand 312 and then a phase of decreasing demand 316 before reaching a peak demand 314. Similar to the ramp-up load profile 202, the computing demand 310 for the bell curve load profile 302 can be adjusted by the feedback layer 138. For example, adjusting the number of threads 306 results in a different peak demand 314. For example, an increasing number of threads results in an increasing peak demand 314. In addition, adjusting the number of containers 304 changes the duration of the computing demand accordingly.

[0052] Go to Figure 4A, performance measurements 402 synthesized from the execution of a representative workload 114 are shown and described. As discussed above, the analysis layer 130 can perform analysis on the performance metrics 128 extracted from the model deployment layer 118 to gain insight into the model behavior under the conditions of the representative workload 114. An example of a performance measurement 402 is to visualize the relationship between latency 404 and data throughput 406. Logically, as data throughput 406 increases, latency 404 also increases. By objectively visualizing this relationship, the performance measurement 402 enables the service provider to identify operating conditions that provide the best balance between the various performance metrics 128.

[0053] Several aspects of the latency 404 and data throughput 406 relationship are of particular interest to service providers. First, the performance measurements 402 identify a steady state 408 in which the computing resources 124 and associated artificial intelligence models 126 maintain a consistent data throughput 406 without significantly increasing latency 404. Under normal circumstances, a service provider can expect most workloads to reside in the steady state 408, in which the failure rate is relatively low for a given level of data throughput 406. Thus, the steady state 408 can be considered a "happy state" that indicates an optimal performance range and / or a most desired user experience.

[0054] Secondly, the performance measurement 402 identifies an inflection point 410 where the computing resources 124 are heavily utilized and latency begins to increase significantly. Beyond the inflection point 410, there is an exponential increase in latency 404 for a relatively small gain in data throughput 406. When the computing resources 124 used for the artificial intelligence model 126 reach the inflection point 410, an automatic resource scaling process is triggered. In other words, the inflection point 410 defines a threshold latency 404 and data throughput 406 at which additional computing resources 124 are allocated to the artificial intelligence model 126 for processing various workloads. In addition, the inflection point 410 may also be referred to as an "elbow state," indicating a region of data throughput 406 where the latency 404 increases exponentially.

[0055] Now go to Figure 4B , another example of a performance measure 412 is shown and described. The performance measure 412 visualizes the relationship between an operating cost 414 and a data throughput 406 associated with operating a cloud computing platform. In various examples, the performance measure 412 is derived from a performance measure 402 representing latency 404 versus data throughput 406. Thus, the performance measure 412 includes a steady state 408 and an inflection point 410 identified by the previous performance measure 402. By plotting the operating cost 414 relative to the data throughput 406, the performance measure 412 identifies various cost boundaries 414A and 414B.

[0056] The first cost boundary 414A is associated with the steady state 408. As described above, the steady state 408 represents the most desired user experience because the latency 404 is minimized for a relatively high data throughput 406. However, the operating cost 414 at the first cost boundary 414A is higher relative to the second cost boundary 414B because low latency 404 requires low utilization of the computing resources 124. In contrast, the second cost boundary 414B at the inflection point 410 is lower as the utilization of the computing resources 124 increases. In this way, the performance measurement 412 can enable the service provider to intelligently price cloud computing services to various users (e.g., customers) based on performance. There are levers that can be adjusted here because the cost associated with the computing resources is inversely proportional to the expected throughput along the cost curve until it reaches the point of a bottleneck.

[0057] Now go to Figure 4C , a comparison between performance measurements 416A to 416C is shown and described. As discussed above, system 100 enables consistent and repeatable load testing and / or performance benchmarking. Therefore, multiple iterations of artificial intelligence models 418A to 418C can be compared to each other in a fair and representative manner. This enables the service provider to select the most ideal artificial intelligence model iteration 418 for deployment to users. Figure 4C As shown, latency 420 for each artificial intelligence model iteration 418 is plotted relative to normalized data throughput 422 to ensure a fair performance comparison.

[0058] From the comparison of performance measurement 416A to performance measurement 416C, it is determined that artificial intelligence model 418B exhibits the most desirable behavior when normalized data throughput 422 increases. Performance measurement 416B experiences an inflection point 424 at a relatively greater normalized data throughput 422 than performance measurement 416A and performance measurement 416C. In addition, performance measurement 416B also experiences relatively less severe degradation in latency 420 after inflection point 424. Therefore, artificial intelligence model iteration 418B is selected for deployment from candidate artificial intelligence model iteration 418A to candidate artificial intelligence model iteration 418C.

[0059] Furthermore, as described above, comparing performance measurements 416A-416C for various artificial intelligence model iterations 418A-418C enables system 100 to construct quality gates 136 based on performance trends across iterations, behavior trends under various representative workloads 114, and other factors. For example, quality gate 136 defines a threshold latency 420 for some normalized data throughput 422. Thus, artificial intelligence model iterations 418 that exceed the threshold latency are prevented from being deployed to users.

[0060] Now go to Figure 5 , illustrates and describes aspects of a routine 500 for implementing load testing and / or performance benchmarking for large language models using a cloud computing platform. Figure 5 , the routine 500 begins at operation 502, where the system selects a load profile that defines a plurality of workload parameters.

[0061] Next, at operation 504, the system generates a representative workload according to the parameters defined by the load profile. The representative workload is configured to evaluate the performance of the artificial intelligence model for the cloud computing platform under the context represented by the load profile.

[0062] Then, at operation 506 , the system executes the representative workload on the computing resources using the artificial intelligence model.

[0063] Next, at operation 508 , after executing the representative workload on the computing resource using the artificial intelligence model, the system extracts a plurality of performance metrics from the computing resource.

[0064] Then, at operation 510, the system synthesizes a performance measurement based on the plurality of performance metrics.

[0065] Then, at operation 512, the load profile is updated using the performance measurements by modifying a number of parameters defined by the load profile.

[0066] Finally, at operation 514, the system generates an updated representative workload configured to further evaluate the artificial intelligence model performance of the cloud computing platform under different contexts.

[0067] For ease of understanding, the processes discussed in this disclosure are depicted as separate operations represented as independent blocks. However, these separately depicted operations should not be interpreted as necessarily being sequence-dependent in their performance. The order in which the processes are described is not intended to be interpreted as limiting, and any number of the described process blocks can be combined in any order to implement the process or an alternative process. In addition, it is also possible to modify or omit one or more of the operations provided.

[0068] The specific implementation of the technology disclosed herein is a selection problem that depends on the performance and other requirements of the computing device. Therefore, the logical operation described herein is referred to as state, operation, structural device, action or module in different ways. These states, operations, structural devices, actions and modules can be realized with hardware, software, firmware, special digital logic, and any combination thereof. It should be understood that more or less operations than those shown in the figure and described herein can be performed. These operations can also be performed in an order different from the order described herein.

[0069] It should also be understood that the illustrated method can end at any time and does not need to be performed as a whole. As defined below, some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions included on a computer storage medium. The term "computer-readable instructions" as used in the specification and claims and its variants are used in this article to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, etc.

[0070] It should be understood, therefore, that the logical operations described herein are implemented as (1) a sequence of computer-implemented actions or program modules running on a computing system, and / or (2) interconnected machine logic circuits or circuit modules within the computing system. Implementation is a matter of choice depending on the performance and other requirements of the computing system. Thus, the logical operations described herein are variously referred to as states, operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules may be implemented in software, firmware, dedicated digital logic, and any combination thereof.

[0071] For example, the operations of routine 500 may be implemented at least in part by modules that run the features disclosed herein, which may be dynamic link libraries (DLLs), static link libraries, functions generated by application programming interfaces (APIs), compiled programs, interpreted programs, scripts, or any other executable instruction sets. Data may be stored in data structures in one or more memory components. Data may be retrieved from a data structure by addressing a link or reference to the data structure.

[0072] Although the diagram may refer to the components of the drawings, it should be understood that the operations of routine 500 may also be implemented in other ways. In addition, one or more operations of the operations of routine 500 may alternatively or additionally be implemented at least in part by a chipset that works alone or in combination with other software modules. In the examples described below, one or more modules of the computing system may receive and / or process the data disclosed herein. Any service, circuit or application suitable for providing the technology disclosed herein may be used in the operations described herein.

[0073] Figure 6Additional details of an example computer architecture 600 for a device such as a computer or server configured as part of the cloud-based platform or system 100 capable of executing computer instructions (eg, the modules or program components described herein) are shown. Figure 6 The computer architecture 600 illustrated in FIG. 6 includes a processing system 602, a system memory 604 including a random access memory (RAM) 606 and a read-only memory (ROM) 608, and a system bus 610 coupling the memory 604 to the processing system 602. The processing system 602 includes (multiple) processing units. In various examples, the (multiple) processing units of the processing system 602 are distributed. In other words, one processing unit of the processing system 602 can be located at a first location (e.g., a rack within a data center), while another processing unit of the processing system 602 is located at a second location separate from the first location.

[0074] Processing unit(s), such as the processing unit(s) of processing system 602 may represent, for example, a CPU-type processing unit, a GPU-type processing unit, a field programmable gate array (FPGA), another type of digital signal processor (DSP), or other hardware logic components that may be driven by a CPU in some instances. For example, illustrative types of hardware logic components that may be used include application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0075] A basic input / output system, containing the basic routines that help to transfer information between elements within the computer architecture 600, such as during start-up, is stored in ROM 608. The computer architecture 600 also includes a mass storage device 612 for storing an operating system 614, application(s) 616, modules 618, and other data described herein.

[0076] The mass storage device 612 is connected to the processing system 602 through a mass storage controller connected to the bus 610. The mass storage device 612 and its associated computer-readable media provide non-volatile storage to the computer architecture 600. Although the descriptions of computer-readable media contained herein refer to mass storage devices, the computer-readable media may be any available computer-readable storage media or communication media that can be accessed by the computer architecture 600.

[0077] Computer-readable media include computer-readable storage media and / or communication media. Computer-readable storage media include one or more of volatile memory, non-volatile memory, and / or other persistent and / or secondary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storing (information such as computer-readable instructions, data structures, program modules or other data). Therefore, computer storage media include media in tangible and / or physical form included in a device and / or as a part of a device or in a hardware component external to the device, including RAM, static RAM (SRAM), dynamic RAM (DRAM), phase change memory (PCM), ROM, erasable programmable ROM (EPROM), electrical EPROM (EEPROM), flash memory, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), optical card or other optical storage medium, magnetic cassette, magnetic tape, magnetic disk storage, magnetic card or other magnetic storage device or medium, solid-state storage device, storage array, network attached storage, storage area network, hosted computer storage or any other storage memory, storage device and / or storage medium that can be used to store and maintain information for access by a computing device.

[0078] In contrast to computer-readable storage media, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer storage media does not include communication media. That is, computer-readable storage media itself does not include communication media consisting only of a modulated data signal, a carrier wave, or a propagation signal.

[0079] According to various configurations, the computer architecture 600 can operate in a networked environment using logical connections to remote computers through a network 620. The computer architecture 600 can be connected to the network 620 through a network interface unit 622 connected to the bus 610. The computer architecture 600 can also include an input / output controller 624 for receiving and processing input from a number of other devices, including a keyboard, a mouse, a touch or electronic stylus or pen. Similarly, the input / output controller 624 can provide output to a display screen, a printer, or other type of output device.

[0080] The software components described herein, when loaded into the processing system 602 and executed, can convert the processing system 602 and the entire computer architecture 600 from a general-purpose computing system to a special-purpose computing system, which is customized to facilitate the functions presented herein. The processing system 602 can be constructed from any number of transistors or other discrete circuit element configurations, which can present any number of states individually or collectively. More specifically, in response to executable instructions contained in the software modules disclosed herein, the processing system 602 can operate as a finite state machine. These computer executable instructions can convert the processing system 602 by specifying how the processing system 602 transitions between states, thereby converting the transistors or other discrete hardware elements that constitute the processing system 602.

[0081] Figure 7 An illustrative distributed computing environment 700 capable of executing the software components described herein is depicted. Figure 7 The distributed computing environment 700 illustrated in FIG. 700 can be used to execute any aspects of the software components presented herein. For example, the distributed computing environment 700 can be used to execute various aspects of the software components described herein.

[0082] Thus, the distributed computing environment 700 may include a computing environment 702 operating on, communicating with, or being part of a network 704. The network 704 may include various access networks. One or more client devices 706A to 706N (hereinafter collectively and / or generally referred to as "computing devices 706") may communicate with the computing environment 702 via the network 704. In one illustrated configuration, the computing devices 706 include a computing device 706A such as a laptop, desktop computer, or other computing device; a notebook or tablet computing device ("tablet computing device") 706B; a mobile computing device 706C such as a mobile phone, smart phone, or other mobile computing device; a server computer 706D; and / or other devices 706N. It should be understood that any number of computing devices 706 may communicate with the computing environment 702.

[0083] In various examples, computing environment 702 includes server 708, data store 610, and one or more network interfaces 712. Server 708 can host various services, virtual machines, portals, and / or other resources. In the illustrated configuration, server 708 hosts virtual machine 714, web portal 716, mailbox service 718, storage service 720, and / or social network service 722. Figure 7 As shown, server 708 may also host other services, applications, portals, and / or other resources (“other resources”) 724 .

[0084] As described above, the computing environment 702 may include a data store 710. According to various implementations, the functionality of the data store 710 is provided by one or more databases operating on or in communication with the network 704. The functionality of the data store 710 may also be provided by one or more servers configured to host data for the computing environment 700. The data store 710 may include, host, or provide one or more real or virtual data stores 726A to 726N (hereinafter collectively and / or generally referred to as "data stores 726"). The data store 726 is configured to host data used or created by the server 808 and / or other data. That is, the data store 726 may also host or store web documents, word documents, presentation documents, data structures, algorithms for execution by the recommendation engine, and / or other data utilized by any application. Various aspects of the data store 726 may be associated with a service for storing files.

[0085] The computing environment 702 can communicate with or be accessed by a network interface 712. The network interface 712 can include various types of network hardware and software for supporting communications between two or more computing devices including computing devices and servers. It should be understood that the network interface 712 can also be used to connect to other types of networks and / or computer systems.

[0086] It should be understood that the distributed computing environment 700 described herein can provide any aspect of the software components described herein with any number of virtual computing resources and / or other distributed computing functions that can be configured to perform any aspect of the software components disclosed herein. According to various implementations of the concepts and technologies disclosed herein, the distributed computing environment 700 provides the software functions described herein as services to computing devices. It should be understood that computing devices can include real or virtual machines, including server computers, web servers, personal computers, mobile computing devices, smart phones, and / or other devices. Thus, various configurations of the concepts and technologies disclosed herein enable any device configured to access the distributed computing environment 700 to utilize the functions described herein for providing the technologies and other aspects disclosed herein.

[0087] The disclosure presented herein also covers the subject matter set forth in the following clauses.

[0088] Example Clause A, a method for evaluating the performance of an artificial intelligence model for a cloud computing platform, the method comprising: selecting a load profile that defines multiple parameters; generating a representative workload, the representative workload being configured to evaluate the performance of the artificial intelligence model for the cloud computing platform under the computing demands imposed by the workload context represented by the load profile; utilizing the artificial intelligence model to execute the representative workload on computing resources; extracting multiple performance metrics from the computing resources after utilizing the artificial intelligence model to execute the representative workload on the computing resources; synthesizing performance measurements based on the multiple performance metrics extracted from the computing resources; updating the load profile using the performance measurements by modifying multiple parameters defined by the load profile; and generating an updated representative workload using the modified multiple parameters in the updated load profile, the updated representative workload being configured to further evaluate the performance of the artificial intelligence model of the cloud computing platform under different computing demands imposed by the updated workload context.

[0089] Example clause B, the method of example clause A, wherein the load profile is selected from a plurality of load profiles established for different workload contexts.

[0090] Example Item C, the method of Example Item A or Example Item B, wherein the workload context represented by the load profile utilizes an artificial intelligence model to capture peak computing demands arriving at the computing resources.

[0091] Example clause D, the method of any of Example clauses A to C, wherein the workload context represented by the load profile utilizes an artificial intelligence model to capture incremental and subsequently decreasing computational demands at the computing resources.

[0092] Example Clause E, a method of any one of Example Clauses A to Example Clauses D, wherein the computing resources also include a model endpoint for simulating a user access point to the artificial intelligence model at the cloud computing platform.

[0093] Example Clause F, the method of any one of Example Clause A to Example Clause E, wherein the representative workload is generated by multiple virtual machine instances.

[0094] Example Clause G, the method of any one of Example Clauses A to Example Clauses F, further comprising generating a quality gate defining an operating window based on a comparison between a performance measurement of the artificial intelligence model for a representative workload and a performance measurement of another iteration of the artificial intelligence model for the representative workload.

[0095] Example clause H, the method of any of Example clause A to Example clause G, wherein the plurality of parameters includes a number of containers and a number of threads per container.

[0096] Example clause I, the method of any of example clauses A to H, wherein the performance measurement includes at least one of latency or data throughput.

[0097] Example Clause J, a system for evaluating the performance of an artificial intelligence model for a cloud computing platform, the method comprising: a processing system; a computer-readable medium having instructions encoded thereon, which when executed by the processing system causes the system to: select a load profile defining multiple parameters; generate a representative workload, which is configured to evaluate the performance of the artificial intelligence model for the cloud computing platform under the workload context represented by the load profile; use the artificial intelligence model to execute the representative workload on computing resources; after using the artificial intelligence model to execute the representative workload on the computing resources, extract multiple performance metrics from the computing resources; synthesize performance measurements based on the multiple performance metrics extracted from the computing resources; update the load profile using the performance measurements by modifying multiple parameters defined by the load profile; and generate an updated representative workload using the modified multiple parameters in the updated load profile, the updated representative workload being configured to further evaluate the performance of the artificial intelligence model of the cloud computing platform under different workload contexts.

[0098] Example Clause K, the system of Example Clause J, wherein the load profile is selected from a plurality of load profiles established for different workload contexts.

[0099] The system of Example Clause L, Example Clause J, or Example Clause K, wherein the workload context represented by the load profile utilizes an artificial intelligence model to capture peak demand on computing resources.

[0100] The system of any of Example Clause M, Example Clause J to Example Clause L, wherein the workload context represented by the load profile utilizes an artificial intelligence model to capture incremental and subsequently decremental demand on computing resources.

[0101] The system of any of Example Clause N, Example Clause J-Example Clause M, wherein the plurality of parameters includes a number of containers and a number of threads per container.

[0102] A system of any of Example Clause O, Example Clause J to Example Clause N, wherein the computer-readable instructions further cause the system to generate a quality gate defining an operating window based on a comparison between a performance measurement of the artificial intelligence model for a representative workload and a performance measurement of another iteration of the artificial intelligence model for the representative workload.

[0103] Example clause P, a computer-readable storage medium having computer-readable instructions encoded thereon, which, when executed by a processing system, causes the system to: select a load profile defining multiple parameters; generate a representative workload, the representative workload being configured to evaluate the performance of an artificial intelligence model for a cloud computing platform under the workload context represented by the load profile; utilize the artificial intelligence model to execute the representative workload on a computing resource; after utilizing the artificial intelligence model to execute the representative workload on the computing resource, extract multiple performance metrics from the computing resource; synthesize performance measurements based on the multiple performance metrics extracted from the computing resource; and update the load profile using the performance measurements by modifying multiple parameters defined by the load profile; generate an updated representative workload using the modified multiple parameters in the updated load profile, the updated representative workload being configured to further evaluate the performance of the artificial intelligence model of the cloud computing platform under different workload contexts.

[0104] Example clause Q, the computer-readable storage medium of example clause P, wherein the load profile is selected from a plurality of load profiles established for different workload contexts.

[0105] The computer-readable storage medium of example clause R, example clause P, or example clause Q, wherein the workload context represented by the load profile utilizes an artificial intelligence model to capture peak demand arriving at computing resources.

[0106] Example clause S, computer-readable storage medium of any one of example clauses P to example clause R, wherein the workload context represented by the load profile utilizes an artificial intelligence model to capture incremental and subsequently decremental demands on computing resources.

[0107] Computer-readable storage medium of any one of Example Clause T, Example Clause P to Example Clause S, wherein the computer-readable instructions further cause the system to generate a quality gate defining an operating window based on a comparison between a performance measurement of the artificial intelligence model for a representative workload and a performance measurement of another iteration of the artificial intelligence model for the representative workload.

[0108] Unless expressly stated otherwise, conditional language such as "can," "can," "might," or "may" is understood in context to mean that certain examples include certain features, elements, and / or steps while other examples do not. Thus, such conditional language is generally not intended to imply that one or more examples in any way require certain features, elements, and / or steps, or that one or more examples must include logic for deciding whether to include or perform certain features, elements, and / or steps in any specific example with or without user input or prompting. Unless expressly stated otherwise, connective language such as the phrase "at least one of X, Y, or Z" should be understood to mean that an item, term, etc. can be X, Y, or Z, or a combination thereof.

[0109] Unless otherwise indicated herein or clearly contradicted by context, the terms "a", "an", "the" and similar referents used in the context of describing the invention (especially in the context of the claims below) should be interpreted to cover both the singular and the plural. Unless otherwise indicated herein or clearly contradicted by context, the terms "based on", "according to" and similar referents should be interpreted to mean "based at least in part on", which includes "based in part on" and "based entirely on".

[0110] In addition, any reference to "first", "second", etc. elements in the invention content and / or detailed description is not intended to and should not be interpreted as necessarily corresponding to any reference to "first", "second", etc. elements in the claims. On the contrary, any use of "first" and "second" in the invention content, detailed description and / or claims can be used to distinguish two different instances of the same element (e.g., two different artificial intelligence models).

[0111] Finally, although various configurations have been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the accompanying representations is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. A method for evaluating the performance of an artificial intelligence model for a cloud computing platform, the method comprising: Select a load profile that defines multiple parameters; generating a representative workload configured to evaluate the performance of the artificial intelligence model for the cloud computing platform under the context represented by the load profile; Utilizing the artificial intelligence model to execute the representative workload on computing resources; extracting a plurality of performance metrics from the computing resource after executing the representative workload on the computing resource using the artificial intelligence model; synthesizing a performance measure based on the plurality of performance metrics extracted from the computing resources; as well as updating the load profile using the performance measurements by modifying the plurality of parameters defined by the load profile; as well as An updated representative workload is generated using the modified multiple parameters in the updated load profile, and the updated representative workload is configured to further evaluate the performance of the artificial intelligence model of the cloud computing platform under different contexts.

2. The method of claim 1, wherein the load profile is selected from a plurality of load profiles established for different contexts.

3. A method according to claim 2, wherein the context represented by the load profile utilizes the machine learning model to capture peak demand reaching the computing resources.

4. The method of claim 2, wherein the context represented by the load profile utilizes the machine learning model to capture increasing and then decreasing demand at the computing resource.

5. A method according to any one of claims 1 to claim 4, wherein the computing resources also include a model endpoint for simulating a user access point to the machine learning model at the cloud computing platform.

6. The method according to any one of claims 1 to claim 5, wherein the representative workload is generated by multiple virtual machine instances.

7. The method according to any one of claims 1 to claim 6 also includes generating a quality gate defining an operating window based on a comparison between the performance measurement of the machine learning model for the representative workload and the performance measurement of another iteration of the machine learning model for the representative workload. 8 . The method according to claim 1 , wherein the plurality of parameters comprises the number of containers and the number of threads per container.

9. The method of any one of claims 1 to 8, wherein the performance measure comprises at least one of latency or data throughput.

10. A system for evaluating the performance of an artificial intelligence model for a cloud computing platform, the system comprising: Processing systems; A computer readable medium having encoded thereon instructions that, when executed by the processing system, cause the system to: Select a load profile that defines multiple parameters; generating a representative workload configured to evaluate the performance of the machine learning model for the cloud computing platform under the context represented by the load profile; executing the representative workload on computing resources using the machine learning model; extracting a plurality of performance metrics from the computing resource after executing the representative workload on the computing resource using the machine learning model; synthesizing a performance measure based on the plurality of performance metrics extracted from the computing resources; as well as updating the load profile using the performance measurements by modifying the plurality of parameters defined by the load profile; An updated representative workload is generated, wherein the updated representative workload is configured to further evaluate the performance of the machine learning model of the cloud computing platform under a different context.

11. The system of claim 10, wherein the context represented by the load profile utilizes the artificial intelligence model to capture peak demand arriving at the computing resources.

12. A system according to claim 10 or claim 11, wherein the context represented by the load profile utilizes the artificial intelligence model to capture increasing and then decreasing demand at the computing resources.

13. The system according to any one of claims 10 to 12, wherein the plurality of parameters include the number of containers and the number of threads per container.

14. A system according to any one of claims 10 to claim 13, wherein the computer-readable instructions further cause the system to generate a quality gate defining an operating window based on a comparison between the performance measurement of the artificial intelligence model for the representative workload and the performance measurement of another iteration of the artificial intelligence model for the representative workload.

15. A computer-readable storage medium having encoded thereon computer-readable instructions that, when executed by a processing system, cause the system to: Select a load profile that defines multiple parameters; generating a representative workload configured to evaluate the performance of the machine learning model for the cloud computing platform under the context represented by the load profile; executing the representative workload on computing resources using the machine learning model; extracting a plurality of performance metrics from the computing resource after executing the representative workload on the computing resource using the machine learning model; synthesizing a performance measure based on the plurality of performance metrics extracted from the computing resources; as well as updating the load profile using the performance measurements by modifying the plurality of parameters defined by the load profile; An updated representative workload is generated, wherein the updated representative workload is configured to further evaluate the performance of the machine learning model of the cloud computing platform under a different context.

Images