Predictable distribution of program instructions

Abstract

Problems can arise when highly engaged customers do not receive reliable feature updates on a predictable timeline. For example, releasing and updating features can burden cloud infrastructure and / or cause it to malfunction. Unpredictable feature updates not only relate to inconsistent customer messaging and potential service disruptions, but can also lead to random end-user experiences and customer dissatisfaction. This approach and system address these issues by dynamically ranking tenants and distributing updates and releases to tenants with lower feature usage before distributing them to those with higher usage. Tenant ranking can dynamically change based on current usage. In this way, higher-ranked tenants can receive access to new and updated features after additional regression testing. This quality-oriented, phased feature release enables load-level management of compute resources and provides a predictable feature experience for end users.

Description

Background Technology

[0001] Using cloud computing environments for software products and services has become a popular trend for consumers and businesses. Enterprise end-users use application services on cloud computing environments to upload and share files, and perform other operations, such as collaborative tasks. A cloud computing environment involves a set of distributed resources, including storage, processing units, and data storage interconnected via secure network communications. Distributed computing resources can be physically and virtually integrated. The virtual integration of servers enables the dynamic allocation of resources to tenants in a cloud computing environment. In this way, service providers enable tenants to seamlessly and dynamically scale up or down their usage by continuously evaluating and reallocating resources among the servers hosting their tenants. However, service providers also rely on cloud resources to distribute new and updated software features. Some tenants may be more sensitive to service disruptions associated with feature releases than others, especially if the tenants are actively using these features. For example, service disruptions may involve new or updated features becoming unavailable or performing unpredictably for end-users. Unpredictable feature activation or updates can lead to randomness in the end-user experience. Furthermore, such inconsistent deployments can result in mismatches between communication with customers and the available features, such as through customer management portals. Unreliable communication with customers can lead to customer dissatisfaction. Therefore, when distributing code associated with new and updated features in a cloud computing environment, there is a need to minimize service disruptions and customer dissatisfaction while also improving the customer experience.

[0002] It is with regard to these and other general considerations that the aspects disclosed herein have been made. Furthermore, while relatively specific problems may be discussed, it should be understood that the examples are not limited to solving specific problems identified in the background of this disclosure or elsewhere. Summary of the Invention

[0003] According to this disclosure, the aforementioned and other issues are addressed by distributing feature-associated program instructions to tenants based on dynamic ranking of each tenant. The "tenant" or customer of the application service can be hosted by the software provider on physical or virtual servers in a cloud computing environment. A "feature" can be associated with an operation performed by the application service, such as uploading a file, creating a list, sharing files, making authentication requests, creating a web part, etc. Traditionally, feature rollout consists of two parts: code "rollout" and code "activation." That is, the code associated with the feature is first deployed (rollout) and then enabled (activated or turned on). Furthermore, for randomly deployed sets of code, features can be enabled in stages or by clicking to stop enabling features. For example, features can be enabled for 1%, then 5%, then 10%, and so on, of tenants. After each stage, the enabled features can be evaluated to ensure code stability and user satisfaction. As used herein, the term "assignment" or "distribution" of program instructions (or code) refers to enabling or turning on program instructions (or code) deployed to a tenant.

[0004] It should be recognized that the early stages of code distribution are often unstable before functional evaluation and testing are completed. In some respects, instead of distributing program instructions to randomly selected tenants, this disclosure distributes program instructions based on a dynamic ranking of tenants, where code is distributed to lower-ranked tenants before being distributed to higher-ranked tenants who may be more sensitive to service disruptions (e.g., due to high feature usage). Therefore, the systems and methods described herein address the problems of service disruptions and customer dissatisfaction associated with distributing new and updated software features to tenants in cloud computing environments.

[0005] The present invention is provided to introduce the selection of concepts in a simplified form, which will be further described in the detailed description below. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Other aspects, features, and / or advantages of the examples will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0006] Non-restrictive and non-exhaustive examples are described with reference to the following figures.

[0007] Figure 1 An overview of an example system for distributing program instructions according to aspects of this disclosure is illustrated.

[0008] Figure 2 An example method of distributing program instructions according to aspects of this disclosure is illustrated.

[0009] Figure 3An example method for ranking tenants for distributing program instructions according to aspects of this disclosure is illustrated.

[0010] Figures 4A-4B An example of ranking tenants according to aspects of this disclosure for use in distributing program instructions is illustrated.

[0011] Figures 5A-5C An example of parameters for ranking tenants according to aspects of this disclosure is illustrated.

[0012] Figure 6 The illustration shows an example distribution system based on an example system, which can be used to practice the present disclosure according to aspects thereof.

[0013] Figure 7 This is a block diagram illustrating example physical components of a computing device that can be used to practice aspects of this disclosure.

[0014] Figure 8A This is a simplified diagram of a mobile computing device that can implement aspects of this disclosure.

[0015] Figure 8B This is another simplified block diagram of a mobile computing device that can implement aspects of this disclosure. Detailed Implementation

[0016] Various aspects of this disclosure are described more fully below with reference to the accompanying drawings, which form part of this disclosure and illustrate specific example aspects. However, different aspects of this disclosure may be implemented in many different ways and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided to make this disclosure thorough and complete and to fully communicate the scope of these aspects to those skilled in the art. The aspects may be practiced as methods, systems, or apparatuses. Thus, aspects may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Therefore, the detailed description below should not be construed as limiting.

[0017] This disclosure relates to systems and methods for distributing program instructions to tenants in a cloud computing environment. A cloud computing environment is a distributed system in which application functions, storage, data storage and retrieval, and various processing functions can operate remotely to each other via a network (e.g., the Internet or a secure intranet). The "tenant" or customer of the application service can be hosted by a software provider on physical or virtual servers within the distributed system. Software program instructions (or code) for accessing the application service can then be distributed (or enabled) by the software provider to the server system hosting the tenant. In fact, cloud-based application services have been widely used as and relied upon by enterprise productivity tools. In this way, enterprise users associated with enterprise customers can collaborate online from anywhere in the world (e.g., storing / sharing documents, accessing internal sites, lists / libraries). Users can access the service using various devices such as mobile phones, laptops, and tablets. The application service is available 24 / 7, 365 days a year, with minimal downtime for user maintenance.

[0018] Typically, customers of application services have come to expect frequent updates providing access to the latest new features and rapid distribution of necessary fixes. Unlike standalone applications whose program instructions reside on the end user's local computing device, application services can be continuously updated by distributing program instructions (such as code) to tenants hosted on servers in the cloud. For example, these servers could collectively serve application services to millions of users. However, problems can arise when the distribution of program instructions burdens the cloud infrastructure and / or when the distributed code contains errors. These errors can suddenly disrupt application services for a large number of users.

[0019] In various ways, code distribution and activation processes can be used to continuously distribute new features and fixes to customers (e.g., tenants). However, distributed features are generally not enabled for all tenants simultaneously; instead, for a randomly distributed set of code, each feature is enabled in stages, or click-to-stop. For example, a feature might be enabled for 1%, 5%, 10%, etc. After each stage, the enabled features can be evaluated to ensure code stability and user satisfaction. In some respects, target tenants can be randomly selected for the early stages of enabling each feature. However, random selection of tenants can lead to unpredictable feature activation for tenants, resulting in customer dissatisfaction with the application service. This problem becomes even more complex when software providers distribute new features and fixes to millions of users almost continuously.

[0020] This disclosure addresses the aforementioned issues by dynamically ranking tenants based on multiple criteria, including tenant priority, tenant usage, and feature type, to prevent service disruptions and customer dissatisfaction during feature releases and updates. Typically, a "feature" can be associated with an action, such as uploading a file, creating a list, sharing a file, verifying a request, or creating a web part. Therefore, service disruptions associated with feature releases or updates may involve interruptions to the ability of end users to perform actions such as uploading or sharing files. Such disruptions have a greater impact on tenants who actively use a particular feature (e.g., an action) than on tenants who are less active in using that feature. Therefore, this method and system seek to distribute new and updated features to lower-priority tenants with lower feature usage before distributing them to higher-priority tenants with higher feature usage. Furthermore, tenant rankings can change depending on the specific feature being released or updated. That is, while a tenant may be ranked low for one feature (e.g., file sharing), that same tenant may be ranked high for another feature (e.g., creating a web part). Therefore, lower-ranked tenants have a lower service priority and are less active in using specific features of the application service compared to higher-ranked tenants. The instruction set associated with this feature is first distributed to lower-ranked tenants. This way, in the event of any service outages, lower-ranked tenants with lower usage of these features will not be significantly affected, thus reducing customer dissatisfaction. After verifying that the new or updated feature functions correctly, the instruction set is distributed to higher-ranked tenants. This allows higher-ranked tenants to access the new and updated features after the instruction sets undergo additional regression testing.

[0021] Figure 1 The illustration shows an overview of an example system for distributing program instructions according to aspects of this disclosure. System 100 may represent a system for distributing program instructions to one or more tenants of a cloud computing environment 140.

[0022] Developer 102 can develop program instructions to perform functions. Functions can be associated with operations, such as uploading files, creating lists, sharing files, verifying requests, creating web parts, etc. Based on the operations performed, functions can have different function types (e.g., upload type, share type, web part type, etc.). In various ways, developer 102 can provide program instructions to program instruction storage device or code storage device 120. After publishing the function, developer 102 can analyze telemetry data 170, which records how end user 104 interacts with the function. In some cases, developer 102 can update the program instructions associated with the function in response to the analysis.

[0023] End user 104 can use computing devices to access software products and services provided by software provider 148 via cloud computing environment 140. For example, each end user 104 may be associated with a tenant hosted by software provider 148 on cloud computing environment 140 (e.g., tenant A 106a, tenant B 106b, and tenant C 106c). Cloud computing environment 140 includes at least one server farm 146. Server farm 146 may contain virtual servers 142a / b and physical servers 144. Server farm 146 provides application services, file sharing, and workflow services to end user 104, for example. In some aspects, millions of end users 104 may use various application services available through cloud computing environment 140.

[0024] Cloud computing environment 140 is an interconnected cloud of servers. The servers collectively provide application services to end user 104. Alternatively, a group of servers forms server farm 146. Server farm 146 may include one or more application services. Server farm 146 may include virtual servers 142a / b and physical servers 144. Virtual servers 142a / b partitioned on one or more physical servers 144 may provide dedicated server functionality (e.g., processing and storage) to one or more tenants within the physical servers. For example, tenant A 106A may use one virtual server 142a from a group of virtual servers 142a / b, and tenant B 106B may use another virtual server 142b from the same group.

[0025] Virtual servers 142a / b and physical servers 144 together form server farm 146. The memory in each virtual server 142a / b and physical server 144 can store program instructions (not shown) for execution by a central processing unit (CPU). Execution of the program instructions makes functions (e.g., operations) available to end user 104. New program instructions (e.g., new code) can be distributed and installed to provide new functionality. Therefore, server farm 146 provides end user 104 with access to software products and services managed by software provider 148, such as uploading files, sharing files, authentication requests, creating web parts, etc. In some respects, millions of end users 104 associated with thousands of tenants can access a variety of application services available through cloud computing environment 140.

[0026] Code recognizer 110 can identify program instruction sets (e.g., code) associated with features used for distribution to tenants. In various aspects, developer 102 registers code for storage in code storage device 120. Additionally, the developer can identify the "type" of the feature. For example, a feature may include "load type," "shared type," "create type," "web part type," etc. Developer 102 can also request code distribution for program instruction sets, which may encode, for example, new features or updates to features. Subsequently, developer 102 can request code distribution for program instruction sets (e.g., enabling or activating). Code recognizer 110 can assign code IDs to program instruction sets. Therefore, code recognizer 110 can identify program instruction sets and feature types based on code IDs.

[0027] In all aspects, tenant ranking 112 assigns rankings to tenants hosted by software provider 148. Rankings are based on parameter values ​​associated with various parameters. These parameters may relate to, but are not limited to, how tenants use the application service, how tenants use specific features that are being updated or released, and how sensitive each tenant is to new or updated features associated with the application service. Tenant database 122 stores such tenant information. For example, tenant information includes tenant identifiers, tenant names, farm IDs, server IDs, tenant service priority levels, and computing resources allocated to the tenant. In all aspects, the tenant information in tenant database 122 can be relatively static information (e.g., static parameters), i.e., information that can only be updated occasionally (e.g., if the service contract between the tenant and software provider 148 is renewed).

[0028] Furthermore, the tenant ranking system 112 assigns rankings to tenants based on more frequently updated (or changed) dynamic parameters. For example, the server usage database 126 may store parameters indicating how end users of each tenant use the application services. The types of dynamic parameters may include, but are not limited to: (1) the number of monthly active users per tenant, (2) the number of requests made per second (or other period) by users of each tenant, (3) the peak usage periods for each tenant, (4) the CPU utilization of each tenant, and (5) the number of end users for each tenant who have not completed support queries. In each respect, the system 100 dynamically updates the parameter values ​​stored in the server usage database 126 over time (e.g., daily, every 5 days, every 15 days, etc.). Thus, the ranking assigned to a tenant by the tenant ranking system 112 can dynamically change based on changes in the dynamic parameter values ​​stored in the server usage database 126.

[0029] As described herein, code recognizer 10 specifies the code (or program instructions) to be distributed to tenants. In various aspects, code recognizer 110 may also specify the function type for the implemented functionality based on the executable code. In various aspects, code recognizer 110 identifies the code to be distributed in the order in which instructions for distributing code are received from developer 102. In some aspects, developer 102 may store the code in code storage device 120. Code storage device 120 may store the code by indexing it based on the storage order of the code.

[0030] Tenant ranker 112 assigns rankings to tenants based on information including, but not limited to, the following: (1) feature information (e.g., feature type) associated with a set of program instructions stored in code storage device 120; (2) tenant information in tenant database 122; (3) weighted values ​​of parameters stored in parameter weight table 124; and (4) server usage information for each tenant from server usage database 126. In each respect, tenant ranker 112 identifies features (and / or feature types) associated with a set of program instructions. This set of program instructions may introduce new features; alternatively, for example, the set of program instructions may update existing features by fixing associated faults, enhancing or changing features, or removing features. Based on the feature (and / or feature type) and tenant attributes, tenant ranker 112 may assign higher rankings to tenants who use the feature (or feature type) more frequently, and lower rankings to tenants who use the feature (or feature type) less frequently. Distributing code to lower-ranked tenants first protects higher-ranked tenants who use the application service more actively. Phased code distribution provides time to validate changes in the code and correct issues discovered in regression testing during early distributions. In this way, higher-ranking tenants who participate more in these features receive more consistent code distributions.

[0031] In some respects, the tenant ranking unit 112 can apply various criteria to rank tenants. Criteria (e.g., algorithm-based) can involve comparing tenants by assigning the value and weight of each parameter in a set of parameters to each tenant. Then, based on the parameter values ​​and weights, the tenant ranking unit 112 can rank each tenant on a continuum from low (or cold) to high (or hot). In various respects, the values ​​of parameters can be measured or assigned according to the parameters. For example, parameters can include, but are not limited to: service priority level per tenant, average monthly users per tenant, CPU utilization per tenant, etc. CPU utilization or CPU utilization level can be an estimated utilization level based on feature type. In various respects, a value for the service priority level can be assigned to each tenant, assigning higher values ​​to higher priority tenants (e.g., VIP or “white glove” tenants with thousands of users and special service agreements), and lower values ​​to lower priority tenants (e.g., those without special service agreements). Alternatively, average monthly users can be measured for each tenant. You can use a measure of average monthly users, or you can use an assigned value (e.g., a value relative to the total average monthly users across all tenants or the average monthly users of other tenants). The weight of each parameter can be assigned based on how much it might affect a tenant's sensitivity to feature distribution. This way, when assigning rankings to each tenant, a higher weight value can emphasize a particular parameter compared to others. In some cases, parameters can even be assigned "negative" weights. In this case, if the parameter exists (i.e., the code distribution fixes a feature bug encountered by the tenant), this will lower the tenant's ranking, causing the tenant to receive the code distribution faster.

[0032] Based on the assigned ranking, tenant ranker 112 can generate a ranked tenant list 128. In some cases, a ranked tenant list 128 can be generated for each new or updated feature to determine how the code will be distributed to tenants. In other cases, the ranked tenant list 128 can be used for a period of time (e.g., 1 day, 5 days, 15 days, 30 days) to determine how the code for the new or updated feature will be distributed to tenants during that period. It should be understood that tenants with higher priority service levels (e.g., those with service agreements guaranteeing a certain level of service quality) can rank higher than tenants with lower priority service levels. Thus, tenants with higher priority in avoiding service interruptions may receive code distribution after additional evaluation and testing, and once the program instruction set is more stable. Similarly, tenants with higher average monthly users can rank higher than other tenants with lower average monthly users, thus receiving more stable code distribution. That is, tenants with lower usage may be associated with a lower chance of experiencing service interruptions. Furthermore, if a feature fails early in the code distribution phase, tenants with lower usage are less likely to file a complaint. Similarly, when the feature becomes more stable in the later stages of distribution, tenants with higher usage are less likely to file complaints. In either case, customer dissatisfaction with the application service decreases.

[0033] Code distributor 114 uses a ranked tenant list 128 to provide the program instruction set to one or more virtual servers 142a / b and / or physical servers 144 associated with a selected group of tenants. In various ways, code distributor 114 can identify the servers hosting specific tenants in order to distribute program instructions based on the ranked tenant list 128. For a particular program instruction set, code distributor 114 can iteratively distribute (e.g., enable) the same program instruction set to different groups of tenants, grouped in ascending order of ranking. For example, there could be a first group of tenants (lower ranking), a second group of tenants (medium ranking), and a third group of tenants (higher ranking). Code distributor 114 distributes the program instruction set to the first group of tenants, then to the second group of tenants. The code distributor can determine whether the first group of tenants successfully uses the distributed program instruction set. If successful, code distributor 114 can distribute the program instruction set to the second group of tenants. If unsuccessful, code distributor 114 can postpone the distribution of the program instruction set until the developer is able to evaluate and update the program instruction set.

[0034] In various respects, if the distribution of the instruction set to the first group of tenants is unsuccessful, developer 102 can update the distributed instruction set. For example, in some cases, the functionality associated with the instruction set may not be performing correctly for the first group of tenants; in other cases, the first group of tenants may be dissatisfied with the functionality for other reasons (e.g., based on user preference rather than code failure). In either case, the instruction set can be updated to improve functionality performance or user satisfaction with the functionality. In some cases, for example, updates to the instruction set can be developed based on the analysis of telemetry data. In this case, code distributor 114 can interrupt the phased distribution of the instruction set and can subsequently continue the phased distribution of the updated instruction set to the second group of tenants. Furthermore, the updated instruction set can be distributed to the first group of tenants.

[0035] Telemetry data receiver 116 receives status information from servers in server farm 146. This status information measures or describes how a tenant's end user 104 interacts with the functionality of the application service. For example, telemetry data 170 may include logs related to how the end user accesses the application service (e.g., click data, input queries and output results, success reports or error reports, etc.). In various aspects, the telemetry data may be updated or used in conjunction with dynamic parameters stored in server usage database 126. Alternatively, telemetry data 170 may include error logs or exceptions associated with program instructions that perform the functionality. In various aspects, developer 102 may receive telemetry data 170 to analyze the performance of program instructions. Developer 102 may update the program instructions to correct any issues before further distribution.

[0036] As should be realized, regarding Figure 1 The various methods, devices, applications, functions, etc., described are not intended to limit System 100 to being performed by the specific applications and functions described. Therefore, additional controller configurations may be used to practice the methods and systems described herein and / or to exclude the described functions and applications without departing from the methods and systems disclosed herein.

[0037] Figure 2 An example method of distributing program instructions according to aspects of this disclosure is illustrated.

[0038] The general order of operations in method 200 is as follows: Figure 2 As shown in the diagram. Typically, method 200 begins with start operation 202 and can be repeated after selection operation 218. Method 200 may include more or fewer steps and / or may be combined with... Figure 2The steps shown are performed sequentially. Method 200 can be executed as a set of computer-executable instructions that are executed and encoded by a computer system or stored on a computer-readable medium. Furthermore, method 200 can be executed by gates or circuits associated with a processor, ASIC, FPGA, SOC, or other hardware device. In the following, method 200 will be referred to in conjunction with... Figure 1 , Figure 3 , Figures 4A-4B , Figures 5A-5C , Figure 6 , Figure 7 , Figure 8A and Figure 8B The description is used to explain the system, components, devices, modules, software, data structures, data function representations, signaling diagrams, methods, etc.

[0039] In identification operation 204, program instruction sets (i.e., code) can be identified for distribution. By distributing program instruction sets to tenants, functionality associated with the application service can be published, enhanced, or replaced for end-user access. In various aspects, developers can provide program instruction sets to code storage device 120. Code recognizer 110 can identify program instruction sets and enable tenant ranker 112 to rank tenants to determine one or more tenants to whom code distributor 114 should distribute program instruction sets. In various aspects, identification operation 204 can further identify the functionality (or functionality type) involved in the program instruction set. For example, the program instruction set may involve a file upload function (or operation) of the application service. Another program instruction set may involve a file sharing function (or operation). Function types may include, but are not limited to, creating a new file, updating a file, sharing a file, editing a file simultaneously, moving a file, renaming a file, publishing a file, deleting a file, and unpublishing a file. By identifying the functionality type, tenant ranking can be based on the functionality of the application service to more targetedly distribute program instruction sets, preventing premature distribution to tenants who actively use the functionality or specific functionality types.

[0040] In operation 206, parameter values ​​for one or more parameters for each of the multiple tenants of the application service can be determined. In each respect, one or more parameters and their corresponding values ​​can be stored in at least one of the tenant database 122 or the server usage database 126.

[0041] In comparison operation 208, tenants can be compared based at least on the parameter value of each of one or more parameters for each tenant. In addition to parameter values, each parameter can also be associated with a weight value. Thus, different parameters can have varying importance when ranking tenants. For example, a higher weighted parameter can have a greater impact on tenant ranking than a lower weighted parameter. When comparing tenants, parameter values ​​can be combined with the weight value of each of one or more parameters for each tenant. Parameter weight table 124 can store one or more parameters with corresponding weight values. Therefore, each tenant can be associated with a "score" (or total tenant value) representing the sum of the parameter values ​​of one or more parameters (as modified by the corresponding weight value). It should be understood that each tenant's score can change dynamically based on the function (or function type) to be distributed and any changes to the parameter values ​​of dynamic parameters.

[0042] In operation 210, tenant rankings can be determined and assigned. Tenant rankings can be determined based on a comparison of total tenant values ​​(or scores) of one or more parameters (modified by weight values) stored in tenant database 122, parameter weight table 124, and server usage database 126. For example, tenant database 122 can provide various levels of service priority to individual tenants. The service priority level of each tenant can be predetermined by determining the tenant's sensitivity to errors and / or interruptions in the application service. Alternatively, some tenants can contract for service levels based on specific guarantees from the software provider. The more sensitive a tenant is to interruptions in the application service, the higher its service priority level. These high-priority tenants are ranked higher, causing the distribution of program instructions to be delayed until the program instruction set stabilizes after use and testing by other tenants. In all respects, the program instruction set can be considered stable after multiple distribution iterations (e.g., more than two). In all respects, the number of tenants increases with each distribution iteration. In other respects, the total number of end users increases with each distribution iteration. The program instruction set can be updated after each distribution iteration. In this way, problems in program instructions can be identified and corrected in early iterations when the total load on features and / or compute resources is still low. Iterative distribution, along with telemetry data collection, ensures early detection and problem resolution. Therefore, issues can be resolved before high-ranking tenants with greater involvement in feature usage receive the distribution. In some respects, the goal of iterative distribution is to first update features to the 1% of low-ranking (e.g., low-usage) tenants (e.g., generating 1% of the total load on feature usage), rather than updating features to the 1% of high-ranking (e.g., high-usage) tenants who might generate 10% of the total load. In this way, problems can be identified and corrected with minimal impact on customer experience. In fact, due to low involvement in new or updated features, low-ranking tenants may not even notice the problem until it is resolved. In other respects, some tenants may request new or updated features early in the iterative distribution. For such tenants, the service priority level is set to low, and program instruction sets can be distributed to these tenants during early iterations. By determining and assigning tenant rankings on a continuum from low to high, operation 210 further determines the order in which codes are distributed to tenants from low to high.

[0043] In all aspects, operation 210 dynamically updates the tenant-determined ranking as the values ​​of various parameters change over time during iterations of distributing the instruction set. Each iteration of distributing and testing the instruction set can be time-consuming. For example, regression testing can last more than 24 hours. For example, during regression testing, some different parameters, service priority levels, peak season status, CPU utilization, support uptime, and tenant feature-specific interests can change. For example, a tenant showing a specific interest in a feature (e.g., high usage) may no longer be interested in that feature (e.g., low usage). Such a tenant should have been ranked lower to receive instruction set distribution earlier in the iteration, but these iterations may have passed. In this case, the tenant can be selected for distribution in the next iteration (regardless of whether the tenant would have been selected for that iteration based on the initial ranking). Tenant groupings can be updated based on the updated rankings. By dynamically updating tenant rankings, subsequent iterations of distribution can include any tenants whose rankings dropped during previous iterations.

[0044] In some respects, determining the service priority level used at Operation 210 can be based on a list of preferred (VIP) tenants or a whitelist of customers. Some tenants may be more sensitive to service disruptions than others. For example, tenants in specific industries such as finance and public services may be more sensitive to data interruptions and loss due to their impact on their mission-critical operations. In some respects, the weight of the preferred tenant list can be greater than the weight of statistical usage information values. That is, preventing unnecessary disruptions to preferred tenants can be a priority for the software provider.

[0045] In some respects, operation 210 can rank tenants with higher monthly average user (MAU) values ​​higher. At least after the second iteration of distribution, higher-ranked tenants will receive program instructions. This reduces the risk of service interruptions for higher-ranked tenants. Server usage database 126 provides statistical usage information for tenants of servers and other distributed resources. These parameters can include MAU, which indicates the average number of end-users a tenant uses over a one-month period. MAU can also indicate the average number of end-users a tenant uses for a specific function or function type (e.g., file sharing, file upload) over a one-month period. Furthermore, server usage database 126 can store the number of requests per second (RPS) made by a tenant. For example, a tenant's end-users can make requests to the application service to search for and download files. The number of requests made over a period of time indicates the level of activity a tenant uses the application service.

[0046] Operation 210 can also use the RPS value to assign a higher rank to tenants with higher RPS values. This allows tenants with more active usage to receive code distribution when the code is more stable. In some respects, the server usage database 126 stores the RPS for each function (or function type) of the application service. Thus, operation 210 can assign different rank to tenants based on the specific function (or function type) associated with the program instruction set used for distribution. Alternatively, MAU and RPS values ​​can be stored in the tenant database 122 and / or the server usage database 126.

[0047] Operation 210 can also use weight values ​​to modify the parameter values ​​of individual parameters. In some respects, parameter weight table 124 provides the weight values ​​of each parameter. Some parameters (such as tenant service priority level) may be more important than others (such as CPU usage). Weight values ​​can be predetermined and set manually. Alternatively, weight values ​​can be automatically updated based on relevance. For example, relevance could be a function of the application service used to distribute program instruction sets, MAU, or a time period of one day, one month, or one year. In the insurance industry, the MAU of all tenants across all servers (virtual or physical) can increase significantly after an incidental event (such as a severe earthquake, fire, or flood in multiple locations). In this case, agents acting as end users (such as tenants) of various insurance companies will need to update documents, claims records, etc. This surge in usage can change the tenant's MAU. Even so, despite the surge in usage, the MAU (average) can remain at a low weight value, and affected tenants can still be ranked lower in the rankings, which will result in receiving code distribution earlier in the iteration during the usage surge. In this case, when this situation is detected in certain parts of the tenant (in this case, the tenant in the insurance section), the weight value can be automatically increased.

[0048] At distribution operation 212, a set of program instructions (or code) can be distributed to one or more tenants based on the ranking assigned by determination operation 206. For example, distribution operation 212 can enable a set of program instructions on one or more virtual servers 142 for a first group of target tenants (e.g., tenant A106A). Once the code has been distributed, end users of the first group of target tenants can use new or updated functionality associated with the application service. In some respects, distribution operation 212 can be paused before subsequent distributions begin. The duration of the pause can be predetermined based on the number of iterations of distributing the program instruction set. The pause between two distributions of the program instruction set can determine the throttle (i.e., rate) of the distribution.

[0049] In receiving operation 214, telemetry data associated with the interaction of an end user with a function encoded by the instruction set can be received from one or more server farms where one or more target tenants reside. The telemetry data may include logs of the execution of the instruction set (or code). For example, telemetry data may be collected after the code distributor 114 has distributed the instruction set to server farm 146. In some aspects, the telemetry data can be analyzed to identify specific portions of the instruction set that are causing problems when the instruction set is executed.

[0050] In update operation 216, the instruction set can be updated based on telemetry data. When the instruction set is executed in the application service, issues can be identified for forms of errors experienced by end users (e.g., unpredictable performance, buffering or slow performance, inability to perform operations, etc.). Alternatively, the code may execute correctly, but the user may be dissatisfied with the functionality (e.g., multiple clicks are required to perform an operation, the completion notification is red instead of green, etc.). In either case, developer 102 can update portions of the instruction set to address the issues identified in the telemetry data (or other data such as user complaints). The updated instruction set can then be identified for subsequent distribution of the feature. Update operation 216 is optional. In various respects, the telemetry data may indicate that an update to the instruction set is not required. Therefore, the instruction set can remain unchanged for further distribution.

[0051] In operation 218, a second group of target tenants can be selected for the next distribution of this instruction set. When telemetry data indicates that an update to the instruction set is not required, a broader group of higher-ranked tenants can be selected as the second group of target tenants to receive the distribution. Alternatively, telemetry data may indicate that an update to the instruction set is needed due to user dissatisfaction with the corresponding functionality or errors encountered while performing that functionality. Once the instruction set is updated, the updated instruction set can be distributed to the lowest-ranked first group of target tenants to restart regression testing. Higher-ranked tenants can receive the distribution after earlier iterations.

[0052] It should be understood that operations 202-218 are described for the purpose of illustrating the method and system and are not intended to limit this disclosure to a particular sequence of steps. For example, the steps may be performed in a different order, additional steps may be performed, and the steps disclosed may be excluded without departing from this disclosure.

[0053] Figure 3 The illustration shows an example method for assigning rankings to tenants in accordance with aspects of this disclosure.

[0054] The general order of operations in method 300 is as follows: Figure 3As shown in the diagram. Typically, method 300 begins with start operation 302 and ends with end operation 318. Method 300 may include more or fewer steps, or may be combined with... Figure 3 The different methods shown arrange the order of steps. Method 300 relates to a method for assigning rankings to tenants. Method 300 relates to... Figure 2 The determination operation 210 is as follows. In some respects, method 300 is repetitive. As the tenant uses the application service, the tenant's parameter values ​​and ranking can be continuously updated.

[0055] Method 300 can be executed as a set of computer-executable instructions that are executed and encoded by a computer system or stored on a computer-readable medium. Furthermore, method 300 can be executed by gates or circuits associated with a processor, ASIC, FPGA, SOC, or other hardware device. In the following, method 300 will be referred to in conjunction with... Figure 1 , Figure 2 , Figures 4A-4B , Figures 5A-5C , Figure 6 , Figure 7 , Figure 8A and Figure 8B Explain the described systems, components, devices, modules, software, data structures, data feature representations, signaling diagrams, methods, etc.

[0056] In selection operation 304, a tenant can be selected from the tenant database 122. In some aspects, selection operation 304 can select a tenant each time an instruction for distributing a function is received. In other aspects, selection operation 304 can select a tenant if parameter values ​​have changed since the last tenant ranking was assigned. The tenant database 122 may include an indicator for each tenant indicating whether any parameter values ​​have changed since the last tenant ranking was assigned. In some aspects, the function or function type can be determined based on the identified set of program instructions used for distribution.

[0057] In retrieval operation 306, the service priority level of a tenant can be retrieved from tenant database 122. Service priority levels for each tenant can be pre-determined by determining their sensitivity to functional failures and / or service interruptions. The more sensitive a tenant is to application service interruptions, the higher the service priority level assigned to that tenant. Higher-priority tenants require higher ranking, causing the distribution of program instructions to be delayed until the program instruction set is more likely to stabilize after use and testing by other tenants.

[0058] At retrieval operation 308, the monthly active users (MAU) of a tenant can be retrieved from tenant database 122. In some respects, the higher a tenant's MAU, the more stable the application service should be. Therefore, program instructions should only be distributed to tenants after iterative regression testing has demonstrated execution stability. In some respects, a tenant's MAU may be related to the received features or feature types that determine each tenant's ranking.

[0059] In retrieval operation 310, the requests per second (RPS) and CPU utilization of the servers being used by the tenant can be retrieved. RPS and CPU utilization indicate the load level of the server farm. High RPS and high CPU utilization indicate that the tenant is heavily using the computing resources of the application service. Therefore, these tenants can be ranked higher to prevent them from receiving distributions until after several iterations of the distribution, in order to minimize service interruptions. In some aspects, a tenant's RPS and CPU utilization can relate to the received functions or function types processed to determine its ranking for each tenant.

[0060] In retrieval operation 312, a value indicating whether a tenant is currently in a peak usage season (peak season) can be retrieved from tenant database 122. For example, this value can be binary, with "1" indicating a peak usage season and "0" indicating an off-peak season. In various respects, any seasonal information regarding tenant usage can be predetermined and set in tenant database 122. For example, tenants in accounting services may have seasonal cycles during months such as January (e.g., a typical month for completing year-end financials), April (e.g., a typical month for completing first-quarter financials), July (e.g., a typical month for completing second-quarter financials), and so on. Alternatively, March can be a peak season due to tax preparation work. When the distribution time matches a peak season value (e.g., "1"), it can be determined that a tenant is in a peak usage season. In other respects, the peak season value can be automatically set and updated by system 100 based on tenant server usage information in server usage database 126. In some respects, the value indicating the peak season parameter can relate to the tenant's received function or function type ranking for each tenant.

[0061] In retrieval operation 314, tenant status information can be retrieved from tenant database 122. For example, status information may indicate whether a tenant is experiencing a problem with a feature related to a program instruction set used for distribution. When a tenant is experiencing a problem with a feature associated with code distribution, the tenant is likely to want the problem resolved as quickly as possible. Therefore, regardless of the tenant's ranking, when the program instruction to be distributed is in response to a tenant's request to resolve a problem (e.g., the program instruction set codes for a fix on a feature), the program instruction set that corrects the problem will be distributed to the tenant in an earlier iteration. If this parameter exists, the tenant's ranking can be overridden. Alternatively, this parameter can be associated with a "negative" weight value, causing the tenant's ranking to be lowered so that they receive code distribution in an earlier iteration. It is important to note that regardless of the tenant's ranking, when the program instruction to be distributed is in response to a tenant's request to resolve a problem (e.g., the program instruction set codes for a fix on a feature), the program instruction set that corrects the problem will be distributed to the tenant in an earlier iteration. RPS and CPU utilization values ​​can be retrieved from server usage database 126.

[0062] In operation 316, tenant rankings can be determined and assigned. Rankings are determined by combining information about various parameters, as described above, along with the applied weight values ​​for each parameter. For example, information such as the service priority level, MAU, RPS, CPU utilization, peak season information, and tenant status information related to the function can be normalized. For example, the normalized value for each parameter can be a value between 0 and 100. The normalized value can then be multiplied by the weight value of each parameter, and a score can be assigned to the tenant, for example. Tenant rankings can be assigned by comparing and sorting the resulting scores among tenants.

[0063] In some aspects, operation 316 uses weighted values ​​of various parameters for comparing and ranking tenants. Parameters for comparison may include, but are not limited to, each tenant's service priority level, each tenant's monthly average users (MAU), and each tenant's CPU utilization. Each parameter used for comparison may have a different weight value to emphasize a particular parameter more than others when assigning a ranking to each tenant. Based on the assigned ranking, tenant ranker 112 generates a ranked tenant list 128. For example, tenants with higher service priority levels may be ranked higher than tenants with lower service priority levels. This allows tenants with a higher priority in avoiding service issues to be ranked higher and receive program instructions in later iterations of distribution. In later iterations of distribution, after more regression testing, the program instruction set is more stable and more likely to meet customer expectations for the feature design. Similarly, tenants with higher monthly average users (MAU) for application services may be ranked higher than tenants with lower MAU to achieve a more stable code distribution. In contrast, tenants with lower usage are less likely to experience service outages in earlier iterations of code distribution. Furthermore, tenants with low usage are less likely to file complaints, for example, before receiving fixes for functional issues with the application service. Similarly, tenants with low usage are less likely to file complaints because subsequent distribution iterations are likely to include fixes for any functional issues. In either case, service outages and customer dissatisfaction will be reduced.

[0064] In some respects, operation 316 corresponds to Figure 2 The determination operation 210 is then performed. Therefore, comparison operation 208 proceeds to determination operation 210 (or 316) to rank each tenant. Determination operation 210 (or 316) proceeds to... Figure 2 The distribution operation 212 is performed within this process. In other aspects, the determination operation 316 returns to the selection operation 304, where the method continues to select, evaluate, and rank tenants dynamically based on changing circumstances (e.g., tenant usage). The repeated steps of method 300 allow for dynamic updating or reassignment of rankings to tenants.

[0065] It should be understood that operations 302-318 are described for the purpose of illustrating the method and system and are not intended to limit the disclosure to a particular sequence of steps. For example, the steps may be performed in a different order, additional steps may be performed, and the disclosed steps may be excluded without departing from the disclosure.

[0066] Figures 4A-4B A sample data structure for a tenant database and a schematic diagram of tenant rankings for distributing program instructions are shown according to aspects of this disclosure.

[0067] Figure 4A The illustration shows an example list of ranked tenants (e.g., list 128). Each row of the ranking list 400A corresponds to one tenant. As illustrated, the table entries are sorted based on the tenant's ranking. Column 402 indicates the ranking of each tenant. In all respects, there can be 1000 tenants, ranging from the highest "1" to the lowest "1000". In some cases, tenants can be assigned to a group. As shown in the list of ranked tenants, there are ten groups of tenants. A ranking value of 1 (a) depicts the highest priority of the corresponding tenant, thus receiving the distribution of the program instruction set (i.e., code) later in the iteration. Column 404 indicates the tenant's group identifier. Tenants with the same group identifier belong to the same group. For example, group 9 includes four tenants: such as "A1-Industry", "HealthTech", "P School", and "CC College". Group 1 includes the tenant "Critical-Research", which has the highest ranking tenant. Column 406 indicates the tenant's identifier. Column 408 indicates the tenant's name. The Farm ID column 410 indicates the identifier of the server farm (e.g., server farm 146) assigned to the tenant. For example, the Farm ID "0100" is assigned to the tenant named "Y Finance" with a tenant ID of "40". The Machine ID column 412 indicates the identifier of the server or group of servers assigned to the tenant. For example, the Server ID "0123" is assigned to the tenant "Y Finance" with a tenant ID of "40".

[0068] Tenant rankings can be assigned based on a combination of one or more parameter values, including service priority level, MAU, RPS, CPU utilization, and whether the tenant is in peak season. The parameters are not limited to these. In some cases, each parameter can have a different weight value, indicating the importance of each parameter relative to others in determining tenant ranking.

[0069] Figure 4BThe diagram illustrates a ranking of tenants for sorting the distribution of program instructions according to aspects of this disclosure. The horizontal bar 400B ranges from “COLD” (i.e., lower ranking) to “HOT” (i.e., higher ranking). The numbers within the horizontal bar indicate the spectrum of rankings 402 for one thousand tenants, from ranking 1000th (lowest) on the left to ranking 1st (highest) on the right. Tenants with lower rankings are “cold” (e.g., due to less usage of the application service). Less activity from tenants translates to a lower risk of being affected by potential errors resulting from the distribution of program instructions. Conversely, tenants with higher rankings are “hot” (e.g., due to higher activity in the application service or higher sensitivity to errors caused by the distribution of program instruction sets). As indicated in the parentheses next to group 404, tenants with a ranking value of “1” have group ID “1”, tenants with ranking values ​​of “2” and “3” have group ID “2”, and so on. Since the distribution of instruction sets occurs from low-risk tenants to high-risk tenants, tenants with group ID "10" (lowest rank) receive the instruction set distribution first. Tenants with group ID "1" (highest rank) receive the distribution during the last iteration of the distribution.

[0070] Figures 5A-5C An example of a data structure for distributing a set of program instructions according to an example system based on aspects of this disclosure is illustrated.

[0071] Figure 5A The illustration shows a sample tenant table in a tenant database (e.g., tenant database 122) of an example system according to aspects of this disclosure. Tenant table 500A includes columns for Tenant ID (502), Tenant Name (504), Farm ID (506), Server ID (508), Priority (510), MAU (512), RPS (514), and Peak Season (516). Each row of the table corresponds to a distinct tenant.

[0072] For example, Tenant ID 502 indicates the tenant's identifier (e.g., 03). Tenant Name 504 indicates the tenant's name (e.g., "PSchool"). Farm ID 506 indicates the identifier of the server farm assigned to the tenant (e.g., "6500"). Server ID 508 indicates the identifier of the server assigned to the tenant (e.g., "6532"). The server can be a virtual server or a physical server, depending on the server farm's resource allocation. Priority 510 indicates the tenant's service priority level (e.g., "1"). A higher level can indicate a higher priority for the service provided to the tenant. MAU 512 indicates the tenant's monthly active users (e.g., "110"). Tenants with a higher MAU indicate higher usage of the application service. RPS 514 indicates the number of requests per second made by the tenant's end users (e.g., "180"). Tenants with a higher RPS indicate more frequent use of the application service. Peak Season 516 indicates whether the tenant is currently in a peak season with high end-user usage (e.g., "Yes" or "1") or a low season with low end-user usage (e.g., "No" or "0").

[0073] Figure 5B The illustration shows an example data structure for weight values ​​of various parameters (e.g., parameter weight table 124) according to an example system based on aspects of this disclosure. In each aspect, the weighted parameter table 500B includes a parameter ID column 522, a weight value column 524, and a parameter name column 526. The parameter ID 522 indicates the identifier of the parameter (e.g., "1"). The weight value 524 indicates the weight value (e.g., "1") used to apply the importance of the parameter when assigning rankings to tenants. In each aspect, a parameter with a higher weight value indicates that the parameter has more influence on ranking tenants than other parameters with lower weight values. The parameter name 526 indicates the name of the parameter (e.g., "tenant service priority level"). In each aspect, the parameters in the weighted parameter table 500B can correspond to, as in... Figure 3 The parameters described in operations 306-316 are for retrieving the parameters.

[0074] The following describes an example of how to combine parameter values ​​and weight values ​​to assign rankings to tenants. For example, in Figure 4A In the rankings, tenant "YFinance" ranks lowest in the 400A list. (For example, in...) Figure 5A As described in tenant table 500A, tenant "Y Finance" has a low service priority level ("1"), with low MAU and RPS of 8 and 1 respectively. The tenant is not currently in peak season. Alternatively, for example, tenant "247Security" would rank higher in ranking list 400A, at number 9. Figure 5AAs described in tenant table 500A, tenant "247Security" is not in peak season. However, it has the highest service priority level, at 10. Service priority level values ​​have significant weight when combining these values ​​to assign rankings. In fact, the weight value of a tenant's service priority level is... Figure 5B It has the highest weight in the 500B weight table.

[0075] Figure 5C The illustration shows an example data structure for the distribution of a scheduler instruction set according to an example system based on aspects of this disclosure. The example scheduling table 500C includes columns for distribution ID (530), code ID (532), tenant group ID (534), start time (536), and pause duration (538). Each row of the table may correspond to the occurrence of a program instruction set distribution.

[0076] Distribution ID 530 indicates an identifier for the distribution phase (e.g., "1"). Code ID 532 indicates an identifier for the program instruction set (e.g., "1"). This program instruction set may be stored in code storage device 120. Tenant group ID 534 indicates an identifier for the tenant group (e.g., "1"). Tenant group ID 534 may correspond to the value of group 404 in ranking list 400A. Start time 536 indicates the time when the first distribution phase begins for the program instruction set with code ID "1" (e.g., "202004100200"). Pause duration 538 indicates a pause or time period (e.g., 120 minutes) that occurs between the completion time of the current distribution and the start time of the next distribution. In each respect, the count of pause duration corresponds to... Figure 6 The use of timer 610. During the pause period after distribution, end users can use the distributed program instruction set to use the server application.

[0077] Therefore, example scheduling table 500C includes scheduling for three distributions with program instructions having distribution IDs 1, 2, and 3. The first distribution enables the program instruction set with code ID = 1 to tenants in tenant group ID = 10, starting at 2:00 AM on April 10, 2020, with a 120-minute pause after distribution. The second distribution enables the same program instruction set with code ID = 1 to tenants in tenant group ID = 9, starting at 3:00 AM on April 11, 2020, with a pause duration of 240 minutes. The third distribution uses the program instruction set with code ID = 2 to tenants in tenant group ID = 10, starting at 2:00 AM on April 12, with a pause duration of 480 minutes. In all respects, the longer 480-minute pause period is scheduled for the third distribution because code ID = 2 corresponds to the updated program instruction set, which will resolve the issues discovered based on using code ID = 1 with distribution ID = 2.

[0078] As should be realized, regarding Figures 5A-5C The various methods, devices, applications, functions, etc., described are not intended to restrict the data structures to be performed by the specific applications and functions described. Therefore, additional controller configurations may be used to practice the methods and systems described herein and / or exclude the described functions and applications without departing from the methods and systems disclosed herein.

[0079] Figure 6 The illustration depicts a schematic example of distributing a set of program instructions according to an example process 600 based on aspects of this disclosure. In some aspects, process 600 processes code distribution to three distinct tenant groups: tenant group 10, tenant group 9, and finally tenant group 1. These groups can rank tenants based on specific functionality (or functionality type). Process 600 illustrates multiple iterations (or stages) of code distribution. In each aspect, tenant group 10 corresponds to... Figure 4A Tenant ID 40 corresponds to group ID 10. Tenant group 9 corresponds to tenant with group ID 9. Tenant group 8 corresponds to... Figure 4A Tenants in group ID 8 are grouped together. In various ways, tenant grouping is based on tenant ranking related to the functionality or feature type of the application service being updated. For example, tenants in tenant group 10 are the least active in using the functionality of the total tenants. In contrast, those tenants in tenant group 1 are the most active in using the functionality (or feature type) of the application service. Tenants in tenant group 9 can receive updates to the application instruction set after two iterations of distributing the product instruction set. In some other ways, tenant ranking and grouping can be based on various parameters across all functionality of the application service.

[0080] At code distribution operation 602 during its first iteration, a set of program instructions (i.e., code) can be distributed to tenant groups. As illustrated in a group of three tenants, the set of program instructions can first be distributed to tenants in tenant group 10. After distribution, end users in tenant group 10 can use the functions associated with the distributed program instructions. Timer 610 instructs application usage 604 to continue for a predetermined period. Telemetry data can be collected when application usage 604 occurs. Telemetry data can include logs of requests and responses made between the end user and the application service on the tenant's server. The logs can include information about errors that occurred.

[0081] When the predetermined time of timer 610 has elapsed, telemetry data analysis 606 can be performed. The telemetry data can be analyzed in various ways to identify problems in the distributed program instruction set. The results of the telemetry data analysis can be provided to the developer.

[0082] In code update operation 608, developers can update the program instruction set. The updated instruction set can resolve problems identified through analysis of telemetry data. In some respects, code update operation 608 is optional.

[0083] During the second iteration, in code distribution operation 602, the (updated) program instruction set is distributed to the combination of tenant group 1 and tenant group 2. Thus, each iteration of code distribution covers an increasing number of tenants. Similar to the operation during the first iteration, end users of tenants in groups 1 and 2 that receive the updated program instruction set use the service application (i.e., application usage 604). Telemetry data is collected and analyzed (606). Furthermore, in code update operation 608, the program instruction set can be updated. During the third iteration, the combination of tenant groups 1, 2, and 3 receives the distribution of program instructions. In all respects, each iteration of program instruction set distribution and usage constitutes a "loop" or iteration. Therefore, as... Figure 6 As shown, there are three "rings".

[0084] In all aspects, the service application continues its operation, and the tenant's end users continue to use the service application by accessing servers in the cloud. Therefore, statistics in the tenant database (e.g., tenant database 122) and server usage database (e.g., server usage database 126) can be dynamically updated. Consequently, tenant rankings can be continuously updated. Tenant groups used to receive new and / or updated features of the service application can change dynamically. Iterative distribution cumulatively increases tenant coverage. Thus, because tenant rankings change dynamically with each distribution iteration, no tenant is unintentionally missed.

[0085] As should be realized, regarding Figure 6 The various methods, devices, applications, functions, etc., described are not intended to limit process 600 to being performed by the specific applications and functions described. Therefore, additional controller configurations may be used to practice the methods and systems described herein and / or may exclude the described features and applications without departing from the methods and systems disclosed herein.

[0086] Figure 7This is a block diagram illustrating the physical components (e.g., hardware) of a computing device 700 that can be used to practice aspects of this disclosure. The computing device components described below may be applicable to the aforementioned computing device. In a basic configuration, the computing device 700 may include at least one processing unit 702 and system memory 704. Depending on the configuration and type of the computing device, system memory 704 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memory. System memory 704 may include an operating system 705 and one or more program tools 706 adapted to perform the aspects disclosed herein. For example, operating system 705 may be adapted to control the operation of computing device 700. Furthermore, aspects of this disclosure may be implemented in conjunction with graphics libraries, other operating systems, or any other applications and are not limited to any particular application or system. This basic configuration in Figure 7 The components within the dashed line 708 are shown. The computing device 700 may have additional features or functionalities. For example, the computing device 700 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 7 The image is shown by a removable storage device 709 and a non-removable storage device 710.

[0087] As described above, numerous program tools and data files can be stored in system memory 704. When executed on processing unit 702, program tool 706 (e.g., code distribution application 720) can perform processes including, but not limited to, aspects as described herein. Code distribution application 720 includes code recognizer 730, tenant ranker 732, code distributor 734, and telemetry data receiver 736, as per [reference to...]. Figure 1 More detailed description. Other program tools that may be used according to aspects of this disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, presentation applications, drawing or computer-aided applications, etc.

[0088] Furthermore, aspects of this disclosure can be implemented in circuits including discrete electronic components, in packages or integrated electronic chips containing logic gates, in circuits utilizing microprocessors, or on a single chip containing electronic components or a microprocessor. For example, aspects of this disclosure can be practiced using a system-on-a-chip (SOC), wherein... Figure 7Each or many of the components shown can be integrated onto a single integrated circuit. Such a SoC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “programmed”) onto a chip substrate as a single integrated circuit. When operating via the SoC, the capabilities described herein regarding the client switching protocol can be operated via application-specific logic integrated onto a single integrated circuit (chip) along with other components of the computing device 700. Aspects of this disclosure can also be practiced using other techniques capable of performing logical operations, such as AND, OR, and NOT, including but not limited to mechanical, optical, fluid, and quantum technologies. Furthermore, aspects of this disclosure can be implemented in a general-purpose computer or any other circuit or system.

[0089] The computing device 700 may also have one or more input devices 712, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. Output devices 714, such as a monitor, speaker, printer, etc., may also be included. The above devices are examples and other devices may be used. The computing device 700 may include one or more communication connections 716 that allow communication with other computing devices 750. Examples of communication connections 716 include, but are not limited to, radio frequency (RF) transmitters, receivers and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0090] As used herein, the term computer-readable medium can include computer storage media (or multiple computer storage media). Computer storage media can include volatile and non-volatile, removable and non-removable media implemented with any method or technology for storing information such as computer-readable instructions, data structures, or program tools. System memory 704, removable storage device 709, and non-removable storage device 710 are examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 700. Any such computer storage medium may be part of computing device 700. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0091] Communication media can be embodied in computer-readable instructions, data structures, program tools, or other data in modulated data signals, such as carrier waves or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" can describe a signal having one or more functions set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media can include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0092] Figure 8A and Figure 8B A computing device or mobile computing device 800 is shown that can implement various aspects of this disclosure, such as a mobile phone, smartphone, wearable computer (e.g., smartwatch), tablet computer, laptop computer, etc. In some aspects, the client (e.g., Figure 1 The computing device of end user 104 in the reference can be a mobile computing device. Figure 8A This illustration shows one aspect of a mobile computing device 800 used to implement these aspects. In a basic configuration, the mobile computing device 800 is a handheld computer with input and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow users to input information into the mobile computing device 800. The display 805 of the mobile computing device 800 can also be used as an input device (e.g., a touchscreen display). If included, optional side input elements 815 allow for further user input. The side input elements 815 can be rotary switches, buttons, or any other type of manual input element. Alternatively, the mobile computing device 800 can incorporate more or fewer input elements. For example, the display 805 may not be a touchscreen in some respects. In yet another alternative, the mobile computing device 800 is a portable telephone system, such as a cellular phone. The mobile computing device 800 may also include an optional keypad 835. The optional keypad 835 can be a physical keypad or a “soft” keypad generated on a touchscreen display. In various aspects, output elements include a display 805 for displaying a graphical user interface (GUI), a visual indicator 820 (e.g., a light-emitting diode), and / or an audio transducer 825 (e.g., a speaker). In some aspects, the mobile computing device 800 incorporates a vibration transducer for providing haptic feedback to the user. In another aspect, the mobile computing device 800 incorporates input and / or output ports for sending signals to or receiving signals from external devices, such as audio inputs (e.g., a microphone jack), audio outputs (e.g., a headphone jack), and video outputs (e.g., a port).

[0093] Figure 8BThis is a block diagram illustrating one aspect of the architecture of a computing device, server (e.g., virtual server 142a or physical server 144), mobile computing device, etc. That is, the computing device 800 can be combined with a system (e.g., architecture) 802 to implement some aspects. The system 802 can be implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 802 is integrated as a computing device, such as integrating a digital assistant (PDA) and a wireless phone.

[0094] One or more applications 866 may be loaded into memory 862 and run on or associated with operating system 864. Examples of applications include telephone dialers, email programs, PIM (Personal Information Management) programs, word processing programs, spreadsheet programs, internet browser programs, messaging programs, etc. System 802 also includes a non-volatile storage area 868 within memory 862. The non-volatile storage area 868 can be used to store persistent information that should not be lost when system 802 is powered off. Applications 866 may use and store information, such as emails or other messages used by email applications, in the non-volatile storage area 868. A synchronization application (not shown) also resides on system 802 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep information stored in the non-volatile storage area 868 synchronized with corresponding information stored on the host computer. It should be appreciated that other applications may be loaded into memory 862 and run on the mobile computing device 800 described herein.

[0095] System 802 has a power supply 870, which can be implemented as one or more batteries. The power supply 870 may also include an external power source, such as an AC adapter or a power docking station for replenishing or charging the batteries.

[0096] System 802 may also include a radio interface layer 872, which performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 872 facilitates wireless connectivity between system 802 and the "outside world" through a communications operator or service provider. Transmissions to and from radio interface layer 872 are conducted under the control of operating system 864. In other words, communications received by radio interface layer 872 can be propagated to application program 866 via operating system 864, and vice versa.

[0097] A visual indicator 820 may be used to provide visual notifications, and / or an audio interface 874 may be used to generate auditory notifications via an audio transducer 825. In the illustrated configuration, the visual indicator 820 is a light-emitting diode (LED), and the audio transducer (not shown) is a speaker. These devices may be directly coupled to a power supply 870 such that when activated, they remain on for the duration specified by the notification mechanism, even if the processor 860 and other components may be turned off to conserve battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 874 is used to provide and receive audible signals to and from the user. For example, in addition to being coupled to an audio transducer (not shown), the audio interface 874 may also be coupled to a microphone to receive audible input, such as to facilitate telephone conversations. According to aspects of this disclosure, the microphone may also be used as an audio sensor to facilitate control of notifications, as described below. System 802 may also include a video interface 876, which enables an onboard camera 830 to operate to record still images, video streams, etc.

[0098] The mobile computing device 800 implementing system 802 may have additional features or functions. For example, the mobile computing device 800 may also include additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 8B The non-volatile storage region 868 is shown in the middle.

[0099] As described above, data / information generated or captured by mobile computing device 800 and stored through system 802 can be stored locally on mobile computing device 800, or the data can be stored on any number of storage media and accessed by the device via radio interface layer 872 or via a wired connection between mobile computing device 800 and a separate computing device associated with mobile computing device 800, such as a server computer in a distributed computing network (e.g., the Internet). It should be appreciated that such data / information can be accessed by mobile computing device 800 via radio interface layer 872 or via a distributed computing network. Similarly, based on well-known data / information transmission and storage methods, including email and collaborative data / information sharing systems, such data / information can be easily transferred between computing devices for storage and use.

[0100] The descriptions and illustrations of one or more aspects provided in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey ownership and enable others to make and use the claimed disclosure in the best possible manner. The claimed disclosure should not be construed as limited to any aspect, such as, or the details provided in this application. Whether shown and described in combination or separately, various features (structures and methods) are intended to be selectively included or omitted to produce embodiments with a particular set of features. Given the descriptions and illustrations provided in this application, those skilled in the art can contemplate variations, modifications, and alternatives that fall within the spirit of the broader aspects of the general inventive concept embodied in this application, without departing from the broader scope of the claimed disclosure.

[0101] In some aspects, a computer-implemented method for distributing program instructions is provided. The method includes identifying a set of program instructions for distribution, wherein the set of program instructions is associated with a function. The method also includes determining a parameter value for at least one parameter for each of a plurality of tenants associated with a cloud computing environment, and comparing the plurality of tenants based on the parameter value of at least one parameter for each tenant. Based on the comparison, the method includes assigning a rank to each of the plurality of tenants, wherein at least a first tenant is assigned a lower rank than at least a second tenant. Based on the assigned rank, the method includes: distributing the program instruction set to at least the first tenant, and receiving telemetry data regarding the execution of the program instruction set by at least the first tenant. Based on the telemetry data, the method includes updating the program instruction set and distributing the updated program instruction set to at least the second tenant.

[0102] In another aspect, a system is provided. The system includes at least one processor and at least one memory storing computer-executable instructions that, when executed by the at least one processor, cause the system to perform operations. These operations include identifying a set of program instructions for distribution, wherein the set of program instructions is associated with a function. The operations also include determining a parameter value for at least one parameter for each of a plurality of tenants associated with a cloud computing environment, and applying weight values ​​to the parameter value of at least one parameter for each of the plurality of tenants. Furthermore, the operations include combining the parameter values ​​and weight values ​​to form a combined value for at least one parameter for each tenant, and comparing the plurality of tenants based on the combined value of at least one parameter for each tenant. Based on this comparison, the operations include assigning a ranking to each of the plurality of tenants, wherein at least a first tenant is assigned a lower ranking than at least a second tenant. Based on the assigned ranking, the operations further include distributing a set of program instructions to at least the first tenant.

[0103] In another aspect, a computer storage medium is provided. The computer storage medium stores computer-executable instructions that, when executed by a processor, cause a computer system to perform operations. These operations include identifying a program instruction set for distribution, wherein the program instruction set is associated with a function. The operations also include determining a parameter value for at least one parameter for each of a plurality of tenants associated with a cloud computing environment, and applying weight values ​​to the parameter value of at least one parameter for each of the plurality of tenants. Furthermore, the operations include combining the parameter value and the weight value to form a combined value for at least one parameter for each tenant, and comparing the plurality of tenants based on the combined value of at least one parameter for each tenant. Based on this comparison, the operations include assigning a ranking to each of the plurality of tenants, wherein at least a first tenant is assigned a ranking lower than that of at least a second tenant. Based on the assigned ranking, the operations further include distributing the program instruction set to at least the first tenant.

[0104] Any combination of one or more of the above aspects with any other one or more aspects. Any one of the one or more aspects described herein.

Claims

1. A computer-implemented method for distributing program instructions, the method comprising: Identify a program instruction set for distribution, wherein the program instruction set is associated with a function; For each of the multiple tenants associated with the cloud computing environment, determine the value of at least one parameter; The plurality of tenants are compared based on the parameter value of at least one parameter for each tenant; Based on the comparison, a ranking is assigned to each of the plurality of tenants, wherein at least the first tenant is assigned a lower ranking than at least the second tenant; Based on the assigned ranking, the program instruction set is distributed to at least the first tenant; Receive telemetry data regarding the execution of the program instruction set by at least the first tenant; Based on the telemetry data, update the program instruction set; and The updated program instruction set is distributed to at least the second tenant.

2. The computer-implemented method as described in claim 1, wherein, The at least one parameter includes at least one of the following: The number of active users within a given time period; The number of requests for the function during the said time period; Service priority level; Peak usage periods falling within or outside the stated time period; or The number of issues reported.

3. The computer-implemented method as described in claim 1, further comprising: It was determined that the set of program instructions resolved the problem experienced by the second tenant; as well as Regardless of the ranking of the second tenant's assignment, the program instruction set will be distributed to the second tenant.

4. The computer-implemented method as described in claim 1, wherein, The at least one parameter is associated with a weight value, and assigning the ranking further includes: The parameter values ​​and the weight values ​​are combined to form a combined value for at least one parameter for each tenant; The plurality of tenants are compared based on the combined values ​​of the at least one parameter for each tenant; and Based on the comparison, the ranking is assigned to each of the plurality of tenants.

5. The computer-implemented method of claim 1, further comprising: For each of the plurality of tenants, determine the usage level of the functionality provided by the program instruction set.

6. The computer-implemented method of claim 1, further comprising: For each of the plurality of tenants, update the parameter value of the at least one parameter; The plurality of tenants are compared based on the updated parameter value of at least one parameter for each tenant; Based on the comparison, a ranking is reassigned to each of the plurality of tenants; as well as Based on the ranking of the reassignment, the program instruction set is distributed to at least the second tenant.

7. The computer-implemented method as described in claim 1, wherein, The first parameter value of the first tenant's at least one parameter is less than the second parameter value of the second tenant's at least one parameter.

8. A system for distributing program instructions, comprising: At least one processor; as well as At least one memory storing computer-executable instructions that, when executed by the at least one processor, cause the system to: Identify a program instruction set for distribution, wherein the program instruction set is associated with a function; For each of the multiple tenants associated with the cloud computing environment, determine the value of at least one parameter; A weight value is applied to the parameter value of at least one parameter for each of the plurality of tenants; The parameter values ​​and the weight values ​​are combined to form a combined value for at least one parameter for each tenant; The plurality of tenants are compared based on the combined values ​​of the at least one parameter for each tenant; Based on the comparison, a ranking is assigned to each of the plurality of tenants, wherein at least the first tenant is assigned a lower ranking than at least the second tenant; and Based on the assigned ranking, the program instruction set is distributed to at least the first tenant.

9. The system of claim 8, wherein the computer-executable instructions, when executed, further cause the system to: Receive telemetry data regarding the execution of the program instruction set by at least the first tenant; Based on the telemetry data, update the program instruction set; and The updated program instruction set is distributed to at least the second tenant.

10. The system of claim 8, wherein, The at least one parameter includes at least one of the following: The number of active users within a given time period; The number of requests for the function during the said time period; Service priority level; Peak usage periods falling within or outside the stated time period; or The number of issues reported.

11. The system of claim 8, wherein the computer-executable instructions, when executed, further cause the system to: It was determined that the program instruction set resolved the problem of the functionality experienced by the second tenant; and Regardless of the ranking of the second tenant's assignment, the program instruction set will be distributed to the second tenant.

12. The system of claim 8, wherein the computer-executable instructions, when executed, further cause the system to: The first tenant is notified before the program instruction set is distributed.

13. A computer storage medium storing computer-executable instructions, which, when executed by a processor, cause a computer system to: Identify the program instruction set for distribution, among which, The program instruction set is associated with the function; For each of the multiple tenants associated with the cloud computing environment, determine the value of at least one parameter; A weight value is applied to the parameter value of at least one parameter for each of the plurality of tenants; The parameter values ​​and the weight values ​​are combined to form a combined value for at least one parameter for each tenant; The plurality of tenants are compared based on the combined values ​​of the at least one parameter for each tenant; Based on the comparison, a ranking is assigned to each of the plurality of tenants, wherein at least the first tenant is assigned a lower ranking than at least the second tenant; as well as Based on the assigned ranking, the program instruction set is distributed to at least the first tenant.

14. The computer storage medium of claim 13, wherein the computer-executable instructions, when executed, further cause the computer system to: In the first phase, the program instruction set is internally distributed to a set of users associated with the software provider; In the second phase, based on the lower ranking, the program instruction set is distributed to at least the first tenant; and In the third phase, the updated program instruction set is distributed to at least the second tenant.

15. The computer storage medium of claim 13, wherein the computer-executable instructions, when executed, further cause the computer system to: For each of the plurality of tenants, update the parameter value of the at least one parameter; The updated parameter values ​​and weight values ​​are combined to form an updated combined value for each tenant's at least one parameter; The plurality of tenants are compared based on the updated combined value of at least one parameter for each tenant; Based on the comparison, a ranking is reassigned to each of the plurality of tenants; as well as Based on the ranking of the reassignment, the program instruction set is distributed to at least the second tenant.

Images