Deploy packages to devices in a cluster in a hierarchical manner

Through machine learning-driven scanning and hierarchical deployment methods, the security and reliability challenges of firmware package deployment in global distributed data centers are solved, achieving efficient and low-impact deployment results.

CN113692573BActive Publication Date: 2024-11-12MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202080027645.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-04-11
Filing Date
2020-03-26
Publication Date
2024-11-12
Estimated Expiration
2040-03-26

AI Technical Summary

Technical Problem

Deploying firmware or other low-level system code packages to various devices in globally distributed data centers is challenging and requires a secure, secure and reliable deployment method, especially in the face of multiple external factors and high levels of Influence potential deployment.

Method used

Using a machine learning-based approach, generates a minimal scan tree to cover selected diversity targets by scanning hardware, verifying and testing payloads, deploying packages in hierarchical to manage explosion radius, ensuring security and reliability of deployments, and using deployment monitoring The device tracks deployment progress and performance metrics.

Benefits of technology

The secure, secure and reliable deployment of firmware or low-level system code packages in globally distributed data centers reduces the impact on customer workloads and improves deployment efficiency and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113692573B_ABST
    Figure CN113692573B_ABST
Patent Text Reader

Abstract

Systems and methods for staged deployment of packages to devices in a cluster are provided. A method includes: performing a first scan of hardware in a first level that is configured to handle functions unrelated to customer workloads to determine whether a selected diversity goal for deploying packages is met. The method also includes: if the selected diversity goal for deploying packages is not met, scanning hardware in a second level that is configured to handle at least a subset of customer workloads to determine whether the selected diversity goal for deploying packages is met. The method also includes: if the selected diversity goal for deploying packages is met after the scan, using a processor, deploying the package to a first subset of a group of devices in the first level and a second subset of a group of devices in the second level.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] It is difficult to deploy packages including firmware or other low-level system code to components in a cluster of machines including hardware corresponding to the cloud. Public clouds include a global network of servers that perform various functions, including storing and managing data, running applications, and delivering content or services such as streaming video, email, office productivity software, or social media. Servers and other components can be located in data centers around the world. Although public clouds provide services to the public over the Internet, enterprises can use private clouds or hybrid clouds. Both private and hybrid clouds also include a network of servers located in a data center.

[0002] Data centers include not only servers, but also other components such as network switches, routers, and other devices. Servers and other components may be provided by different vendors and may include different types or versions of motherboards, CPUs, memory, and other devices. In addition to computing, network, and storage components, data centers also include other components such as chassis, racks, power supply units, and other such components.

[0003] Each of these devices may require low-level system code, including firmware. It is challenging to deploy packages to a variety of devices that may be distributed across many data centers around the world. Therefore, there is a need for methods and systems for deploying packages to devices in a fleet. Summary of the invention

[0004] In one example, the present disclosure relates to a method for deploying a package to a group of devices in a cluster, the cluster including hardware configurable to perform functions related to a customer workload. The method may include: performing a first scan of hardware in a first stage to determine whether a selected diversity goal for deploying the package is met, wherein the first stage includes hardware configured to handle functions unrelated to the customer workload. The method may also include: if after the first scan, the selected diversity goal for deploying the package is met, then using a processor to deploy the package to a first subset of the first stage of the group of devices based on instructions associated with the package. The method may also include: if after the first scan, the selected diversity goal for deploying the package is not met, then scanning the hardware in a second stage to determine whether the selected diversity goal for deploying the package is met, wherein the second stage includes hardware configured to handle at least a subset of the customer workload. The method may also include: if after the scan, the selected diversity goal for deploying the package is met, then using a processor to deploy the package to a first subset of the first stage of the group of devices and a second subset of the second stage of the group of devices based on instructions associated with the package.

[0005] In another example, the present disclosure relates to a method for deploying a package to a group of devices in a cluster, the cluster including hardware configurable to perform functions related to customer workloads. The method may include: performing a first scan of hardware in a first level to determine whether a selected diversity goal for deploying the package is met, wherein the first level includes hardware configured to not handle any workloads. The method may also include: if after the first scan, the selected diversity goal for deploying the package is not met, scanning the hardware in a second level to determine whether the selected diversity goal for deploying the package is met, wherein the second level includes hardware configured to handle a first selected number of customer workloads. The method may also include: if after the scan, the selected diversity goal for deploying the package is not met, scanning a third level to determine whether the selected diversity goal for deploying the package is met, wherein the third level includes hardware configured to handle a second selected number of customer workloads, wherein the second selected number is greater than the first selected number. The method may also include: if after scanning the third level scan, the selected diversity goal for deploying the package is met, using the processor, based on instructions associated with the package, deploying the package to a first subset of the group of devices in the first level, a second subset of the group of devices in the second level, and a third subset of the group of devices in the third level.

[0006] In yet another example, the present disclosure relates to a system for deploying a package to a group of devices in a cluster, the cluster including hardware that can be configured to perform functions related to customer workloads. The system can be configured to: perform a first scan of the hardware in the first level to determine whether the selected diversity goal for deploying the package is met, wherein the first level includes hardware configured to handle functions unrelated to the customer workload. The system can also be configured to: if after the first scan, the selected diversity goal for deploying the package is met, then based on instructions associated with the package, deploy the package to a first subset of the first level of the group of devices. The system can also be configured to: if after the first scan, the selected diversity goal for deploying the package is not met, scan the hardware in the second level to determine whether the selected diversity goal for deploying the package is met, wherein the second level includes hardware configured to handle at least a subset of customer workloads. The system can also be configured to: if after the scan, the selected diversity goal for deploying the package is met, then based on instructions associated with the package, deploy the package to a first subset of the first level of the group of devices and a second subset of the second level of the group of devices.

[0007] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure is illustrated by way of example and is not limited by the accompanying drawings, in which like reference numerals represent like elements. For simplicity and clarity, the elements shown in the drawings are not necessarily drawn to scale.

[0009] Figure 1 A schematic diagram of a system environment for deploying packages to devices in a cluster including hardware in a cloud according to one example is shown;

[0010] Figure 2 is a block diagram of a system including deployment and monitoring of a cluster according to an example;

[0011] Figure 3 is a block diagram of a cluster in a data center according to an example;

[0012] Figure 4 A block diagram of deployment and monitoring according to an example is shown;

[0013] Figure 5 shows a schematic diagram of a memory 500 including modules having instructions for performing operations associated with deployment and monitoring;

[0014] Figure 6 shows a portion of a package according to an example;

[0015] Figure 7 An example of a hierarchical deployment according to one example is shown;

[0016] Figure 8 A flow chart of a method for deploying a package according to an example is shown;

[0017] Fig. 9 A schematic diagram of a scan tree for hardware in a stage according to one example is shown;

[0018] Fig.10 A schematic diagram showing a scan tree of hardware in another stage according to one example;

[0019] Fig.11A and 11B A flow chart of a method for deploying a package according to an example is shown;

[0020] Fig.12 shows an impact table according to one example;

[0021] Fig.13 A machine learning system according to an example is shown;

[0022] Fig.14 According to one example, a method for Fig.13memory for instructions and data used with machine learning systems;

[0023] Fig.15 A flowchart of a method for creating a package including a payload for deployment to a group of devices according to one example is shown;

[0024] Fig.16 A flowchart showing a method for deploying a package to a cluster according to one example; and

[0025] Fig.17 A deployment dashboard according to one example is shown. DETAILED DESCRIPTION

[0026] The examples described in this disclosure involve creating a package including a payload and deploying it to a cluster. Certain examples involve creating and deploying packages based on operations derived from a machine learning model. It is difficult to deploy packages including firmware or other low-level system code to components in a cluster including hardware in the cloud. Public clouds include a global network of servers that perform various functions, including storing and managing data, running applications, and delivering content or services, such as streaming video, email, office productivity software, or social media. Servers and other components can be located in data centers around the world. Although public clouds provide services to the public over the Internet, enterprises can use private clouds or hybrid clouds. Both private and hybrid clouds also include a network of servers located in a data center.

[0027] Data centers include not only servers, but also other components such as network switches, routers, and other devices. Servers and other components may be provided by different vendors and may include different types or versions of motherboards, CPUs, memory, and other devices.

[0028] Each of these devices may require low-level system code, including firmware. It is challenging to deploy packages to a variety of devices that may be distributed in many data centers around the world. This is because the deployment of packages needs to be done safely, securely, and reliably. There are several external factors that affect safety, security, and reliability goals. For example, there are often more deployments than can be managed at a time, especially when some deployments have high impact potential. Certain types of changes or goals require explicit consent from all parties to other gated deployments (e.g., potential capacity or performance impacts). In addition, the impact of deployed packages may need to be monitored to ensure safe and reliable deployment. Finally, payloads often include firmware or other code originating from other companies and therefore must be evaluated and tested to ensure security.

[0029] To ensure safe, secure, and reliable deployment of packages, certain examples of the present disclosure involve ensuring high-quality payloads, proper validation and testing, and monitoring the impact on the fleet. Certain examples involve using machine learning to improve package creation and deployment.

[0030] Figure 1 A diagram of a system environment 100 for deploying packages to devices in a cluster of hardware in a cloud according to one example is shown. Examples of devices include, but are not limited to, baseboard management controllers (BMCs), CPUs, GPUs, FPGAs, FPGA instances, rack managers / controllers, chassis managers / controllers, power supply unit controllers, storage devices (e.g., SSDs or HDDs), networking devices (e.g., switches, routers, firewalls, and bridges), or any other device in a data center that may require an update. The package may include a payload, which may include instructions, low-level system code, firmware, settings, configurations, or other information that may need to be updated. The system environment 100 may include payload submission 102, package creation 110, package repository 120, deployment and monitoring 140, compute / storage / network 160, control plane 170, data plane 180. Machine learning or other automation techniques may be used for package creation 110, deployment and monitoring 140, as well as compute / storage / network 160, control plane 170, data plane 180.

[0031] Continue to refer Figure 1 , package creation 110 may include scanning 112, testing 114 and packages 116. In this example, a payload may be received via payload submission 102. Payload submission 102 may be implemented using a self-service portal of a payload engineering team. Therefore, in this example, payload submission 102 may provide a graphical user interface, via which any payload may be submitted. As part of payload submission 102, various relevant information may be acquired and stored in a database (e.g., a database 212 associated with deployment and monitoring). Information may be acquired by presenting a questionnaire to the submitter. Alternatively or additionally, this information may be included as metadata associated with the payload. This information may include deployment, change, test and impact related information. Payload related information or any information derived from the submitted information is referred to as parameters associated with the payload.

[0032] Still reference Figure 1, scanning 112 may include scanning the payload for various parameters associated with the payload. For example, as part of the process, metadata associated with the payload may be scanned and extracted. The extracted information may be processed to determine whether it meets the submission criteria. Meeting the submission criteria may indicate that the submission is a valid submission. After the submission is confirmed, the extracted information may be stored as a record in a database associated with deployment and monitoring 140. The information may also be timestamped. Scanning 112 may also include scanning the payload for any viruses or other undesirable artifacts. Testing 114 may include updating certain target devices with the payload to test the payload to ensure that the payload will work as expected when installed. Packaging 116 may include packaging the payload. Additional steps involved as part of the process are described in other parts of this disclosure.

[0033] Continue to refer Figure 1 , as shown in dashed box 130, machine learning can be used to deploy machine learning models associated with deployment and monitoring 140. Similarly, as shown in dashed box 150, machine learning can be used to deploy machine learning associated with compute / storage / network 160, control plane 170, and data plane 180. Other parts of this disclosure provide additional details on the use of such machine learning models.

[0034] Figure 2 is based on an example including deployment and monitoring (e.g., Figure 12 and 3. The block diagram of a system 200 of a deployment and monitoring 140 and a cluster 210. As used in the present disclosure, the term cluster may include, but is not limited to, some or all of the data centers owned by a cloud service provider, some or all of the data centers owned and operated by a cloud service provider, some or all of the data centers owned by a cloud service provider and operated by a customer of the service provider, any other combination of data centers, a single data center, or even some clusters in a particular data center. The deployment and monitoring 140 may be coupled to the cluster 210 via a data plane 220. The deployment and monitoring 140 may be coupled to the cluster 210 via a control plane 230. The cluster 210 may include one or more data centers, which in turn may include clusters of computing / storage / network devices. Thus, in this example, the cluster 210 may include cluster 1 240, cluster 2 260, and cluster Q 280, where Q may be an integer greater than 1. Cluster 1 240 may be coupled to deployment and monitoring 140 via bus 222; cluster 2 260 may be coupled to deployment and monitoring 140 via bus 224; and cluster Q 280 may be coupled to deployment and monitoring 140 via bus 226. Cluster 1 240 may be coupled to deployment and monitoring 140 via bus 232; cluster 2 260 may be coupled to deployment and monitoring 140 via bus 234; and cluster Q 280 may be coupled to deployment and monitoring 140 via bus 236. Figure 2 As shown, machine learning can be coupled to deployment and monitoring 140. Additional details related to machine learning are provided later in this disclosure. Figure 2 1, but the deployment and monitoring 140 may be coupled to the network via any number of intervening networks (e.g., wide area networks, optical networks, wireless networks, wired networks, or other types of networks). Figure 2 Thus, the term bus as used in this disclosure includes, but is not limited to, signal lines, signal lines coupled via routers, switches, other networking equipment, signal lines coupled via any type of network, wireless connections, combinations of signal lines and wireless connections, switching fabrics, etc. Although Figure 2 Cluster 210 is shown with a specific number of clusters arranged in a specific manner, but cluster 210 may include more or fewer clusters. Figure 2 A particular arrangement of deployment and monitoring 140 with respect to cluster 210 is shown, but deployment and monitoring 140 may be arranged differently, including being distributed in parts over several locations and interconnected via different types of networks or buses.

[0035] Figure 33 is a block diagram of a cluster 300 in a data center according to an example. Cluster 300 can be one of the clusters included in a cluster managed using deployment and monitoring 140. In this example, cluster 300 can include a top-of-rack (TOR) switch, which can be used to interconnect racks of hardware components. Thus, cluster 300 can include TOR switches 302, 304, and 306. Each TOR switch can couple at least some components in cluster 300. For example, TOR switch 304 can interconnect rack 310, rack 330, and rack 350. Each rack can include a rack manager and several chassis, which can include components such as servers, network storage devices, networking hardware (e.g., switches, routers, and bridges). Rack 310 can include several chassis, including, for example, chassis 314, 316, 318, and 320. Similarly, rack 330 can include several chassis, including, for example, chassis 334, 336, 338, and 340. Likewise, rack 350 may include several chassis, including, for example, chassis 354, 356, 358, and 360. Each rack may include a rack manager configured to interface with a deployment and monitoring system, such as deployment and monitoring 140. Thus, rack 310 may include rack manager 312, rack 330 may include rack manager 332, and rack 350 may include rack manager 352. Each chassis may include servers, such as blades, organized in groups. Some or all of the chassis may also include networking and storage devices. Each chassis may also include fans to provide cooling air to the servers or other components housed within the chassis. Although Figure 3 A specific arrangement of racks, switches, chassis, and components within the chassis is shown, but the systems and methods disclosed herein are not limited to any specific arrangement. Therefore, the systems and methods are applicable to any organization of a data center with respect to both data and control planes.

[0036] Figure 4 Deployment and monitoring 400 according to one example is shown (e.g., Figure 1400). Deployment and monitoring 400 may include a processor 402, an I / O device 404, a memory 406, a display 408, a sensor 410, a deployment database 412, and a networking interface 414, which may be interconnected via a bus 420. The processor 402 may execute instructions stored in the memory 406. The I / O device 404 may include components such as a keyboard, a mouse, a voice recognition processor, or a touch screen. The memory 406 may be any combination of non-volatile memory or volatile memory (e.g., flash memory, DRAM, SRAM, or other types of memory). The display 408 may be any type of display, such as an LCD, LED, or other type of display. The sensor 410 may include a telemetry or other type of sensor configured to detect and / or receive information (e.g., a condition associated with a device). The sensor 410 may include a sensor configured to sense a condition associated with a CPU, a memory or other storage component, an FPGA, a motherboard, a baseboard management controller, and the like. Sensors 410 may also include sensors configured to sense conditions associated with racks, chassis, fans, power supply units (PSUs), etc. Sensors 410 may also include sensors configured to sense conditions associated with network interface controllers (NICs), top-of-rack (TOR) switches, middle-of-row (MOR) switches, routers, power distribution units (PDUs), rack-level uninterruptible power supply (UPS) systems, etc.

[0037] Continue to refer Figure 4 , the sensor 410 can be implemented in hardware, software, or a combination of hardware and software. Some sensors 410 can be implemented using a sensor API, which can allow the sensor 410 to receive information via the sensor API. Software configured to detect or monitor specific conditions or events can transmit any conditions associated with the equipment monitored by the deployment and monitoring 400 via the sensor API. Remote sensors or other telemetry devices can be incorporated into the data center to sense conditions associated with components installed in the data center. Remote sensors or other telemetry technologies can also be used to monitor other adverse signals in the data center and feed information to deployment and monitoring. For example, if a fan that is cooling a rack stops working, the situation can be sensed by a sensor and reported to the deployment and monitoring function. This type of monitoring can ensure that any second-order effects of the deployment are detected, reported, and corrected.

[0038] Still reference Figure 4 , the deployment database 412 can be used to store records related to payload submissions and packages. In addition, the deployment database 412 can also store data used to generate reports related to deployment. Other parts of this disclosure provide additional details about the functions of the deployment database 412.

[0039] Networking interface 414 may include a communication interface such as Ethernet, cellular radio, Bluetooth radio, UWB radio, or other types of wireless or wired communication interfaces. Bus 420 may be coupled to both the control plane and the data plane. Figure 4 Deployment and monitoring 400 is shown as including a specific number of components arranged and coupled in a specific manner, but deployment and monitoring 400 may include fewer or more components arranged and coupled in different manners. In addition, the functionality associated with deployment and monitoring 400 may be distributed as desired.

[0040] Figure 5 A memory 500 (eg, Figure 4 406), the memory 500 includes a memory for executing and deploying and monitoring 400 (also Figure 1 Memory 500 may include a pre-scan module 502, a planning module 504, a packaging module 506, a verification module 508, a deployment module 510, and a deployment monitor 512. Pre-scan module 502 may evaluate payload parameters provided as part of a payload submission (e.g., payload submission 102). Pre-scan module 502 may evaluate the impact of deployment of any package that includes the payload (e.g., using Figure 4 The results generated by the pre-scan module 502 may be recorded in Figure 4 The planning module 504 can be built on the work performed by the pre-scanning module 502 and determine any additional planning information or steps that will be needed, and record the results back in the database 412. The planning module 504 can determine the risk factors associated with the planned deployment. The planning module 504 can further determine the gates and watchdogs that may be needed to ensure a safe and reliable deployment. The details related to the gates and watchdogs can be recorded in the database 412. Based on all of this information, the planning module 504 can evaluate the coverage that can be provided by the various levels for deployment and verification given the current configuration of the relevant areas of the cluster. By deploying a portion of the package confirmation to the primary level (e.g., a level including nodes without workloads), then the secondary level (e.g., a level including nodes with non-customer workloads), then the minimum scan tree (described below), and then the cluster deployment, the automatic deployment can be performed, and its scope can vary based on the package and payload. Taking into account the current use and composition of the cluster, the minimum scan tree of the cluster can be automatically generated to achieve the target coverage. Fig. 9 and 10Additional details are provided about the minimum scan tree that generates the cluster. Finally, if the parameters associated with the payload or package indicate that additional staging is recommended (e.g., a generation before another generation of equipment), a recommended staging plan can be generated. The generated information can be stored in database 412.

[0041] Still reference Figure 5 , the packaging module 506 can process the submitted payload and construct a deployment package. As part of this process, the packaging module 506 can generate deployment instructions based on the payload parameters. Before further promoting (rolling out) the deployment, the packaging module 506 can specify a group of production test (TIP) devices that can be targeted for deployment. In one example, the TIP mechanism can be used to perform minimum coverage testing to confirm packages in key configurations. Figure 6 600. Thus, example package 600 may include one or more payloads for deployment. In this example, package 600 may include payload 1 602, payload 2 604, and payload N 606. N may be an integer greater than 1. Package 600 may also include health monitor 608. Health monitor 608 may include information about what to monitor and trigger thresholds associated with the monitored information. Package 600 may also include package deployment instructions 610. In one example, package deployment instructions 610 may include operations or actions that specify a deployment plan. Package deployment instructions may also include instructions about monitoring second-order effects at a more general level.

[0042] refer to Figure 5 , the verification module 508 can receive the completed package and confirm the package. The process can include initiating a production test (TIP) deployment to the target device. The deployment monitor 512 can be implemented as a logical service for monitoring the progress of active deployments. The status of each submission and subsequent deployment can be tracked, including the start time of the deployment, the end time of the deployment, the running time of the deployment, and any delays in the deployment. The deployment monitor 512 can also track performance indicators related to the deployment. Table 1 below shows some of the main key performance indicators (KPIs) that can be tracked by the deployment monitor 512.

[0043]

[0044]

[0045] Table 1

[0046] Deployment and monitoring 512 may also track additional KPIs, which are referred to as secondary KPIs in Table 2 below.

[0047]

[0048]

[0049] Table 2

[0050] Figure 7 An example of hierarchical deployment 700 according to an example is shown. Package deployment can affect hundreds of unique components, each of which can have a corresponding SKU or unique identifier to identify the type of component, thereby distinguishing the component from other component types. An example component can be a specific CPU version produced by Intel. In this example, deploying firmware to an Intel CPU can affect dozens of unique CPU versions produced by Intel, which can be deployed as part of receiving firmware updates for a cluster. A specific CPU version can have a corresponding SKU to identify the version of the CPU. A cluster can include thousands of CPUs with this specific version of the CPU. The cluster can include many other versions of Intel CPUs, and thousands (or less) of CPUs for each of these other versions. In this example, deployment can be safely implemented by following a secure deployment process. An example secure deployment process can include first scanning the cluster to determine the diversity of SKUs in the cluster. The process can include deploying and monitoring 140 to continuously scan the cluster and track each unique triple or 3-tuple, including the generation associated with the hardware, the manufacturer associated with the hardware, and the SKU associated with the device. Figure 4 The deployment database 412 may include a table to track unique triples.

[0051] Continue to refer Figure 7In one example, a deployment test setup (e.g., cluster portion 710) may be organized in levels that conceptually correspond to a blast radius. Thus, each level may include an increasing diversity of hardware such that package deployments to an increasing number of levels may cover an increasing diversity of hardware in the cluster. Thus, level 1 720 may include hardware (e.g., servers) that are configured to handle only synthetic workloads. For example, a server organized as part of level 1 720 may host a virtual machine that does not serve any customer workload or other workload that may adversely affect a customer or another user. In this example, using the blast radius analogy, the blast radius may be zero because any package deployment to level 1 720 will not affect any real workload. Level 2 730 may include hardware (e.g., servers) that are configured to handle real-time workloads that, if affected, may adversely affect the workload of at least some users, but will not affect the workload of any customer. Thus, in this example, using the blast radius analogy, level 2 730 may include at least some number of servers that, if affected by package deployment, may affect at least a small amount of a user's workload. One way to assess the impact on customer workloads is to classify servers according to container counts or virtual machine (VM) counts. Thus, in one example, as part of level 2 730, only those servers with container counts or VM counts below a threshold (e.g., two containers per server or two VMs per server) may be targeted for deployment. Level 3 740 may include a greater number of servers that, if affected, may adversely affect at least a small portion of a customer's workload. In one example, level 3 740 may include servers with a greater per-server container count or a greater per-server VM count. Level 4 750 and level 5 760 may include an increasing number and diversity of servers that, if affected, may adversely affect an increasing number of customer workloads. Again, in one example, an increasing number of customer workloads may correspond to an increasing number of per-server container counts or per-server VM counts. Of course, other markers of the customer's workload can also be used to determine which servers or other hardware to include in one tier or another. In one example, by first deploying the package to as few tiers as possible, the blast radius of the deployment can be managed. Figure 7 A specific number of stages are shown for a secure deployment, but additional or fewer stages may be used.

[0052] Figure 8A flowchart 800 of a method for deploying a package according to an example is shown. Step 802 may include receiving a submission. After receiving the submission, some of the processing steps that follow may be performed in parallel. Thus, in this example, one path may involve creating a package and validating the package, and a parallel path may involve pre-scanning and planning deployment. As part of the packaging path, the package may be automatically created and validated. Thus, step 804 may include creating a package that may include (multiple) payloads, package deployment instructions (e.g., configuration instructions or settings), and health monitors (e.g., Figure 6 In this example, when the Figure 5 The instructions corresponding to the packaging module 506 are Figure 4 This step may be performed when the processor 402 of the package is executed. The package may also include a set of production test (TIP) targets identified for verification of the package. Step 808 may include verifying the package. In this example, when the Figure 5 The instruction corresponding to the verification module 502 is Figure 4 This step may be performed when the processor 402 of the embodiment of the present invention is executed. Packet verification may include automatically processing packets as part of TIP. Any results obtained through these steps may be stored in Figure 4 The deployment database 412 is in the deployment database 412.

[0053] Continue to refer Figure 8 , planning the path may involve automatically determining risk factors and any gates that may be required. Thus, in this example, step 806 may include pre-scanning the payload and any other submission parameters to determine risk factors associated with deploying any package including the submitted payload(s). Figure 4 Risk factors and gates are tracked in the deployment database 412 of Figure 5 The instructions corresponding to the pre-scan module 502 are Figure 4 Based on these parameters, given the current configuration of the cluster, Figure 7 Level 1 of 720 and / or Figure 7 An assessment can be made of the coverage that a smaller number of servers or other types of devices organized in level 2 730 will provide. As part of this step, deployment and monitoring (e.g., Figure 1The deployment and monitoring 140) may provide a suggested deployment plan (step 810). The deployment plan may include the degree of parallelism that can be achieved during packaging and deployment. An example of parallelism during deployment may involve whether all servers in a rack that are part of a single cluster can receive packages in parallel. The deployment plan may also include the level of testing and validation required before the package is deployed to the cluster. The deployment plan may also take into account the importance of the customer and their workloads. Therefore, some customers may have devices that are critical to their operations, and deploying the software package to these devices may require additional sign-off. The details associated with the deployment plan may be stored in Figure 4 In this example, when the Figure 5 The corresponding instructions of the planning module 504 are Figure 4 This step may be performed when the processor 402 of is executed. In step 812, the deployment plan may be approved or not approved. In one example, the decision may be made by an administrator associated with the cluster.

[0054] Still reference Figure 8 If the deployment plan is approved at step 812, then at step 814, the package(s) may be deployed. Figure 5 The deployment module 510 corresponds to the instruction Figure 4 When the processor 402 executes, this step may be performed. Fig. 9 and 10 Additional details are provided regarding deployment using the minimum scanning tree method. Alternatively, if, in step 812, the deployment is not approved, then, in step 816, the submission can be rejected. As part of this step, deployment and monitoring can record the submission rejection. Although Figure 8 A specific number of steps are shown as being performed in a specific order, but more or fewer steps may be performed in the same order or a different order.

[0055] Fig. 9 According to an example, Figure 7 FIG. 9 is a diagram of a scan tree 900 of hardware in level 1 720 of FIG. As previously described, level 1 720 may include hardware (e.g., servers) that are configured to handle only synthetic workloads. For example, a server organized as part of level 1 720 may host a virtual machine that does not handle any customer's workload or other workload that may adversely affect a customer or another user. In this example, Fig. 9There are two generations (G1 910 and G2 940) of hardware in level 1 720. Each device with a unique SKU (or some other item identifier) ​​can be represented as an edge of the scan tree, and the device can be the target of a payload (e.g., firmware) deployment. In this example, each device can be identified by a triple (3-tuple) that includes: (1) the generation of hardware in the data center where the device is located, (2) the manufacturer of the server or other equipment that has the device, and (3) the SKU associated with the device. Thus, in this example, level 1 720 may include the following triplets: G1 910, M1 912, SKU 1 922; G1 910, M1 912, SKU 2 924; G1 910, M2 914, SKU 3 926; G1 910, MM 916, SKU 7 928; and G1 910, MM 916, SKU 9 930. Level 1 720 may also include the following triplets: G2 940, M1 942, SKU 1 952; G2 940, M1 942, SKU 2 954; G2 940, M3 944, SKU 5 956; and G2 940, MM 946, SKU 8 958. Information about the triples in scan tree 900 corresponding to level 1 720 may be stored in a database (e.g., Figure 4 deployment database 412). Fig. 9 A scan tree with only two generations of hardware is shown, but there may be additional scan trees with other generations of hardware. Fig. 9 A specific number of manufacturers and a specific number of SKUs are shown, but there may be additional manufacturers and SKUs.

[0056] Fig.10 According to an example, Figure 7 FIG. 1000 is a diagram of a scan tree 1000 of hardware in level 2 730 of FIG. As previously described, level 2 730 may include at least some number of servers that, if affected by package deployment, may affect at least a small amount of active workload for users. In this example, Fig.10 There are two generations (G1 1010 and G2 1040) of hardware in level 2 730. Each device with a unique SKU (or some other identifier) ​​can be represented as an edge of the scan tree, which can be the target of a payload (e.g., firmware) deployment. In this example, similar to the example about Fig. 9In the described example, each device can be identified by a triplet (3-tuple) that includes: (1) the generation of hardware in the data center where the device is located, (2) the manufacturer of the server or other device that has the device, and (3) the SKU associated with the device. 1034; G1 1010, MM 1016, SKU 8 1036; and G1 1010, MM 1016, SKU 9 1038. Level 2 730 may also include the following triplets: G2 1040, M1 1042, SKU 1 1052; G2 1040, M1 1042, SKU 2 1054; G2 1040, M1 1042, SKU 3 1056; G2 1040, M3 1044, SKU 4 1058; G2 1040, M3 1044, SKU 5 1060; G2 1040, MM1046, SKU 7 1062; and G2 1040, MM 1046, SKU 8 1064. In this example, the scan tree for level 2 730 includes components of a higher diversity. For example, the scan tree for level 2 730 includes SKU 3 1026 as part of the G1 generation hardware provided by manufacturer M1. In addition, the scan tree for level 2 730 includes additional types of components from the G1 generation provided by manufacturer MM. Similarly, with respect to the G2 generation hardware, the scan tree includes additional types of components. Information related to the triples in the scan tree 1000 corresponding to level 2 730 may be stored in a database (e.g., Figure 4 deployment database 412). Fig.10 A scan tree with only two generations of hardware is shown, but there may be additional scan trees with other generations of hardware. Fig.10 A specific number of manufacturers and a specific number of SKUs are shown, but there may be additional manufacturers and SKUs.

[0057] Fig.11A and 11B A flow chart of a method for deploying a package according to an example is shown. Step 1102 may include scanning hardware in a cluster to obtain information about the hardware. In one example, step 1102 may include Figure 5 The pre-scan module 502 scans the hardware in the cluster to obtain information about the hardware components deployed in the cluster. The information obtained by scanning the hardware can be stored in Figure 4 The scanned information may include information about the generation, manufacturer, and SKU associated with each of the hardware components in the fleet. This information may be organized in one or more tables and stored in Figure 4 The deployment database 412 is in the deployment database 412.

[0058] Step 1104 may include classifying the hardware in the cluster into deployment categories by volume. In one example, classifying the hardware in the cluster into deployment categories by volume may include Figure 5 The planning module 504 processes the scanned information about the hardware. A deployment category may include at least one component type (or category) that may require a deployment package. Thus, in one example, each deployment category may include those components that are capable of receiving the same or similar payloads. The classification information may be stored in Figure 4 The planning module 504 may also determine risk factors associated with the planned deployment. For example, the planning module 504 may determine the types of customers that may be affected by the deployment. The planning module 504 may further determine gates and watchdogs that may be required to ensure a safe and secure deployment. Details related to gates and watchdogs may also be recorded in Figure 4 The deployment database 412 of FIG. Based on all of this information, the planning module 504 can evaluate the coverage that can be provided by various levels for deployment and verification given the current configuration of the relevant areas of the fleet.

[0059] Step 1106 may include mapping the packages to the devices selected for deployment. As part of this step, the planning module 504 may create information (e.g., a table or a set of tables) that maps the packages to the devices selected for package deployment. This information may be stored in Figure 4 The deployment database 412 is in the deployment database 412.

[0060] Step 1108 may include hardware in scan stage 1 to determine whether the selected diversity goal is met. If the selected diversity goal is met, the process may proceed to process stage A 1110. Otherwise, the process may proceed to process stage B 1112. In one example, as part of this step, the planning module 504 may construct (or process an existing) minimum scan tree, such as with respect to Fig. 9 and 10The goal may be to obtain a reasonable amount of confidence that the package is deployed in a manner consistent with operational characteristics, such as with minimal disruption to customer workloads. For example, if the package is deployed to FPGAs with five different SKUs in a cluster, the minimum scan tree may include the following: Fig. 9 The selected set of triplets described in level 1 allows for covering a selected diversity target. The selected diversity target can be the percentage of different types of SKUs that can receive packages as part of the deployment planning level. Therefore, in this example, as long as 80% of the different types of FPGAs receive packages, then this may be sufficient to meet the selected diversity target. As part of this example, assuming that level 1 only includes 50% of the different types of FPGAs, the planning module 504 can conclude that the selected diversity target is not met and processing can continue to processing stage B 1112. Alternatively, if level 1 includes 80% of the different types of FPGAs, the planning module 504 can proceed to processing stage A 1110.

[0061] about Fig. 11B If the hardware in scan stage 1 does not determine that the selected diversity goal is met, then the process can continue from process stage B 1112. Therefore, step 1114 can include scanning the hardware in stage 2 to determine whether the selected diversity goal is met. In this example, the planning module 504 can construct (or process an existing) minimum scan tree, such as Fig.10 Similar to Fig.11A For example, if the combination of SKUs for FPGAs in level 1 and level 2 includes 80% of the SKUs for FPGAs, the planning module 504 can proceed to process stage A 1110. Otherwise, the process can proceed to step 1116.

[0062] Step 1116 may include scanning the hardware in stage 3 to determine whether the selected diversity goal is met. In this example, the planning module 504 may perform a similar operation to the reference Fig.10 The described method constructs (or processes an existing) minimum scan tree for level 3. Fig.11A For example, if the combination of SKUs of FPGAs in levels 1, 2, and 3 includes 80% of the SKUs for FPGAs, the planning module 504 can proceed to process stage A 1110. Otherwise, the process can proceed to step 1118.

[0063] Step 1118 may include continuing to scan additional levels until the selected diversity goal is met or all remaining levels have been scanned. In one example, the selected diversity goal may be selected based on the package type. Alternatively or additionally, the selected diversity goal may be selected based on the impact type. Thus, for a particular package type, the selected diversity goal may be 75% of the SKUs, while for another package type, the selected diversity goal may be 90% of the SKUs.

[0064] Once the state of the fleet has been determined and the minimum scan tree has been determined for the deployment of a particular package or group of packages, processing can proceed to the next steps. These steps may include determining the speed of the deployment. In one example, the speed of the deployment can be related to the number of gates that the deployment includes. Each gate can correspond to a wait period (e.g., a particular number of hours, days, or months) that specifies the time that the deployment can be delayed after each step of the deployment process. For example, for the deployment of a particular package to a CPU, the deployment can be gated for 24 hours after the deployment of the minimum scan tree; after the 24 hours have passed, the package can be deployed to the CPU with the associated SKU in the rest of the fleet. In one example, when the deployment involves a control plane (e.g., Figure 2 230) is handled by the device, as opposed to deploying devices that involve access via the data plane (e.g., Figure 2 For devices processed by the data plane 220), the gate can specify a longer waiting period.

[0065] In another example, the speed of a deployment may be related to the impact of the deployment. Thus, the number of gates and the waiting time periods specified by the gates may depend on the impact of the deployment on the fleet. For example, certain deployments may be characterized as having an impact, while other deployments may be characterized as having no impact. Deployments may also be characterized along a sliding scale between having no impact and having an impact. The process may include the planning module 504 taking into account both the package type and the impact type of the package. Information corresponding to the impact (including the impact type and the package type) may be stored in a database (e.g., Figure 4 In a table in the deployment database 412).

[0066] Fig.12 An impact table 1200 according to one example is shown. The impact table 1200 can be used to track the impact of package deployment on various devices in the cluster. For example, the impact table 1200 can be used to classify the impact on the device into an impact type and to classify the package into a package type. In one example, the impact table 1200 can be stored in Figure 412. In this example, the impact table 1200 may include information organized in rows and columns, including impact types 1210 in the rows and package types 1240 in the columns. The impact types 1210 may include CPU pause 1212, storage pause 1214, network pause 1216, FPGA pause 1218, restart 1220, power increase 1222, performance decrease 1224, and thermal impact 1226. The package types 1240 may include microcode 1242, universal extensible firmware interface (UEFI) / basic input / output system (BIOS) 1244, baseboard management controller (BMC) 1246, solid state drive (SSD) 1248, hard disk drive (HDD) 1250, and FPGA 1252. FPGA 1252 may also include two subtypes of packages: FPGA platform 1254 and FPGA image 1256. Although the impact table 1200 shows certain information organized in a particular manner, more or less information may be included and organized in a different manner. In addition, the information in the impact table 1200 can be encapsulated in other types of data structures, including linked lists or other structures. As another example, the information in the impact table can be distributed, so that the impact information for each package can be included with the package as metadata or other data structures associated with the package.

[0067] Continue to refer Fig.12 In this example, as shown in the table, deploying a package with microcode can result in less than X seconds (Xs) of CPU pauses; deploying a package with UEFI / BIOS can result in less than X seconds (Xs) of CPU pauses; and deploying a package to an FPGA platform can result in less than X seconds (Xs) of CPU pauses, where X is a number. In addition, in this example, deploying a package to an SSD can result in Y seconds (Ys) of storage pauses, and deploying a package to an HDD can result in Z seconds (Zs) of storage pauses, where each of Y and Z is a number. In addition, deploying a package to an FGPA platform can result in less than F seconds (Fs) of network pauses. Deploying a package to an FPGA platform can result in less than P seconds (Ps) of FPGA pauses. On the other hand, deploying a package to an FPGA image can be image-specific.

[0068] Still reference Fig.12, in this example, the impact table 1200 may include information about the types of packets that may cause a reboot. Thus, in this example, deploying UEFI / BIOS packages and deploying packages to FPGA platforms may always cause a reboot; however, deploying software packages to SSDs or HDDs may only cause a reboot at certain times. The impact table 1200 may also include information about power consumption changes (e.g., power increases) based on deployment of certain package types. For microcode, UEFI / BIOS, SSD, and HDD package types, the power increase may be low; however, for FPGA platform package types, the power increase may be moderate, and for FPGA images, the power increase may be image-specific. The impact table 1200 may also include information about performance changes (e.g., performance degradation) based on deployment of certain package types. For microcode, UEFI / BIOS, SSD, and HDD package types, the performance degradation may be low. The impact table 1200 may also include information about thermal impact changes (e.g., higher or lower thermal impact) based on deployment of certain package types. For microcode, UEFI / BIOS, and BMC package types, the thermal impact may be low.

[0069] Although the impact table 1200 includes information about certain packet types and impact types, the impact table 1200 may include additional or less information about each of the packet types and impact types. For example, the impact table 1200 may include information about the impact of deploying a packet to a network interface controller (NIC), a top-of-rack (TOR) switch, a middle-of-row (MOR) switch, a router, a power distribution unit (PDU), and a rack-level uninterruptible power supply (UPS) system.

[0070] Fig.13 A machine learning system 1300 according to an example is shown. The machine learning system 1300 may include a processor 1302, an I / O device 1304, a memory 1306, a sensor 1310, a display 1320, and a networking interface 1322, which may be interconnected via a bus system 1330. The bus system 1330 may be coupled to a data plane (e.g., Figure 2 data plane 220) and a control plane (e.g., Figure 2 The processor 1302 may execute instructions stored in the memory 1306. The memory 1306 may be any combination of non-volatile storage or volatile storage (e.g., flash memory, DRAM, SRAM, or other types of memory). The sensor 1310 may include a telemetry or other type of sensor configured to detect and / or receive information (e.g., a condition associated with a device).

[0071] Continue to refer Fig.13, the sensors 1310 may include sensors configured to sense conditions associated with a CPU, memory or other storage component, FPGA, motherboard, baseboard management controller, etc. The sensors 1310 may also include sensors configured to sense conditions associated with a rack, chassis, fan, power supply unit (PSU), etc. The sensors 1310 may also include sensors configured to sense conditions associated with a network interface controller (NIC), a top-of-rack (TOR) switch, a middle-of-row (MOR) switch, a router, a power distribution unit (PDU), a rack-level uninterruptible power supply (UPS) system, etc. The sensors 1310 may be implemented using a sensor API, which may allow the sensors 1310 to receive information via the sensor API. Software configured to detect or listen for specific conditions or events may communicate any conditions associated with the equipment monitored by the deployment and monitoring 400 via the sensor API. Telemetry sensors or other telemetry devices incorporated into the data center to sense conditions associated with components installed in the data center may sense conditions and provide information to the sensors 1310 or the processor 1302. Additionally, the deployment and monitoring may also transmit data related to the event or condition to the sensor 1310 or the processor 1302. For example, Figure 4 Any events or conditions sensed by sensor 410 may be provided to processor 1302 as needed.

[0072] Display 1320 may be any type of display, such as an LCD, LED, or other type of display. Network interface 1322 may include a communication interface, such as Ethernet, cellular radio, Bluetooth radio, UWB radio, or other type of wireless or wired communication interface. Fig.13 The machine learning system 1300 is shown as including a specific number of components arranged and coupled in a specific manner, but it may include fewer or additional components arranged and coupled in a different manner. In addition, the functionality associated with the machine learning system 1300 may be distributed as desired.

[0073] Fig.14 14 shows a memory 1400 (e.g., Fig.13 1406). In this example, instructions may be organized in memory 1400 in blocks or modules that include code, data, or both. In this example, memory 1400 may include a learning-based analyzer (LBA) 1410, training data 1420, a machine learning (ML) model 1430, an impact table 1440, payload parameters 1450, package parameters 1460, deployment parameters 1470, and fleet parameters 1480. Although Fig.14Instructions and data are shown organized in a particular manner, but the instructions and data may be combined or distributed in various ways.

[0074] Continue to refer Fig.14 , a learning-based analyzer (LBA) 1410 can implement a supervised learning algorithm that can be trained based on input data, and once the algorithm is trained, it can make predictions or prescriptions based on the training. In this example, LBA 1410 can implement techniques such as linear regression, a support vector machine (SVM) set for regression, a random forest set for regression, a gradient boosted tree set for regression, and a neural network. Linear regression can include modeling the past relationship between an independent variable and an output dependent variable. A neural network can include artificial neurons for creating an input layer, one or more hidden layers, and an output layer. Each layer can be encoded as a weight matrix or vector represented in the form of coefficients or constants, which may have been obtained by training the neural network offline. The neural network can be implemented as a recursive neural network, a long short-term memory (LSTM) neural network, or a gated recursive unit (GRU). All information required by a supervised learning-based model can be converted into a vector representation corresponding to any of these techniques. Taking LSTM as an example, an LSTM network can include a series of repeated RNN layers or other types of layers. Each layer of the LSTM network can consume input at a given time step (e.g., the state of the layer from the previous time step) and can produce a new set of outputs or states. In the case of using LSTM, a single block of content can be encoded into a single vector or multiple vectors. For example, a word or a combination of words (e.g., a phrase, sentence, or paragraph) can be encoded into a single vector. Each block can be encoded into an individual layer (e.g., a specific time step) of the LSTM network. A set of equations can be used to describe the LSTM layer, such as the following:

[0075] i t =σ(W xi xt+W hi h t-1 +W ci c t-1 +b i

[0076] f t =σ(W xf x t +W hf h t-1 +W cf c t-1 +b f )

[0077] c t =f t ct-1 i t tanh(W xc x t +W hc h t-1 +b c )

[0078] o t =σ(W xo x t +W ho h t-1 +W co c t +b o )

[0079] h t =o t tanh(c t )

[0080] In this example, inside each LSTM layer, the input and hidden states can be processed using a combination of vector operations (e.g., dot products, inner products, or vector additions) or nonlinear operations, if necessary.

[0081] although Fig.14 LBA 1410 is described as including instructions, but the instructions may be encoded as hardware corresponding to the A / I processor. In this case, some or all of the functionality associated with the learning-based analyzer may be hard-coded or otherwise provided as part of the A / I processor. For example, an FPGA having the necessary functionality may be used to implement the A / I processor.

[0082] The training data 1420 may be data that may be used to train a neural network model or a similar machine learning model. In one example, the training data 1420 may be used to train a machine learning model to minimize an error function associated with the deployment of a package. In one example, minimization of the error function may be obtained by obtaining user feedback about various payload and package parameters and determining appropriate weights for convolution operations or other types of operations to be performed as part of machine-based learning. For example, a set of pre-selected mapping functions with known payload and package parameters may be provided to a user in a test environment and the user may be asked to select a mapping function that he or she prefers.

[0083] ML model 1430 may include a machine language model that may be used as part of machine learning system 1300. ML model 1430 may include a model created by a training process. In this example, training data 1420 may include target attributes, such as a selected diversity target for deploying a package. An appropriate machine learning algorithm included as part of LBA 1410 may find patterns in training data 1420 that map a given set of input parameters (e.g., payload parameters and package parameters) to the selected diversity target for deploying a package. In another example, a machine learning algorithm may find patterns in training data 1420 that map input parameters to deployment classifications. Example deployment classifications may include at least two categories: influential or non-influential. Other machine language models may also be used. For example, training data 1420 may be used to train a machine language model that maps input packet types to any impact associated with the deployment of a package. The impact may be represented in a form similar to that described with respect to impact table 1440. Thus, impact table 1440 may be used with Fig.12 The impact table 1200 is similar or identical.

[0084] The payload parameters 1450 may include parameters associated with the payload. In one example, the payload parameters may include the type of payload, the target SKU of the payload, the degree of change caused by the deployment of the payload, any prerequisites, any known impacts, and the required deployment time. The payload parameters 1450 may be extracted from metadata associated with the payload, or otherwise obtained through the submission process as previously described.

[0085] The package parameters 1460 may include parameters associated with a package including a payload. In one example, the package parameters 1460 may include information about the type of health monitoring included with the package. The package parameters 1460 may also include the package type and the gates and watchdogs required to deploy the package.

[0086] Deployment parameters 1470 may include information about the rollout plan. For example, deployment parameters 1470 may include an assessment of target conditions that the deployment will require. These conditions may include information about whether any of a device reset, node restart, node resurfacing, power cycle, or disk reformatting is required. These parameters may be included as part of the instructions and / or metadata associated with the package.

[0087] The fleet parameters 1480 may include information about an entire fleet or a subset of fleets that may be targeted for deployment. The fleet parameters may include information about item types (e.g., SKUs) associated with data centers in the fleet or subset of fleets. The information may include the number of each SKU in the SKUs. In addition, the fleet parameters 1480 may include additional details about the data centers included in the fleet or subset of fleets. For example, information about the data centers may include location information, AC voltage supply in the data center (e.g., 120 volts or 240 volts), operator information (e.g., whether the data center is operated by a service provider or a customer of the service provider). Figure 5 The deployment module 510 evaluates the cluster parameters 1480. Some of the cluster parameters 1480 may be stored in Figure 4 The deployment database 412 is in the deployment database 412.

[0088] The ML model 1430 may include a model trained to prioritize targets with the least impact. Thus, in one example, the ML model may learn that when a node is required to restart, empty nodes should be deployed first because they are not running any workloads. The ML model 1430 may also include a model that can be trained to receive parameters associated with payloads, packages, deployments, and clusters as inputs, and determine whether some of the deployment steps can be executed in parallel. In addition, the ML model 1430 may include a model that can be trained to receive parameters associated with payloads, packages, deployments, and clusters as inputs, and determine specific gates and watchdogs that may be required during deployment of the cluster. In addition, the ML model 1430 may include a model that can be trained to receive parameters associated with payloads, packages, deployments, and clusters as inputs, and determine the type of health monitoring that should be included as part of the package deployment. Finally, other automated feedback models may also be used. For example, such an automated feedback model may not rely on machine learning; instead, it may rely on other feedback mechanisms to allow automatic creation of packages for deployment, or to allow automatic creation of deployment plans for deploying packages to a fleet of machines. Regardless, in some cases, the automated feedback model may use a machine language model for learning, such as a reinforcement learning model.

[0089] Fig.15 A flowchart 1500 of a method for creating a package including a payload for deployment to a group of devices is shown according to an example. Step 1502 may include receiving a payload, wherein the payload has an associated set of payload parameters for deploying the payload to the group of devices. As previously explained, the payload may be received via a submission portal or another manner. As described in reference Fig.14As explained, the payload parameter may be payload parameter 1450 .

[0090] Step 1504 may include using a processor to automatically create a package for deployment to a set of devices, wherein the package includes instructions for deploying a payload to the set of devices, and wherein the instructions specify at least one of a plurality of operations derived from a machine learning model based on at least a subset of the associated set of payload parameters. In this example, processor 1302 may execute instructions stored in memory 1306 (e.g., instructions corresponding to learning-based analyzer 1410) to perform this step. The instructions for deploying the payload may specify operations, such as the number of gates and / or watchdogs required for deployment. The operations may involve deployment parameters associated with deploying the package to the set of devices (e.g., Fig.14 The operations may also include health monitoring information for package deployment. The health monitoring information may include what to monitor and trigger thresholds associated with the monitored information.

[0091] In one example, automatically creating a package for deployment to a set of devices may include processing metadata or other submission parameters associated with a payload. The machine learning model may be trained based on training data that includes a mapping between at least a subset of the associated set of payload parameters and a set of labels that classify the impact of deploying the payload to the set of devices. In one example, the set of labels may include a first label that classifies the impact as having an impact and a second label that classifies the impact as having no impact. Fig.14 Any of the described ML models 1430 may be trained and used as previously described.

[0092] Fig.16 A flowchart 1600 of a method for deploying a package to a cluster is shown according to an example. Step 1602 may include evaluating the cluster to determine a set of cluster parameters associated with deploying the package to the cluster. In this example, processor 1302 may execute instructions stored in memory 1306 (e.g., instructions corresponding to learning-based analyzer 1410) to perform this step. Evaluating the cluster may include processing metadata associated with the cluster. The metadata may include information about the use and composition of the cluster. The metadata may be stored in Figure 4 In one example, evaluating the cluster may also include scanning hardware associated with the cluster, as previously described with respect to Fig.11A and 11B Explained.

[0093] Step 1604 may include: using a processor, automatically creating a deployment plan for deploying the package to the cluster, wherein the deployment plan includes instructions for deploying the package to the cluster, and wherein the instructions specify at least one of a plurality of operations derived from the machine learning model based on at least a subset of the set of cluster parameters. In this example, the processor 1302 may execute instructions stored in the memory 1306 (e.g., instructions corresponding to the learning-based analyzer 1410) to perform the step. The machine learning model may be trained based on the following training data: the training data includes a mapping between at least a subset of the cluster parameters and at least one label associated with the deployment plan. In addition, the machine learning model may be trained based on feedback regarding the deployment of the package to the cluster. The plurality of operations may include actions corresponding to monitoring the deployment of the package to the cluster. Thus, as previously explained, the deployment monitor may monitor the deployment of the cluster. Regarding Figure 5 Deployment monitor 512 provides additional details about the deployment monitor. These operations may include information about what to monitor and trigger thresholds associated with the monitored information. In addition, the operation may include an action corresponding to generating information about a minimum scanning tree including a set of devices in the cluster. Fig.11A and 11B Additional details are provided about generating information about the minimum scan tree.

[0094] In one example, rolling out a package across a fleet can be staged in a way that minimizes impact on customer workloads. Thus, first the package can be deployed to empty nodes (e.g., nodes that do not host any workloads). Next, the package can be deployed to those nodes with the smallest number (e.g., two) of workloads (determined, for example, based on container counts or the number of virtual machines being supported by the node). Next, the package can be deployed to nodes with a slightly higher number of workloads, and so on. This can limit the blast radius if the deployment causes hardware outages and help contain any harm to customer workloads.

[0095] A logical reporting service can also be implemented to track deployments in real time. This service can access the Figure 4 The data in the deployment database 412 and the data stored in other sources are automatically generated and reports can be generated. Business analysis tools such as Power BI) to implement reporting services. You can also deploy dashboards.

[0096] Fig.17 A deployment dashboard 1700 is shown according to an example. The deployment dashboard 1700 includes information about the current state of the fleet, which is related to the deployment of various package types. The information can be displayed and tracked in real time. The information about the current state of the fleet can be stored in Figure 4The deployment database 412 and can be Figure 5 The deployment monitor 512 of the embodiment of the present invention is retrieved and displayed via the deployment dashboard 1700. In this example, the deployment dashboard 1700 may include freshness information indicating the current state of the deployment of the fleet with respect to certain package types. In this example, the deployment dashboard 1700 may include a meter that indicates the current state of the fleet by package type. The deployment dashboard 1700 may also indicate whether the meter is associated with a control plane device or a data plane device. In addition, the deployment dashboard 1700 may indicate whether the deployment being tracked is non-impactful or impactful. Thus, the meter 1710 may indicate the current state of the package deployment associated with the chassis manager. The meter 1720 may indicate the current state of the package deployment associated with the power supply unit (PSU). The meter 1730 may indicate the current state of the package deployment associated with the CPU microcode. The meter 1730 may indicate the current state of the package deployment associated with the CPU microcode. The meter 1740 may indicate the current state of the package deployment associated with the baseboard management controller (BMC). The meter 1750 may indicate the current state of the package deployment associated with the UEFI / BIOS. Meter 1760 may indicate the current status of package deployment associated with the HDD / SDD. Fig.17 The deployment dashboard 1700 is shown with a particular number of meters organized in a particular manner, but the deployment dashboard 1700 may include additional or fewer meters that may be organized differently. For example, the deployment dashboard 1700 may include meters that track other package types, including, for example, FPGA platforms, FPGA images, network interface controllers (NICs), top-of-rack (TOR) switches, middle-of-row (MOR) switches, routers, power distribution units (PDUs), and rack-level uninterruptible power supply (UPS) systems.

[0097] In addition, other dashboards may be provided, including a dashboard that tracks each active deployment. Each such dashboard may display the progress of deployments, including the current deployment rate and estimated completion time. In addition to active deployments, pending deployments may also be displayed conditionally. For pending deployments, the dashboard may include the status of the deployment, such as Submitted, Packaged, Testing, Pending, Aborted, or Completed. Additional details about each deployment (active or pending) may be displayed by a deployment monitor (e.g., Figure 5 In addition to the dashboard, the deployment monitor 512 can also provide reports on key results, such as deployment safety, deployment time, detection efficiency, deployment impact, and deployment parallelism.

[0098] In summary, the present disclosure relates to a method for deploying a package to a group of devices in a cluster, the cluster including hardware configurable to perform functions related to customer workloads. The method may include: performing a first scan of hardware in a first stage to determine whether a selected diversity goal for deploying the package is met, wherein the first stage includes hardware configured to handle functions unrelated to customer workloads. The method may also include: if the selected diversity goal for deploying the package is met after the first scan, then using a processor, based on instructions associated with the package, deploying the package to a first subset of the first stage of the group of devices. The method may also include: if the selected diversity goal for deploying the package is not met after the first scan, scanning hardware in a second stage to determine whether the selected diversity goal for deploying the package is met, wherein the second stage includes hardware configured to handle at least a subset of customer workloads. The method may also include: if the selected diversity goal for deploying the package is met after the scan, then using a processor, based on instructions associated with the package, deploying the package to a first subset of the first stage of the group of devices and a second subset of the second stage of the group of devices.

[0099] The method may also include scanning hardware in the cluster to obtain information about the hardware, and storing the information about the hardware in a database. The method may also include classifying the hardware in the cluster into deployment categories by capacity. The method may also include generating first information about a first minimum scan tree, the first minimum scan tree including a first subset of the set of devices. The method may also include generating second information about a second minimum scan tree, the second minimum scan tree including a second subset of the set of devices.

[0100] The fleet may include a set of devices of a predetermined number of types, including devices of the first type, and wherein the selected diversity goal specifies a percentage of devices of the first type. The method may also include: if the selected diversity goal for the deployment package is not met after the scanning, continuing to scan any remaining levels until the selected diversity goal for the deployment package is met or all remaining levels have been scanned.

[0101] In another example, the present disclosure relates to a method for deploying a package to a group of devices in a cluster, the cluster including hardware configurable to perform functions related to customer workloads. The method may include performing a first scan of hardware in a first level to determine whether a selected diversity goal for deploying the package is met, wherein the first level includes hardware configured to not handle any workloads. The method may also include: if the selected diversity goal for deploying the package is not met after the first scan, scanning the hardware in a second level to determine whether the selected diversity goal for deploying the package is met, wherein the second level includes hardware configured to handle a first selected number of customer workloads. The method may also include: if the selected diversity goal for deploying the package is not met after the scan, scanning a third level to determine whether the selected diversity goal for deploying the package is met, wherein the third level includes hardware configured to handle a second selected number of customer workloads, wherein the second selected number is greater than the first selected number. The method may also include deploying the package to a first subset of the group of devices in the first level, a second subset of the group of devices in the second level, and a third subset of the group of devices in the third level, using a processor based on instructions associated with the package if a selected diversity target for deploying the package is met after scanning the third level.

[0102] The method may also include scanning hardware in the cluster to obtain information about the hardware, and storing the information about the hardware in a database. The method may also include classifying the hardware in the cluster into deployment categories by capacity. The method may also include generating first information about a first minimum scan tree, the first minimum scan tree including a first subset of the set of devices.

[0103] The method may also include generating second information about a second minimum scan tree, the second minimum scan tree including a second subset of the set of devices. The method may also include generating third information about a third minimum scan tree, the third minimum scan tree including a third subset of the set of devices. The method may also include continuing to scan any remaining levels if the selected diversity goal for the deployment package is not met after the scan until the selected diversity goal for the deployment package is met or all remaining levels have been scanned.

[0104] In yet another example, the present disclosure relates to a system for deploying a package to a group of devices in a cluster, the cluster including hardware that can be configured to perform functions related to customer workloads. The system can be configured to: perform a first scan of the hardware in the first level to determine whether the selected diversity goal for deploying the package is met, wherein the first level includes hardware configured to handle functions unrelated to the customer workload. The system can also be configured to: if the selected diversity goal for deploying the package is met after the first scan, then based on instructions associated with the package, deploy the package to a first subset of the first level of the group of devices. The system can also be configured to: if the selected diversity goal for deploying the package is not met after the first scan, scan the hardware in the second level to determine whether the selected diversity goal for deploying the package is met, wherein the second level includes hardware configured to handle at least a subset of customer workloads. The system can also be configured to, if the selected diversity goal for deploying the package is met after the scan, then based on instructions associated with the package, deploy the package to a first subset of the first level of the group of devices and a second subset of the second level of the group of devices.

[0105] The system may also be configured to scan hardware in the cluster to obtain information about the hardware and store the information about the hardware in a database. The system may also be configured to classify the hardware in the cluster into deployment categories by capacity. The system may also be configured to generate first information about a first minimum scan tree, the first minimum scan tree including a first subset of the set of devices.

[0106] The system may also be configured to generate second information about a second minimum scan tree, the second minimum scan tree including a second subset of the set of devices. The system may also be configured to, if the selected diversity goal for the deployment package is not met after the scan, continue scanning any remaining levels until the selected diversity goal for the deployment package is met or all remaining levels have been scanned.

[0107] It should be understood that the methods, modules and components described herein are merely exemplary. Alternatively or additionally, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc. In an abstract but still clear sense, any arrangement of components that achieve the same functionality is effectively "associated" so that the desired functionality is achieved. Therefore, any two components that are combined to achieve a specific functionality herein may be considered to be "associated" with each other so that the desired functionality is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated may also be considered to be "operably connected" or "coupled" to each other to achieve the desired functionality.

[0108] Functionality associated with some examples described in this disclosure may also include instructions stored in non-transient media. The term "non-transient media" as used herein refers to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Exemplary non-transient media include non-volatile media and / or volatile media. Non-volatile media include, for example, hard disks, solid-state drives, magnetic disks or tapes, optical disks or tapes, flash memory, EPROM, NVRAM, PRAM or other such media, or networked versions of such media. Volatile media include, for example, dynamic memories such as DRAM, SRAM, cache or other such media. Non-transient media are different from transmission media, but can be used in conjunction with transmission media. Transmission media are used to transmit data and / or instructions to or from a machine. Exemplary transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media (such as radio waves).

[0109] In addition, those skilled in the art will recognize that the boundaries between the functionality of the above operations are merely illustrative. The functionality of multiple operations can be combined into a single operation, and / or the functionality of a single operation can be distributed in additional operations. In addition, alternative embodiments can include multiple instances of a particular operation, and the order of the operations can be changed in various other embodiments.

[0110] Although the present disclosure provides specific examples, various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the appended claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure. Any benefits, advantages, or solutions to problems described herein with respect to specific examples are not intended to be construed as key, required, or essential features or elements of any or all claims.

[0111] Furthermore, the terms "a" or "an" as used herein are defined as one or more than one. Furthermore, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be interpreted as implying that the introduction of another claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to an invention containing only one such element, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an". The same is true for the use of definite articles.

[0112] Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements these terms describe. Therefore, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.

Claims

1. A method for deploying a package to a set of devices in a cluster, the cluster comprising hardware configurable to perform functions associated with a customer workload, the method comprising: performing a first scan of hardware in a first stage to determine whether a selected diversity goal for deploying the package is satisfied, wherein the first scan further comprises generating first information about a first minimum scan tree, the first minimum scan tree comprising a first subset of a set of devices in the first stage, wherein the first stage comprises hardware configured to handle functions unrelated to the customer workload, and wherein the selected diversity goal is based on at least a package type associated with the package and an impact type associated with the package; deploying, with a processor, the package to at least the first subset of the set of devices in the first stage based on instructions associated with the package if the selected diversity goal for deploying the package after the first scan is met; if the selected diversity goal for deploying the package is not met after the first scan, second scanning hardware in a second stage to determine whether the selected diversity goal for deploying the package is met, wherein the second stage includes hardware configured to handle at least a subset of the customer workloads; as well as If the selected diversity goal for deploying the package is met after the second scan, deploying the package to the first subset of the group of devices in the first level and to a second subset of the group of devices in the second level, using the processor, based on the instructions associated with the package.

2. The method according to claim 1, further comprising: Scanning the hardware in the cluster to obtain information about the hardware, and storing the information about the hardware in a database; The information at least includes the packet type associated with the packet and the impact type associated with the packet. 3 . The method of claim 1 , further comprising classifying the hardware in the cluster into deployment categories by capacity. 4 . The method of claim 1 , further comprising generating second information about a second minimum scan tree, the second minimum scan tree including the second subset of the set of devices.

5. The method of claim 1, wherein the fleet includes a predetermined number of types of the set of devices, the predetermined number of types of the set of devices including a first type of devices, and wherein the selected diversity target specifies a percentage of devices of the first type.

6. The method according to claim 1, further comprising: If the selected diversity goal for deploying the package is not met after the second scan, scanning any remaining stages continues until the selected diversity goal for deploying the package is met or all of the remaining stages have been scanned.

7. A method for deploying a package to a set of devices in a cluster, the cluster comprising hardware configurable to perform functions associated with a customer workload, the method comprising: performing a first scan of hardware in a first stage to determine whether a selected diversity goal for deploying the package is satisfied, wherein the first scan further comprises generating first information about a first minimum scan tree, the first minimum scan tree comprising a first subset of a set of devices in the first stage, wherein the first stage comprises hardware configured not to handle any workloads, and wherein the selected diversity goal is based on at least a package type associated with the package and an impact type associated with the package; if the selected diversity goal for deploying the package is not met after the first scan, second scanning hardware in a second stage to determine whether the selected diversity goal for deploying the package is met, wherein the second stage includes hardware configured to handle a first selected number of the customer workloads; scanning a third stage to determine whether the selected diversity goal for deploying the package is met if the selected diversity goal for deploying the package is not met after the second scan, wherein the third stage includes hardware configured to handle a second selected number of the customer workloads, wherein the second selected number is greater than the first selected number; as well as If the selected diversity target for deploying the package is met after the third scan of the third level, using a processor, based on instructions associated with the package, deploy the package to at least the first subset of the group of devices in the first level, the second subset of the group of devices in the second level, and the third subset of the group of devices in the third level.

8. The method according to claim 7, further comprising: The hardware in the cluster is scanned to obtain information about the hardware, and the information about the hardware is stored in a database.

9. The method of claim 7, further comprising classifying the hardware in the fleet into deployment categories by capacity.

10. The method of claim 7, further comprising generating second information about a second minimum scan tree, the second minimum scan tree including the second subset of the set of devices.

11. The method of claim 10, further comprising generating third information about a third minimum scan tree, the third minimum scan tree including the third subset of the set of devices.

12. The method according to claim 7, further comprising: If the selected diversity goal for deploying the package is not met after the third scan, scanning any remaining stages continues until the selected diversity goal for deploying the package is met or all of the remaining stages have been scanned.

13. A system for deploying a package to a set of devices in a cluster, the cluster comprising hardware configurable to perform functions associated with a customer workload, wherein the system comprises at least one processor and a set of instructions stored in at least one memory, the set of instructions, when executed by the at least one processor, being configured to: performing a first scan of hardware in a first stage to determine whether a selected diversity goal for deploying the package is satisfied, wherein at least a first subset of the set of instructions is further configured to generate first information about a first minimum scan tree, the first minimum scan tree comprising the first subset of a set of devices in the first stage, wherein the first stage comprises hardware configured to handle functions unrelated to the customer workload, and wherein the selected diversity goal is based on at least a package type associated with the package and an impact type associated with the package; deploying the package to at least a first subset of the set of devices in the first stage based on instructions associated with the package if the selected diversity goal for deploying the package is met after the first scan; if the selected diversity goal for deploying the package is not met after the first scan, second scanning hardware in a second stage to determine whether the selected diversity goal for deploying the package is met, wherein the second stage includes hardware configured to handle at least a subset of the customer workloads; as well as If the selected diversity goal for deploying the package is met after the second scan, deploying the package to the first subset of the set of devices in the first level and to a second subset of the set of devices in the second level based on the instructions associated with the package.

14. The system of claim 13, wherein at least a second subset of the set of instructions is further configured to: scan the hardware in the cluster to obtain information about the hardware, and store the information about the hardware in a database.

15. The system of claim 13, wherein at least a second subset of the set of instructions is further configured to: classify the hardware in the cluster into deployment categories by capacity.

16. The system of claim 13, wherein at least a second subset of the set of instructions is further configured to generate second information about a second minimum scan tree, the second minimum scan tree including the second subset of the set of devices.

17. A system according to claim 16, wherein at least a third subset of the set of instructions is further configured to: if the selected diversity goal for deploying the package is not met after the second scan, continue scanning any remaining levels until the selected diversity goal for deploying the package is met or all of the remaining levels have been scanned.

Citation Information

Patent Citations

  • Server cluster unit system with single system image

    CN101271401A

  • Domestic CPU (central processing unit) and operating system based software and hardware fault alarming system and method

    CN106649055A

  • Nonintrusive dynamically-scalable network load generation

    CN107005438A

  • Network Port Profile Representation in Open Virtualization Format Package

    US20130132951A1