Hyper-converged infrastructure network management card assisted shutdown

Abstract

Various techniques for network management card assisted shutdown for hyperconverged infrastructure are disclosed. A network management card includes a network interface communicatively coupled with a hyperconverged infrastructure environment, one or more processors, and one or more non-transitory computer-readable media storing a plurality of instructions. When executed by the one or more processors, the plurality of instructions cause the one or more processors to perform a plurality of operations including receiving, from the hyperconverged infrastructure environment via the network interface, a selection of a shutdown instruction set from a plurality of shutdown instruction sets supported by the network management card, the plurality of shutdown instruction sets configured to support shutdown procedures for at least two different hyperconverged infrastructure platforms, detecting that the hyperconverged infrastructure environment is performing a shutdown, and completing the shutdown at least by executing the shutdown instruction set.

Description

Technical Field

[0001] This disclosure generally relates to the shutdown of hyperconverged infrastructure. Background Technology

[0002] Hyperconverged infrastructure (HCI) is a data center infrastructure that forms a "converged" data center stack. Specifically, in HCI, hardware services including compute, storage, and networking are virtualized and run through a hypervisor. Virtualization allows HCI to run on a range of underlying hardware, including but not limited to commercial off-the-shelf servers. Virtualization also allows HCI to emulate storage and networking functions in software that would otherwise require dedicated hardware.

[0003] The methods described in this section were not necessarily conceived and / or performed prior to the filing of this application. Therefore, unless otherwise stated, the methods described in this section should not be construed as prior art. Summary of the Invention

[0004] Typically, in one aspect, a Network Management Card (NMC) includes: a network interface communicatively coupled to a hyperconverged infrastructure (HCI) environment; one or more processors; and one or more non-transitory computer-readable media storing a plurality of instructions. When executed by the one or more processors, the plurality of instructions cause the one or more processors to perform a plurality of operations, the plurality of operations including: receiving, via the network interface, a selection of a shutdown instruction set from a plurality of shutdown instruction sets supported by the NMC, the plurality of shutdown instruction sets being configured to support a plurality of shutdown procedures for at least two different hyperconverged infrastructure platforms; detecting that the hyperconverged infrastructure environment is performing a shutdown; and completing the shutdown at least by executing the shutdown instruction set. The step of completing the shutdown may include performing one or more shutdown operations after all virtual machines in the hyperconverged infrastructure environment have been shut down. The NMC may be coupled to an uninterruptible power supply (UPS). The step of completing the shutdown may include turning off the power supply, which is provided to the hyperconverged infrastructure environment via the UPS. The step of detecting that the hyperconverged infrastructure environment is performing a shutdown may include receiving a shutdown command from a virtual shutdown agent operating in a virtual machine within the hyperconverged infrastructure environment. The step of receiving a selection of the shutdown command set includes receiving multiple portions of the shutdown command set from the hyperconverged infrastructure environment via the network interface. Prior to receiving the selection of the shutdown command set, the shutdown command set may have already been stored in the network management card.

[0005] Generally, in one aspect, one or more non-transitory computer-readable media storing a plurality of instructions, when executed by one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations including: selecting a shutdown instruction set from a plurality of shutdown instruction sets supported by a network management card communicatively coupled to the hyperconverged infrastructure environment via a shutdown agent operating in a hyperconverged infrastructure environment, the plurality of shutdown instruction sets being configured to support a plurality of shutdown procedures for at least two different hyperconverged infrastructure platforms; sending the shutdown instruction set from the hyperconverged infrastructure environment to the network management card; and during a shutdown of the hyperconverged infrastructure environment, sending an instruction to the network management card to complete the shutdown, wherein the network management card is configured to respond to the instruction to complete the shutdown by at least executing the shutdown instruction set. Selecting the shutdown instruction set may include: detecting a specific hyperconverged infrastructure platform used by the hyperconverged infrastructure environment; and selecting the shutdown instruction set at least based on the shutdown instruction sets supporting the specific hyperconverged infrastructure platform. Selecting the shutdown instruction set may include: presenting the plurality of multiple shutdown instruction sets in a user interface; and receiving user input through the user interface to select the shutdown instruction set. The plurality of operations may also include: modifying the shutdown instruction set in response to user input corresponding to at least one custom instruction. The shutdown agent may be a virtual shutdown agent running in a virtual machine within a hyperconverged infrastructure environment. The plurality of operations may also include: after sending the instruction to the network management card to complete the shutdown, the virtual shutdown agent terminates the virtual machine it is running. The plurality of operations may also include: selecting an alternative shutdown instruction set from the plurality of shutdown instruction sets through the shutdown agent; and sending the alternative shutdown instruction set from the hyperconverged infrastructure environment to the network management card. The virtual shutdown agent may be configured to communicate with the network management card using encrypted communication.

[0006] Generally, in one aspect, one or more non-transitory computer-readable media storing a plurality of instructions, when executed by one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations including: receiving, via a network management card (NMC), a selection of a shutdown instruction set from a plurality of shutdown instruction sets supported by the network management card (NMC), the network management card (NMC) being communicatively coupled to a hyperconverged infrastructure (HCI) environment, the plurality of shutdown instruction sets being configured to support a plurality of shutdown procedures for at least two different hyperconverged infrastructure platforms; detecting, via the network management card, that the hyperconverged infrastructure environment is performing a shutdown; and, via the network management card, completing the shutdown at least by executing the shutdown instruction set. The step of completing the shutdown may include performing one or more shutdown operations after all virtual machines in the hyperconverged infrastructure environment have been shut down. The step of completing the shutdown may include turning off power, the power being supplied to the hyperconverged infrastructure environment via an uninterruptible power supply (UPS). The step of detecting that the hyperconverged infrastructure environment is performing a shutdown may include receiving a shutdown command from a virtual shutdown agent operating in a virtual machine within the hyperconverged infrastructure environment. The selection of receiving the shutdown command set may include multiple elements of receiving the shutdown command set from the hyperconverged infrastructure environment.

[0007] One or more embodiments described in this specification and / or listed in the claims may not be included in this overview section. Attached Figure Description

[0008] At least one embodiment will now be discussed with reference to the accompanying drawings, which are not intended to be drawn to scale. The drawings are included to provide illustration and further understanding of the aspects and embodiments, and are incorporated in and form part of this specification, but are not intended to be limiting of the invention. In the drawings, each identical or substantially identical component shown in the various figures is designated by the same reference numerals. For clarity, some components may not be labeled in every figure. In the drawings:

[0009] Figure 1 This is a block diagram of an example system according to an embodiment;

[0010] Figure 2 This is an example flowchart of the operation of assisted shutdown of a network management card for hyperconverged infrastructure according to the embodiments;

[0011] Figures 3A-3E This is a block diagram of an example according to an embodiment;

[0012] Figure 4This is an example flowchart of the operation of assisted shutdown of a network management card for hyperconverged infrastructure according to the embodiments;

[0013] Figure 5 This is an example flowchart of an operation for assisted shutdown of a network management card for hyperconverged infrastructure according to an embodiment; and

[0014] Figure 6 This is a block diagram of an example computer system according to an embodiment. Detailed Implementation

[0015] Due to the technical difficulties associated with virtual machine shutdown, current hyperconverged infrastructure environments typically do not allow for automatic shutdown. For example, when the shutdown agent itself is running within a virtual machine, hyperconverged infrastructure environments present technical challenges in providing an abstraction layer for virtual machine shutdown at the underlying hardware level. Furthermore, current hyperconverged infrastructure environments contain shutdown hardware and programs limited to a single hyperconverged infrastructure platform, thus locking hardware purchasers to that specific platform. These are technical problems in the field of data center technology, and more specifically, technical problems related to shutdown of hyperconverged infrastructure. In at least one embodiment described herein, network management card (NMC)-assisted shutdown for hyperconverged infrastructure incorporates improvements over previous systems and aims to address these technical problems by providing an automatic shutdown service capable of shutting down the entire hyperconverged infrastructure environment without deploying a shutdown agent to a physical computer outside the hyperconverged infrastructure environment. Furthermore, one or more embodiments described herein use a set of shutdown instructions (e.g., scripts and / or other types of instructions) that allows for cross-platform compatibility and reconfiguration of shutdown operations as needed. Additionally, one or more embodiments described herein can be implemented using existing network management card hardware, thereby improving the functionality of existing systems without requiring the purchase of new hardware. These technical solutions are not found in current systems and are therefore not conventional or traditional. These technical solutions are practical applications of network management card technology, solving the aforementioned technical problems and constituting an improvement in this technical field. In at least one embodiment described herein, network management card-assisted shutdown of a hyperconverged infrastructure includes: a hyperconverged infrastructure having a network interface communicatively coupled to a hyperconverged infrastructure environment; receiving, through the network interface, a selection of a shutdown instruction set from a plurality of shutdown instruction sets supported by the network management card from the hyperconverged infrastructure environment, the plurality of shutdown instruction sets being configured to support multiple shutdown procedures for at least two different hyperconverged infrastructure platforms; detecting that the hyperconverged infrastructure environment is performing a shutdown; and completing the shutdown at least by executing the shutdown instruction set. This technical solution is not found in current systems and is therefore not conventional or traditional. This technical solution is a practical application of network management card technology, solving the aforementioned technical problems and constituting an improvement in this technical field.In at least one embodiment described herein, a network management card-assisted shutdown of a hyperconverged infrastructure includes: selecting a shutdown instruction set from a plurality of shutdown instruction sets supported by a network management card communicatively coupled to the hyperconverged infrastructure environment via a shutdown agent operating in a hyperconverged infrastructure environment, the plurality of shutdown instruction sets being configured to support multiple shutdown procedures for at least two different hyperconverged infrastructure platforms; sending the shutdown instruction set from the hyperconverged infrastructure environment to the network management card; and during a shutdown in the hyperconverged infrastructure environment, sending an instruction to the network management card to complete the shutdown, wherein the network management card is configured to respond to the instruction to complete the shutdown by at least executing the shutdown instruction set. This technical solution is not found in current systems and is therefore not conventional or traditional. This technical solution is a practical application of network management card technology, solves the aforementioned technical problems, and constitutes an improvement in the technical field. In at least one embodiment described herein, a network management card-assisted shutdown of a hyperconverged infrastructure includes: receiving, via a network management card communicatively coupled to a hyperconverged infrastructure environment, a selection of a shutdown instruction set from a plurality of shutdown instruction sets supported by the network management card, the plurality of shutdown instruction sets being configured to support multiple shutdown procedures for at least two different hyperconverged infrastructure platforms; detecting, via the network management card, that the hyperconverged infrastructure environment is performing a shutdown; and completing the shutdown via the network management card, at least by executing the shutdown instruction set. This technical solution is not found in current systems and is therefore not conventional or traditional. This technical solution is a practical application of network management card technology, solves the aforementioned technical problems, and constitutes an improvement in the technical field.

[0016] Figure 1 This is a block diagram of an example of system 100 according to an embodiment. In one embodiment, system 100 may include more than Figure 1 The components shown may have more or fewer components. Figure 1 The components shown can be local or remote. Figure 1 The components shown can be implemented in software and / or hardware. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined into a single application and / or machine. Operations described with respect to one component can be performed by another component.

[0017] like Figure 1 As shown, the hyperconverged infrastructure (HCI) environment 104 contains one or more virtual machines 106. Figure 1Only one virtual machine 106 is shown in this document. A hyperconverged infrastructure environment 104 typically contains multiple virtual machines that provide multiple software abstraction layers for compute, storage, and networking services on the underlying hardware (not shown). In this example, at least one virtual machine 106 includes a Virtual Shutdown Agent (VSA) 108. The Virtual Shutdown Agent 108 is a software agent that executes within a virtual machine 106 and is configured to perform services such as virtual machine migration and / or prioritized virtual machine shutdown. For example, the Virtual Shutdown Agent 108 may be a version of PowerChuteAgent developed by Schneider Electric, or another agent. In one embodiment, the Virtual Shutdown Agent 108 is configured to shut down the hyperconverged infrastructure environment 104 with the assistance of a Network Management Card (NMC) 112, as discussed in further detail herein.

[0018] In one embodiment, the network management card 112 is installed in an expansion port of an uninterruptible power supply (UPS) 110. The UPS 110 is configured to help ensure uninterrupted power supply to the hyperconverged infrastructure environment 104 in the event of a main power supply (not shown) failure. The UPS 110 may also be configured to accommodate modular hardware via one or more expansion ports. For example, the UPS 110 may be a model from the Smart UPS series manufactured by Schneider Electric, which includes a SmartSlot port for an optional interface card. The network management card 112 may be a Network Management Card 3 or another model of a network management card manufactured by Schneider Electric. Alternatively, the UPS 110 may be a UPS of another brand or model that supports a network management card 112, and / or the network management card 112 may be a card of another brand or model that provides shutdown assistance as described herein. Alternatively, the network management card 112 may be installed in a server chassis, a standalone rack, or some other component of the system 100. As described herein, installing the network management card 112 in the uninterruptible power supply 110, or otherwise communicatively coupling the network management card 112 to the uninterruptible power supply 110, allows the network management card 112 to instruct the uninterruptible power supply 110 to shut down the hyperconverged infrastructure environment 104 as part of a shutdown procedure. The network management card 112 includes software and hardware configured to manage the characteristics of the hyperconverged infrastructure environment 104. Specifically, the network management card 112 is configured to assist in shutting down the hyperconverged infrastructure environment 104, as further described in detail herein.

[0019] like Figure 1 As shown, the network management card 112 may include a network interface 114. The network interface 114 may include an Ethernet port, a Wi-Fi radio, and... A transmitter and / or another network interface or a combination thereof. The network interface 114 allows the network management card 112 to communicate with one or more components of the system 100 via a network. The network management card 112 includes one or more processors 116 that allow the network management card 112 to execute software and / or firmware instructions (not shown) to assist in shutting down the hyperconverged infrastructure environment 104. Furthermore, the network management card 112 includes a memory 118. The memory 118 can be any type of storage medium or combination thereof configured to store one or more shutdown instruction sets 120. The same memory 118 and / or other memories (not shown) may be configured to store multiple operating instructions (e.g., firmware and / or a software operating system) for the network management card 112.

[0020] A shutdown instruction set 120 refers to a series of instructions used to assist in shutting down the hyperconverged infrastructure environment 104. For example, a shutdown instruction set 120 may include instructions for shutting down multiple services, shutting down multiple virtual machines, logging multiple shutdown events, shutting down the hyperconverged infrastructure environment 104, and / or performing another shutdown action, or a combination thereof. A shutdown instruction set 120 may include a secure shell (SSH) script, another script, and / or another set of instructions, or a combination thereof. In one example, the shutdown instruction set includes code to execute Representational State Transfer (REST) ​​application programming interface (API) calls to one or more virtual machines 106 (e.g., securely via Hypertext Transfer Protocol (HTTPS)). The network management card 112 may be configured to support multiple shutdown instruction sets. For example, different shutdown instruction sets may be required depending on the hyperconverged infrastructure platform executing in the hyperconverged infrastructure environment 104. As used herein, a hyperconverged infrastructure platform is a vendor-specific implementation of hyperconverged infrastructure technology, and each hyperconverged infrastructure platform may be incompatible with hyperconverged infrastructure platforms implemented by other vendors. If the system 100 includes multiple shutdown agents, multiple shutdown instruction sets can be configured based on each agent. Optionally or additionally, configuring a shutdown instruction set 120 may include specifying a maximum amount of time (e.g., "5 minutes or another predetermined maximum duration") that the shutdown instruction set 120 is allowed to run.

[0021] The network management card 112 can be configured to install and / or select from a plurality of available shutdown instruction sets to execute the correct shutdown instruction set for the specific hyperconverged infrastructure environment 104 using the network management card 112. The network management card 112 can be configured to store a plurality of selectable shutdown instruction sets. Optionally or additionally, the network management card 112 can be configured to receive a shutdown instruction set 120 from the virtual shutdown agent 108 or another component of the system 100.

[0022] Table 1 below contains an example of a shutdown instruction set (in this example, a script) for the Nutanix hyperconverged infrastructure platform according to one embodiment.

[0023]

[0024]

[0025] Table 2 below contains an example of a shutdown command set (in this example, a script) for a VSAN hyperconverged infrastructure platform according to one embodiment.

[0026]

[0027] In one example, to trigger a shutdown action, the virtual shutdown agent 108 sends an HTTP POST request to the network management card 112. For example, the virtual shutdown agent 108 can send a request such as:

[0028] / Forms / macontrol1_control_run_shutdown_action=[pcns_agent_ip],[delay]

[0029] Where [pcns_agent_ip] is the Internet Protocol (IP) address of the registered virtual shutdown agent 108, and [delay] is a delay (e.g., seconds or another metric) before the network management card 112 should begin executing the shutdown instruction set 120. This delay allows the virtual shutdown agent 108 sufficient time to power off the virtual machine 106 before any services (e.g., multiple cluster services) that should not be shut down before being stopped by the network management card 112.

[0030] In one embodiment, a user interface 102 refers to hardware and / or software configured to facilitate multiple communications between a user and multiple management features of the hyperconverged infrastructure environment 104. For example, through the user interface 102, a user can provide user input selected from multiple sets of available shutdown instructions, and / or a user can provide one or more user-specific instructions (e.g., a user-specific command of a custom script) for a shutdown instruction set.

[0031] Typically, a user interface 102 provides multiple user interface elements and receives input via these elements. A user interface 102 can be a graphical user interface (GUI), a command-line interface (CLI), a haptic interface, a voice command interface, and / or any other type of interface or a combination thereof. Examples of multiple user interface elements include multiple checkboxes, multiple radio buttons, multiple drop-down lists, multiple list boxes, multiple buttons, multiple toggle keys, multiple text boxes, multiple date and time pickers, multiple command lines, multiple sliders, multiple pages, and multiple forms. Different components of the user interface 102 can be specified as different languages. The behavior of the multiple user interface elements can be specified as a dynamic programming language (such as JavaScript). The content of the multiple user interface elements can be specified as a markup language, such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), or XML User Interface Language (XUL). The layout of the multiple user interface elements can be specified as a stylesheet language, such as Cascading Style Sheets (CSS). Optionally or additionally, multiple aspects of a user interface 102 can be specified as one or more other languages, such as Java, Python, Perl, C++, and / or any other language or a combination thereof.

[0032] In one embodiment, one or more components of the system 100 are implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device may refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, feature-specific hardware devices, hardware routers, hardware switches, hardware firewalls, hardware network address translation (NAT), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handheld devices, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management devices, routers, switches, controllers, access points, and / or client devices.

[0033] Figure 2This is a flowchart illustrating an example of an operation for assisted shutdown of a network management card for a hyperconverged infrastructure, according to one embodiment. Figure 2 One or more operations shown can be modified together, rearranged, or omitted. Therefore, Figure 2 The specific sequence of operations shown should not be construed as limiting the scope of one or more embodiments.

[0034] In one embodiment, the network management card is installed in an uninterruptible power supply (operation 202). As described above, installing the network management card may involve inserting the network management card into an expansion port. Alternatively, as described above, the network management card may be installed in another component or as a standalone component. A Virtual Shutdown Agent (VSA) is installed in the hyperconverged infrastructure environment (operation 204). Installing the Virtual Shutdown Agent may involve installing a software agent in a virtual machine.

[0035] As described above, the network management card uses a shutdown instruction set to assist in shutting down the hyperconverged infrastructure environment. A shutdown instruction set can be selected for the network management card in several ways (operation 206). For example, the shutdown instruction set can be selected based on the hyperconverged infrastructure platform in use. A component (e.g., the virtual shutdown agent, the network management card, or another system component) can detect the hyperconverged infrastructure platform used by the hyperconverged infrastructure environment (operation 208) and select a shutdown instruction set designed for the hyperconverged infrastructure platform. For example, a virtual shutdown agent can query a controller VM to determine which hyperconverged infrastructure platform is being used. Alternatively, a component (e.g., the virtual shutdown agent, the network management card, or another system component) can receive user input to select a specific shutdown instruction set to use (operation 210). The system can also receive user input that modifies a shutdown instruction set (operation 212) or provides a shutdown instruction set entirely specified by the user. If a shutdown command set that is not yet installed on the network management card is selected, the shutdown command set can be transmitted to the network management card via a network or other type of connection (operation 214).

[0036] To shut down the hyperconverged infrastructure environment, shutdown is first initiated (operation 216). Shutdown may be initiated in response to user input and / or by an automated program that detects a shutdown condition (e.g., a security vulnerability, an unsafe operating condition, running on an uninterruptible power supply for a predetermined threshold time, and / or another predetermined rule for initiating shutdown of the hyperconverged infrastructure environment). Initiating shutdown causes the virtual shutdown agent to instruct one or more other virtual machines (VMs) operating within the hyperconverged infrastructure environment to shut down.

[0037] In one embodiment, the virtual shutdown agent does not instruct all virtual machines to shut down. One or more virtual machines (e.g., one or more controller virtual machines and / or one or more virtual machines managing cluster services) may need to remain running until the virtual shutdown agent shuts down its virtual machines, at which point the virtual shutdown agent is no longer available to complete the shutdown procedure. Without delegating the final steps of the shutdown procedure outside the virtual machines, the hyperconverged infrastructure environment cannot be shut down "cleanly," meaning each virtual machine must undergo a complete shutdown procedure and be shut down in the correct order. Terminating multiple virtual machines without fully shutting them down can lead to problems such as data loss and / or corruption, unexpected system behavior, etc.

[0038] This document provides a system and method for performing shutdown of a hyperconverged infrastructure environment with the assistance of a network management card, allowing all virtual machines to be shut down cleanly. During the shutdown procedure, the virtual shutdown agent sends a shutdown command to the network management card (operation 218). The virtual machines of the virtual shutdown agent then shut themselves down (operation 220), and the network management card completes the shutdown procedure (operation 222) without requiring the virtual shutdown agent to remain running. Completing the shutdown procedure may involve various operations, such as shutting down one or more controller virtual machines, shutting down one or more virtual machines managing cluster services, and / or other operations depending on the hyperconverged infrastructure platform and configuration. In one embodiment, the network management card negotiates a connection with each virtual machine to be shut down and instructs each virtual machine to shut down via the connection. This approach allows the multiple virtual machines to be shut down cleanly / gracefully, using appropriate multiple commands / syntaxes for the hyperconverged infrastructure platform in use. The connection may be an SSH connection, a REST API call, or other type of connection, depending on the hyperconverged infrastructure platform. In one example of using SSH, the network management card can use an SSH key pair to negotiate an SSH connection. An SSH key pair allows passwordless login, eliminating the need for the network management card to store multiple login credentials. Alternatively, another authentication and / or connection security protocol can be used. When all virtual machines are shut down, the network management card can instruct the uninterruptible power supply (UPS) to power off the hyperconverged infrastructure environment, at which point the hyperconverged infrastructure environment is completely shut down.

[0039] In one embodiment, the use of multiple shutdown instruction sets allows for the replacement and / or reconfiguration of the multiple shutdown operations performed by the network management card. Specifically, some or all of the shutdown instruction sets used by the network management card can be replaced (operation 224). One or more specific instructions can be replaced, or the entire shutdown instruction set can be replaced. For example, a patch can be applied to replace part or all of the shutdown instruction set. As another example, the network management card can be physically moved to different hyperconverged infrastructure environments using different hyperconverged infrastructure platforms, and the shutdown instruction set can be replaced with a shutdown instruction set that supports different hyperconverged infrastructure platforms. Using multiple shutdown instruction sets makes the network management card compatible with hyperconverged infrastructure platforms that did not even exist when the network management card was manufactured, thereby increasing the functionality and lifespan of the network management card.

[0040] For clarity, detailed examples will be described below. The components and / or operations described below should be understood as examples that may not be applicable to one or more embodiments. Therefore, the components and / or operations described below should not be construed as limiting the scope of one or more embodiments.

[0041] From Figure 3A In the initial example, a hyperconverged infrastructure environment 302 contains three virtual machines (VMs). VM 304 is running a virtualization storage service 305. VM 306 is running a cluster service 307. VM 308 is running a Virtual Shutdown Agent (VSA) 310. The hyperconverged infrastructure environment 302 is coupled to an uninterruptible power supply 312 in which a network management card 314 is installed. In a shutdown procedure, as shown in Figure 3B, the virtual shutdown agent 310 first sends a shutdown command to VM 304. The virtual shutdown agent 310 does not send a shutdown command to VM 306 because the shutdown cluster service 307 is needed after the virtual shutdown agent 310 shuts down. After shutting down VM 304, the virtual shutdown agent 310 instructs the network management card 314 to complete the shutdown procedure, and then, as... Figure 3C As shown, it shuts itself down. The network management card 314 executes its shutdown instruction set, which includes (a) shutting down the controller virtual machine 306, as shown. Figure 3D As shown, and then (b) the power supply to the hyperconverged infrastructure environment 302 is turned off using the uninterruptible power supply 312, as... Figure 3E As shown.

[0042] Figure 4 This is a flowchart illustrating an example of an operation for assisted shutdown of a network management card for a hyperconverged infrastructure, according to one embodiment. Figure 4 One or more operations shown can be modified together, rearranged, or omitted. Therefore, Figure 4 The specific sequence of operations shown should not be construed as limiting the scope of one or more embodiments.

[0043] In one embodiment, Figure 4 One or more operations shown and described herein are performed by a network management card communicatively coupled to a hyperconverged infrastructure environment. The network management card can receive, via a network interface of the network management card, a selection of a shutdown instruction set supported by the network management card (operation 402) from the hyperconverged infrastructure environment. The shutdown instruction set can be one of multiple shutdown instruction sets supported by the network management card and can be configured to support shutdown procedures for at least two different hyperconverged infrastructure platforms. The network management card can receive the selected shutdown instruction set from the hyperconverged infrastructure environment via the network interface. Alternatively, the selected shutdown instruction set may be stored in the network management card at the time of selection.

[0044] Upon receiving the selection of the shutdown command set, the network management card can detect that the hyperconverged infrastructure environment is performing a shutdown (operation 402). For example, the network management card may receive a shutdown command from a Virtual Shutdown Agent (VSA) operating in a virtual machine within the hyperconverged infrastructure environment.

[0045] In response to the detection that the hyperconverged infrastructure environment is performing a shutdown, the network management card can complete the shutdown (operation 406) at least by executing the selected shutdown instruction set. As described herein, the network management card can perform one or more shutdown operations after all virtual machines in the hyperconverged infrastructure environment have been shut down. The steps of completing the shutdown may include turning off power, which is supplied to the hyperconverged infrastructure environment through an uninterruptible power supply (UPS) (configured in the network management card).

[0046] Figure 5 This is a flowchart illustrating an example of an operation for assisted shutdown of a network management card for a hyperconverged infrastructure, according to one embodiment. Figure 5 One or more operations shown can be modified together, rearranged, or omitted. Therefore, Figure 5 The specific sequence of operations shown should not be construed as limiting the scope of one or more embodiments.

[0047] In one embodiment, Figure 5One or more operations illustrated herein and those described herein are performed by a shutdown agent (e.g., a virtual shutdown agent) and / or one or more other programs executed in the hyperconverged infrastructure environment. These programs can be collectively referred to as "HCI-side processes" compared to network management card-side processes executed in a network management card. The HCI-side processes can select a shutdown instruction set supported by a network management card communicatively coupled to the hyperconverged infrastructure environment (operation 502). The selected shutdown instruction set can be one of multiple shutdown instruction sets supported by the network management card and can be configured to support shutdown procedures for at least two different hyperconverged infrastructure platforms. To select the shutdown instruction set, the HCI-side process can detect a specific hyperconverged infrastructure platform used by the hyperconverged infrastructure environment and select a shutdown instruction set that supports the detected hyperconverged infrastructure platform. Alternatively, the HCI-side process can present multiple shutdown instruction sets in a user interface and receive a specific shutdown instruction set selected by user input through the user interface. Optionally, the hyperconverged infrastructure side program may further receive user input corresponding to one or more customized instructions, and modify the instruction set in response to the user input (operation 504).

[0048] The hyperconverged infrastructure side program may send the shutdown command set from the hyperconverged infrastructure environment to the network management card (operation 506). Alternatively, the hyperconverged infrastructure side program may send a selection of a shutdown command set already stored in the network management card, causing the network management card to mark the shutdown command set as selected. During shutdown of the hyperconverged infrastructure environment, the hyperconverged infrastructure side program may send a command to the network management card to complete the shutdown (operation 508). Sending the command to the network management card may cause the network management card to respond to the command to complete the shutdown by at least executing the selected shutdown command set.

[0049] In one embodiment, the shutdown agent (VSA) is a virtual shutdown agent running within a virtual machine in a hyperconverged infrastructure environment. Optionally, after sending the instruction to the network management card to complete the shutdown, the hyperconverged infrastructure-side program will terminate the virtual machine running by the virtual shutdown agent (operation 510).

[0050] As described above, the network management card can support multiple shutdown command sets, including multiple shutdown commands not yet stored in the network management card. Optionally, the hyperconverged infrastructure side program can select an alternative shutdown command set and send the alternative shutdown command set from the hyperconverged infrastructure environment to the network management card (operation 512), thereby enabling the network management card to continue using the alternative shutdown command set.

[0051] In one embodiment, a system includes one or more means, the one or more means including one or more hardware processors configured to perform any of the operations described herein and / or any of the claims.

[0052] In one embodiment, one or more non-transitory computer-readable storage media store a plurality of instructions that, when executed by the one or more hardware processors, cause the one or more processors to perform a plurality of operations as described herein and / or in any of the claims.

[0053] Any combination of the various features and functions described herein may be used according to one embodiment. Various embodiments have been described in the preceding specification with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive. The unique and exclusive references to the scope of the invention, and the content that the applicant intends to be within the scope of the invention, are the textual and equivalent scope of the set of claims set forth in this application, taking into account any subsequent corrections.

[0054] In one embodiment, the techniques described herein are implemented by one or more dedicated computing devices (i.e., computing devices specifically configured to perform specific functions). These dedicated computing devices may be hard-wired to execute multiple technologies and / or may include multiple digital electronic devices, such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or network processing units (NPUs), continuously programmed to execute the multiple technologies. Alternatively or additionally, a computing device may include one or more general-purpose hardware processors programmed to execute the multiple technologies according to program instructions in firmware, memory, and / or other memories. Alternatively or additionally, a dedicated computing device may combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to implement the multiple technologies. A dedicated computing device may include a desktop computer system, a portable computer system, a handheld device, a network device, and / or any other device containing hard-wired and / or program logic to implement the multiple technologies.

[0055] For example, Figure 6This is a block diagram of an example of a computer system 600 according to an embodiment. The computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled to the bus 602 for processing information. The hardware processor 604 may be a general-purpose microprocessor.

[0056] Computer system 600 also includes a main memory 606, such as random access memory (RAM) or other dynamic storage device, coupled to bus 602, for storing information and instructions to be executed by processor 604. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. When these instructions are stored in one or more non-transitory storage media accessible to processor 604, computer system 600 is presented in a dedicated machine customized to perform multiple operations specified in the instructions.

[0057] The computer system 600 also includes a read-only memory (ROM) 608 or other static storage device coupled to the bus 602 for storing static information and multiple instructions for the processor 604. A storage device 610 (e.g., a magnetic disk or optical disk) is provided and coupled to the bus 602 for storing information and multiple instructions.

[0058] Computer system 600 may be coupled to a display 612 via bus 602, such as a liquid crystal display (LCD), plasma display, electronic ink display, cathode ray tube (CRT) monitor, or any other type of device for displaying information to a computer user. An input device 614 (including alphanumeric keys and other keys) may be coupled to bus 602 for transmitting information and multiple command selections to processor 604. Alternatively or additionally, computer system 600 may receive user input via a cursor control 616, such as a mouse, trackball, trackpad, or cursor arrow keys, for transmitting directional information and multiple command selections to processor 604 and for controlling cursor movement on display 612. This input device typically has two degrees of freedom on two axes, a first axis (e.g., x) and a second axis (e.g., y), to allow the device to specify a position in a plane. Alternatively or additionally, computer system 600 may include a touchscreen. Display 612 may be configured to receive user input via one or more pressure sensors, multi-touch sensors, and / or gesture sensors. Alternatively or additionally, the computer system 600 may receive user input via a microphone, camera, and / or some other type of user input device (not shown).

[0059] Computer system 600 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This hardware logic, combined with other components of computer system 600, enables or programs computer system 600 to become a dedicated machine. According to one embodiment, computer system 600 performs the techniques described herein in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. These instructions may be read into main memory 606 from another storage medium (e.g., storage device 610). Executing the sequence of instructions contained in main memory 606 causes processor 604 to perform the processing steps described herein. Alternatively, hardwired circuitry may be used in place of or in combination with software instructions.

[0060] As used herein, the term "storage medium" refers to one or more non-transitory media used to store data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or other magnetic data storage media, CD-ROMs or any other optical data storage media, any physical media with a perforated pattern, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), flash memory-EPROM, non-volatile random access memory (NVRAM), any other memory chip or magnetic tape, content-addressable memory (CAM), and tri-state content-addressable memory (TCAM).

[0061] Storage media differ from transmission media, but can be used in conjunction with transmission media. Transmission media participate in the information transmission between storage media. Examples of transmission media include coaxial cables, copper wires, and optical fibers, containing the conductors that constitute bus 602. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0062] Various forms of media may be involved when transferring one or more sequences of one or more instructions to processor 604 for execution. For example, the multiple instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a network via a network interface controller (NIC) (such as an Ethernet controller or a Wi-Fi controller). A local NIC of computer system 600 may receive data from the network and place the data on bus 602. Bus 602 transfers the data to main memory 606, from which processor 604 retrieves and executes the instructions. Instructions received by main memory 606 may optionally be stored on storage device 610 before or after execution by processor 604.

[0063] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides bidirectional data communication coupling to network link 620 connected to local network 622. For example, communication interface 618 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide data communication connectivity to the appropriate type of telephone line. As another example, communication interface 618 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN. It may also be implemented as a wireless link. In any such implementation, communication interface 618 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0064] Network link 620 typically provides data communication to other data devices via one or more networks. For example, network link 620 may provide connection to host computer 624 or to data devices operated by Internet service provider (ISP) 626 via local network 622. ISP 626, in turn, provides data communication services via a global packet data communication network now commonly referred to as the “Internet” 628. Both local network 622 and Internet 628 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, signals on network link 620, and signals through communication interface 618 (which carry digital data to or from computer system 600) are example forms of transmission media.

[0065] Computer system 600 can send and receive messages, including program code, through multiple networks, network links 620, and communication interfaces 618. In the Internet example, server 630 can send application request codes through the Internet 628, Internet service provider 626, local network 622, and communication interface 618.

[0066] The received code can be executed by processor 604 upon receipt and / or stored in storage device 610 or other non-volatile memory for later execution.

[0067] In one embodiment, a computer network operates software that provides connectivity between a set of multiple nodes, the software utilizing the techniques described herein. Nodes may be local to each other and / or remote. Nodes are connected by a set of links. Examples of links include coaxial cable, unshielded twisted pair, copper cable, fiber optic cable, and virtual links.

[0068] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and Network Address Translation (NAT). Another subset of nodes uses a computer network. Such nodes (also called "hosts") can execute client programs and / or server programs. Client programs make requests for computing services (e.g., requests to execute a specific application and / or retrieve a specific dataset). Server programs respond by performing the requested services and / or returning the corresponding data.

[0069] A computer network can be a physical network, comprising physical nodes connected by physical links. A physical node is any digital device. A physical node can be a function-specific hardware device. Examples of function-specific hardware devices include hardware switches, hardware routers, hardware firewalls, and hardware NAT. Alternatively or additionally, a physical node can be any physical resource that provides computing power to perform tasks, such as a physical resource configured to run various virtual machines and / or applications performing corresponding functions. A physical link is the physical medium connecting two or more physical nodes. Examples of links include coaxial cable, unshielded stranded cable, copper cable, and fiber optic cable.

[0070] Computer networks can be overlay networks. An overlay network is a logical network implemented on top of another network (e.g., a physical network). Each node in the overlay network corresponds to a corresponding node in the underlying network. Therefore, each node in the overlay network is associated with both an overlay address (to address the overlay node) and an underlying address (to address the underlying node that implements the overlay node). Overlay nodes can be digital devices and / or software programs (e.g., virtual machines, application instances, or threads). Links connecting overlay nodes are implemented as tunnels through the underlying network. Overlay nodes at both ends of the tunnel can treat the underlying multi-hop path between them as a single logical link. Tunneling is performed through encapsulation and decapsulation.

[0071] In this embodiment, the client may be located locally on the computer network and / or remotely on the computer network. The client may access the computer network via other computer networks, such as a private network or the Internet. The client may use a communication protocol, such as Hypertext Transfer Protocol (HTTP), to send requests to the computer network. These requests are communicated through interfaces such as client interfaces (e.g., web browsers), program interfaces, or application programming interfaces (APIs).

[0072] In this embodiment, a computer network provides connectivity between clients and network resources. Network resources include hardware and / or software configured to execute server programs. Examples of network resources include processors, data storage, virtual machines, containers, and / or software applications. Network resources are shared among multiple clients. Clients independently request computing services from the computer network. Network resources are dynamically allocated to requesting and / or clients as needed. The network resources allocated to each requesting and / or client may be scaled up or down based on, for example, (a) computing services requested by a specific client, (b) aggregated computing services requested by a specific tenant, and / or (c) the requested aggregated computing services of the computer network. Such a computer network may be referred to as a "cloud network."

[0073] In this embodiment, the service provider offers a cloud network to one or more end users. The cloud network can implement various service models, including but not limited to Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider provides end users with the ability to use the service provider's applications, which execute on network resources. In PaaS, the service provider provides end users with the ability to deploy custom applications onto network resources. Custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides end users with the ability to provision processing, storage, networking, and other basic computing resources provided by the network resources. Any application, including operating systems, can be deployed on network resources.

[0074] In embodiments, various deployment models can be implemented using computer networks, including but not limited to private clouds, public clouds, and hybrid clouds. In a private cloud, network resources are provided to a specific group of one or more entities (the term "entity" as used herein refers to a company, organization, individual, or other entity). Network resources can be local and / or remote for a specific group of entities. In a public cloud, cloud resources are provided to multiple entities (also referred to as "tenants" or "customers") that are independent of each other. In a hybrid cloud, the computer network includes both private and public clouds. The interface between the private and public clouds allows for the portability of data and applications. Data stored in the private cloud and data stored in the public cloud can be exchanged through the interface. Applications implemented in the private cloud and applications implemented in the public cloud may depend on each other. Calls from applications in the private cloud to applications in the public cloud (and vice versa) can be performed through the interface.

[0075] In one embodiment, the system supports multiple tenants. A tenant is a company, organization, enterprise, business unit, employee, or other entity that accesses shared computing resources (e.g., shared computing resources in a public cloud). One tenant may be separate from another tenant (by operations, tenant-specific practices, employees, and / or external identification). The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Several tenants may use the same specific network resources at different times and / or at the same time. Network resources may be local to the tenant's premises and / or remote. Different tenants may have different network requirements for the computer network. Examples of network requirements include processing speed, data storage capacity, security requirements, performance requirements, throughput requirements, latency requirements, resilience requirements, quality of service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may need to meet the different network requirements requested by different tenants.

[0076] In one or more embodiments, tenant isolation is implemented in a multi-tenant computer network to ensure that applications and / or data from different tenants are not shared with each other. Various tenant isolation methods can be used. In one embodiment, each tenant is associated with a tenant ID. Applications implemented by the computer network are tagged with tenant IDs. Additionally or alternatively, data structures and / or datasets stored by the computer network are tagged with tenant IDs. A tenant is allowed access to a specific application, data structure, and / or dataset only when the tenant and the specific application, data structure, and / or dataset are associated with the same tenant ID. For example, each database implemented by a multi-tenant computer network can be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in the specific database. As another example, each entry in a database implemented by a multi-tenant computer network can be tagged with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in the specific entry. However, the database can be shared by multiple tenants. Subscription lists can indicate which tenants are authorized to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is allowed access to a specific application only when their tenant ID is included in the subscription list corresponding to that specific application.

[0077] In this embodiment, network resources (e.g., digital devices, virtual machines, application instances, and threads) corresponding to different tenants are isolated to tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, data packets from any source device within a tenant overlay network can only be transmitted to other devices within the same tenant overlay network. Encapsulation tunnels are used to prevent any transmission from a source device on one tenant overlay network to devices in other tenant overlay networks. Specifically, data packets received from a source device are encapsulated within an outer data packet. The outer data packet is transmitted from a first encapsulation tunnel endpoint (communicating with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with the destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer data packet to obtain the original data packet transmitted by the source device. The original data packet is transmitted from the second encapsulation tunnel endpoint to the destination device within the same specific overlay network.

Claims

1. A network management card, characterized in that, The network management card includes: One network interface, coupled with a hyperconverged infrastructure environment for communication; One or more processors; and One or more non-transitory computer-readable media storing a plurality of instructions, which, when executed by the one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations including: The network interface receives a selection of one of a plurality of shutdown instruction sets supported by the network management card from the hyperconverged infrastructure environment, the plurality of shutdown instruction sets being configured to support multiple shutdown procedures for at least two different hyperconverged infrastructure platforms. The step of detecting that the hyperconverged infrastructure environment is performing a shutdown includes receiving a shutdown command from a virtual shutdown agent operating in a virtual machine in the hyperconverged infrastructure environment; as well as The shutdown is completed at least by executing the shutdown instruction set, wherein the steps of completing the shutdown include performing one or more shutdown operations after all virtual machines in the hyperconverged infrastructure environment have been shut down.

2. The network management card as described in claim 1, characterized in that, The network management card is coupled to an uninterruptible power supply.

3. The network management card as described in claim 2, characterized in that, The shutdown process includes turning off the power supply, which is provided to the hyperconverged infrastructure environment via the uninterruptible power supply.

4. The network management card as described in claim 1, characterized in that, The step of receiving the selection of the shutdown instruction set includes receiving multiple components of the shutdown instruction set from the hyperconverged infrastructure environment via the network interface.

5. The network management card as described in claim 1, characterized in that, The power-off command set has been stored in the network management card before the selection is received.

6. One or more non-transitory computer-readable media, characterized in that, The one or more non-transitory computer-readable media store a plurality of instructions, which, when executed by one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations including: A shutdown agent operating in a hyperconverged infrastructure environment selects a shutdown instruction set from multiple shutdown instruction sets supported by a network management card communicatively coupled to the hyperconverged infrastructure environment, the multiple shutdown instruction sets being configured to support multiple shutdown procedures for at least two different hyperconverged infrastructure platforms. Send the shutdown command set from the hyperconverged infrastructure environment to the network management card; and During a shutdown process in the hyperconverged infrastructure environment, a command is sent to the network management card to complete the shutdown. The network management card is configured to respond to the instruction to complete the shutdown by executing the shutdown instruction set at least once.

7. The one or more non-transitory computer-readable media as claimed in claim 6, characterized in that, The steps for selecting the shutdown command set include: The system detects a specific hyperconverged infrastructure platform used by the hyperconverged infrastructure environment; and The shutdown instruction set is selected based at least on the shutdown instruction set that supports the specific hyperconverged infrastructure platform.

8. The one or more non-transitory computer-readable media as claimed in claim 6, characterized in that, The steps for selecting the shutdown command set include: The multiple power-off command sets are presented in a user interface; and The user interface receives user input selecting the power-off command set.

9. One or more non-transitory computer-readable media as claimed in claim 6, characterized in that, The plurality of operations further include: The shutdown instruction set is modified in response to user input corresponding to at least one custom instruction.

10. One or more non-transitory computer-readable media as claimed in claim 6, characterized in that, The shutdown agent is a virtual shutdown agent running in a virtual machine within a hyperconverged infrastructure environment.

11. One or more non-transitory computer-readable media as claimed in claim 10, characterized in that, The plurality of operations further include: After sending the instruction to the network management card to complete the shutdown, the virtual shutdown agent will terminate the virtual machine that the virtual shutdown agent is running.

12. The one or more non-transitory computer-readable media as claimed in claim 6, characterized in that, The plurality of operations further include: The shutdown agent selects an alternative shutdown instruction set from the plurality of shutdown instruction sets; and The alternative shutdown command set is sent from the hyperconverged infrastructure environment to the network management card.

13. One or more non-transitory computer-readable media as claimed in claim 12, characterized in that, The shutdown agent is configured to communicate with the network management card using encrypted communication.

14. One or more non-transitory computer-readable media, characterized in that, The one or more non-transitory computer-readable media store a plurality of instructions, which, when executed by one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations including: A network management card, which is communicatively coupled to a hyperconverged infrastructure environment, receives a selection of a power-off instruction set from a plurality of power-off instruction sets supported by the network management card, the plurality of power-off instruction sets being configured to support a plurality of power-off procedures for at least two different hyperconverged infrastructure platforms. Detecting that the hyperconverged infrastructure environment is performing a shutdown via the network management card includes receiving a shutdown command from a virtual shutdown agent operating in a virtual machine within the hyperconverged infrastructure environment; and The shutdown is accomplished by executing the shutdown instruction set via the network management card, wherein the shutdown is performed after all virtual machines in the hyperconverged infrastructure environment have been shut down, and one or more shutdown operations are executed.

15. One or more non-transitory computer-readable media as claimed in claim 14, characterized in that, The shutdown process includes performing one or more shutdown operations after all virtual machines in the hyperconverged infrastructure environment have been shut down.

16. One or more non-transitory computer-readable media as claimed in claim 14, characterized in that, The shutdown process includes turning off the power supply, which is provided to the hyperconverged infrastructure environment without interruption.

17. One or more non-transitory computer-readable media as claimed in claim 14, characterized in that, The step of detecting that the hyperconverged infrastructure environment is performing a shutdown includes receiving a shutdown command from a virtual shutdown agent operating in a virtual machine within the hyperconverged infrastructure environment.

18. One or more non-transitory computer-readable media as claimed in claim 14, characterized in that, The selection of receiving the shutdown instruction set includes multiple aspects of receiving the shutdown instruction set from the hyperconverged infrastructure environment.

Images