System and method for provisioning and management of devices

Abstract

There is provided a system and method for provisioning raw metal devices. The provisioning process includes multiple phases. In a first provisioning phase, an operating system and client application are installed on the metal device. Installing the client application may include storing a private key on the metal device, and a public key on a master device. In a second provisioning phase, software applications defining the metal's functions may be installed on the metal device. Master device may require authentication of metals prior to initiating a provisioning phase. In a third provisioning phase, software updates, security patches and configurations for installed applications may be pushed to metals.

Description

SYSTEM AND METHOD FOR PROVISIONING AND MANAGEMENT OF DEVICES FIELD

[0001] This disclosure relates to configuration of devices with software, and in particular to the configuration of devices from a bare hardware state.BACKGROUND

[0002] The Internet of Things (loT) refers to collective networks of connected devices with sensors, processing ability, and other technologies that connect and exchange data with other devices and systems over communication networks. The efficient configuration of loT devices from a raw, unconfigured state (that is, without any operating system) to a configured state remains a relatively challenging and inflexible process.

[0003] There is a need for a system which can efficiently, securely, and flexibly provision and configure raw devices (referred to herein as “raw metals”) with the software and functionality required for operation within an loT system.SUMMARY

[0004] According to an aspect, there is provided a method of provisioning a raw metal device, the method comprising: creating, at a master device, a metal object corresponding to said raw metal device; receiving, at said master device, a first provisioning request from said raw metal device; initiating, by said master device, a first provisioning phase, said first provisioning phase comprising: providing a first glaze image to said raw metal device, said glaze image containing an operating system and a client application; initializing said raw metal device to a configured metal device, said initializing comprising installing said operating system and said client application on said raw metal device; authenticating said configured metal device based on said client application; after authenticating said configured metal device, initiating, by said master device, a second provisioning phase comprising: selecting one or more glaze images for provisioning to said configured metal device, said one or more glaze imagescontaining one or more applications; providing said selected one or more glaze images to said configured metal device; and installing, by said client application, said one or more applications based on said selected one or more glaze images.

[0005] According to another aspect, there is provided a system for provisioning a raw metal device, the system comprising: one or more processors; a non-transitory computer-readable storage medium having stored thereon processor executable instructions that, when executed by said one or more processors, cause said one or more processors to perform a method comprising: creating, at a master device, a metal object corresponding to said raw metal device; receiving, at said master device, a first provisioning request from said raw metal device; initiating, by said master device, a first provisioning phase, said first provisioning phase comprising: providing a first glaze image to said raw metal device, said glaze image containing an operating system and a client application; initializing said raw metal device to a configured metal device, said initializing comprising installing said operating system and said client application on said raw metal device; authenticating said configured metal device based on said client application; after authenticating said configured metal device, initiating, by said master device, a second provisioning phase comprising: selecting one or more glaze images for provisioning to said configured metal device, said one or more glaze images containing one or more applications; providing said selected one or more glaze images to said configured metal device; and installing, by said client application, said one or more applications based on said selected one or more glaze images.

[0006] According to still another aspect, there is provided a non-transitory computer-readable storage medium having stored thereon processor executable instructions that, when executed by one or more processors, cause said one or more processors to perform a method comprising: creating, at a master device, a metal object corresponding to said raw metal device; receiving, at said master device, a first provisioning request from said raw metal device; initiating, by said master device, a first provisioning phase, said first provisioning phase comprising: providing a first glaze image to said raw metal device, said glaze image containing an operating system and a client application; initializing said raw metal device to a configured metal device, saidinitializing comprising installing said operating system and said client application on said raw metal device; authenticating said configured metal device based on said client application; after authenticating said configured metal device, initiating, by said master device, a second provisioning phase comprising: selecting one or more glaze images for provisioning to said configured metal device, said one or more glaze images containing one or more applications; providing said selected one or more glaze images to said configured metal device; and installing, by said client application, said one or more applications based on said selected one or more glaze images.

[0007] Other features will become apparent from the drawings in conjunction with the following description.BRIEF DESCRIPTION OF DRAWINGS

[0008] In the figures which illustrate example embodiments,

[0009] FIG. 1 is a block diagram depicting components of an example computing system;

[0010] FIG. 2 is a block diagram depicting components of an example computing device;

[0011] FIG. 3 depicts a simplified arrangement of software at computing device;

[0012] FIG. 4 depicts an example process of provisioning a device, in accordance with some embodiments;

[0013] FIG. 5 depicts an example metal provisioning and configuration system, in accordance with some embodiments;

[0014] FIG. 6 depicts an example configuration of metals assigned to spaces, in accordance with some embodiments;

[0015] FIGs. 7A and 7B depict an example network address translation (NAT) tables, in accordance with some embodiments;

[0016] FIG. 8 depicts an example process of converting a workflow to glaze images, in accordance with some embodiments;

[0017] FIG. 9 depicts an example process of phase 1 and phase 2 provisioning, in accordance with some embodiments;

[0018] FIG. 10A depicts example provisioning times for provisioning a metal using various strategies, in accordance with some embodiments; and

[0019] FIG. 10B depicts example power consumption figures for provisioning a metal using various strategies, in accordance with some embodiments.DETAILED DESCRIPTION

[0020] Traditional approaches for provisioning metals and managing loT devices over their life cycle typically involve painstaking, manual configuration of the metal’s hardware before final deployment in an loT system. When maintenance and / or updates are required, field visits are typically required. Such an approach is time-consuming, presents security risks, and becomes increasingly inefficient with scale.

[0021] Some existing approaches use unattended configuration mechanisms, in which pre-configured devices (i.e. , devices which already have an operating system installed) can download and install software updates (e.g. security patches, operating system kernel upgrades, user programs, and the like) over-the-air (OTA) from a remote sever using protocols such as HTTPS and MQTT. However, the unattended configuration of raw metals (i.e., having no operating system) requires a separate metal hardware provisioning system.

[0022] FIG. 4 depicts an example process of provisioning a device, in accordance with some embodiments. As depicted, at block 1 , raw metal device 402 may be configured to boot from a network and / or software stack which enables network booting. For example, some embodiments make use of the Preboot Execution Environment (PXE) network stack. In some embodiments, the PXE is available only via wiredethernet connections 403. In other embodiments, network / software stacks which support network booting may be possible using wired and / or wireless connections.

[0023] In some embodiments, network booting may create a client-server execution environment which allows metals 402 to obtain (at block 2) an IP address and an address of a remote hardware provisioner 404 (using, for example, the Bootstrap Protocol (BOOTP) and / or the Dynamic Host Configuration Protocol (DHCP)). At block 3, the metal 402 may apply its assigned IP address. Once metal 402 has an IP address configured, the network booting stack may download a Network Boot Program (NBP) to the memory of the metal 402 using a protocol (e.g., Trivial File Transfer Protocol (TFTP)). The NBP may then enter a second stage of booting, in which metal 402 uses the address of the remote hardware provisioner 404 to request and obtain pre-defined workflow images. Such workflow images may include one or more of an operating system, drivers, binaries and developer libraries, and sometimes user programs and packages defining the functions of the loT metal 402.

[0024] Conventional loT configuration and life cycle management systems typically provide a combination of OTA and metal provisioning services to meet the software needs of the metal 402 at various configuration stages (e.g., in a raw state, and when updates are required). However, conventional systems present significant security risks. For example, when provisioning raw metal 402, it may be difficult to avoid or distinguish between legitimate software and malicious software (due to the metal 402 not having any operating system or bootable software stack which would otherwise be used for authentication once configured).

[0025] In some embodiments, there is provided a multi-phase metal provisioning system which may address some or all of the security challenges noted above, and provide greater flexibility and customizability to loT metals 402. A multi-phase provisioning model may include a first phase which includes provisioning a metal from a raw state to a configured state including the minimum functionality necessary for operation and installation of software programs (phase 1), a second phase which includes providing a configured metal with subsequent binaries specifically for the taskthe metal will be performing in the loT system (phase 2), and a third phase in which one or more of software updates, security patches, and / or configuration changes (e.g. disabling a feature in a program executing on the metal device) are provided to functioning loT metals (phase 3). In some embodiments, each phase in the multi-phase metal provisioning system described herein may require authentication and / or authorization. Although various example embodiments described herein relate to provisioning raw metals for use in Internet of Things systems, it is contemplated that various embodiments described herein are applicable to raw metals of any kind.

[0026] Various embodiments described herein make use of interconnected computer networks and components. FIG. 1 is a block diagram depicting components of an example computing system 100. As depicted, the system 100 includes a variety of clients incorporating and / or incorporated into a variety of computing devices 102 which may communicate with other computing devices 102 via one or more networks 110. For example, a client 102 may incorporate and / or be incorporated into client application implemented at least in part by one or more computing devices.

[0027] Example computing devices may include, for example, at least one server 102 with a data storage 118 such as a hard drive, an array of hard drives, network-accessible storage, or the like; at least one web server 106, and a plurality of client computing devices 108. Server 102, web server 106, and client computing devices 108 may be in communication by way of a network 110. More or fewer of each device are possible relative to the example configuration depicted in FIG. 1. In some embodiments, one or more computing devices may be logically internal to an organization 10 (depicted in FIG. 1 as devices 102, 109, 108 and 106 being internal to organization 10).

[0028] Network 110 may include one or more local-area networks or wide-area networks, such as IPv4, IPv6, X.25, IPX compliant, or similar networks, including one or more wired or wireless access points. The networks may include one or more localarea networks (LANs) or wide-area networks (WANs), such as the internet. In some embodiments, the networks are connected with other communications networks, such as GSM / GPRS / 3G / 4G / LTE / 5G networks.

[0029] FIG. 2 is a block diagram depicting components of an example computing device, such as a desktop computing device 102, metal 402, server 108, mobile computing device, and the like. As depicted, an example computing device may include a processor 114, memory 116, persistent storage 118, network interface 120, and input / output interface 122.

[0030] Processor 114 may be an Intel or AMD x86 or x64, PowerPC, ARM processor, a microcontroller, or the like. Processor 114 may operate under the control of software loaded in memory 116. Network interface 120 connects the computing device to network 110. Network interface 120 may support domain-specific networking protocols for certain peripherals or hardware elements. I / O interface 122 connects the computing device to one or more storage devices and peripherals such as keyboards, mice, pointing devices, USB devices, disc drives, display devices 124, and the like.

[0031] In some embodiments, I / O interface 122 may connect various hardware and software devices used in connection with the systems and methods described herein to processor 114 and / or to other computing devices. In some embodiments, I / O interface 122 may be compatible with protocols such as WiFi, Bluetooth, and other communication protocols.

[0032] Software may be loaded onto one or more computing devices. Such software may be executed using processor 114.

[0033] FIG. 3 depicts a simplified arrangement of software at an example computing device. The software may include an operating system 128 and application software, such as metal provisioning and management system 126. It will be appreciated that in distributed computing environments, implementation, and administration of a service such as system 126 may be distributed amongst a plurality of separate computing devices, and FIG. 3 is intended to depict a simplified logical separation between an operating system 128 and an application executing on an example computing device(s).

[0034] FIG. 5 depicts an example multi-phase metal provisioning and configuration system 500 (also referred to herein as the Toronto Infrastructure Provisioning System (TIPS)), in accordance with some embodiments. In some embodiments, system 500 may be cloud-based. System 500 may be a secure, multiphase metal provisioning system which provides increased control over the metal provisioning process, and improves the security of raw metal provisioning systems.

[0035] Some embodiments described herein relate to a multi-phase provisioning model for loT metals. In some embodiments, the model includes three phases. In the first phase (or phase 1), raw metals 402 may undergo a hardware provisioning process that transforms raw metal 402 to a configured state (i.e. , a state in which an operating system and / or other applications such as client application 530 is installed thereon).

[0036] In the second phase, configured metals 402 may be provided with additional provisioning of user-specific packages and programs. In some embodiments, the second phase may prepared a configured metal 402 to perform the tasks that define the metal’s role in an loT system (e.g., sensing, actuating, and the like).

[0037] In the third phase, configured and functioning loT metals 402 from the second phase may be provided with additional support, such as security patches and / or software updates. Advantageously, in some embodiments, system 500 may require authentication before any of the first, second and third phases are permitted to be performed, which may significantly enhance overall security.

[0038] As depicted in FIG. 5, system 500 includes a master device 510 (denoted as TIPS-M), a gateway device 520 (denoted as TIPS-G) communicatively coupled to the master device 510 via a network 110 (such as the internet), and a client application 530 (denoted as TIPS-C) executing on loT metals 402a, 402b, 402c, 402d. In some embodiments, master device 510 may be integrated within a distributed computing framework, such as a cloud computing environment. In other embodiments, master device 510 functionality may be containerized for deployment on virtual machines and cloud physical servers. As such, in some embodiments, master device 510 may be aphysical device, whereas in other embodiments, master device 510 may be implemented as a node or software on a virtual machine.

[0039] In some embodiments, master device 510 includes a discovery module 511 (denoted as TIPS-D), an updater module 516 (denoted as TIPS-ll) and a controller 519. In some embodiments, the updater module 516 is configured to provision metals 402 with software in the initial phase (configuring raw metal 402) and the third phase (support and / or updates). In some embodiments, controller 519 is configured to perform user management 610, management 620 of loT metals 402, preparation of installation packages for metals (referred to herein as Glaze images), and may provide an application programming interface (API) to enable user interaction with master device 510.

[0040] As depicted in FIG. 5, gateway device 520 is located in the same network environment where metals 402 are deployed or being deployed. In some embodiments, gateway device 520 may perform one or more of metal 402 discovery and registration, acting as the default gateway for loT metals 402 when interacting with master device 520 and other external networks, and acting as a Network Address Translation (NAT) server for loT metals within the same local network. As depicted, gateway device 520 is connected to master device 510 via a network such as the internet 110. It is contemplated that gateway device 520 may support numerous networking connections and protocols, including WiFi and cellular data connections (e.g., 5G, LTE, satellite, and the like).

[0041] As depicted, client application 530 is software which runs on loT metals 402. In some embodiments, client application 530 may be installed on an loT metal during the first phase of the provisioning process along with the operating system. Once installed, client application 530 enables the metal to receive, verify and / or deploy software updates during the second and third phases of the multi-phase provisioning system 500.

[0042] In some embodiments, prior to phase 1 provisioning, a user may create a metal object 602 for a raw metals 402 intended to be provisioned. The metal object 602may be stored on master device 510. A metal object can be created by providing a hardware address (e.g., a MAC address) which corresponds to the raw metal 402. In some embodiments, a metal object 602a is an object (i.e. , a logical metal / device twin component) maintained by master device 510 which contains information about a particular loT metal 402a, including but not limited to one or more of the metal 402a’s network interfaces 624a (as shown, for example, in FIG. 6). A metal object 602a may also be assigned a globally unique identifier 626 (depicted as TIPS ID in FIG. 6). As depicted in FIG. 6, in the case of metals having more than one network interface, the metal object 602 may store hardware addresses for some or all of the network interfaces 624 that can be used for provisioning. In some embodiments, the creation of a metal object 602 may have the effect of entering a network interface of a metal to a “whitelist” maintained by master device 510, such that when metal 402 is powered on in a raw state in the vicinity of gateway device 520, phase 1 of the provisioning process may begin automatically.

[0043] In some embodiments, metal objects 602 may be grouped into logical groupings referred to as spaces 622 (depicted as TIPS Spaces in FIG. 6). For example, a space 622 may contain one metal object 602 (e.g. space 622n) or a plurality of metal objects 602 (e.g. space 622c), and a metal object 602 may be a member of more than one space (for example, metal 402a is a member of space 622a via a network interface 1 , and of space 622c via network interface 2). In some embodiments, a space may be accessible by a plurality of network interfaces 624 belonging to the same metal (e.g., metal 402a can access space 622a via network interface 1 , and network interface 2). Finally, a metal object 602 may belong to a plurality of spaces using the same network interface (e.g., metal object 602a belongs to space 622a and space 622c using network interface 2).

[0044] In some embodiments, a space 622 is a logical provisioning management entity which allows for users to apply software configurations to groupings of loT metals 402, and for different provisioning phases (e.g. phase 1, phase 2, and phase 3). In some embodiments, when a metal object belongs to more than one space 622, a default space may be designated. In some embodiments, the default space is a phase 1space. Once a metal has been configured with client 530, software provisioning for phase 2 and phase 3 may be performed by applying configurations from one or more phase 2 spaces 622, and one or more phase 3 spaces 622, respectively.

[0045] The use of spaces 622 may make the management of loT metals 402 more flexible and customizable. For example, a user can create separate spaces 622 for different programs that are to be deployed on loT metals 402 in phase 2 and phase 3 to define the role and functionality of loT metal 402. When a metal object 602 is a member of multiple phase 2 and phase 3 spaces 622, the loT metal 402 corresponding to the metal object 602 may receive user programs and software updates from all spaces 622 that metal object 602 belongs to. In this manner, users can better manage and control the provisioning of user programs on their loT devices 402.

[0046] In some embodiments, a metal 402 may be a member of a phase 1 space 622 (e.g., a default space) without being a member of both a phase 2 space 622 and a phase 3 space 622. For example, if a metal 402 is a member of a phase 1 space and a phase 3 space, then provisioning using system 500 may be customized such that after phase 1 installation, no phase 2 space workflow is performed, and phase 3 workflows may be performed if and when there is an update, security patch, or configuration change.

[0047] It is further contemplated that a metal 402 may be a member of a phase 1 space 622 and a phase 2 space 622, without being a member of a phase 3 space. In this configuration, phase 1 provisioning may be performed, with user applications subsequently installed in a phase 2 provisioning. However, metal 402 might not receive any updates, security patches, or configuration changes as a result of not being a member of a phase 3 space 622. This may be advantageous in scenarios in which the maximum reliability or continuity of operation is desired (i.e. , no updates or patches are allowed, and user programs on metal 402 function in accordance with builds which are known to be stable and reliable, without the risk of instability or interruptions in performance which are associated with updates). Of course, it is contemplated that a user may subsequently assign a metal object 602 to an additional space afterdeployment (or remove metal object 602 from one or more spaces), and as such, system 500 offers the flexibility to change device management over time, without being forced into a static configuration.

[0048] As noted above, a metal 402 may belong to a space 622 using a network interface 624. In some embodiments, provisioning of a metal 402 is performed using the same network interface 624 that links the corresponding metal object 602 to a space 622.

[0049] Advantageously, in some embodiments, when an loT metal 402 has more than one network interface 624 and is linked to a space 622 by more than one network interface (e.g., metal 402a linked to space 622a by network interface 1 and network interface 2), multi-homing functionality may be utilized, which allows metal 402a to receive data over two or more of the registered network interfaces 624 for the space 622a. The use of a plurality of network interfaces for data transmission and reception may allow for data to be transferred to metal 402 more quickly, which may reduce the total amount of time required for provisioning metal 402 with software programs. In some embodiments, phase 1 provisioning may require the use of a single interwork interface (depending upon the functionality supported by the network booting stack on metal 402a - some network booting stacks may support multi-homing in phase 1 provisioning).

[0050] As depicted in FIG. 5, controller 519 includes a data store which is managed a device and user management (DLIRM) module 612, and stores metal object 602 data and space 622 assignments for each metal object 602. In some embodiments, the data store may include one or more of metal object identifiers 626, metal object network interfaces 624, metal object 402 hardware addresses, and identifiers for spaces 622.

[0051] Returning to FIG. 5, once metal objects 602 and spaces 622 have been created by the user, a raw metal 402 may begin the provisioning process. In some embodiments, provisioning begins with metal discovery 550. For example, metal 402 may begin broadcasting so-called ‘Hello’ messages (e.g., broadcastingDHCPDISCOVER messages) which can be received and processed by a nearby gateway device 520. In the case of metals 402 capable of wireless communications, joining the wireless SSID of a nearby gateway device 520 may include the transmission and receipt of a DHCPDISCOVER message.

[0052] When a gateway device 520 (or gateway instance) requires a DHCPDISCOVER message from a raw loT metal, gateway device 520 may respond by transmitting a DHCPOFFER message, which provides the metal 402 with a private IP address. Upon metal 402 accepting the private IP address offer, metal 402 may transmit a DHCPREQUEST message to gateway 520 requesting configuration. In response, gateway 520 may reply to metal 402 with a DHCPACK acknowledgment message which assigns the IP address to the metal’s network interface 624, configures the gateway 520 as the default network gateway for metal 402, and configures metal 402 to contact master device 510 for additional configurations.

[0053] In some embodiments, gateway device 520 stores the metal 402’s network interface type, hardware address, status, and assigned IP address to the gateway’s NAT table 700. NAT table may enable metal 402 to communicate with one or more of master device 510 and external networks. An example NAT table 700 is depicted in FIG. 7. As depicted, a metal 402a having a unique identifier 626a (e.g.299910) may have multiple entries in NAT table 700, which indicates that the metal 402 has multiple network interfaces 624.

[0054] Upon configuring the metal 402’s network interface with an IP address, the status of the metal’s network interface may be in a “pending” state. In some embodiments, a network interface in the pending state or a blocked state is not authorized to communicate directly with master device 510 or external networks. The pending state is a temporary state which remains in effect for a network interface of metal 402 while the discovery and registration process 552 is in progress.

[0055] While the network interface of metal 402 is in the pending state, gateway device 520 may transmit data about the metal 402 and network interface to discovery module 511 to begin the process of metal registration 552. Once discovery module 511has received the new metal’s information, discovery module 511 may query DURM module 612 at block 554 to determine whether the metal 402 has a corresponding metal object 602. In this manner, the process of registration 552 may ensure that only metals 402 which have had corresponding metal objects 602 created in master device 510 can proceed further with provisioning. For example, discovery module 511 may determine whether the network interface 624 hardware address of metal 402 matches the an existing metal object 602. If so, it is established that metal 402 has an owner and is preauthorized for provisioning in phase 1. For example, if a metal object 602 has a default space 622 configured, then the metal 402 corresponding to metal object 602 is preauthorized for phase 1 provisioning.

[0056] In some embodiments, a metal object 602 may be assigned to only one space 622 which is not the metal’s default space set by the metal’s owner. In this situation, master device 510 does not allow metal 402 to proceed to phase 1 provisioning, as described in further detail below.

[0057] When metal 402 is pre-authorized for phase 1 provisioning, discovery module 511 may confirm the metal’s registration for gateway 520 by transmitting the metal’s globally unique identifier 626 to gateway 520. Upon receipt, the gateway 520 may set the state of the metal 402’s network interface from a pending state to an active state. Once in the active state, metal 402 has permission to communicate directly with master device 510 and / or external networks.

[0058] Returning to block 554, if the metal information shared by discovery module 511 with DURM module 612 is not confirmed to match any metal object 602, then discovery module 511 may reject the registration attempt by metal 402. Discovery module 511 my further transmit a notification to gateway 520 of an unapproved attempt to register using the network interface of metal 402, optionally including the identifier of the metal 402. In some embodiments, gateway 520 may set the metal’s network interface state from the pending state to a blocked state, to prevent further attempts to register by the network interface of metal 402. In some embodiments, system 500 can control access for individual network interfaces, and as such, it is contemplated that asingle metal 402 may have different network interfaces that are in different registration states. For example, as depicted in FIG. 7, the metal having an identifier of 299910 has a Wi-Fi network interface which is blocked, while having an Ethernet network interface which is active.

[0059] In some embodiments, records of registration attempts may be stored in memory, and accessible using a master device API 614. It is contemplated that records of failed registration attempts, and / or records of both successful and failed registration attempts may be used to develop artificial intelligence (Al) or machine learning (ML) models for detecting anomalous behaviour.

[0060] In some embodiments, a network interface 624 with a blocked status may be unblocked by a user. The use may unblock the network interface by adding the blocked interface’s hardware address to the metal object 602 corresponding to the metal 402 having the blocked network interface, and then restart the network interface on the metal. After resetting the network interface, the discovery module 511 may check for the network address in its records. If the network interface address is found to have been previously blocked, the discovery service 511 may release the block and transmit a command to the corresponding gateway 520 to clear the entry from NAT 700 for the blocked network interface, at which point the metal 402 may attempt to join using the now un-blocked network interface 624 through registration process 552 outlined above.

[0061] After the process of metal discovery and registration is complete, metal 402 may be provisioned with one or more software programs. In some embodiments, system 500 may provision software to metals in the form of a container referred to herein as a glaze. A glaze may be conceptualized as a unit of a software update, in that a workflow image containing all software components for configuring an loT metal 402 may be subdivided into a plurality of separate glaze images that can be used during separate provisioning phases.

[0062] FIG. 8 is a block diagram depicting a decomposition of an example workflow from a single binary image 810 into a plurality of glaze images 850, in accordance with some embodiments. As depicted, a glaze image 850 is a self-executable, self-descriptive binary file that includes at least a glaze name 852 which describes the glaze image, the binary data 854 that presents the image content, and a hash 856 of the image data, which may be used by metals 402 in various provisioning phases to verify the integrity of the glaze image 800 data. In some embodiments, hash 856 is an SHA-256 hash of the content in the image. In some embodiments, a glaze image 850 may include a glaze image identifier which provides a unique identifying code to the glaze image.

[0063] In some embodiments, glaze images 850 are required to be provided to a glaze image repository 5161 on master device 510. Without adding a glaze image 850 to glaze image repository 5161 , system 500 may be unable to send the glaze image to metal 402. A glaze image 850 may be added to repository 5161 by using API 614 to prepare and insert glaze images 850. In some embodiments, when a user creates a glaze image 850 for phase 1 provisioning, a private key of the user may be included in the glaze image 850. In some embodiments, a public key associated with the corresponding private key may be stored in DLIRM module 612. The use of public and private keys may provide enhanced security and facilitate authentication for phase 2 and phase 3 provisioning.

[0064] In some embodiments, a user may specify that a particular space 622 can use a particular glaze image 850 for provisioning, and system 500 may cause each metal 402 which belongs to the particular space to be provisioned with a glaze image 850. In this manner, provisioning may be coordinated among groups and / or subgroups of metals 402 and batch updates may be performed.

[0065] As noted above, system 500 may provision an loT metal 402 in multiple phases. In phase 1, the metal 402 is provided with the minimum software pieces required to boot and become available to loT applications. Contrastingly, conventional provisioning systems feature a large scale workflow image 800 deployment which contains all software components needed to configure loT metal 402 all at once. A large scale workflow deployment may require more time relative to a phase 1 installation. Rather than installing all software components at the same time, system 500 may firstperform phase 1 provisioning, and then metal 402 may continue to receive software in phases 2 and 3 after metal 402 is able to boot and be available to loT applications.

[0066] In some embodiments, phase 1 (or the first phase) of provisioning relates to provisioning raw or non-configured metals 402 (i.e. , metals which do not have an operating system). Provisioning of a raw metal 402 may begin with the discovery and registration processes described above. After the discovery and registration process is complete, a metal 402 will have been registered with the discovery module 511, and will have entries in NAT 700 of its corresponding gateway 520 which enable the metal 402 to connect to master device 510 and begin the provisioning process 556.

[0067] In some embodiments, discovery module 511 is configured to determine whether metal 402 has previously connected to master device 510 (e.g., in scenarios in which metal 402 is attempting to connect to a different gateway device 520 than the gateway device 520 which performed the original discovery and registration process -e.g., if metal 402 has been moved to a different location). As such, it is contemplated that a plurality of gateway devices 520 of system 500 may be deployed, and that metal 402 need not be re-configured in order to communicate with a different gateway node or device 520. Since master device 510 stores a copy of the NAT tables 700, 750 of various gateway devices, and discovery module 511 and DLIRM module 612 maintain records of metal objects 602 and previously registered metals 402, system 500 offers security advantages over existing systems, as a plurality of different gateway devices 520 in an environment will be configured to authenticate and verify metals 402 in the same manner.

[0068] FIG. 9 depicts an example workflow of phase 1 and phase 2 provisioning, in accordance with some embodiments. As depicted, provisioner 910 in master device 510 receives a provisioning request from metal 402. Upon receiving the request, provisioner 910 may create a request handler for the metal 402 which sent the request. The provisioner may task the update manager 616 in controller 519 to determine and select the appropriate glaze images 850 from glaze image repository 5161 based on thedefault space for metal 402. The update manager 616 may then transmit, to provisioner 910, an identifier for the for the selected glaze image(s).

[0069] In some embodiments, while master device 510 is processing the provisioning request from metal 402, metal 402 may be receiving initial configurations from provisioner 910. As depicted, provisioner 910 is a Tinkerbell hardware provisioner, which uses the Operating System Install Environment (OSIE) 912 to provide initial configurations to metal 402. In some embodiments, the initial configurations are used by metal 402 to prepare an in-memory execution environment for the provisioning of the glaze images 850 which will be received subsequently.

[0070] As noted above, in some embodiments, provisioner 910 may be a Tinkerbell hardware provisioner, which is an open-source provisioner engine. It is contemplated that system 500 does not require the use of a particular provisioner 910 engine for installing glaze images 850. Instead, some embodiments of system 500 may be configured to work with a variety of underlying metal hardware provisioned. System 500 may provide additional services, components, and mechanisms for guiding various embodiments of the provisioning process described herein, while ensuring the flexibility to use any metal hardware provisioning engine which meets the requirements of a particular organization and use case. As such, system 500 may be considered to be agnostic in terms of which hardware provisioning system is used. For example, some embodiments of system 500 may include a plurality of metal provisioning engines which may work in parallel to meet the needs of different metal 402 hardware technologies. For example, some metal provisioner engines can only work with particular hardware configurations (e.g., ARM CPUs). It should be appreciated that in embodiments using hardware provisioned other than the Tinkerbell provisioning system depicted in FIG. 5, OSIE 912 might not be present (as this is a component of the Tinkerbell system).

[0071] Returning to FIG. 9, after the initial in-memory configuration is complete, the provisioning process continues with the metal 402 downloading the phase 1 glaze image 850 for installation from provisioner 910. As depicted, the glaze image 850 is downloaded from a Tink server, and it will be appreciated that this is an exampleembodiment specific to the Tinkerbell provisioner engine and not intended to be limiting, and that various provisioner engines with different file transfer mechanisms are contemplated for use with system 500.

[0072] After the glaze image 850 has been downloaded by metal 402, the metal 402 installs the image binary and may boot from the image. In some embodiments, the image may install a full operating system with all necessary components (e.g., drivers, bins, developer libraries, and the like), and also includes client application 530. In some embodiments, client application 530 may include the private key that is embedded in the glaze image 850 installed on the metal 402 during phase 1 provisioning. As described above, phase 2 and phase 3 provisioning may be performed using client application 530, which is configured to interact with master device 510 to obtain subsequent glaze images with increased security and flexibility.

[0073] In some embodiments, phase 2 provisioning begins with client application 530 on metal 402 transmitting a signal to master device 510. In some embodiments, the signal from client application 530 may be a phase 2 provisioning request. In some embodiments, the signal from client application 530 may be a signal to master device 510 that client application 530 is operational, which master device 510 may automatically interpret as a request for phase 2 provisioning. In some embodiments, master device 510 may authenticate the client application 530 on metal 402 using the embedded private key together with the corresponding public key which is stored in DLIRM module 612 of master device 510. In some embodiments, different private and public keys may be used for subsequent phase 2 and / or phase 3 installation authentication.

[0074] Once authenticated using the associated keys, system 500 may allow metal 402 to proceed with phase 2 provisioning. In some embodiments, metal 402 may be assigned to one or more phase 2 spaces 622, due to the corresponding metal object 602 having been associated with spaces 622 by the user at the time of metal object creation.

[0075] When metal 402 proceeds to phase 2 provisioning, a new request handler is created by master device 510 and transmitted to update manager 616. Update manager 616 may select the glaze images 850 that will be delivered to metal 402 for installation. In some embodiments, the update manager 616 may obtain data for metal 402 from DURM module 612, which stores the metal 402’s space configurations and the glaze images 850 associated with each space 622.

[0076] In some embodiments, update manager 616 may share the selected glaze image identifiers with the OTA updater 920 of updater module 516 of master device 510. Metal 402 may then download phase 2 glaze images 850 from OTA updater 920, and install the glaze images 850 using the locally stored client application 530.

[0077] After phase 1 and phase 2 provisioning is complete, system 500 enters the third phase of provisioning. In phase 3, fully configured loT metals 402 may continue to receive software updates (e.g., security patches and / or program updates) as long as metals 402 are registered and active. In contrast to phases 1 and 2, phase 3 provisioning may be initiated by master device 510 rather than metal 402. In some embodiments, client application 530 executing on a metal 402 may subscribe to master device 510 for phase 3 updates.

[0078] In some embodiments, master device 510 may transmit a notification to metals 402 when there is a software update available. In some embodiments, a software update may be associated with a space 622, which may cause software updates to be pushed to each metal 402 assigned to that space 622. During the phase 3 update process, the identity of metal 402 may be verified, and authentication may be performed using the private key embedded in client application 530 and the public key stored in DURM module 612, which may ensure that only authorized metals 402 can receive further software updates. In some embodiments, master device 510 may be configured to maintain a maximum length of time for authentication for a give metal 402. For example, if the previous successful authentication for metal 402 was outside of a threshold period of time (e.g., 120 seconds), master device 510 may require an additional authentication before new updates are transmitted.

[0079] In some embodiments, authenticated metals 402 may receive a notification when there is a phase 3 update available in any of the spaces 622 in which a metal is registered. In some embodiments, phase 3 update notifications may include instructions on how to obtain a software update (e.g., glaze image identifiers, update schedules, and the like) from master device 510.

[0080] In some embodiments, metals 402 may be limited to using only a wired ethernet network interface during phase 1 provisioning, depending on the network / software stack used for network booting on metal 402. For example, a PXE may require a wired connection. In other embodiments, the network / booting stack of metal 402 may support wired and / or wireless connections. In some embodiments, client application 530 may support phase 2 and 3 provisioning using other types of network interfaces, including but not limited to Wi-Fi, Bluetooth, and Long Range WAN (LoRa). FIG. 7B is an example NAT table 750 which includes Bluetooth and LoRa interfaces for various metals 402.

[0081] In some embodiments, metals 402 having a Bluetooth interface may pair with a gateway 520 instance in physical proximity. To be able to pair with gateway 520, the Bluetooth interface may have to be in discovery mode. Once gateway 520 and the Bluetooth interface are detected, a handshake may be performed using unique physical addresses (e.g. MAC addresses), and communication may begin using an appropriate communication protocol (e.g., the Logical Link Control and Adaptation Layer Protocol (L2CAP), ora protocol offering similar functionality). In some embodiments, gateway 520 may act as a protocol proxy which handles communicate between the Bluetooth interface and master device 510.

[0082] In some embodiments, metals 402 having a LoRa interface may communicate with gateway 520 using an appropriate protocol (e.g., the LoRaWAN protocol, or a protocol offering similar functionality). When devices with a LoRa interface are detected by gateway 520, gateway 520 may act as a LoRa network server which handles data exchanges between LoRa interfaces and master device 510. In some embodiments, functioning as a LoRa network server may include the use of existingLoRa server software. Some examples of LoRa server software may include Loriot, Thingsboard, Chirpstack, and the like. However, it is contemplated that customized network server software may be created for gateway device 520 to implement a LoRa network server in some embodiments.

[0083] Bluetooth and LoRa interfaces operate differently from IP-compatible network interfaces (e.g. Ethernet and Wi-Fi), since Bluetooth and LoRa do not have associated IP addresses. Gateway device 520 when acting as a NAT server may work as a protocol proxy which encapsulates one or both of Bluetooth and LoRa frames in application layer messages to communicate with master device 510. Likewise, responses from master device 510 may be unpacked and converted to Bluetooth and / or LoRa frames for delivery to metals 402 by gateway 520.

[0084] In some embodiments, LoRa and / or Bluetooth network interfaces may initially be in the pending state in the NAT table, until authorized by master device 510. After communication is established between gateway 520 and a metal 402 using Bluetooth or LoRa interfaces, the interface will remain in the pending state, and gateway device 520 may send a message to discovery module 511 containing the network interface address information. Discovery module 511 may then verify the interface hardware address with DURM module 612.

[0085] If the interface hardware address is validated by DURM module 612, then the unique metal identifier may be provided to gateway 520 for association with the interface in NAT table 750, and the interface state may be set to Active rather than Pending, and metal 402 can use the interface for communicating with master device 510 and external networks. If discovery module 511 does not validate the network interface’s hardware address, then the interface state will be set to “Blocked”, which prevents communication by the interface with master device 510 and external networks.

[0086] As noted above, gateway device 520 may act as an NAT server for metals 402. FIGs. 7A and 7B depict example NAT tables 700, 750 which may be maintained at gateway device 520. It should be noted that although Ethernet, Wi-Fi, and Bluetoothinterfaces use 48-bit MAC addresses, LoRa interfaces may use the 64-bit DevELII identifier.

[0087] In some embodiments, a metal 402 may belong to more than one space (as depicted, for example, in FIG. 6). After phase 1 provisioning is complete, it is contemplated that metal 402 may receive phase 2 glaze images 850 from a plurality of spaces 622 and / or via a plurality of network interfaces. In some embodiments, metal 402 may receive glaze images contemporaneously from different spaces. For example, metal 402d may receive a phase 2 glaze image 850 associated with its registration in space 622c via network interface 1 , and may also receive a phase 2 glaze image 850 associated with space 622b in parallel via network interface 2.

[0088] In some embodiments, when a metal 402 has a plurality of network interfaces associated with a same space 622, glaze images 850 may be transmitted over a plurality of network interfaces to increase bandwidth. For example, metal 402a may receive the data for a glaze image 850 from space 622a over both network interface 1 and network interface 2. The increase in available bandwidth by using two network interfaces (also referred to as “multi-homing”) may result in faster provisioning times, as more data can be transmitted to the metal 402 in a shorter period of time. For example, software updates can be delivered to loT metal 402 over all available network interfaces that have been registered with master device 510 and are enabled in the NAT table of gateway device 520. Moreover, the ability to use multiple network interfaces of metal 402 may improve the reliability of the provisioning process, as the failure of one network interface might not necessarily interrupt or end the provisioning process when other network interfaces are still operational.

[0089] In some embodiments, the selection of which network interface to use for a phase 2 or 3 glaze image 850 transfer may be determined at least in part based on the size of the glaze image 850 and / or the distance of metal 402 from master device 510. For example, a relatively small-sized update might be transmitted over one or both of LoRa and Bluetooth interfaces for energy saving purposes. Likewise, if the metal 402is a far distance from master device 510, the LoRa interface may be usable for distances up to 15-20km, in some embodiments.

[0090] It should be noted that long range functionality may provide benefits beyond provisioning of glaze images 850. For example, if a smart vehicle is stolen and driven a distance away from its original location, a LoRa interface might be useful for disabling one or more applications executing on metal 402 or network interfaces of metal 402, which may, for example, disable functionality to allow the vehicle to be started. Such a command to disable the vehicle may be in the form of a phase 3 update, in which the owner of the vehicle may use master device API 614 to request system 500 to disable the functionality of software that allows the engine to start on the vehicle. Such a request may be carried out by disabling the particular software functionality on the phase 3 space 622, which would result in master device 510 notifying metal 402 (via client application 530) that an update is available, and the metal 402 executing the change of configuration to disable engine starts.

[0091] In some embodiments, the multi-phased provisioning approach described herein may provide performance advantages, both in terms of average provisioning time and energy consumption, relative to approaches in which all of the software to be installed on metal 402 is provided at one time in a single package. FIGs. 10A and 10B depict experimental results comparing the average provisioning time and energy consumption level, respectively, observed using a single, 4GB full workflow image 800 containing all software to be installed, and three glaze images 850 of the same workflow divided into a phase 1 operating system glaze image (3GB in size), and two phase 2 glaze images containing user-specific packages and programs (500MB per glaze image). In this example, the metals 402 were Raspberry Pi 4 Model B with 8GB of RAM, the gateway device 520 was a server with a Core i5 CPU and 8GB of RAM, and the master device 510 was virtualized in a cloud server, using a virtual machine with 16 vCPUs and 16GB of RAM. In the experiment, metals 402 were connected to gateway 520 using a gigabit ethernet switch.

[0092] As depicted in FIG. 10A, the provisioning time for the phase 1 glaze packet (283 seconds) for 5 metals 402 was lower than the provisioning time required for the full workflow image (332 seconds), and the total time for phase 1 and phase 2 provisioning was 361 seconds. Experimental data is also depicted for systems with 10 metals and 15 metals. As can be seen, loT metals 402 can become available to loT applications about 15-18% faster using a phase 1 provisioning compared to a full workflow image install. The time savings may be useful for allowing loT metals 402 to receive another round of authentication and authorization before being programmed to perform the actual loT system tasks. In some embodiments, an additional round of authentication and authorization may enhance the security of the system, as the risk of nefarious software being installed in phase 2 is lower than in the case of full workflow image installs.

[0093] As depicted in FIG. 10B, it can be observed that the power consumption associated with a full workflow image 800 install compared to the total power consumption of a phase 1 and phase 2 install is marginally lower. As such, there appears to be a marginal power consumption overhead increase in exchange for the significant improvement in security and flexibility provided by some embodiments of system 500.

[0094] Various embodiments described herein may offer numerous advantages and benefits relative to conventional provisioning systems. For example, conventional systems require that the entire software stack for a metal be installed at one time, which presents a risk of a tampered software stack being installed on metal 402.Contrastingly, system 500 may divide the provisioning process into separate phases, in which each phase begins with an authentication and / or authorization step which prevents unauthorized details from being provisioned. Moreover, system 500 provides the opportunity to interrupt or otherwise cancel the provisioning process for a metal 402, whereas conventional systems do not allow for the safe interruption of the full workflow image provisioning process. Moreover, during the provisioning process, a user may, via master device 510’s API, block or disable individual network interfaces on anyregistered metal device 402 without having to manually interact with individual metals 402.

[0095] The multi-phase provisioning approach may offer numerous security benefits that improve the overall security of metal provisioning and life cycle management. For example, system 500 may improve the responsiveness of an loT system to changes in security requirements. In systems which require tight security requirements which can change depending on system conditions (e.g. smart transportation, smart grids, utility networks, and the like), the provisioning of new loT metals 402 is a time-sensitive process which can introduce delays in applying updated and new security measures and policies. However, using system 500, loT metals 402 can be made available to loT systems and applications much sooner, which may allow new security policies to be applied on new loT metals 402 (in some cases, security policies can be applied even before metal 402 has obtained the phase 2 provisioning which provides the programs that define the metal’s role in the loT system).

[0096] Some embodiments of system 500 may protect loT systems from unauthorized operating systems and / or mass deployment of malicious software stacks on metals 402 in loT systems. By verifying the metal 402’s access to workflow images and the programs the metal may receive during each provisioning phase, the unauthorized deployment of malicious software stacks may be prevented. For example, since phase 1 only includes operating system installation and client application 530 installation, and is only performed when a metal object 602 corresponding to metal 402 has been created by the user, no program can be installed on metal 402 without having been provisioned by the master device 510. In some embodiments, if a program is manually installed on metal 402 (i.e. after phase 1 provisioning is complete), client application 530 inform master device 510, which make take action in response. For example, master device 510 may block all of the metal 402’s network interfaces from accessing external networks (e.g. by altering NAT table 700 in gateway 520).

[0097] Of course, the above-described embodiments are intended to be illustrative only and in no way limiting. For example, although this disclosure includessome example embodiments relating to network devices, it should be appreciated that systems and methods described herein are applicable to many other types of devices, including computing devices more broadly. The described embodiments are susceptible to many modifications of form, arrangement of parts, details, and order of operation. The invention is intended to encompass all such modifications within its scope, as defined by the claims.

Claims

WHAT IS CLAIMED IS:

1. A method of provisioning a raw metal device, the method comprising:creating, at a master device, a metal object corresponding to said raw metal device;receiving, at said master device, a first provisioning request from said raw metal device;initiating, by said master device, a first provisioning phase, said first provisioning phase comprising:providing a first glaze image to said raw metal device, said glaze image containing an operating system and a client application;initializing said raw metal device to a configured metal device, said initializing comprising installing said operating system and said client application on said raw metal device;authenticating said configured metal device based on said client application;after authenticating said configured metal device, initiating, by said master device, a second provisioning phase comprising:selecting one or more glaze images for provisioning to said configured metal device, said one or more glaze images containing one or more applications;providing said selected one or more glaze images to said configured metal device; andinstalling, by said client application, said one or more applications based on said selected one or more glaze images.

2. The method of claim 1 , wherein said metal object includes a unique identifier for said raw metal device and an address for at least one network interface of said raw metal device, and wherein said method further comprises:prior to said receiving said first provisioning request, receiving, at a gateway device, a request for provisioning from a network interface of said raw metal device;determining, by said master device, whether an address of said network interface of said raw metal device matches said address for said at least one network interface in said metal object;authorizing communication between said raw metal device and said master device via said gateway device, based on whether said network interface address matches said metal object.

3. The method of claim 1 , wherein said metal object is associated with a space defining a configuration for metals in said space.

4. The method of claim 3, wherein said metal object is associated with said space by a first network interface.

5. The method of claim 4, wherein said metal object is associated with said space by a second network interface.

6. The method of claim 1 , wherein said metal object is associated with a first space of said spaces by a first network interface, and wherein said metal object is associated with a second space of said spaces by said first network interface.

7. The method of claim 3, wherein said space defines one or more glaze images to be provisioned to members of said space, and wherein said selecting said one or more glaze images for provisioning to said configured metal device in said second provisioning phase is based on said configuration defined by said space associated with said metal object.

8. The method of claim 4, wherein said providing said selected one or more glaze images comprises transmitting said selected one or more glaze images to said client application using said first network interface.

9. The method of claim 5, wherein said providing said selected one or more glaze images comprises transmitting said selected one or more glaze images to said client application using said first network interface and said second network interface in parallel.

10. The method of claim 1 , further comprising maintaining, by a gateway device communicatively coupled to said master device, a network address translation (NAT) table including one or more network interfaces associated with said metal device.

11. The method of claim 10, wherein said one or more network interfaces include at least one of ethernet, Wi-Fi, Bluetooth, and Long Range (LoRa).

12. The method of claim 10, wherein said NAT table includes a state for each of said one or more network interfaces, said state including one of pending, active, and blocked.

13. The method of claim 1 , wherein said first glaze image comprises glaze data corresponding to a program, a glaze name, and a hash code.

14. The method of claim 1 , further comprising:receiving, at said configured metal device, a notification from said master device that an update is available, said notification including an identifier for a glaze image containing said update;receiving, at said master device, a request for provisioning of glaze image corresponding to said identifier;transmitting, by said master device, said glaze image corresponding to said identifier to said configured metal device; andinstalling said update based on said glaze image.

15. The method of claim 1 , wherein authenticating said client application of said configured metal device comprises comparing a private key associated with said client application on said configured metal device to a public key associated with said client application on said master device.

16. The method of claim 1 , further comprising, prior to said installing, by said client device, said one or more applications, validating said selected one or more glaze images based on a hash code contained in each of said glaze images.

17. The method of claim 1 , further comprising, prior to authenticating said configured metal device, receiving a signal from said client application.

18. A system for provisioning a raw metal device, the system comprising:one or more processors;a non-transitory computer-readable storage medium having stored thereon processor executable instructions that, when executed by said one or more processors, cause said one or more processors to perform a method comprising:creating, at a master device, a metal object corresponding to said raw metal device;receiving, at said master device, a first provisioning request from said raw metal device;initiating, by said master device, a first provisioning phase, said first provisioning phase comprising:providing a first glaze image to said raw metal device, said glaze image containing an operating system and a client application;initializing said raw metal device to a configured metal device, said initializing comprising installing said operating system and said client application on said raw metal device;authenticating said configured metal device based on said client application;after authenticating said configured metal device, initiating, by said master device, a second provisioning phase comprising:selecting one or more glaze images for provisioning to said configured metal device, said one or more glaze images containing one or more applications;providing said selected one or more glaze images to said configured metal device; andinstalling, by said client application, said one or more applications based on said selected one or more glaze images.

19. A non-transitory computer-readable storage medium having stored thereon processor executable instructions that, when executed by one or more processors, cause said one or more processors to perform a method comprising:creating, at a master device, a metal object corresponding to said raw metal device;receiving, at said master device, a first provisioning request from said raw metal device;initiating, by said master device, a first provisioning phase, said first provisioning phase comprising:providing a first glaze image to said raw metal device, said glaze image containing an operating system and a client application;initializing said raw metal device to a configured metal device, said initializing comprising installing said operating system and said client application on said raw metal device;authenticating said configured metal device based on said client application; after authenticating said configured metal device, initiating, by said master device, a second provisioning phase comprising:selecting one or more glaze images for provisioning to said configured metal device, said one or more glaze images containing one or more applications;providing said selected one or more glaze images to said configured metal device; andinstalling, by said client application, said one or more applications based on said selected one or more glaze images.

Images