Allocating messages using guaranteed or synchronized delivery times

Abstract

Various embodiments of devices and methods for multicast, multiple unicast, and unicast allocation of messages using time synchronized delivery are described. In some embodiments, the disclosed systems and methods include a reference timer that provides a reference clock to one or more host computing devices. The one or more host computing devices host compute instances and also contain respective isolated timing hardware that is uncontrolled by the compute instances. The isolated timing hardware of the one or more host computing devices then receives respective packets and obtains the same time to deliver the respective packets. Upon determining that the same specified time for delivering the packet has occurred, each isolation timing hardware provides the packet or information for accessing the packet to its respective destination compute instance. Thus, the respective packets are delivered to one or more destination compute instances almost simultaneously.

Description

Background Art

[0001] In order to improve the utilization of computing resources such as in a data center, virtualization technology can be used to allow a single physical computing device to host one or more virtual machine instances, which appear to the user as independent computing devices and operate as independent computing devices. Using virtualization, a single physical computing device can dynamically create, maintain, delete, or otherwise manage virtual machines. Users of virtualized computing services can then request computer resources, including configurations of single computing devices or networked computing devices, and obtain different quantities of virtual machine resources. These virtual machines can perform various functions that might originally be implemented on non-virtualized hardware devices, such as calling network-accessible services, performing data processing, etc.

[0002] In some cases, the sender of information (such as a packet) may wish to perform multicast or group communication. In multicast, data transmission is addressed to a group of destination computers so that packets are sent from the sending entity or group of entities (e.g., a computer or group of computers) simultaneously or nearly simultaneously, but may arrive at different destination entities (e.g., computers) at different times, depending on network conditions between the sending entity and the receiving entity and other factors. Multicast can be application layer multicast or network-assisted multicast, where the latter allows the sender to efficiently send to the group in a single transmission. For example, in network-assisted multicast, replicas are automatically created in other network elements such as routers, switches, and cellular network base stations, but only in the network segments that currently contain group members. Network-assisted multicast can be implemented at the data link layer or the Internet layer using IP multicast. In IP multicast, the implementation of the multicast concept is performed at the IP routing level, where routers create the best distribution path for datagrams sent to the multicast destination address. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A Depicted is a logical model of a sender sending a group of multicast packets to multiple receivers according to some embodiments, wherein the multicast packets include or are associated with specified times for delivering respective packets, wherein the packets arrive at multiple isolated timing hardware elements associated with the receivers at different times, and the respective isolated hardware elements release respective packets to respective receivers at the same specified times based on the delivery times specified for the multicast packets, such that the receivers receive the packets at approximately the same time.

[0004] Figure 1BDepicted is a logical model of a sender sending a group of multiple unicast packets to multiple receivers according to some embodiments, wherein the multiple unicast packets include or are associated with specified times for delivering respective packets, wherein the packets arrive at multiple isolated timing hardware elements associated with the receivers at different times, and the respective isolated hardware elements release respective packets to respective receivers at the same specified times based on the delivery times specified for the multiple unicast packets, such that the receivers receive the packets at approximately the same time.

[0005] Figure 1C Depicted is a logical model of a sender sending a unicast packet to a receiver according to some embodiments, wherein the unicast packet includes or is associated with a specified time for delivering the packet, wherein the packet arrives at an isolated timing hardware element associated with the receiver, and the isolated hardware element releases the packet to the receiver based on the delivery time specified for the unicast packet.

[0006] Figure 2 A logical model of a receiving host computing device is depicted, the receiving host computing device being configured to participate in multicast, multi-unicast, and / or unicast distribution of messages using time-guaranteed and / or synchronized delivery, wherein the receiving host computing device uses a set of computing resources to host a computing instance and also includes isolated timing hardware that is not controlled by the computing instance. In addition, a time synchronization agent uses a dedicated timing network to synchronize a hardware clock of the isolated timing hardware. Moreover, the isolated timing hardware includes packets and packet delivery time receivers configured to receive packets from a data network and provide the packets to a packet storage data structure. In addition, according to some embodiments, a packet delivery determination component of the isolated timing hardware is configured to determine when to deliver a packet based on the hardware clock and a specified delivery time indicated for the packet, and to notify a data structure manager / packet provider to deliver the packet from the data structure to the computing instance hosted on the host computing device at the specified delivery time indicated for the packet.

[0007] Figure 3A logical model of an alternative configuration of a receiving host computing device is depicted, the receiving host computing device being configured to participate in multicast, multi-unicast, and / or unicast distribution of messages using time-guaranteed and / or synchronized delivery, wherein the receiving host computing device uses a set of computing resources to host a computing instance and also includes isolated timing hardware that is not controlled by the computing instance. For example, delivery of a packet can be guaranteed to be performed at a delivery destination at a guaranteed time, wherein the delivery time is guaranteed to be highly accurate based on a synchronization time indicated at an isolated timing hardware clock at or near the destination location, wherein the isolated timing hardware releases the packet or checks the information required for the packet at the specified "guaranteed" time. In addition, in some embodiments, the delivery of multiple packets can be synchronized to be performed simultaneously at different destinations, such as at synchronized guaranteed delivery times. Moreover, in some embodiments, synchronized delivery can be performed at synchronized times determined by a packet delivery mechanism, and the sender does not have to specify the synchronized delivery time, but only specifies that the packet group will be delivered in a synchronized manner at the synchronized time. In addition, the time synchronization agent uses a dedicated timing network to synchronize the hardware clock of the isolated timing hardware. Furthermore, the isolated timing hardware includes a packet and a packet delivery time receiver configured to receive packets from a data network and provide the packets to a packet encryptor that encrypts the packets, wherein the packet encryptor provides the encrypted packets to a storage manager / packet provider / key provider and provides the encryption key to an encryption key storage device. According to some embodiments, the packet provider can provide the packets to the computing instance without delay, and the encryption key delivery determination component determines when to deliver the encryption key based on the specified delivery time of the packet, and notifies the storage manager / key provider to deliver the encryption key from the encryption key storage device at the specified time.

[0008] Figure 4 A logical model of another configuration of a receiving computing device is depicted, the receiving computing device being configured to receive packets with guaranteed and / or synchronized delivery times. The computing device (which may be a single computer or a host hosting multiple computing instances) is communicatively coupled to a network switch, the network switch containing isolated timing hardware and not under the control of the receiving computing device. A time synchronization agent of the isolated timing hardware of the network switch uses a dedicated timing network to synchronize the hardware clock of the isolated timing hardware. The isolated timing hardware of the network switch also includes packets and packet delivery time receivers configured to receive packets from a data network at specified delivery times and provide the packets to a packet storage data structure. According to some embodiments, the isolated timing hardware of the network switch also includes a packet delivery determination component configured to determine when to deliver a packet based on the hardware clock and the specified delivery time of the packet, and configured to notify a data structure manager / packet provider to deliver the packet from the data structure at the specified time. In some embodiments, similar to Figure 2The encryption / encryption key delivery system shown can be used in Figure 4 The isolated timing hardware of the network switch shown is implemented in FIG.

[0009] Figure 5 An example system environment is depicted according to some embodiments including a provider network in which embodiments of the present disclosure may be implemented, wherein client computing device multicast, multi-unicast, and / or unicast packets are sent to multiple instances at multiple host computing devices, and wherein highly accurate time information is provided from a reference timer to isolated timing hardware of the host computing devices using a data or dedicated timing network.

[0010] Figure 6 A logical model of a computing instance is depicted, the computing instance being configured to perform multicast, multi-unicast, and / or unicast distribution of messages using guaranteed and / or synchronized delivery times, wherein a future delivery time calculation component is implemented as a service of a provider network or as a component of isolated timing hardware that is communicatively coupled to a sending computing instance. A sending application of the sending computing instance is configured to communicate with the service or isolated timing hardware via an API processed by an API processor, wherein a packet and delivery time sender provides outgoing multicast, multi-unicast, and / or unicast packets and delivery time information to a receiving computing instance when sending a given multicast, multi-unicast, and / or unicast packet. Furthermore, according to some embodiments, the sending computing instance or its associated isolated timing hardware includes a statistics receiver, the statistics receiver being configured to receive incoming statistics from one or more receiving host computing devices and use the statistics to adjust future delivery time calculations to deliver additional multicast, multi-unicast, and / or unicast packets.

[0011] Figure 7 A logical model of a sending resource according to some embodiments is depicted, wherein a future delivery time calculation component is implemented in software on a computing instance or in an OS application hosted on the sending resource, wherein a packet and delivery time sender provides outgoing multicast, multi-unicast and / or unicast packets with outgoing packets and delivery time information to a receiving computing instance through receiver isolation timing hardware, and wherein a statistics receiver receives incoming statistics from one or more receiving host computing devices and uses the statistics to adjust the future delivery time calculation to deliver additional multicast, multi-unicast and / or unicast packets.

[0012] Figures 8A-8DDepicted is a logical model of a progression of events for a sender sending a set of multicast and / or multi-unicast packets to multiple receivers according to some embodiments, wherein the set of multicast and / or multi-unicast packets include specified times for delivery of respective packets, wherein the respective packets arrive at a plurality of isolated timing hardware elements at different times, wherein each isolated timing hardware is associated with a receiver, the sender then receiving statistical information regarding reception of the packets from the isolated timing hardware of the receivers, sending a second multicast and / or multi-unicast packet to the receivers, the second multicast and / or multi-unicast packet including a different specified time for delivery of the packets at a longer time increment (e.g., a future delivery time), and receiving additional statistical information regarding reception of the second packet from the isolated timing hardware of the receivers.

[0013] Figures 9A-9D Different options are presented for sender isolated timing hardware or provider network services to package multicast, multi-unicast and / or unicast packets received from a sending resource along with future delivery times in order to provide the packet and delivery time information to a data network for transmission to multicast, multi-unicast and / or unicast packet recipients in accordance with some embodiments.

[0014] Fig.10 is a flow chart of an illustrative method for multicasting, multi-unicasting, and / or unicasting distribution of messages using time-guaranteed and / or synchronized delivery, wherein a networked computing resource receives a packet from a sender at a first reception time, wherein the packet is not accessible to a particular computing instance, obtains a specified time for delivery of the packet from the packet or from information separate from the packet, wherein the specified time for delivery of the packet is not under the control of the particular computing instance, and after determining that the specified time for delivery of the packet has occurred, provides the packet or information for accessing the packet to the particular computing instance.

[0015] Fig.11 is a flow chart of an example method according to some embodiments, which can be used to implement multicast, multi-unicast and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization, wherein according to some embodiments, a networked computing resource receives a packet from a sender at a first reception time, wherein the packet is not accessible to a particular computing instance, obtains a specified time for delivery of the packet from the packet or from information separate from the packet, wherein the specified time for delivery of the packet is not under the control of the particular computing instance, determines whether the packet or encryption key was delivered to the destination at the specified time, and performs actions commensurate with the appropriate sub-flowchart based on the answer.

[0016] Fig.12is a flow chart of an example method for a sender of multicast, multi-unicast and / or unicast packets according to some embodiments, which example method can be used to implement multicast, multi-unicast and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization, wherein the sender receives information from a receiving host computing device and adjusts timing details of future sent packets according to some embodiments.

[0017] Fig.13 Depicts an example topology of a dedicated time network according to some embodiments, such as Figure 5 as shown in the options in .

[0018] Fig.14 is a block diagram of an example computer system that may be used for a trusted and / or certified packet timestamping service in accordance with some embodiments.

[0019] Although embodiments are described herein for several embodiments and illustrative drawings by way of example, it will be appreciated by those skilled in the art that embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and their detailed description are not intended to limit the embodiments to the specific forms disclosed, but on the contrary, it is intended to cover all modifications, equivalents and alternatives falling within the spirit and scope defined by the appended claims. The titles used herein are only for organizational purposes and are not intended to be used to limit the scope of this specification or claims. As used throughout this application, the word "may" is used in a permissible sense (i.e., meaning "possible") rather than a mandatory sense (i.e., meaning "must"). Similarly, the word "include, including and includes" means including but not limited to.

[0020] In addition, in the following sections, reference will now be made in detail to embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, many specific details are set forth in order to thoroughly understand the present disclosure. However, it will be apparent to one of ordinary skill in the art that some embodiments may be practiced without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks are not described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0021] This specification includes references to "one embodiment" or "an embodiment." The appearances of the phrases "in one embodiment" or "in an embodiment" are not necessarily referring to the same embodiment. The particular features, structures, or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0022] "Comprising". This term is open ended. As used in the appended claims, this term does not exclude additional structures or steps. Consider an embodiment that recites: "A device comprising one or more processor units..." Such a claim does not exclude the device from including additional components (e.g., a network interface unit, graphics circuitry, etc.).

[0023] "Configured to". Various units, circuits, or other components may be described or referred to as being "configured to" perform one or more tasks. In such contexts, "configured to" is used to represent a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit system) that performs these one or more tasks during operation. Thus, it can be said that the unit / circuit / component is configured to perform a task even when the specified unit / circuit / component is not currently working (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware - for example, circuits, memories storing executable program instructions to implement operations, etc. The statement that a unit / circuit / component is "configured to" perform one or more tasks is explicitly intended not to invoke 35 U.S.C. §112, sixth paragraph, for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit system) that is manipulated by software and / or firmware (e.g., a network interface card (NIC) or a general processor executing software) to operate in a manner capable of performing the tasks in question. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor manufacturing facility) to manufacture a device (eg, an integrated circuit) adapted to perform or execute one or more tasks.

[0024] "Based on". As used herein, this term is used to describe one or more factors that influence a determination. The term does not exclude additional factors that may influence the determination. That is, the determination may be based solely on those factors or at least in part on those factors. Consider the phrase "A is determined based on B". While in this case, B is a factor that influences the determination of A, such a phrase does not exclude that the determination of A is also based on C. In other cases, A may be determined based solely on B.

[0025] It will also be understood that, although the terms first, second, etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the expected scope, the first contact can be referred to as the second contact, and similarly, the second contact can be referred to as the first contact. The first contact and the second contact are both contacts, but they are not the same contact. As used herein, these terms are used as labels for the nouns in front of them, and do not imply any type of ordering (e.g., space, time, logic, etc.). For example, a buffer circuit can be described in this article as performing a write operation on a "first" and "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0026] The terms used in the specification of this article are only used to describe specific embodiments and are not intended to be restrictive. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more of the associated enumerated items. It will be further understood that when used in this specification, the terms "include", "including", "comprise" and / or "comprising" specify the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0027] As used herein, the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [a stated condition or event]" or "in response to detecting [a specified condition or event]," depending on the context. DETAILED DESCRIPTION

[0028] As described herein, embodiments of the system and method implement multicast, multi-unicast and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization. The embodiments provide delivery time for multicast, multi-unicast and / or unicast packets, and the delivery time defines the time when the packet should be forwarded to each user at one or more receivers. Multicast in the present application is a broad concept, which covers both application layer multicast and network-assisted multicast, as well as any other type of near real-time group communication of electronic messages and / or packets. In some embodiments, a networked computing resource (such as isolated timing hardware) associated with one of the receivers receives the multicast, multi-unicast and / or unicast packets so that the packets are not controlled by destination computing resources (such as destination computing instances). The networked computing resources use the delivery time information to deliver the packets to users at specific delivery times. In multicast and multi-unicast embodiments, other networked computing resources associated with other receivers perform the same action so that within the time tolerance of the same specific delivery time, multiple packets are provided by the networked computing resources to multiple users at multiple destination computing instances so that in some embodiments they are delivered almost simultaneously.

[0029] In some embodiments of the systems and methods described herein, a virtualized computing service can multicast, multi-unicast, and / or unicast messages using a guaranteed delivery time and / or a delivery time synchronization service running across user accounts of a provider network. Some of these embodiments provide features or services for embedding delivery time information provided by a provider network into multicast, multi-unicast, and / or unicast packets sent by a sender (e.g., from services such as virtual switching). In other embodiments, a sender application or OS associated with the sender embeds the delivery time information into the multicast, multi-unicast, and / or unicast packets sent by the sender. In some embodiments, on the receiving side, a mechanism for accessing the delivery time information and comparing it with an accurate time source is provided to isolated timing hardware associated with a receiving host computing device. Because the isolated timing hardware is included in the receiving host computing device, or is physically located adjacent to the receiving host computing device, and is further synchronized with a highly accurate reference timing device, a virtualized computing service that includes the isolated timing hardware can multicast, multi-unicast, and / or unicast distribution of messages from ports of a receiving computing instance implemented on a given receiving host computing device at a time that very accurately reflects the delivery time information of a given packet using guaranteed delivery times and / or delivery time synchronization. The isolated timing hardware can store the packet (as well as other packets sent from other senders with delivery time information) so that the packets can be ordered based on the delivery time information and delivered using highly accurate delivery times to ensure that the received packets are received nearly simultaneously across multiple receiving host computing devices.

[0030] The isolation timing hardware can be physically and logically positioned so that the received multicast, multi-unicast and / or unicast packets processed by the isolation timing hardware do not enter the control of the computing instance and therefore cannot be changed by the user to which the computing instance is assigned. For example, a service provider providing virtualized computing services can have exclusive access to multicast, multi-unicast and / or unicast packets received by the isolation timing hardware associated with the destination computing instance, so that the service provider can provide a guarantee or "proof" that the destination computing instance cannot access the packet before the packet is provided to the destination computing instance. The destination computing instance can only access the packet when the isolation timing hardware provides the packet or information for accessing the packet to the destination computing instance. The isolation timing hardware, as well as the isolation timing hardware of other receivers, can use the delivery time associated with the packet to deliver the packet to the intended destination at a specific delivery time. However, in order to accurately deliver the packet, in some embodiments, the isolation timing hardware requires a fairly accurate clock, or at least a clock that is accurately synchronized with other clocks of other recipients to be within any time tolerance of delivery variability.

[0031] However, a known and long-standing problem in distributed computing systems is that it is difficult to synchronize time information between devices. Many computing devices use crystal oscillators for timing. In theory, these oscillators operate at a known frequency, and therefore the time can be measured by the signal provided by the oscillator. However, in practice, the frequency of the oscillator varies based on various factors such as operating temperature. Although the frequency change may be small, it is enough to cause substantial desynchronization between two different devices. Another source of error in time synchronization is the mechanism by which the two devices are synchronized. In general, as the devices move farther, synchronization becomes less accurate because the laws of physics dictate the maximum speed at which time information can move between devices, which is further reduced due to various limitations of the equipment used to transmit data between devices. Although time synchronization technology can attempt to overcome this problem by adjusting the transmission time, this approach has its limitations. For example, it is generally unable to solve asymmetric delays, in which the round-trip communication time between two devices is unevenly divided between the outgoing and return communication paths. In this case, the first device can receive time information from the second device, but it may not be certain how much time has passed during the transmission of information from the second device to the first device. Furthermore, such techniques are generally unable to account for differences in transmission times, which may occur, for example, due to congestion on the network. As a result, the time information is generally inaccurate.

[0032] One large-scale application in which accurate time information may be desired is in hosted computing environments (sometimes referred to as "cloud computing" environments). A typical tenant of such environments is hardware commoditization: users of such environments are typically provided with a general overview of the hardware on which hosted computing instances will execute, but are not required to select and maintain individual hardware. In the context of cloud computing, time synchronization becomes particularly problematic because hosted computing environments typically span multiple data centers within a given geographic region and multiple geographic regions around the world. If such environments attempted to provide synchronization across all hosted computing instances, it would be desirable that such synchronization not be dependent on the specific hardware of the hosted instances, the specific location of said hardware, etc. Therefore, given the above-mentioned limitations on time synchronization, providing accurate time information in a hosted computing environment is very difficult and inaccuracies may occur.

[0033] Highly accurate clocks are essential for various applications such as financial transactions that require accurate timestamps to support and audit fair and equal access regulatory requirements. For example, the EU MiFID II regulations require accuracy within 100 microseconds for many trading applications. In addition, a typical requirement for stock exchanges is that they provide fair and equal access to market information. However, in current multicast and / or multi-unicast implementation schemes, information sent from the sender such as packets arrives at each packet receiver at different times. For example, the network delay, network configuration, geographic distance, etc. that vary between the packet sending the computing resource and the packet receiving the computing resource make the multicast and / or multi-unicast packets arrive at the first packet receiver at a time different from the time when the multicast and / or multi-unicast packets arrive at the second packet receiver. In some applications, such time arrival variability may be insignificant. However, in other applications, such differences may be important. For example, financial markets require (or are required by regulatory agencies) to ensure that information arrives at market participants in an almost synchronous manner. Currently, financial markets make great efforts to physically equalize network delays between market participants, to the point of ensuring that network cables are equal lengths for market participants to ensure that one market participant does not receive information before another, thereby giving them an undue advantage. This requires great cost and effort, and requires highly specialized computer networks and systems that are tightly controlled and cannot be ported to other networks or systems. In data centers and provider networks, these mechanisms need to be replaced by something that works in a more loosely coupled virtual environment.

[0034] One mechanism to address these issues is to equip each computing device with a highly accurate reference timing device. For example, each device may have an atomic clock installed inside it that is manually synchronized with an initial reference point (e.g., by physically co-locating the atomic clock and the reference point). Alternatively, each device may have a GNSS receiver installed that is configured to determine the current time by reference to the GNSS. Neither approach is suitable for large-scale application due to the cost and complexity of these timing devices and the logistical issues of setting up and maintaining such devices.

[0035] Some embodiments of the present disclosure solve the above problems by providing highly accurate time information at the hosted computing instance in a manner that does not require a reference timing device based on the instance and maintains the flexibility of the hosted computing environment by requiring little or no modification to the operation of the hosted computing instance. These embodiments can provide highly accurate time information so that the clock of a piece of isolated timing hardware included in (or associated with) the host computing device (hosting the computing instance) can be used to determine when to deliver a packet to the destination computing instance (for example, at a specified delivery time associated with the received packet). In addition, the service provider network can synchronize the isolated timing hardware across multiple host computing devices with the reference timing device within a few microseconds or nanoseconds (and therefore, within a few microseconds or nanoseconds of other isolated timing hardware included in or associated with other host computing devices that are also synchronized with the reference timing device). In some embodiments, the time tolerance for future delivery of multicast and / or multi-unicast packets can be determined by the tightness of the clock synchronization of various isolated timing hardware associated with various destination host computing devices.

[0036] More specifically, some embodiments of the present disclosure synchronize one or more reference timing devices within a data center with isolated timing hardware of (or associated with) a host computing device in a data center by using a dedicated time information network specifically for transmitting time information. The dedicated time network can provide highly accurate (e.g., nanosecond level) time information to the isolated timing host hardware of the host computing device. An embodiment provides a delivery time for multicast, multi-unicast and / or unicast packets, which defines the time when the packet should be forwarded to each user at one or more receivers. In some embodiments, a networked computing resource (such as isolated timing hardware) associated with one of the receivers receives the multicast, multi-unicast and / or unicast packet so that the packet is not controlled by a destination computing resource (such as a destination computing instance). In some embodiments, the networked computing resource can be located on a network interface card (NIC) of the host hardware. The networked computing resource uses the delivery time to deliver the packet to the user at a specific delivery time. In multicast and multi-unicast embodiments, other networked computing resources associated with other receivers perform the same action (e.g., at the same specified delivery time) such that multiple packets are provided by the networked computing resources to multiple users at multiple destination computing instances within a time tolerance of the same specific delivery time such that they are delivered nearly simultaneously in some embodiments.

[0037] Depending on the embodiment, the computing resources associated with the sender for calculating the delivery time of multicast, multi-unicast and / or unicast packets may be different. In some embodiments, it can be a sending application or an OS associated with the sending application. In other embodiments, it can be an isolated timing hardware associated with a sending host computing device that hosts a sending computing instance of sending multicast, multi-unicast and / or unicast packets. In still other embodiments, it can be an application that sends multicast, multi-unicast and / or unicast packets, such as a service that communicates with it through an API, such as a service of a provider network. In some embodiments, a computing resource can determine the time increment from the current time to the future time of the delivery time, and other computing resources can add the time increment to the current time to set specific future delivery time information. For example, the sending application can determine the future increment to deliver the packet, and the increment can be transmitted to the related isolated timing hardware. The isolated timing hardware can then calculate the delivery time by adding the increment to the current time from its accurate hardware clock to determine the future time (e.g., delivery time) indicating the delivery of the packet at the receiver device. In other embodiments, the computing resources associated with the sender that calculate the delivery time of the packet can be configured in various ways, and the above examples should not be construed as limiting. In some embodiments, the sending computing resource can calculate the delivery time of the multicast, multi-unicast and / or unicast packets by using the longest network delay of multiple recipients and then adding additional time to account for any network variability in the delivery time, and then adding the total delay to the current clock time to get the "future" delivery time.

[0038] There is no guarantee that a multicast, multi-unicast, and / or unicast packet will reach all intended recipients. A packet may be dropped, may be damaged, or the queue of the receiving device may be full when the packet arrives. Therefore, in some embodiments, the sending computing resource may also obtain feedback information about whether any of the previously sent multicast, multi-unicast, and / or unicast packets were delayed in arriving at any intended recipient (or never arrived). The feedback mechanism may be implemented in the system or exposed to the client. For example, if a packet is sent at t=0, where the delivery time is at time t=10, and most recipients receive the packet at least at t=3, but one recipient gets its packet at t=11, then the feedback mechanism may inform the sending computing resource that one of the multicast, multi-unicast, and / or unicast packets sent was recently delayed in arriving. In some of these embodiments, the delivery rate may be monitored very closely, and the delivery time value may be adjusted for future multicast, multi-unicast, and / or unicast packets based on the feedback information.

[0039] The receiver may provide feedback to the sending computing resource for each packet or periodically or after processing multiple packets. The feedback may include statistical information. The feedback mechanism may allow users such as financial markets to guarantee market participants that a high percentage of market messages will be delivered to all market participants simultaneously. In some embodiments, the feedback mechanism may be an optional feature, and in some of these embodiments, may be enabled through an API.

[0040] However, in some embodiments, the isolation timing hardware on the receiving side can deliver the packet to its intended recipient without waiting to exceed the specified delivery time. In these embodiments, for example, there is no delivery confirmation message sent back from the recipient to the sender to confirm that the message was received before it was published. These confirmation messages introduce further delays and more complexity about how long to wait for confirmation messages and when to determine that the absent recipient is no longer part of the group, which these embodiments avoid by not adopting confirmation message conditions for releasing packets. Instead, feedback is given about the timing of the multicast, multi-unicast and / or unicast packets actually delivered, including statistical information and whether any packets arrive after the delivery time. However, in some embodiments, confirmation messages can be used without adjusting the delivery time when such confirmation messages are received.

[0041] The sending computing resource does not need to be part of the same provider network as the receiving computing device. The packet can be simply sent in a format that is recognized by the isolated timing hardware associated with the receiving host computing device as a multicast, multi-unicast and / or unicast packet with delivery time information associated with the multicast, multi-unicast and / or unicast packet. For example, the packet can be sent to an agreed destination port at the receiving host computing device so as to be recognized as a multicast, multi-unicast and / or unicast packet with delivery time information associated with the multicast, multi-unicast and / or unicast packet. Depending on the embodiment, the delivery time information may include the packet, or be sent separately from the packet. The isolated timing hardware (such as associated with the receiving host computing device) does not have to be part of a network controlled by a trusted institution, however, in some embodiments, it can be. For example, in a financial market use case, market participants (such as brokers) receive market updates from the financial market on a regular basis. In this use case, the computing device associated with the trusted institution ensures that market participants cannot access these updates in advance. Packets associated with updates or information used to access inaccessible packets (such as keys to encrypt packets) must wait within a secure system (such as isolated timing hardware associated with the receiving host computing device) until the designated delivery time is reached.

[0042] When the time of delivering these packets has not yet arrived, the networked computing resources (such as isolation timing hardware) associated with the receiving side can implement data structures such as queues to save packets, or save information for accessing inaccessible packets (such as encrypted packets) that have been delivered or are about to be delivered. Data structures such as queues may have relevant managers to manage entries in the data structure. Depending on the embodiment, the data structure can be implemented in hardware or software. Before the packet (or the information for accessing inaccessible packets) is placed in the data structure, the identifier can identify these types of multicast, multi-unicast and / or unicast packets. In some embodiments, the packet is encrypted (such as by a networked computing resource) and delivered to the destination before the delivery time, wherein the decryption key is stored in the data structure and is provided to the destination at the delivery time. A time monitor can monitor time, and can deliver stored packets (or storage information for accessing packets), or can signal a data structure or a data structure manager to deliver stored packets or information at the delivery time.

[0043] In some embodiments, the hardware associated with each host computing device for providing time information, receiving multicast, multi-unicast and / or unicast packets, obtaining a specified time for delivering packets, and providing packets or information for accessing packets at a specified time is different from the hardware used to host any receiving computing instance. For example, the hardware (e.g., isolated timing hardware) for providing time information, receiving multicast, multi-unicast and / or unicast packets, obtaining a specified time for delivering packets, and providing packets or information for accessing packets at a specified time can be a physical offload card connected to other hardware of the host computing device via a peripheral component interconnect (PCI) bus. The offload card can include independent computing resources, such as independent processors and memories, to avoid competing for resources with the hosted computing instance. In other embodiments, the hardware can be a switch communicatively coupled to the host computing device, such as a top-of-rack ("TOR") switch. The TOR switch can include its own independent computing resources, such as independent processors and memories. In some cases, a portion of these resources can be dedicated to processing multicast, multi-unicast and / or unicast packets to further avoid potential problems associated with resource contention. In other embodiments, the final receiver may be external to the network with hardware for providing time information, receiving multicast, multi-unicast and / or unicast packets, obtaining the specified time for delivery of the packet, and providing the packet or information for accessing the packet at the specified time. In these cases, the hardware may be located in an edge router, and the multicast, multi-unicast and / or unicast packets are released from the edge router to the final destination at the delivery time.

[0044] Some embodiments of the systems and methods provide highly accurate time information for multicast, multi-unicast, and / or unicast distribution of messages by using a time synchronization method using isolated timing hardware that improves local clock accuracy to within microseconds (or nanoseconds) of a given time (e.g., UTC time). With reliable, microsecond-level clock accuracy and network packet time synchronization in the provider's network, users can use cloud computing providers to migrate and update their most time-sensitive applications. In addition, users can easily audit whether any system that provides highly accurate time information for time synchronization is working and providing the expected microsecond-level accuracy.

[0045] More specifically, some embodiments of the present disclosure provide a data center including two different communication networks: a first packet data network that supports standard network communications of instances, and a second network dedicated to providing highly accurate synchronized time information, such as a pulse per second signal. The dedicated time network may include one or more reference timing devices configured to provide highly accurate time information, such as by synchronizing with a global navigation satellite system (GNSS) such as a global positioning system (GPS). The host computing devices in the data center may be equipped with isolated timing hardware that receives time information from the dedicated time network and uses the time information to maintain highly accurate time relative to the reference timer. The isolated timing hardware may further provide a trusted mechanism for timestamping packets using highly accurate time information. Because the time information is provided by hardware local to the hosted instance and does not need to pass through a network external to the host, the time information may be highly accurate, such as within 100 microseconds, within 20 microseconds, or within a shorter time, relative to traditional NTP. Further, embodiments of the present disclosure may provide multiple dedicated time networks, and select a network for time information based on analysis of these networks, thereby further improving the accuracy of timing.

[0046] As disclosed herein, a data center may include multiple host computing devices interconnected by a non-dedicated packet-switched communication network. Each host computing device may host one or more computing instances, which may include virtual computing instances (e.g., virtualized hardware provided by software executed on the host device, such as a hypervisor) or bare metal instances (e.g., instances that directly access the hardware of the host computing device). The computing instances may communicate through a non-dedicated network and using a wider network (e.g., the Internet). In addition, in some embodiments, a data center may include one or more dedicated networks that carry time information. In other embodiments, the time information may be carried by a conventional data network of the data center. Illustratively, each dedicated time network may be connected to a reference timer device, such as a GNSS-connected timing device. The reference timer device may transmit a PPS signal (or other fixed-width signal) through a dedicated time network, which may therefore be broadcast to a receiving device on the network. The dedicated time network of some embodiments may be configured to maintain an extremely high level of accuracy of time information, potentially on the order of tens of microseconds or nanoseconds. For example, the dedicated time network may implement the Synchronous Ethernet (SyncE) standard to maintain synchronization between devices of the network, or may implement the design and protocol of the White Rabbit Project to facilitate the distribution of time information, which is a network design known in the art that itself utilizes SyncE. Each host computing device may include hardware that engages with the dedicated time network and receives signals from the dedicated time network, thereby facilitating synchronization between the host computing device and the reference timing device. For example, such hardware may include networked computing resources, such as isolated network hardware, which is included in the host computing device or physically adjacent to the host computing device. In some embodiments, the isolated timing hardware may include a network interface card (NIC) and one or more sockets that perform packet encapsulation and / or packet encryption. In one embodiment, the dedicated time network specifically carries a PPS (or other fixed width) signal from a reference timer, which is aligned with the relevant time boundary (e.g., the signal pulses at the correct second boundary). Therefore, in order to determine the current time, the hardware (e.g., the isolated timing hardware) may also need to obtain a reference point indicating which second is currently. Because the reference point does not need to be highly accurate in order to achieve synchronization, the hardware (e.g., isolated timing hardware) can obtain the reference point through a non-dedicated network, such as through communication with an NTP server on a non-dedicated network. The hardware (e.g., isolated timing hardware) can then determine the current time with high accuracy by combining the reference point with the second boundaries indicated by the PPS signal. In addition, the hardware (e.g., isolated timing hardware) can provide an indication of the current time to the timestamp creation component for use in timestamping packets transmitted by one or more computing instances implemented on the host computing device.

[0047] In some embodiments, a dedicated time network may include multiple reference timing devices, each of which transmits a PPS (or other fixed width) signal, for example, in a portion of the network. One or more devices within the dedicated time network may be configured to select from the signals provided by each timing device in order to identify the "leading" signal to be followed as a time source. Illustratively, the host computing devices in a data center may be physically arranged into units, such as racks. Each rack may include a switch on a dedicated time network, the switch having logic to select from multiple signals, each provided by a different reference timing device. The switch may include hardware such as a field programmable gate array (FPGA), which may be configured to provide a selected signal to each host computing device on the unit with very low latency. Therefore, the switch may check the available signals, select the appropriate signal, and reconfigure the FPGA to provide a signal to the device of the unit. To facilitate seamless operation, the switch may include multiple FPGAs so that the first FPGA may continue to send time information from the previous reference timing device, while the second FPGA is configured to send such time information from the new reference timing device.

[0048] In one embodiment, the switch selects a reference time signal based on a consensus algorithm. For example, if a majority (or other consensus threshold) of signals appear within a threshold period, the switch may select a signal from a signal group (e.g., earliest, latest, closest to average, etc.) to provide to the host computing device of the unit. In another embodiment, the switch selects a reference timing device based on metadata about a reference timing device, a dedicated time network, or a combination thereof. For example, a data center may include a computing device that is configured to perform timing analysis to detect delays or other problems that may prevent the reference timing device from delivering an accurate signal. The analysis device may deliver such information to each unit switch to facilitate signal selection. Although signal selection is discussed herein for each unit switch, in some cases, other elements of the dedicated time network (including isolated timing hardware of the host computing device) may perform such signal selection.

[0049] According to the present disclosure, it will be understood by those skilled in the art that the embodiments disclosed herein improve the ability of a computing system to multicast, multi-unicast and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization by providing a trusted receiving network computing device with the delivery time of the message. The package and delivery time are not controlled by the receiving computing instance, and therefore are not controlled by the user, so that other users or other entities outside the data center or provider network can "believe" that the user cannot access the message before the delivery time occurs. In addition, the embodiments disclosed herein improve the ability of a computing system to provide highly accurate time information to a hosted computing instance without the need to specifically configure such a hosted computing instance. In addition, the embodiments disclosed herein solve the inherent technical problems in computing systems; specifically, the difficulty of establishing a highly accurate concept of common time within a distributed computing system, the difficulty of providing highly accurate time information in a column of distributed computing devices, and the difficulty of multicasting, multi-unicasting and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization across user accounts. These technical problems are solved by various technical solutions described herein, including (a) using isolated timing hardware associated with a host computing device to synchronize with a reference timer device using a data network or a dedicated time network; (b) in some embodiments, using isolated timing hardware to provide packets (or access to inaccessible packets) to multiple users at multiple destination computing instances only within a time tolerance of the same specific delivery time, so that they are delivered nearly simultaneously; (c) using isolated timing hardware to keep packets (or information for accessing packets) out of the control of any computing instance, and therefore out of the control of any user until the delivery time occurs, and (d) using isolated timing hardware to provide a local interface for time information to the hosted computing instance. Therefore, the present disclosure generally represents improvements in host devices and computing systems.

[0050] The foregoing aspects and many of the attendant advantages of the present disclosure will become more readily appreciated as the same becomes better understood by reference to the following description when taken in conjunction with the accompanying drawings.

[0051] Figure 1A A logical model is depicted in which a sender 140 sends a multicast packet 160 to multiple receivers (110A-110D). The multicast packet 160 is sent at 12:00:01 PM and includes a specified time for delivery of the packet, which is specified at Figure 1A In this example, it is 12:00:10 PM. For simplicity, the times in this diagram are very coarse and are specified only in seconds, but in practice they may be specified to microseconds or even nanoseconds. Figure 1AA plurality of isolated timing hardware devices (120A-120D) are shown, each of which is associated with its respective receiver (110A-110D). As previously explained, the isolated timing hardware devices (120A-120D) may be respective physical offload cards of other hardware connected to the respective host computing devices via a peripheral component interconnect (PCI) express bus. In other embodiments, the isolated timing hardware devices (120A-120D) may be included in one or more switches, such as a top-of-rack ("TOR") switch, which is communicatively coupled to the receiving host computing device hosting the receiver. In other embodiments, the ultimate receiver (110A-110D) may be outside of the provider network that will include the isolated timing hardware (120A-120D). In these cases, the isolated timing hardware (120A-120D) may be located in an edge router of the provider network, and the multicast packets will be released from the edge router to the ultimate receiver (110A-110D) at the delivery time.

[0052] refer to Figure 1A , a multicast packet 160 arrives at multiple isolation timing hardware (120A-120D) at different times. For example, the multicast packet 160 arrives at the isolation timing hardware 120A at 12:00:06 PM, arrives at 120B at 12:00:02 PM, arrives at 120C at 12:00:04 PM, and arrives at 120D at 12:00:02 PM. The isolation timing hardware (120A-120D) receives the multicast packet 160, obtains the specified time for delivery of the packet, in this case 12:00:10 PM, and provides the packet or information for accessing the packet at the specified time. Therefore, each isolation timing hardware (120A-120D) releases the multicast packet to its corresponding recipient at the same specified time for delivery of the packet, 12:00:10 PM, so that the recipients receive the packets almost simultaneously at 12:00:10 PM because the delay between the isolation timing hardware and the recipients is within the time tolerance. Thus, in this embodiment, multiple isolated timing hardware provide multicast packets to their respective recipients within a time tolerance of the same future specified time of delivery of the packet.

[0053] Figure 1B A logical model is depicted in which a sender 140 sends multiple unicast packets 162 to multiple receivers (112A-112D). The multiple unicast packets 162 are sent at different times (12:00:01 PM, 12:00:02 PM, 12:00:03 PM, 12:00:04 PM) and include a specified time for delivery of the packet, which is specified in Figure 1B In this example, it is 12:00:10 PM. For simplicity, the times in this diagram are very coarse and are specified only in seconds, but in practice they may be specified to microseconds or even nanoseconds. Figure 1BA plurality of isolated timing hardware devices (122A-122D) are shown, each of which is associated with its respective receiver (112A-112D). As previously explained, the isolated timing hardware devices (122A-122D) may be respective physical offload cards of other hardware connected to the respective host computing devices via a peripheral component interconnect (PCI) express bus. In other embodiments, the isolated timing hardware devices (122A-122D) may be included in one or more switches, such as a top-of-rack ("TOR") switch, which is communicatively coupled to the receiving host computing device hosting the receiver. In other embodiments, the ultimate receiver (112A-112D) may be outside of the provider network that will include the isolated timing hardware (122A-122D). In these cases, the isolated timing hardware (122A-122D) may be located in an edge router of the provider network, and the multicast packets will be released from the edge router to the ultimate receiver (112A-112D) at the delivery time.

[0054] refer to Figure 1B , multiple unicast packets 162 arrive at multiple isolation timing hardware (122A-122D) at different times. For example, multiple unicast packet 162 arrives at isolation timing hardware 122A at 12:00:03 PM, arrives at 122B at 12:00:03 PM, arrives at 122C at 12:00:09 PM, and arrives at 122D at 12:00:07 PM. The isolation timing hardware (122A-122D) receives the multiple unicast packets 162, obtains the specified time for delivery of the packet, in this case 12:00:10 PM, and provides the packet or information for accessing the packet at the specified time. Therefore, each isolation timing hardware (122A-122D) releases a corresponding multiple unicast packet in the multiple unicast packets to its corresponding recipient at the same specified time for delivery of the packet, 12:00:10 PM, so that the recipients receive the packets almost simultaneously at 12:00:10 PM because the delay between the isolation timing hardware and the recipients is within the time tolerance. Thus, in this embodiment, multiple isolated timing hardware provide multiple unicast packets to their respective recipients within a time tolerance of the same future designated time of delivery of the packets.

[0055] Figure 1C A logical model is depicted in which a sender 144 sends a unicast packet 164 to a receiver (114). The unicast packet 164 is sent at 12:00:02 PM and includes a specified time for delivery of the packet, which is specified at Figure 1C In this example, it is 12:00:10 PM. For simplicity, the times in this diagram are very coarse and are specified only in seconds, but in practice they may be specified to microseconds or even nanoseconds. Figure 1CAn isolation timing hardware device (124) associated with a receiver (114) is shown. As previously explained, the isolation timing hardware device (124) may be a physical offload card that is connected to other hardware of a host computing device via a peripheral component interconnect (PCI) express bus. In other embodiments, the isolation timing hardware device (124) may be included in one or more switches, such as a top-of-rack ("TOR") switch, that are communicatively coupled to a receiving host computing device that hosts the receiver. In other embodiments, the ultimate receiver (114) may be outside of a provider network that will include the isolation timing hardware (124). In these cases, the isolation timing hardware (124) may be located in an edge router of the provider network, and the unicast packets will be released from the edge router to the ultimate receiver (114) at the delivery time.

[0056] refer to Figure 1C , unicast packet 164 arrives at the isolation timing hardware (124) at 12:00:06 PM. The isolation timing hardware (124) receives the unicast packet 164, obtains the specified time for delivery of the packet, in this case 12:00:10 PM, and provides the packet or information for accessing the packet at the specified time. Therefore, the isolation timing hardware (124) releases the unicast packet to its recipient at the same specified time of delivery of the packet, 12:00:10 PM, so that the recipient receives the packet at approximately 12:00:10 PM because the delay between the isolation timing hardware and the recipient is within the time tolerance. Therefore, in this embodiment, the isolation timing hardware provides the unicast packet to its corresponding recipient within the time tolerance of the same future specified time of delivery of the packet.

[0057] Figure 2 An example host computing device 215 is depicted in which embodiments of the present disclosure may be implemented. Figure 2A logical model for multicasting, multi-unicasting, and / or unicasting distribution of messages by a host computing device 215 using guaranteed delivery time and / or delivery time synchronization at a receiving host computing device is depicted. The host computing device 215 uses a set of computing resources 210 to host a computing instance 216. The host computing device 215 of this embodiment also includes networked computing resources that are not controlled by the computing instance 216, such as isolated timing hardware 220. For example, in some embodiments, the isolated timing hardware 220 is part of the host computing device 215, such as part of a chassis. In some of these embodiments, the isolated timing hardware 215 can be embedded in a network interface card (NIC). In other embodiments, the isolated timing hardware 220 is an entity completely separate from the host computing device 215. The time synchronization agent 228 uses information from a data network or a dedicated timing network 206 to synchronize the hardware clock 224. If 206 is a data network, the data network 206 can be the same data network as 204 or a different data network. If it is a dedicated timing network 206 , then the dedicated timing network 206 is a different network than the data network 204 .

[0058] Packet and Packet Delivery Time Receiver 226 receives packets from data network 204. Receiver 226 provides packets to packet storage data structure 222, possibly via data structure manager 232. The data structure is managed by data structure manager / packet provider 232. In some embodiments, receiver 226 also provides delivery time information to packet delivery determination component 230. Packet delivery determination component 230 determines when to deliver the packet based on the delivery time information and hardware clock 224. According to some embodiments, packet delivery determination component 230 notifies data structure manager / packet provider 232 to deliver the packet from data structure 222 to destination computing instance 216 at the appropriate time.

[0059] Thus, a networked computing resource, such as isolated timing hardware 220, is coupled to a host computing device 215 that hosts one or more computing instances 216. A packet receiver 226 of the networked computing resource receives a packet from a sender via a data network 204 at a first reception time. The received packet contains a packet destination for a particular computing instance in the one or more computing instances 216. The particular computing instance cannot access the received packet. The networked computing resource, such as isolated timing hardware 220, obtains a specified time for delivering the packet from the packet or from information separate from the packet, wherein the specified time for delivering the packet is not controlled by the particular computing instance. After the packet delivery determination component 230 determines that the specified time for delivering the packet has occurred, the networked computing resource, such as isolated timing hardware 220, provides the packet or information for accessing the packet to the particular computing instance via a packet provider 232.

[0060] Figure 3An example host computing device 315 is depicted in which embodiments of the present disclosure may be implemented. Figure 3 A logical model of a receiving host computing device 315 is depicted according to some embodiments, where the receiving host computing device 315 uses a set of computing resources 310 to host a computing instance 316. The host computing device 315 also includes isolated timing hardware 320 that is not controlled by the computing instance 316. A time synchronization agent 328 synchronizes the hardware clock 324 using a data network or a dedicated timing network 306.

[0061] The packet and packet delivery time receiver 326 receives the packet from the sender via the data network 304 and provides the packet to the packet encryptor 334. The packet and packet delivery time receiver 326 also obtains the delivery time information associated with the packet and provides it to the encryption key delivery determination component 330. The packet encryptor 334 encrypts the packet. The packet encryptor 334 provides the encrypted packet to the storage manager / packet provider / key provider 332 and provides the encryption key to the encryption key storage device 322. The encryption key storage device 322 is managed by the storage manager 332. The packet provider 332 provides the packet whose content cannot be accessed by the computing instance to the destination computing instance 316. The packet provider 332 can provide the packet before the delivery time occurs because the destination computing instance cannot access the packet. The encryption key delivery determination component 330 then determines when to deliver the encryption key to the destination computing instance 316 based at least on the received delivery time information and the hardware clock 324. The encryption key delivery determination component 330 notifies the storage manager / key provider 332 to deliver the packet from the encryption key storage 322 to the same destination compute instance 316 that received the packet at the appropriate time.

[0062] Figure 4 A logical model is depicted for multicast, multi-unicast, and / or unicast distribution of messages by a receiving host computing device 415 using a guaranteed delivery time and / or delivery time synchronization. The receiving host computing device 415 uses a set of computing resources 410 to host a computing instance 416. A network switch 420 communicatively coupled to the host computing device 415 includes isolated timing hardware that is not controlled by the computing instance 416. Similar to Figure 2, the time synchronization agent 428 uses the data network or dedicated timing network 406 to synchronize the hardware clock 424. The packet and packet delivery time receiver 426 receives the packet from the sender via the data network 404 and provides the packet to the packet storage data structure 422. The receiver 426 also receives the delivery time information from the packet or from the information separate from the packet and provides the information to the packet delivery determination component 430. The packet delivery determination component 430 determines when to deliver the packet based on at least the hardware clock 424 and the received delivery time information. According to some embodiments, the packet delivery determination component 430 notifies the data structure manager / packet provider 432 to deliver the packet from the data structure 422 to the destination computing instance 416 at the appropriate time.

[0063] Figure 5 An example system environment including a provider network is depicted in accordance with some embodiments, in which embodiments of the present disclosure may be implemented to provide highly accurate time information for multicast, multi-unicast, and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization using timing information received from a data or private timing network (506A and 506B). A client computing device 502 may access a provider network 550 via a network 504. A provider network 550 (sometimes referred to simply as a "cloud") refers to a network-accessible pool of computing resources (such as computing, storage, and networking resources, applications, and services) that may be virtualized or bare metal. A provider network may provide convenient, on-demand network access to a shared pool of configurable computing resources that may be programmatically provisioned and released in response to user commands. These resources may be dynamically provisioned and reconfigured to accommodate variable loads. Thus, cloud computing may be viewed as both applications delivered as services over a publicly accessible network (e.g., the Internet or a cellular communications network) and as hardware and software in a provider network data center that provides these services.

[0064] The provider network 550 may implement various computing resources or services, which may include virtual computing services, data processing services (e.g., map reduce, data streaming, and / or other large-scale data processing techniques), data storage services (e.g., object storage services, block-based storage services, or data warehouse storage services), and / or any other type of network-based services (which may include various other types of storage, processing, analysis, communication, event processing, visualization, and security services not shown). The resources required to support the operation of such services (e.g., computing and storage resources) may be provided in an account associated with the provider network, as distinct from resources requested by users of the provider network, which may be provided in a user account.

[0065] exist Figure 5In the example, the provider network 550 includes multiple groups of host computing devices 515, which are connected to the Figure 5 515A, 515B, 515C. Each group may represent a logical grouping of devices 515, such as a physical "rack" of devices. Each device may support one or more hosted machine instances 516A-C. Host instance 516 may be a virtual machine instance, which represents virtualized hardware (supported by the underlying hardware of device 515) that supports, for example, an operating system and applications. Hosted compute instance 516 may further represent a "bare metal" instance, whereby a portion of the computing resources of device 515 directly supports (without virtualization) instance 516. In some cases, instance 516 may be created and maintained on behalf of a user. For example, a client may utilize client computing device 502 to request the creation of instance 516 that executes client-defined software. In other cases, instance 516 may implement functionality of provider network 550 itself. For example, instance 516 may correspond to a block storage server, object storage server, or compute server that, in terms of terminology, provides block storage, object storage, or compute, respectively, to client computing device 502. Each host computing device 515 includes hardware computer memory and / or a processor, an operating system that provides executable program instructions for the general management and operation of the server, and a computer-readable medium storing instructions that, when executed by the server's processor, allow the server to perform its intended functions. Figure 5 Not shown, but provider network 550 may include other computing devices that facilitate the operation of host computing device 515, such as data storage for storing account information, computing devices for implementing logging, monitoring, and billing services, etc.

[0066] The provider network 550 can provide users with an on-demand, scalable computing platform through the network 504, for example, thereby allowing users to use scalable "virtual computing devices" at will through their use instances 516 or services provided by such instances 516. These virtual computing devices have the attributes of personal computing devices, including hardware (various types of processors, local memory, random access memory ("RAM"), hard disk and / or solid state drive ("SSD") storage devices), a choice of operating system, networking capabilities, and pre-loaded application software. Each virtual computing device can also virtualize its console input and output ("I / O") (e.g., keyboard, display, and mouse). This virtualization allows users to connect to their virtual computing devices using computer applications such as browsers, application programming interfaces, software development kits, etc., in order to configure and use their virtual computing devices just as they configure and use personal computing devices. Unlike personal computing devices that have a fixed number of hardware resources available to users, the hardware associated with the virtual computing devices can be scaled up or down according to the resources required by the user. Users can choose to deploy their virtual computing systems to provide network-based services for their own use and / or for use by their users.

[0067] The provider network 550 can be formed into multiple regions, where a region is a separate geographic region in which a cloud provider clusters data centers. Each region can include two or more availability zones connected to each other through a dedicated high-speed network (e.g., a fiber optic communication connection). An availability zone (also referred to as an availability domain, or simply "region") refers to an isolated fault domain that includes one or more data center facilities, which have separate power, separate networks, and separate cooling separated from facilities in another availability zone. A data center refers to a physical building or housing that houses the servers of the provider network and provides power and cooling for them. Preferably, the availability zones within a region are far enough apart from each other that the same natural disaster should not simultaneously take more than one availability zone offline. Users can connect to the availability zone of the provider network via a switching center (TC) through a publicly accessible network (e.g., the Internet, a cellular communication network). TC is the main backbone location that connects users to the provider network, and can be collocated with other network provider facilities (e.g., an Internet service provider, a telecommunications provider) and securely connected (e.g., via a VPN or direct connection) to the availability zone. Two or more TCs can be operated in each region to achieve redundancy. The regions are connected to a global network that includes a dedicated networking infrastructure (e.g., a fiber connection controlled by a cloud provider) that connects each region to at least one other region. The provider network can deliver content from points of presence outside of these regions but networked to these regions through edge locations, regional edge cache servers. This division and geographic distribution of computing hardware enables the provider network to provide users with low-latency resource access with a high degree of fault tolerance and stability on a global scale. In some embodiments, the provider network may include one or more cellular networks managed and provided by a cloud provider. Figure 4 The elements shown illustratively represent a collection of elements that exist in a single region or availability zone. Therefore, these elements can be replicated in each such region or availability zone.

[0068] like Figure 5 As shown, instances 516 can communicate with each other and with client computing devices 502 via network 504. Network 504 can include any suitable network, including an intranet, the Internet, a cellular network, a local area network, or any other such network or combination thereof. Figure 5 In the embodiment of the present invention, network 504 is illustratively a general purpose communication network, such as the Internet. Thus, network 504 may contain packetized information that enables arbitrary data exchange between networked devices. Protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are known to those skilled in the art of computer communications and therefore need not be described in further detail herein.

[0069] Client computing devices 502 may include any network-equipped computing device, such as a desktop computer, laptop computer, smart phone, tablet computer, e-reader, game console, etc. Clients may access provider network 550 through network 504 to view or manage its data and computing resources and use websites and / or applications hosted by provider network 550. Although shown differently in FIG. 1 , elements of provider network 550 may also act as clients to other elements of the network 550. Thus, client device 502 may generally refer to any device that accesses a network-accessible service as a client of the service.

[0070] In some embodiments, a user may have a way to select multicast, multi-unicast, and / or unicast services of a provider network. For example, a user may set a flag to disclose or not disclose multicast, multi-unicast, and / or unicast and time delivery information for received packets. Furthermore, a user may choose to include or not include delivery time information in packets sent from the user's computing instance. The selection may be a property of the computing instance 516 set by the user. In some embodiments, a user may be charged for multicast, multi-unicast, and / or unicast services for the entire service or for each packet or message, depending on the embodiment.

[0071] In addition to the network, the host computing device 515 can be connected to an internal data or dedicated timing network represented as networks 506A, 506B. These internal networks can carry data as well as time information. In some embodiments, one of the internal networks is a dedicated time network that only carries timing information. The internal data and / or dedicated time networks 506A-B can be further connected to one or more reference timers 512, which serve as reference points for time information delivered through the network. For example, each reference timer 512 can be an atomic clock or a GNSS 510 receiver, and can therefore serve as a highly accurate source of time information for the device 515 within the network 406. In one embodiment, each different reference timer 512 is synchronized with each other, and therefore shares a common time with high accuracy. For example, each timer 512 can be synchronized with a public GNSS (such as GPS) with high accuracy (e.g., tens of nanoseconds).

[0072] The internal data and / or dedicated time network may include a set of interconnected devices configured to carry time information from a reference timer 512 to a host computing device 415 with minimal loss of accuracy (e.g., on the order of nanoseconds). For example, network 506 may include devices configured in accordance with the specifications of the White Rabbit Project, a known initiative to develop a highly accurate timing network. In some embodiments, the data and / or dedicated time distribution network may be coupled between a GNSS receiver and each top-of-rack switch ("TOR") in a data center. To avoid congestion or contention issues for network resources, each network 406 may be dedicated to time information and provide such time information with little or no encoding. In other embodiments, the time information may pass through the normal data network of the provider network. In some embodiments, the data network and the dedicated time network may utilize a common but electrically partitioned physical substrate. For example, Figure 5 The cables between the elements of the device may include wiring for the network and different wiring for the dedicated time network. The respective wirings may be electrically isolated from each other so that signals passing through the network do not inhibit signals passing through the dedicated time network. In other embodiments, the time signal and the data signal pass through the same network. The depicted timer 512 and internal network 506 may be used to provide time synchronization services provided by the isolated timing hardware 520. Such services may be used for the isolated timing hardware 520A-520C.

[0073] Illustratively, each network (506A and 506B) may be dedicated to carrying a fixed signal, such as a fixed width pulse (e.g., a pulse per second or PPS signal or other fixed width signal, including a fixed frequency signal) broadcast from a single reference timer 512 to downstream devices. In one embodiment, the time information indicates the passage of time without a reference point for the current time. For example, the time information may indicate the passage of each second, but not which second is passing. In embodiments where multiple timers 512 are present within the internal network 506, a device on the network (e.g., a timing switch) may be operable to select a single upstream signal from a single timer 512 to be used as a source of time information to be propagated to downstream devices, such as a host device 515. In some embodiments, the switch may select a signal based on metadata associated with the signal, as discussed in more detail below. Additionally, switches or other devices within the internal network 406 may utilize other techniques to address potential inaccuracies in the network 406. For example, a switch or other device can account for delays on network 406 due to physical distances between devices, such as by adjusting received time information based on expected delays (e.g., calculated based on known cable lengths, based on measured round-trip communication times, etc.). Because network 506 can be dedicated to carrying time information from reference timer 512, isolated timing hardware 520 of host computing device 515 can be configured to act purely as a consumer of information on network 506, and not to transmit information on network 406.

[0074] In addition to being connected to the data or dedicated time network 506, the reference timer 512 can also be connected to the general communication network 504. Illustratively, being connected to the network 504 can facilitate the collection of metadata about the operation of the reference timer 512, as discussed below. In addition, in some embodiments, the network is used to provide a reference time from the reference timer 512 to the isolated timing hardware 520 of the host computing device 515, the reference time indicating the current time. For example, in the case where the internal network 506 indicates that the second mark appears but does not indicate which second has occurred, the isolated timing hardware 520 of the host computing device 515 may need a reference point to determine which second has occurred. The reference point does not need to share the same accuracy as the time information provided by the internal network 506, because the reference time can be offset by plus or minus half of the period of the time information. For example, if the reference time is four tenths of a second early, the isolated timing hardware 520 of the host computing device 515 can receive the reference time and adjust the reference time to the nearest second mark indicated by the time information of the internal network 506, thereby achieving time synchronization with the accuracy of the time information. In one embodiment, the reference time is transmitted according to a known timing protocol such as NTP over the network 504. While such protocols may not be accurate enough to synchronize time independently, they may be accurate enough to provide a reference time according to embodiments of the present disclosure.

[0075] By utilizing a combination of a reference time (e.g., obtained via network 504) and time information obtained via internal network 506, each instance of isolated timing hardware 520 of host computing devices 515A to 515C can synchronize a local clock with the clock of reference timer 512. According to an embodiment of the present disclosure, the time of the local clock can then be synchronized with the time of reference timer 512. Figure 2 The packet delivery determination component 230 is used to determine when to deliver the packet based on the delivery time information. In one embodiment, each host computing device 515 includes isolated timing hardware 520 isolated from the computing resources of the instance 516, and the hardware is used to maintain the clock of the host computing device 515. For example, the isolated timing hardware 520 can be included in an "offload" card physically installed on the device 515, and the card includes separate computing resources (e.g., different processors, different memories, etc.) separated from those used by the instance 516, where such separate computing resources can be dedicated to instance management tasks, such as virtual machine management, input / output virtualization of network attached storage volumes, local migration management tasks, instance health monitoring, and the disclosed host clock or virtualized hardware clock and certified timestamps. In one embodiment, the card is connected to the resources used by the instance 516 through the peripheral component express interconnect (PCIe) bus of the host computing device 515. Therefore, the instance 516 executed on the different computing resources can communicate with the card (or other networked computing resources, such as the isolated timing hardware 520) through the local interface of the device 515 without going through the network. Furthermore, because such hardware 520 may be isolated from the hardware of instance 516, issues such as resource contention may be avoided that might otherwise inhibit accurate timing.

[0076] According to an embodiment of the present disclosure, the isolated hardware of the host computing device 515 can operate as a Tier 1 server and / or as the isolated timing hardware 520 ( Figure 5 In the example of FIG. 1 , the timing hardware 520A-C is shown as isolated timing hardware 520A-C) operation. In the terminology of some network time protocols, "stratum" indicates the distance from a reference clock. For example, "stratum 0" may indicate a reference clock, and "stratum 1" may indicate a device directly linked to the reference clock. Stratum 2 may indicate a device linked to a stratum 1 device through a network path. Figure 5, each host device 515 obtains time information from a reference timer 512 (e.g., a Tier 0 device) via an internal network 506 (e.g., a data or dedicated time network). This enables internal networks 506A and 506B to act as a "direct link" to reference timer 512, such that time information does not need to flow through a general communications network (e.g., network 504). Thus, each host computing device 515 effectively operates as a Tier 1 server. Furthermore, because instances 516 are co-located on host computing devices 515, instances 516 may never obtain time information via network 504. Thus, each instance 516 can synchronize with the reference timer with very high accuracy.

[0077] Each of the host computing devices 515A-C may serve as Figure 2-4 The receiving host computing device or Figure 6 The host computing device may send and receive packets back and forth over an internal data network, which in some embodiments may be the same network (506A-B) used to transmit the timing information. For example, a computing instance 516C of a host computing device 515C may send a multicast and / or multi-unicast packet to a destination computing instance 516A-B of a host computing device 515A-B. In this case, the isolated timing hardware 520C of the sending host computing device 515C will receive the multicast and / or multi-unicast packet from the sending computing instance 516C and will receive the multicast and / or multi-unicast packet according to the timing information. Figure 6 In some embodiments, the sending computing instance 516C will include the delivery time in the packet. In other embodiments, the delivery time will be determined by the isolated timing hardware 520C of the sending host computing device 515C and added to the packet by the isolated timing hardware 520C, or sent as a different packet, where the delivery time information can be based on Figures 9A-9D The packet will be sent to the destination host computing device 515A-B. The destination host computing device 515A-B will place it according to the different options described in Figure 2 , 34. In some embodiments, the isolated timing hardware 520A-B of the destination host computing device 515A-B receives the multicast and / or multi-unicast packet so that it is not controlled by the destination computing instance 516A-B. The isolated timing hardware 520A-B delivers the packet (or information for accessing the packet) to the destination computing instance 516A-B at a specific delivery time using the delivery time associated with the packet. In some embodiments, the isolated timing hardware 520A-B provides multiple packets to multiple users at the destination computing instance 516A-B within a time tolerance of the same delivery time so that the packets are delivered at approximately the same time. In some embodiments, the time tolerance can be determined by how closely the clocks of the receiving isolated timing hardware 520A-B are synchronized with each other.

[0078] Figure 6 An example sending host computing device 615 is depicted in which embodiments of the present disclosure may be implemented. Figure 6 Depicts a logical model for host computing device 615 to multicast, multi-unicast, and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization at the sending host computing device. Of course, any host computing device can send and receive packets, but although Figure 2-4 Focused on receiving multicast, multi-unicast, and / or unicast packets, obtaining associated delivery time information, and delivering the packets (or information for accessing the packets) to a destination computing instance at a specified delivery time, Figure 6 Instead, the focus is on sending multicast, multi-unicast and / or unicast packets with delivery time information, and then obtaining feedback from the recipient. The provider network 615 includes a host computing device (not shown) that uses a set of computing resources 610 to host a computing instance 616. The host computing device of some embodiments also includes networked computing resources that are not controlled by the computing instance 616, such as isolated timing hardware 620. In other embodiments, the sending application can communicate with the provider network service 620 outside the host computing device to multicast, multi-unicast and / or unicast the message using guaranteed delivery time and / or delivery time synchronization, wherein the sending application will provide the packets to be multicast, multi-unicast and / or unicast (and may provide time increments to calculate future delivery times), and the service 620 will perform the remaining operations, including determining and transmitting the delivery time. The sending application of the sending computing instance 616 can communicate with the service or isolated timing hardware 620 through the API 632 processed by the API processor 634.

[0079] In the sending isolation timing hardware / provider network service 620, the time synchronization agent (not shown) can use the information from the data network or the dedicated timing network to synchronize the hardware clock. The hardware clock can be used to calculate the future delivery time information of the package. The sending computing resource (such as the sending isolation timing hardware or the provider network service 620) can receive multicast, multi-unicast and / or unicast packets from the sending application of the sending computing instance 616. The package can be received directly or through the API 632 processed by the API processor 634. The future delivery time calculation component 630 can calculate the delivery time of the received multicast, multi-unicast and / or unicast packets. In some embodiments, the future delivery time calculation component 630 can calculate the delivery time by using the longest network delay of multiple recipients and then adding additional time to solve any network variability in the delivery time, and then add the total waiting time to the current clock time to obtain the "future" delivery time. The future delivery time calculation component 630 can provide multicast, multi-unicast and / or unicast packets and delivery time information to the package and delivery time transmitter 626. In some embodiments, the packet and delivery time sender 626 can receive the packet directly after receiving it from the sending application, so that the packet itself bypasses the future delivery time calculation component 630. The packet and delivery time sender 626 can add the delivery time information to the packet, or send the delivery time information as a different packet, where the delivery time information can be based on Figures 9A-9D The packet and delivery time sender 626 provides outgoing multicast, multi-unicast and / or unicast packets and delivery time information 616 to the recipient computing instance through the associated recipient isolation timing hardware.

[0080] With multicast, multi-unicast, and / or unicast, there is no guarantee that a packet will reach all or any intended recipients. Packets may be dropped, may be corrupted, or the queue of a receiving device may be full at a certain time. Therefore, in some embodiments, the statistics receiver 622 of the sending computing resource 620 can also obtain feedback information about whether any of the previously sent multicast, multi-unicast, and / or unicast packets are delayed in arriving at any intended recipient, such as statistics of the recipient host computing device 618. Depending on the embodiment, the feedback mechanism can be implemented in the system or exposed to the client. For example, if a packet is sent at t=0, where the delivery time is at time t=10, and most recipients receive the packet at least at t=3, but one recipient gets the packet at t=11, then the feedback mechanism can inform the sending computing resource that one of the multicast, multi-unicast, and / or unicast packets sent was recently delayed in arriving. In some of these embodiments, the delivery rate can be monitored very closely, and the future delivery time calculation component 630 can adjust the delivery time value for future multicast, multi-unicast, and / or unicast packets based on the feedback information.

[0081] The receiver may provide feedback 618 to the multicast statistics receiver 622 of the sending computing resource 620 for each packet or periodically or after processing multiple packets. The feedback may include statistical information. The feedback may be about the timing of the multicast, multi-unicast and / or unicast packets actually delivered, including statistical information and whether any packets arrived after the delivery time. The feedback mechanism may allow users such as financial markets to guarantee market participants that a high percentage of market messages will be delivered to all market participants at the same time. In some embodiments, the feedback mechanism may be an optional feature, and in some of these embodiments, may be enabled through the API 632.

[0082] The sending computing resource 620 need not be part of the same provider network as the receiving computing device. The packet can simply be sent in a format that is recognized by the isolated timing hardware associated with the receiving host computing device as a multicast, multi-unicast and / or unicast packet with delivery time information associated with the multicast, multi-unicast and / or unicast packet. For example, the packet can be sent to an agreed destination port at the receiving host computing device so as to be recognized as a multicast, multi-unicast and / or unicast packet with delivery time information associated with the multicast, multi-unicast and / or unicast packet. Depending on the embodiment, the delivery time information can be included with the packet, or sent separately from the packet.

[0083] Depending on the embodiment, there are multiple ways to implement how the sending host computing device or the sending instance 616 provides delivery time information for the packet sent to the receiving host computing device. In some embodiments, the isolated timing hardware 620 adds the delivery time to the packet (e.g., using the packet and the delivery time transmitter 626). In some of these embodiments, the delivery time or the increment of the calculated delivery time is determined by the sending application, and some of these embodiments may involve modifying the operating system to allow the delivery time information to be passed from the computing instance. In some embodiments, the control channel can be used to pass the delivery time information. The delivery time information can be metadata on the data of the packet itself. In some embodiments, the delivery time can be received from the computing instance by software, such as through the API 632, and can then be added to the packet by the hardware 626. In some embodiments, the delivery time information can be provided in the packet as part of the NIC interface, or can be provided as an additional data field in the descriptor, so that the driver and software stack do not need to be modified. In these embodiments, if a tunnel is used to transmit packets between host computing devices, it is transparent to the user and is implemented instead by the control plane of the isolated timing hardware and / or the computing instance.

[0084] In other embodiments, the packet can be completely encapsulated as part of the tunnel. In some of these embodiments, the user of the computing instance understands and creates the tunnel to form an instance-to-instance tunnel. In some of these embodiments, when the packet is transmitted through the isolation timing hardware, the isolation timing hardware uses the delivery time information to rewrite a portion of the encapsulation packet (which is encapsulated at the instance). In some embodiments, the delivery time information and regular data of the packet are in two different paths in terms of how they can be retrieved. The receiving application running on the computing instance can retrieve the data of the packet, and the delivery time information of the packet can be retrieved as a separate operation (through an operating system such as an API or through tunnel detection). Therefore, one embodiment will need to modify the operating system, while other embodiments do not need to modify the operating system. However, in all cases, in addition to normal data flow, delivery time information also needs to be retrieved.

[0085] In part, because of the way the ecosystem is built, the delivery time of sent packets is inherently guaranteed for the sender. The receiving instance does not have the opportunity to set the delivery time. However, the sender can know in different ways that the receiver receives the packets at the delivery time, and these packets are not under the control of the receiving instance before that. In some embodiments, when a relationship is established between a sender and a receiver (such as a VPC peer), an API can be used to indicate that packets sent over the relationship can have selected characteristics so that the delivery time information is signed, and the control plane and / or isolation timing hardware can implement the signature. In some of these or other embodiments, the control plane and / or isolation timing hardware verifies the sender's signature so that the timestamp is delivered to the destination compute instance at the proven delivery time that the receiver can verify.

[0086] It should be noted that while it has been discussed that the isolated timing hardware 620 may include a hardware clock and a time synchronization agent in order to perform the functions of the future delivery time calculation component 630, in some embodiments, a hardware component that performs delivery time insertion (such as a virtualized offload of a computing instance 616) can perform similar packet sending and delivery time insertion functions without a time synchronization agent and a hardware clock. For example, in some embodiments, virtualized offload hardware (e.g., a hardware component similar to the isolated timing hardware 620) included in a chassis of a host computing device can include a packet and delivery time sender, a future delivery time calculation component 630, a statistics receiver 622, and optionally an API processor 634 and an API 632.

[0087] In some embodiments, a single piece of isolated timing hardware may include components for determining delivery timing information and delivering the information with an outgoing packet when the packet is sent (e.g., Figure 6), and may also include a component for obtaining delivery time information from a received packet (e.g., as Figure 2-4 ). For example, in some embodiments, a single piece of isolated timing hardware 120 / 220 may include a packet and packet delivery time receiver 226, a packet storage data structure 222, a packet delivery determination component 230, a data structure manager / packet provider 230, a hardware clock 224, a time synchronization agent 226, a future delivery time calculation component 630, a packet and delivery time transmitter 626, a statistics receiver 622, an API processor 634, and an API 632. In such embodiments, the isolated timing hardware 220 / 620 may include a single hardware clock 224 that is used to determine the exact time used when calculating the delivery time information of an outgoing packet and may be used to determine when the delivery time of an incoming packet occurs. Moreover, a time synchronization agent (e.g., 228) may be used to synchronize the single hardware clock 224. In other words, the hardware clock 224 and the time synchronization agent 228 may not have to be replicated in the isolated timing hardware that is configured to send and receive multicast, multi-unicast, and / or unicast packets containing delivery time information.

[0088] As described above, in some embodiments, the delivery time calculation / sending may be performed in software rather than by isolated timing hardware (such as isolated timing hardware 620). For example, Figure 7 A logical model of a sending host computing device is depicted, wherein a future delivery time calculation component 730 and a packet and delivery time sender 726 are implemented in software on a computing instance 720 hosted on a sending resource 716, according to some embodiments.

[0089] In some embodiments, incoming multicast, multi-unicast and / or unicast packets from a sending application can be received by a future delivery time calculation component 730 and / or a packet and delivery time transmitter 726, which can be implemented in the operating system 720 of the computing instance 716, and / or can be implemented as an application 720 executed at the computing instance 716. The packet can be received from an OS call, or delivered through a software interface, or delivered through an API 732 processed by an API processor 734. The future delivery time calculation component 730 can calculate the delivery time of the received multicast, multi-unicast and / or unicast packets. In some embodiments, the future delivery time calculation component 730 can calculate the delivery time by using the longest network delay of multiple recipients and then adding additional time to solve any network variability in the delivery time, and then adding this total waiting time to the current clock time to get the "future" delivery time. The future delivery time calculation component 730 can provide the multicast, multi-unicast and / or unicast packets and delivery time information to the packet and delivery time transmitter 726. In some embodiments, the packet and delivery time sender 726 can receive the packet directly after receiving it from the sending application, so that the packet itself bypasses the future delivery time calculation component 730. The packet and delivery time sender 726 can add the delivery time information to the packet, or send the delivery time information as a different packet, where the delivery time information can be based on Figures 9A-9D The packet and delivery time sender 726 provides outgoing multicast, multi-unicast and / or unicast packets and delivery time information 716 to the recipient computing instance through the associated recipient isolation timing hardware.

[0090] With multicast, multi-unicast and / or unicast, there is no guarantee that the packet will reach all intended recipients. The packet may be dropped, may be damaged, or the queue of the receiving device may be full at a certain time. Therefore, in some embodiments, the statistics receiver 722 of the software 720 can also obtain feedback information about whether any of the previously sent packets are delayed to reach any intended recipient, such as the statistics of the recipient host computing device 718. Depending on the embodiment, the feedback mechanism can be implemented in the system or exposed to the client. In some of these embodiments, the delivery rate can be monitored very closely, and the future delivery time calculation component 730 can adjust the delivery time value for future multicast, multi-unicast and / or unicast packets based on the feedback information.

[0091] The receiver may provide feedback 718 to the statistics receiver 722 of the sending computing resource 720 for each packet or periodically or after processing multiple packets. The feedback may include statistical information. The feedback may be about the timing of the multicast, multi-unicast and / or unicast packets actually delivered, including statistical information and whether any packets arrived after the delivery time. This feedback mechanism may allow users such as financial markets to assure market participants that a high percentage of market messages will be delivered to all market participants at the same time. In some embodiments, the feedback mechanism may be an optional feature, and in some of these embodiments, it may be enabled through the API 732. In some embodiments where the calculation and sending of delivery times are implemented in software, the software may additionally include an API processor 734 that makes an API call to the API 732 to generate an API message similar to the one described above with respect to Figure 6 The delivery time information is calculated in the manner described.

[0092] Figures 8A-8D A logical model of the progression of events for a sender (840, 842, 844, 846) to send a multicast and / or multi-unicast packet (860, 864) to multiple receivers associated with isolated timing hardware (820A-C, 822A-C, 824A-C, 826A-C) is depicted in accordance with some embodiments. Fig. 8A In , a sender 840 sends a multicast and / or multi-unicast packet (860) including a specified time for delivering the packet. The packet arrives at a plurality of isolated timing hardware (820A-C) at different times, wherein each isolated timing hardware is associated with a receiver. Figure 8B In , the sender (842) then receives statistical information (852, 854, 856) about packet reception from the receiver's isolated timing hardware (822A-C). Figure 8C In, the sender 844 sends a second multicast and / or multi-unicast packet (864) to the isolated timing hardware (824A-C) receivers, the second multicast and / or multi-unicast packet including different specified times for delivering the packets and longer time increments. Fig.8D , sender 846 then receives additional statistical information (872, 874, 876) regarding the reception of the second packet from the receiver's isolated timing hardware (826A-C).

[0093] More specifically, refer to Fig. 8A, sender 840 sends a multicast and / or multi-unicast packet 860 to the isolated timing hardware (820A-C), where the specified delivery time of the packet is 12:00:14.764300 PM. Both isolated timing hardware (820B-C) receive the packet before the delivery time and therefore release the packet to their respective recipients at the delivery time of 12:00:14.764300 PM. However, isolated timing hardware 820A receives the packet at 12:00:14.764398 PM, which is after the delivery time. Determining that the delivery time has occurred, isolated timing hardware 820A releases the packet to its recipient 1 microsecond later at 12:00:14.764399 PM.

[0094] exist Figure 8B , the receiver's isolated timing hardware (822A-C) sends statistics (852, 854, 856) about packet reception back to the sender (842). The isolated timing hardware 822C sends Δ=348 microseconds (852) to the sender (842). The Δ is the amount of time the packet was received before the delivery time of packet 860. For the isolated timing hardware 822C, the packet was received at 12:00:14.763952 PM, which is 348 microseconds earlier than the delivery time of 12:00:14.764300 PM. Similarly, the isolated timing hardware 822B sends Δ=926 microseconds (854) to the sender (842). For the isolated timing hardware 822B, the packet was received at 12:00:14.763374 PM, which is 926 microseconds earlier than the delivery time of 12:00:14.764300 PM. However, isolated timing hardware 822A sends Δ=-98 microseconds (856) to the sender (842). For isolated timing hardware 822A, the packet was received late. The packet was received at 12:00:14.764398 PM, which is 98 microseconds later than the delivery time of 12:00:14.764300 PM. Therefore, isolated timing hardware 822A sends a negative delta to sender 842 as part of its statistics regarding packet reception 856.

[0095] exist Figure 8C , sender 844 sends a second multicast and / or multi-unicast packet (864) to the isolated timing hardware (824A-C) recipients, the second multicast and / or multi-unicast packet including a different specified time for delivery of the packet and a longer time increment. Sender 844 sends multicast and / or multi-unicast packet 864 to the isolated timing hardware (824A-C), wherein the specified delivery time of the packet is 12:00:15.574800 p.m. All three isolated timing hardware (820A-C) receive the packet before the delivery time and therefore release the packet to its respective recipients at the delivery time 12:00:15.574800 p.m.

[0096] exist Fig.8D , the receiver's isolated timing hardware (826A-C) sends statistics (872, 874, 876) about packet reception back to the sender (846). The isolated timing hardware 826C sends Δ=792 microseconds (872) to the sender (846). The Δ is the amount of time the packet was received before the delivery time of packet 864. For the isolated timing hardware 826C, the packet was received at 12:00:15.574008 PM, which is 792 microseconds earlier than the delivery time of 12:00:15.574800 PM. Similarly, the isolated timing hardware 826B sends Δ=1054 microseconds (874) to the sender (846). For the isolated timing hardware 826B, the packet was received at 12:00:15.573746 PM, which is 1054 microseconds earlier than the delivery time of 12:00:15.574800 PM. Similarly, isolated timing hardware 826A sends Δ=99 microseconds (876) to sender (846).For isolated timing hardware 826A, the packet is received at 12:00:15.574701 PM, which is 99 microseconds earlier than the delivery time of 12:00:15.574800 PM.

[0097] Figures 9A-9D Different options are presented for sender isolation timing hardware or provider network services to package multicast, multi-unicast and / or unicast packets received from a sending resource and their determined future delivery times in order to provide the packets and delivery time information to a data network for transmission to multicast, multi-unicast and / or unicast packet recipients in accordance with some embodiments. For example, Fig. 9A A previous packet 910 is shown, which includes a header 917 and a body 919, where the header includes different kinds of information, including the packet destination 912. Fig. 9A The delivery time 914 is shown appended to the packet to create a modified or new packet 915 that is larger in size than the original packet 910. Although the figure shows the delivery time 914 being appended to the body of the previous packet 919, the delivery time information 914 may also be appended to the header 917 or body 919 of the previous packet and may be placed before or after the header 917 or before or after the body 919 of the previous packet to create the new packet 915. This new or modified packet 915 is then provided to the data network for transmission to the packet destination.

[0098] Fig. 9B Demonstrates rewriting of packet bits using delivery time information. Fig. 9B A packet 920 is shown including a header 927 and a body 929, where the header includes different kinds of information, including the packet destination 922. The packet 920 may also include bits 924 where delivery time information is to be written. Fig. 9BNew delivery time information 926 is shown written to the bits of packet 924 to create a modified packet 920, where the size of the modified packet is the same as the size of the original packet. Fig. 9B The delivery time information bit position 924 is shown in the packet body 929, but the bit of the packet for the timestamp 924 may be located in the header 927 or body 929 of the packet. For example, the timestamp may be embedded in the header 927 of the packet, and it may then be overwritten by the isolated timing hardware of the host computing device upon transmission. The new or modified packet 920 is transmitted to the data network for transmission to the packet destination.

[0099] Fig. 9C It is shown that a separate packet is created with the delivery time information, which is also sent to the packet destination. Fig. 9C A data packet 930 is shown including a header 937 and a body 939 , where the header includes different kinds of information, including the packet destination 922 . Fig. 9C A separate packet 940 is shown created, the separate packet including delivery time information 944, a packet destination 947 that is the same as the data packet destination 932, and information 946 linking the created separate delivery time packet 940 to the data packet 930. The packet and the created separate packet are provided to the data network for transmission to the packet destination. In these embodiments, complexity is introduced at the receiver in order to associate the two packets together and reconstruct the association between the data packet 930 and the delivery time information 944 that matches the data packet 930 in the delivery time packet 940.

[0100] Fig.9D It is shown encapsulating packet 950 and delivery time information 964 according to the tunneling protocol and providing encapsulated packet 960 to the packet destination. Fig.9D A data packet 950 is shown including a header and a body, where the header includes different kinds of information, including the packet destination 952 . Fig.9D It is shown that according to the tunnel protocol, the packet 950 is encapsulated into an encapsulation packet 960, wherein the encapsulation packet includes delivery time information 964. In some embodiments, the encapsulation packet 960 may also include a destination 962 that is the same as the destination of the encapsulation packet 950. The encapsulation packet 960 including the packet 950 and the delivery time information 964 is provided to the data network for transmission to the packet destination. In some embodiments, the packet 950 may be encapsulated as part of the tunnel protocol. In some embodiments of these embodiments, the encapsulation packet may be transmitted in a dedicated and / or new tunnel according to the tunnel protocol. In some other embodiments of these embodiments, the tunnel protocol is, for example, an existing virtual network protocol that has been used between instances of a provider network and / or host computing devices.

[0101] Fig.1010 is a flow chart of an illustrative method for multicasting, multi-unicasting, and / or unicasting distribution of messages using time-guaranteed delivery times and / or delivery synchronization. In block 1010, a networked computing resource is coupled to a host computing device that hosts a computing instance. The networked computing resource receives a packet from a sender at a first reception time, wherein the received packet has a packet destination for a specific computing instance, and wherein the specific computing instance cannot access the packet. The flow chart transitions to block 1020, wherein the networked computing resource obtains a specified time for delivering the packet from the packet itself or from information separate from the packet, wherein the specified time for delivering the packet is not controlled by the specific computing instance. In a final block 1030, after determining that the specified time for delivering the packet has occurred, the networked computing resource provides the packet or information for accessing the packet to the specific computing instance.

[0102] Fig.11 is a flow chart of an example method that can be used to implement multicast, multi-unicast, and / or unicast distribution of messages using guaranteed delivery times and / or delivery time synchronization in accordance with some embodiments. Fig.11 In a beginning block 1110 of the flow chart, a networked computing resource coupled to a host computing device hosting a computing instance receives a packet from a sender at a first reception time, wherein the received packet has a packet destination of a specific computing instance, and wherein the specific computing instance has no access to the received packet. The flow chart then transitions to 1120, wherein the networked computing resource obtains a specified time to deliver the packet from the packet itself or from information separate from the packet, wherein the specified time to deliver the packet is not controlled by the specific computing instance.

[0103] The flowchart then determines at 1130 whether it is a packet or a decryption key that is to be delivered to the destination computing instance at the specified time. If it is a packet that is to be delivered to the destination computing instance at the specified time, the flowchart transitions to box 1140, which stores the packet in a queue entry, determines at box 1142 that the specified time for delivering the packet has occurred, and then after such determination, provides the packet to the specific computing instance at 1144. If it is a decryption key that is to be delivered to the destination computing instance at the specified time, the flowchart transitions to box 1150, which encrypts the packet. The flowchart then transitions to box 152, which provides the encrypted packet to the specific computing instance before the specified time for delivering the packet. Box 1154 stores the decryption key in the queue entry. The flowchart then transitions to 1156, which determines that the specified delivery time for the packet has occurred. After making such a determination, the flowchart transitions to box 1158, where the networked computing resource provides the decryption key to the specific computing instance.

[0104] Fig.121 is a flowchart of an example method for implementing a multicast, multi-unicast, and / or unicast distribution of messages using a guaranteed delivery time and / or delivery time synchronization by a sender of a multicast, multi-unicast, and / or unicast packet according to some embodiments. The flowchart begins at box 1210, where the sender sends a packet to a specific computing instance of a host computing device and also sends a second packet to a second computing instance of a second host computing device. The two packets contain the same designated time for delivery of the packets. The flowchart then transitions to box 1220, where the sender receives information about a first reception time from a networked computing resource associated with the host computing device and receives information about a second reception time from a second networked computing resource associated with the second host computing device. The flowchart then transitions to box 1230, where the sender adjusts a second designated time for delivery of a third packet based at least in part on the received information about the first reception time and the received information about the second reception time to improve the delivery efficiency of the third packet.

[0105] refer to Fig.13 , will describe the dedicated time network ( Figure 2 206 of them Figure 3 Dedicated time network 306, Figure 4 Dedicated time network 406 or Figure 5 An example configuration 1300 of a dedicated time network 506A-B in a provider network 550 may represent, for example, a single dedicated time network ( Figure 2 206 of them Figure 3 306 of Figure 4 406 or Figure 5 506A-B in the configuration 1300 Fig.13 is shown as a directed acyclic graph (DAG) where information is Fig.13 As a reference timer for nodes R1 and R2 (1312) ( Figure 5 512) flows through network 506 (including second level, third level and fourth level network node configurations - 1350, 1360 and 1370) to Fig.13 The host computing device ( Figure 5515A-C in ). Thus, configuration 1300 includes a network device, such as a switch, connecting nodes R1 and R2 to node H. Illustratively, for redundancy and resilience, the configuration includes at least two replicated nodes at each level. For example, the second level of configuration 1350 includes nodes A1 and A2, which in turn communicate with a third level of configuration 1360 designated as nodes B1 and B2. At the third level (which may be referred to as an "assignment" level), nodes are logically grouped into sets 1302, each of which includes nodes B1 and B2. Fig.13 Two such sets 1302 are shown, namely sets 1302A and 1302N. However, the configuration may include any number of sets 1300. Similarly, at a fourth level 1370 (which may be referred to as an "aggregation" level), nodes are logically grouped into level 4 sets 1304, where Fig.13 Each level 4 set 1304 includes two network nodes C1 and C2, which are from a dedicated time network ( Figure 5 506A-B, etc.) are provided to a logical grouping of host computing devices 1315, shown as node H. The logical grouping may represent, for example, a physical rack of such devices. Illustratively, the rack may include a "top-of-rack" switch configured to obtain time information from nodes C1 and C2 and select time information from one of these nodes to be passed to the host computing devices 1315 within the rack. Each third-level collection 1302 may include any number of fourth-level collections 1304. Thus, configuration 1300 enables the two reference timers 1312 represented by nodes R1 and R2 to be shared among a large number of host computing devices 1315.

[0106] As described above, each layer of configuration 1300 may include redundant devices. Fig.13 Two redundant devices are shown, but three or more devices are possible. This redundancy can ensure a dedicated time network ( Figure 5506A-B in configuration 1300) continues to operate, even in the event of a device failure. Although complete failures may occur, in the context of time information, a problematic situation is a partial failure of a device, whereby the device is unable to provide time information with the desired accuracy. For example, problems with a device or an interconnection between devices may cause signal delays. For general communication networks, such delays are predictable and tolerable. However, for time information, such delays may significantly affect synchronization. Therefore, embodiments of the present disclosure may include a timing metadata system 1306 in configuration 1300, which is configured to analyze metadata about the operation of a device (e.g., a node in configuration 1300) to detect potential delays in operation and in some cases attribute such delays to specific nodes or edges within configuration 1300.

[0107] More specifically, the timing metadata system 1306 can obtain metadata about time information received over the network 406 from each device in the configuration 1300 (e.g., each node, including switches in each switching level of the network node configurations (1350, 1360, and 1370), and the host computing device 1315. In one embodiment, the metadata reflects jitter in the signal transmitted over the network 406, as measured relative to the internal clock (e.g., a crystal oscillator) of the measuring device. For example, each reference timer 1312 can be configured to provide a PPS signal with edges aligned with each second. The receiving device (e.g., node A1 in the second level 1350) can determine the duration between the second-aligned edges to determine, for example, whether more or less than one second has passed according to its internal clock. Jitter represents the variation in this period, and generally speaking, the greater the jitter, the less accurate the time information. However, because jitter compares the internal clock to the network signal, it may not be possible to determine the source of such inaccuracy, that is, jitter may occur due to changes in the device's internal clock, changes in the network signal, or both.

[0108] By collecting metadata from multiple points within the network, the timing metadata system 1306 can at least partially eliminate inaccuracies to determine their sources. For example, in the configuration 1300, two nodes B1 and B2 in the third level 1360 receive signals from node A1. An increase in jitter at both nodes B1 and B2 (relative to the signal from A1) can indicate a problem with the operation of A1. An increase in jitter at B1 (relative to the signal from A1) without a corresponding increase in jitter at B2 can indicate a problem with node B1 or the connection between B1 and A1, rather than a problem with A1 itself. Similarly, B1 is connected to both A1 and A2. If the jitter at B2 increases relative to the signal of A1 but does not increase relative to the signal of A2, this can indicate a problem with the connection between B1 and A1 (or alternatively, a problem with A1, which can be ruled out using the above logic). Alternatively, if the jitter at B2 increases relative to the signals of both A1 and A2, this can indicate a problem with B2 (or alternatively, a problem with both A1 and A2, which can be ruled out using the above logic). Thus, the timing metadata system 1306 can perform pairwise comparisons of timing metadata between various nodes at a given level of the configuration 1300 to identify specific nodes or edges that are experiencing low accuracy operation.

[0109] While jitter is provided as an example of metadata regarding low-accuracy operation of a node within the network 506, other types of metadata are possible. For example, in the event of a failure of a node within the network, downstream nodes may continue to retransmit time information further downstream to avoid cascading failures. Illustratively, if node A1 stops receiving signals from nodes R1 and R2, node A1 may still continue to transmit time information downstream to each node B1 and B2, such as by generating a PPS signal based on the internal clock of node A1. Because such signals may not be as accurate as signals locked to a reference timer device, the timing metadata system 1306 may collect metadata indicating that a given node's signal has been decoupled from an upstream node, and may notify downstream nodes that node A1 is therefore operating with low accuracy. Additionally or alternatively, node A1 may modify a signal generated by the node to indicate low accuracy, such as by modifying a signal transmitted downstream in a manner that indicates low-accuracy information.

[0110] Upon detecting a node or edge experiencing low accuracy operation, the timing metadata system 1306 can notify relevant downstream nodes (e.g., nodes connected to the node or edge). These downstream nodes can in turn be configured to select an alternative upstream source of time information. Thus, if node A1, for example, is experiencing low accuracy operation, each of nodes B1 and B2 can be configured to pass time information from node A2, thereby allowing time information from node A1 to be discarded. Although Fig.131306 can communicate with each node via a network. Figure 5 The timed metadata system 1306 may be implemented as software executing within the instance 516 .

[0111] As discussed above, another source of inaccurate time information is clock drift, whereby a crystal oscillator or other time source within a device operates at a varying frequency, causing the device's internal clock to run at a variable rate. A known factor in clock drift is temperature. However, the specific drift experienced by an oscillator at a given temperature may vary depending on the specific oscillator in question. Manufacturers may specify such drifts statistically, guaranteeing, for example, a drift within a certain range at a given temperature. Unfortunately, these statistical measurements are often not accurate enough to calibrate individual oscillators.

[0112] Embodiments of the present disclosure can solve this problem by providing device-specific drift measurements. Specifically, metadata collected by the timing metadata system 1306 can be used to calculate drift measurements relative to temperature on a per-device basis. As described above, each device can periodically report metadata about time information obtained through the network to the timing metadata system 1306. In one embodiment, the metadata includes an incremental metric that indicates the difference between the measured width of the PPS signal and the expected width (e.g., the signal appears before or after the expected nanoseconds according to the internal clock). It can be expected that clock drift at least partially explains such increments. For example, a device can report that the PPS signal appears about 5 nanoseconds in advance at low temperatures and about 5 nanoseconds later at high temperatures. Under these conditions, it can be calculated that there is a drift of 10 nanoseconds between the two temperatures. In fact, such obvious drifts may be difficult to detect very accurately. However, statistical analysis can be applied to the metadata of the device for a long time in order to predict device-specific drifts. For example, the timing metadata system 1306 can correlate the increments with the device temperature (also reported by each device), such as by calculating an average increment as a function of temperature, for example, while excluding outliers caused by devices experiencing low accuracy operation. In some embodiments, drift information (e.g., a drift curve) can be returned from the system 1306 to the respective device and used to calibrate an internal clock. For example, the device can apply an offset to the time information provided by the internal clock, where the offset is determined based on the device temperature and the drift information. In this way, the accuracy of individual devices (and therefore the network 506) is increased.

[0113] In some embodiments, the timing metadata system 1306 may be further configured to estimate the accuracy of the time information for a given host computing device 1315. For example, the timing metadata system 1306 may obtain metadata about the operation of each node and edge between the reference timer 1312 and the host computing device 1315, and analyze the metadata to determine the current accuracy of each node and edge. Potential inaccuracy ranges may be summed in the path from the reference timer 1312 and the host computing device 1315 to determine the effective accuracy range at the host computing device. For example, in the case where two nodes divide the reference timer 1312 and the host computing device 1315, the first node has an accuracy of plus or minus 2 microseconds relative to the reference timer 1312, and the second node has an accuracy of plus or minus 3 microseconds relative to the first node, the expected accuracy of the host computing device 1315 relative to the reference timer 1312 may be 5 microseconds. In some embodiments, this information may be combined with the expected accuracy between the isolated timing hardware 520 and the instance 516 to determine the expected accuracy between the instance 516 and the reference timer 1312. This expected accuracy can then be provided to the instance 516 as metadata about the time information obtained from the isolated timing hardware 520. For example, when processing a request for the current time, the isolated timing hardware 520 can respond with the current time and metadata indicating the accuracy of the current time relative to the reference timer 1312 (e.g., "It is now 00:00:00 UTC January 1, 2022, with an accuracy range of plus or minus 20 microseconds"). Thus, the instance 516 can obtain information indicating the expected accuracy of the time information.

[0114] Fig.14 is a block diagram illustrating an example computer system that can be used to provide highly accurate time information for timestamping packets for hosted computing instances, according to some embodiments.

[0115] In at least some embodiments, a computer implementing part or all of the methods and apparatus for multicasting, multi-unicasting, and / or unicasting distribution of messages using guaranteed delivery times and / or delivery time synchronization as described herein may include a general purpose computer system or computing device that includes or is configured to access one or more computer-accessible media, such as Fig.14 Computer system 1400 is shown. Fig.1414 is a block diagram illustrating an example computer system that may be used in some embodiments. The computer system may be used to multicast, multi-unicast, and / or unicast distribution of messages using guaranteed delivery times and / or delivery time synchronization, or, for example, as a host computing device (215, 315, 415, or 515A-C) executing one or more of a plurality of computing instances (216, 316, 416, or 516A-E). In the illustrated embodiment, the computer system 1400 includes one or more processors 1410 coupled to a system memory 1420 via an input / output (I / O) interface 1430. The computer system 1400 further includes a network interface 1440 coupled to the I / O interface 1430.

[0116] In various embodiments, computer system 1400 may be a uniprocessor system including one processor 1410, or a multiprocessor system including several processors 1410 (e.g., two, four, eight, or another suitable number). Processor 1410 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1410 may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as an x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of processors 1410 may typically, but not necessarily, implement the same ISA.

[0117] The system memory 1420 may be configured to store instructions and data accessible by the processor 1410. In various embodiments, the system memory 1420 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above for multicasting, multi-unicasting, and / or unicasting distribution of messages using guaranteed delivery times and / or delivery time synchronization, are shown as stored within the system memory 1420 as code and data for multicasting, multi-unicasting, and / or unicasting distribution of messages using guaranteed delivery times and / or delivery time synchronization 1422.

[0118] In one embodiment, the I / O interface 1430 may be configured to coordinate I / O traffic between the processor 1410, the system memory 1420, and any peripheral device in the device including the network interface 1440 or other peripheral interface. In some embodiments, the I / O interface 1430 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 1420) into a format suitable for use by another component (e.g., the processor 1410). In some embodiments, for example, the I / O interface 1430 may include support for devices attached via various types of peripheral buses (e.g., variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, for example, the functionality of the I / O interface 1430 may be split into two or more separate components, such as a north bridge and a south bridge. Moreover, in some embodiments, some or all of the functionality of the I / O interface 1430 (e.g., the interface of the system memory 1420) may be directly incorporated into the processor 1410.

[0119] For example, the network interface 1440 can be configured to allow data to be exchanged between the computer system 1400 and other devices 1460 attached to one or more networks 1470, such as other computer systems or devices illustrated in Figures 1, 2, 3, 4, 5, 6, 7, 8, and 13. For example, in various embodiments, the network interface 1440 can support communication over any suitable wired or wireless general data network (such as Ethernet type). In addition, the network interface 1440 can support communication over telecommunication / telephone networks such as analog voice networks or digital fiber optic communication networks, over storage area networks such as Fibre Channel SANs, or over any other suitable type of network and / or protocol.

[0120] In some embodiments, system memory 1420 may be one embodiment of a computer-accessible medium configured to store the computer-accessible medium described above with respect to Figures 2 to 7The described program instructions and data are used to multicast, multi-unicast and / or unicast distribution of messages using guaranteed delivery time and / or delivery time synchronization. However, in other embodiments, program instructions and / or data can be received, sent or stored on different types of computer accessible media. In general, computer accessible media can include non-transitory storage media or memory media (such as magnetic or optical media), for example, a disk or DVD / CD coupled to the computer system 1400 through an I / O interface 1430. Non-transitory computer accessible storage media can also include any volatile or non-volatile media, such as RAM (for example, SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., which can be included in some embodiments of the computer system 1400 as system memory 1420 or another type of memory. Further, the computer accessible medium can include a transmission medium or a signal (such as an electrical signal, an electromagnetic signal or a digital signal), which is transmitted through a communication medium (such as a network and / or a wireless link), such as can be implemented through a network interface 1440.

[0121] Any of a variety of computer systems may be configured to implement processes associated with a provider network, a reference timer, a node of a dedicated timing network, a host computing device, isolated timing hardware, or any other component of the above-described figures. In various embodiments, the provider network, reference timer, node of a dedicated timing network, host computing device, isolated timing hardware, or any other component of any of Figures 1-13 may each include one or more computer systems 1400, such as Fig.14 In embodiments, a provider network, a reference timer, a node of a dedicated timing network, a host computing device, isolated timing hardware, or any other component may include one or more components of computer system 1400 that function in the same or similar manner as described for computer system 1400.

[0122] Various embodiments may further include receiving, sending or storing instructions and / or data implemented according to the foregoing description on a computer-accessible medium. In general, a computer-accessible medium may include a storage medium or memory medium (such as a magnetic or optical medium, for example, a disk or a DVD / CD-ROM), a volatile or non-volatile medium (such as a RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), a ROM, etc.), and a transmission medium or signal (such as an electrical signal, an electromagnetic signal, or a digital signal) conveyed by a communication medium (such as a network and / or a wireless link).

[0123] The various methods shown in the drawings and described herein represent exemplary embodiments of the methods. The methods may be implemented in software, hardware, or a combination thereof. The order of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.

[0124] It will be apparent to those skilled in the art having the benefit of this disclosure that various modifications and changes may occur. It is intended to embrace all such modifications and changes, and therefore, the above description is to be regarded as illustrative rather than restrictive.

[0125] Embodiments of the present disclosure may be described in light of the following terms:

[0126] Clause 1. A system comprising:

[0127] a plurality of host computing devices, wherein individual host computing devices include computing resources for hosting computing instances and are further associated with respective isolation timing hardware;

[0128] wherein first isolated timing hardware associated with a first host computing device of the plurality of host computing devices hosting a first computing instance is configured to:

[0129] receiving a first packet at a first reception time, wherein the first packet includes a first packet destination for the first computing instance, wherein the first packet further includes a same future specified time for delivery of the packet, and wherein the first packet is not under control of the first computing instance;

[0130] determining, based on a first clock of the first isolated timing hardware, that the same future specified time for delivering the packet has occurred; and

[0131] providing the first packet to the first computing instance in response to the determining that the same future specified time for delivering the packet has occurred;

[0132] wherein second isolation timing hardware associated with a second host computing device of the plurality of host computing devices hosting a second computing instance is configured to:

[0133] receiving a second packet at a second reception time different from the first reception time, wherein the second packet includes a second packet destination for the second computing instance, wherein the second packet further includes the same future specified time for delivery of the packet, and wherein the second packet is not under the control of the second computing instance;

[0134] determining, based on a second clock of the second isolated timing hardware, that the same future specified time for delivering the packet has occurred; and

[0135] providing the second packet to the second computing instance in response to the determining that the same future specified time for delivering the packet has occurred; and

[0136] Wherein the first packet is provided to the first computing instance by first isolated timing hardware and the second packet is provided to the second computing instance by second isolated timing hardware within a time tolerance of the same future specified time for delivery of the packets.

[0137] Clause 2. A system according to clause 1,

[0138] Wherein the first isolation timing hardware is further configured to:

[0139] providing information about the first reception time to a single sender of the first packet and the second packet; and

[0140] Wherein the second isolated timing hardware is further configured to:

[0141] Information about the second reception time is provided to the single sender of the first packet and the second packet.

[0142] Clause 3. The system of clause 2, wherein the single sender is configured to:

[0143] sending the first packet to the first computing instance of the first host computing device and sending the second packet to the second computing instance of the second host computing device, wherein the first packet and the second packet include the same future specified time for delivery of the packets;

[0144] receiving information about the first reception time from the first isolated timing hardware and receiving information about the second reception time from the second isolated timing hardware; and

[0145] A second future designated time for delivering the third packet and the fourth packet is adjusted based on the received information about the first reception time and the received information about the second reception time so as to improve the delivery efficiency of the third packet and the fourth packet.

[0146] Clause 4. A system according to any one of clauses 1 to 3,

[0147] wherein the first host computing device comprises the first isolated timing hardware, or a first network switch communicatively coupled to the first host computing device comprises the first isolated timing hardware; and

[0148] The second host computing device comprises the second isolated timing hardware, or a second network switch communicatively coupled to the second host computing device comprises the second isolated timing hardware.

[0149] Clause 5. The system according to any one of clauses 1 to 4, further comprising:

[0150] a reference timer providing a reference clock synchronized with a global navigation satellite system (GNSS);

[0151] Wherein the first isolation timing hardware is further configured to:

[0152] setting a first hardware clock of the first isolated timing hardware based on a signal corresponding to the reference clock, the reference clock being obtained from the reference timer via a dedicated timing network inaccessible to the first computing instance hosted by the first host computing device;

[0153] wherein the first clock of the first isolated timing hardware comprises the first hardware clock; and

[0154] Wherein the second isolated timing hardware is further configured to:

[0155] setting a second hardware clock of the second isolated timing hardware based on a signal corresponding to the reference clock, the reference clock being obtained from the reference timer via a dedicated timing network inaccessible to the second computing instance hosted by the second host computing device;

[0156] The second clock of the second isolated timing hardware includes the second hardware clock.

[0157] Clause 6. A method comprising:

[0158] receiving a packet from a sender at a networked computing resource at a first reception time, wherein the networked computing resource is coupled to a host computing device hosting one or more computing instances, wherein the received packet includes a packet destination for a particular computing instance of the one or more computing instances, and wherein the particular computing instance has no access to the received packet;

[0159] obtaining, at the networked computing resource, from the packet or from information separate from the packet, a specified time for delivery of the packet, wherein the specified time for delivery of the packet is not controlled by the particular computing instance; and

[0160] After determining that the designated time for delivering the packet has occurred, the packet or information for accessing the packet is provided by the networked computing resource to the particular computing instance.

[0161] Clause 7. The method according to clause 6, wherein:

[0162] The sender is a computing instance hosted by a second host computing device of the same computing service as the host computing device; or

[0163] The sender is located outside the computing service including the host computing device.

[0164] Clause 8. The method of clause 6 or clause 7, wherein a network interface card in the host computing device comprises the networked computing resource, or a network switch communicatively coupled to the host computing device comprises the networked computing resource.

[0165] Clause 9. The method according to any one of clauses 6 to 8, further comprising:

[0166] Information about the first reception time is provided to the sender by the networked computing resource.

[0167] Clause 10. The method according to any one of clauses 6 to 9, further comprising:

[0168] receiving a second packet from the sender at a second networked computing resource at a second reception time different from the first reception time, wherein the second networked computing resource is coupled to a second host computing device hosting a second one or more computing instances, wherein the received second packet includes a second packet destination for a second computing instance of the second one or more computing instances, and wherein the received second packet is inaccessible to the second computing instance;

[0169] obtaining, at the second networked computing resource, from the second packet or from information separate from the second packet, a same specified time for delivery of the packet, wherein the same specified time for delivery of the packet is not controlled by the second computing instance; and

[0170] providing, by the second networked computing resource, the second packet or information for accessing the second packet to the second computing instance after determining that the same designated time for delivering the packet has occurred;

[0171] wherein the first packet is provided by the first networked computing resource to the first computing instance and the second packet is provided by the second networked computing resource to the second computing instance within a time tolerance of the same specified time for delivering the packets.

[0172] Clause 11. The method according to clause 10, further comprising:

[0173] sending, by the sender, the packet to the particular computing instance of the host computing device and sending the second packet to the second computing instance of the second host computing device, wherein the packet and the second packet include the same specified time for delivery of the packet;

[0174] receiving, by the sender, information about the first reception time from the networked computing resource and receiving information about the second reception time from the second networked computing resource; and

[0175] A second designated time for delivering a third packet is adjusted by the sender based on the received information about the first reception time and the received information about the second reception time so as to improve the delivery efficiency of the third packet.

[0176] Clause 12. The method according to any one of clauses 6 to 11, further comprising:

[0177] setting a hardware clock of the networked computing resource based on a signal corresponding to a reference clock obtained from a reference timer via a dedicated timing network inaccessible to the one or more computing instances; and

[0178] A determination is made based on the hardware clock that the designated time for delivering the packet has occurred.

[0179] Clause 13. The method according to any one of clauses 6 to 12, further comprising:

[0180] encrypting the package;

[0181] providing, by the networked computing resource, the encrypted packet to the particular computing instance prior to the specified time for delivering the packet; and

[0182] Wherein providing the package or information for accessing the package to the specific computing instance further comprises:

[0183] A key for decrypting the packet is provided to the particular computing instance by the networked computing resource and after the determination that the designated time to deliver the packet has occurred.

[0184] Clause 14. The method according to any one of clauses 6 to 13, further comprising:

[0185] storing, by the networked computing resource, the received packet or the information for accessing the received packet and the specified time for delivering the received packet in a queue entry, wherein the queue contains one or more additional entries; and

[0186] Wherein providing the package or the information for accessing the package to the specific computing instance further comprises:

[0187] After determining that the designated time for delivering the packet has occurred, the received packet stored in the queue or the information for accessing the received packet is provided to the particular computing instance.

[0188] Clause 15. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors of a networked computing resource, cause the one or more processors to:

[0189] receiving a packet from a sender at the networked computing resource at a first reception time, wherein the networked computing resource is coupled to a host computing device hosting one or more computing instances, wherein the received packet includes a packet destination for a particular computing instance of the one or more computing instances, and wherein the particular computing instance is inaccessible to the received packet;

[0190] obtaining, from the packet or from information separate from the packet, a specified time for delivering the packet, wherein the specified time for delivering the packet is not controlled by the particular computing instance; and

[0191] After determining that the designated time for delivery of the packet has occurred, the packet or information for accessing the packet is provided to the particular computing instance.

[0192] Clause 16. The one or more non-transitory computer-readable storage media of Clause 15, wherein the program instructions further cause the one or more processors of the networked computing resources to:

[0193] encrypting the packet; and

[0194] Prior to the specified time for delivering the packet, providing the encrypted packet to the particular computing instance; and

[0195] Wherein providing the package or information for accessing the package to the specific computing instance further comprises:

[0196] After determining that the designated time for delivering the packet has occurred, a key for decrypting the packet is provided to the particular computing instance.

[0197] Clause 17. The one or more non-transitory computer-readable storage media of Clause 15, wherein the program instructions further cause the one or more processors of the networked computing resources to:

[0198] storing the received packet or the information for accessing the received packet and the specified time for delivering the received packet in a queue entry, wherein the queue contains one or more additional entries; and

[0199] Wherein providing the package or the information for accessing the package to the specific computing instance further comprises:

[0200] After determining that the designated time for delivering the packet has occurred, the received packet stored in the queue or the information for accessing the received packet is provided to the particular computing instance.

[0201] Clause 18. One or more non-transitory computer-readable storage media according to any one of clauses 15 to 17, wherein the program instructions further cause the one or more processors of the networked computing resources to:

[0202] Information about the first reception time is provided to the sender.

[0203] Clause 19. One or more non-transitory computer-readable storage media according to any one of clauses 15 to 18, wherein the program instructions further cause the one or more processors of the networked computing resources to:

[0204] Prior to providing the packet or the information for accessing the packet to the particular computing instance, a certificate proving the specified time for delivering the packet is created by the sender.

[0205] Clause 20. One or more non-transitory computer-readable storage media according to any one of clauses 15 to 19, wherein the program instructions further cause the one or more processors of the networked computing resources to:

[0206] setting a hardware clock of the networked computing resource based on a signal corresponding to a reference clock obtained from a reference timer via a dedicated timing network inaccessible to the one or more computing instances; and

[0207] A determination is made based on the hardware clock that the designated time for delivering the packet has occurred.

Claims

1. A system comprising: a host computing device comprising computing resources configured to host a computing instance and further associated with isolation hardware configured to implement a clock for the host computing device on the isolation hardware; The isolation hardware is further configured to: receiving a first packet at a first reception time, wherein the first packet includes a first packet destination for a first computing instance hosted by the host computing device, wherein the first packet further includes a future specified time for delivery of the packet to the first computing instance, and wherein the first packet received at the isolation hardware is not under control of the first computing instance; determining, based on the clock implemented on the isolated hardware, that the future specified time for delivering the packet has occurred; and The first packet is provided to the first computing instance in response to the determining that the future specified time for delivering the packet has occurred.

2. The system of claim 1, further comprising: a second host computing device comprising computing resources configured to host an additional computing instance and further associated with other isolated hardware configured to implement a second clock for the second host computing device on the other isolated hardware; The other isolation hardware is further configured as follows: receiving a second packet at a second reception time different from the first reception time, wherein the second packet includes a second packet destination for a second computing instance hosted by the second host computing device, wherein the second packet further includes the same future specified time for delivery of the packet, and wherein the second packet is not under the control of the second computing instance; determining based on the second clock of the other isolated hardware that the same future specified time for delivering the packet has occurred; and providing the second packet to the second computing instance in response to the determining that the same future specified time for delivering the packet has occurred; and Wherein the first packet is provided to the first computing instance by first isolated timing hardware and the second packet is provided to the second computing instance by second isolated timing hardware within a time tolerance of the same future specified time for delivery of the packets.

3. The system according to claim 2, Wherein the first isolation hardware is further configured to: providing information about the first reception time to a single sender of the first packet and the second packet; and The other isolation hardware is further configured as follows: Information about the second reception time is provided to the single sender of the first packet and the second packet.

4. The system according to any one of claims 1 to 3, wherein the first host computing device comprises the isolation hardware, or a first network switch communicatively coupled to the first host computing device comprises the isolation hardware; and Wherein the second host computing device comprises the other isolation hardware, or a second network switch communicatively coupled to the second host computing device comprises the other isolation hardware.

5. The system according to any one of claims 1 to 4, further comprising: a reference timer providing a reference clock synchronized with a global navigation satellite system (GNSS); The isolation hardware is further configured to: The clock is set based on a signal corresponding to the reference clock, the reference clock being obtained from the reference timer via a dedicated timing network inaccessible to the first computing instance hosted by the first host computing device.

6. A method comprising: receiving a packet from a sender at a networked computing resource at a first reception time, wherein the networked computing resource is coupled to a host computing device hosting one or more computing instances, wherein the received packet includes a packet destination for a particular computing instance of the one or more computing instances, and wherein the particular computing instance has no access to the received packet; obtaining, at the networked computing resource, from the packet or from information separate from the packet, a specified time for delivery of the packet, wherein the specified time for delivery of the packet is not controlled by the particular computing instance; as well as After determining that the designated time for delivering the packet has occurred, the packet or information for accessing the packet is provided by the networked computing resource to the particular computing instance.

7. The method according to claim 6, wherein: The sender is a computing instance hosted by a second host computing device of the same computing service as the host computing device; or The sender is located outside the computing service including the host computing device.

8. The method of claim 6 or claim 7, wherein a network interface card in the host computing device comprises the networked computing resource, or a network switch communicatively coupled to the host computing device comprises the networked computing resource.

9. The method according to any one of claims 6 to 8, further comprising: Information about the first reception time is provided to the sender by the networked computing resource.

10. The method according to any one of claims 6 to 9, further comprising: receiving a second packet from the sender at a second networked computing resource at a second reception time different from the first reception time, wherein the second networked computing resource is coupled to a second host computing device hosting a second one or more computing instances, wherein the received second packet includes a second packet destination for a second computing instance of the second one or more computing instances, and wherein the received second packet is inaccessible to the second computing instance; obtaining, at the second networked computing resource, from the second packet or from information separate from the second packet, a same specified time for delivery of the packet, wherein the same specified time for delivery of the packet is not controlled by the second computing instance; as well as providing, by the second networked computing resource, the second packet or information for accessing the second packet to the second computing instance after determining that the same designated time for delivering the packet has occurred; wherein the first packet is provided by the first networked computing resource to the first computing instance and the second packet is provided by the second networked computing resource to the second computing instance within a time tolerance of the same specified time for delivering the packets.

11. The method according to claim 10, further comprising: sending, by the sender, the packet to the particular computing instance of the host computing device and sending the second packet to the second computing instance of the second host computing device, wherein the packet and the second packet include the same specified time for delivery of the packet; receiving, by the sender, information about the first reception time from the networked computing resource, and receiving information about the second reception time from the second networked computing resource; as well as A second designated time for delivering a third packet is adjusted by the sender based on the received information about the first reception time and the received information about the second reception time so as to improve the delivery efficiency of the third packet.

12. The method according to any one of claims 6 to 11, further comprising: setting a hardware clock of the networked computing resource based on a signal corresponding to a reference clock obtained from a reference timer via a dedicated timing network inaccessible to the one or more computing instances; and A determination is made based on the hardware clock that the designated time for delivering the packet has occurred.

13. The method according to any one of claims 6 to 12, further comprising: encrypting the package; providing, by the networked computing resource, the encrypted packet to the particular computing instance prior to the specified time for delivering the packet; and Wherein providing the package or information for accessing the package to the specific computing instance further comprises: A key for decrypting the packet is provided to the particular computing instance by the networked computing resource and after the determination that the designated time to deliver the packet has occurred.

14. The method according to any one of claims 6 to 13, further comprising: storing, by the networked computing resource, the received packet or the information for accessing the received packet and the specified time for delivering the received packet in a queue entry, wherein the queue contains one or more additional entries; and Wherein providing the package or the information for accessing the package to the specific computing instance further comprises: After determining that the designated time for delivering the packet has occurred, the received packet stored in the queue or the information for accessing the received packet is provided to the particular computing instance.

15. The method according to any one of claims 6 to 14, further comprising Prior to providing the packet or the information for accessing the packet to the particular computing instance, a certificate proving the specified time for delivering the packet is created by the sender.

Images