Providing services with guest VM mobility

Abstract

Some embodiments provide novel methods for performing services for machines running in one or more data centers. For example, for a group of related guest machines (e.g., a group of tenant machines), some embodiments define two different forwarding planes: (1) a guest forwarding plane and (2) a service forwarding plane. The guest forwarding plane is connected to the machines in the group and performs L2 and / or L3 forwarding for the machines. The service forwarding plane (1) is connected to a service node that performs services for data packets sent to and from the machines and (2) forwards the data packets to the service node. In some embodiments, the guest machines are not directly connected to the service forwarding plane. For example, in some embodiments, each forwarding plane is connected to a machine or service node via a port that receives data packets from the machine or service node or provides data packets to the machine or service node. In such embodiments, the service forwarding plane does not have a port that directly receives data packets from any guest machine or provides data packets to any guest machine. Instead, in some such embodiments, data associated with the guest machine is routed to a port proxy module executing on the same host computer, and the other module has a service plane port. In some embodiments, the port proxy module can indirectly connect more than one guest machine on the same host to the service plane (ie, can serve as a port proxy module for more than one guest machine on the same host).

Description

[0001] Rahul Mishra, Camille Lecuyer, Sahil Gokhale, Rajeev Nair, Anuprim Charvadi, Yang Ping, Kantish Mundaraj, Pierluigi Rolando, Jayant Jain, Raju Korganti. Background Art

[0002] Today's data centers use a static, configuration-intensive approach to distribute data packets between different application layers and to different service layers. A common approach today is to configure virtual machines to send packets to a virtual IP (VIP) address, and then configure the forwarding elements and load balancers in the data center with forwarding rules that direct them to forward VIP-addressed packets to the appropriate application layer and / or service layer. Another problem with existing packet distribution schemes is that today's load balancers are often choke points for distributed traffic. Therefore, there is a need in the art for a new method to seamlessly distribute data packets between different application layers and / or service layers in a data center. Ideally, this new approach would allow the distribution scheme to be easily modified without reconfiguring the servers that transmit the data packets. Summary of the invention

[0003] Some embodiments provide novel methods for performing services for machines running in one or more data centers. For example, for a group of related guest machines (e.g., a group of tenant machines), some embodiments define two different forwarding planes: (1) a guest forwarding plane and (2) a service forwarding plane. The guest forwarding plane connects to the machines in the group and performs L2 and / or L3 forwarding for the machines. The service forwarding plane (1) connects to service nodes that perform services for data packets sent to and from the machines and (2) forwards the data packets to the service nodes.

[0004] In some embodiments, the guest machine is not directly connected to the service forwarding plane. For example, in some embodiments, each forwarding plane is connected to a machine or service node via a port that receives data packets from the machine or service node or provides data packets to the machine or service node. In such embodiments, the service forwarding plane does not have a port that directly receives data packets from any guest machine or provides data packets to any guest machine. Instead, in some such embodiments, data associated with the guest machine is routed to a port proxy module executed on the same host computer, and the port proxy module has a service plane port. In some embodiments, the port proxy module can indirectly connect more than one guest machine on the same host to the service plane (i.e., can serve as a port proxy module for more than one guest machine on the same host).

[0005] In some embodiments, a guest machine is any machine other than a service machine or node. A guest machine can be a tenant's machine in a multi-tenant data center, but it does not have to be so. In some embodiments, a guest machine is a guest virtual machine or a guest container. In some embodiments, a service node is a service virtual machine, a service container, or a service appliance. In some embodiments, a service node performs a middlebox service operation, such as a firewall, an intrusion detection system, an intrusion prevention system, a load balancer, an encryptor, a message monitor, a message collector, or any number of other middlebox services. Therefore, in some embodiments, the service used in this document is any type of middlebox service operation.

[0006] The foregoing summary of the invention is intended as a brief introduction to some embodiments of the present invention. It is not meant to be an introduction or overview of all the inventive themes disclosed in this document. The following detailed description and the drawings cited in the detailed description will further describe the embodiments described in the summary of the invention and other embodiments. Therefore, in order to understand all the embodiments described in this document, it is necessary to fully review the summary of the invention, the detailed description, the drawings and the claims. In addition, the subject matter claimed for protection is not limited by the illustrative details in the summary of the invention, the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are set forth in the following drawings.

[0008] Figure 1 An example of a separate guest plane and service plane implemented by two logical forwarding elements in some embodiments is illustrated.

[0009] Figure 2 Illustrated is a data packet between two guest virtual machines (GVMs) being redirected along a service path to be processed by a service virtual machine (SVM) of some embodiments.

[0010] Figure 3 The relationship between a service chain and a set of one or more service paths that implement the service chain in some embodiments is conceptually illustrated.

[0011] Figure 4 An example of a service chain and its associated service paths is illustrated.

[0012] Figure 5 Pictured Figure 4 An example of a reverse service path for the forward service path illustrated in FIG.

[0013] Figure 6An example of an input / output (IO) chain component that implements a service plane in some embodiments is illustrated.

[0014] Figure 7 Illustrated is the process performed by the service index pre-processor and service transport layer invoker of some embodiments.

[0015] Figure 8 The diagram corresponds to Figure 7 An example of the data flow for the process described in .

[0016] Fig. 9 Operation of a port proxy of some embodiments to format a data packet for forwarding by a first service node is illustrated.

[0017] Fig.10 The process of some embodiments for delivering a data packet in a service path to a next hop is conceptually illustrated.

[0018] Fig.11 In some embodiments, Figure 6 A process that is performed by a service agent each time it receives a data packet traversing along an ingress path of a service node.

[0019] Fig.12 Three encapsulation headers of a data packet of some embodiments are conceptually illustrated.

[0020] Fig.13 An exemplary process is conceptually illustrated that the SVM performs in some embodiments each time it receives a data packet to be processed from a service agent.

[0021] Fig.14 A first mapping table of the SVM of some embodiments is illustrated.

[0022] Fig.15 An example of a data packet being forwarded from a first-hop service node to a second-hop service node in some embodiments is illustrated.

[0023] Fig.16 The process that a service agent performs in some embodiments each time it receives a data packet traversing along an egress path of its service node is conceptually illustrated.

[0024] Fig.17 The process initiated by an encapsulation processor on a next-hop computer that receives an encapsulated data packet that needs to be processed by an SVM executing on that computer is conceptually illustrated.

[0025] Fig.18 An example of a data packet being forwarded from a second-hop service node to a third-hop service node in some embodiments is illustrated.

[0026] Fig.19 An example of a data packet being forwarded from a third-hop service node back to a first-hop service node in some embodiments is illustrated.

[0027] Fig. 20 Conceptually illustrates a process performed by a service index post-processor in some embodiments.

[0028] Fig.21 The network service header of some embodiments is illustrated.

[0029] Fig. 22 An example of metadata content stored in a metadata content header of some embodiments is illustrated.

[0030] Figure 23-Figure 24 The diagram shows an example in which the service agent forwards a data packet of the GVM with an encapsulated GRE header to the egress side and the ingress side of the SVM.

[0031] Fig.25 The GRE header format used to store service data for the egress direction in some embodiments is illustrated.

[0032] Fig.26 The GRE header format used in some embodiments to store service data for the ingress direction is illustrated.

[0033] Fig. 27 The diagram illustrates the use of two Geneve encapsulation headers, an outer Geneve header for carrying service transport layer data and an inner Geneve header for carrying service insertion layer metadata.

[0034] Fig.28 Figure 1 shows the combined Geneve header Fig. 27 Two Genece encapsulation headers.

[0035] Fig.29 The object data model of some embodiments is illustrated.

[0036] Fig.30 Conceptually illustrated are several operations that network managers and controllers perform in some embodiments to define rules for service insertion, next service hop forwarding, and service processing.

[0037] Fig.31 Illustrated is how the service path may be modified dynamically in some embodiments.

[0038] Fig.32 Illustrated is a process that some embodiments perform to define a service plane and its associated service nodes for tenants in a multi-tenant data center.

[0039] Fig.33The diagram illustrates a process performed by a source host of a migrated GVM and a process performed by a destination host of a migrated GYM.

[0040] Figure 34A-Figure 34C An example of GVM migration in some embodiments is illustrated.

[0041] Fig.35 Operations performed by the control plane following a GVM migration in some embodiments are conceptually illustrated.

[0042] Fig.36 The diagram illustrates a process performed at a source host of a migrated SVM and a process performed at a destination host of a migrated SVM.

[0043] Figure 37A-Figure 37C An example of SVM migration in some embodiments is illustrated.

[0044] Fig.38 Operations performed by the control plane following SVM migration in some embodiments are conceptually illustrated.

[0045] Fig.39 An example of intra-host SVM migration in some embodiments is illustrated.

[0046] Fig.40 An electronic system for implementing some embodiments of the present invention is conceptually illustrated. DETAILED DESCRIPTION

[0047] In the following detailed description of the invention, many details, examples and embodiments of the invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the embodiments set forth and that the invention can be practiced without some of the specific details and examples discussed.

[0048] Some embodiments provide novel methods for performing services for machines running in one or more data centers. For example, for a group of related guest machines (e.g., a group of tenant machines), some embodiments define two different forwarding planes: (1) a guest forwarding plane and (2) a service forwarding plane. The guest forwarding plane connects to the machines in the group and performs L2 and / or L3 forwarding for the machines. The service forwarding plane (1) connects to service nodes that perform services for data packets sent to and from the machines and (2) forwards the data packets to the service nodes.

[0049] In some embodiments, the guest machine is not directly connected to the service forwarding plane. For example, in some embodiments, each forwarding plane is connected to a machine or service node via a port that receives data packets from the machine or service node or provides data packets to the machine or service node. In such embodiments, the service forwarding plane does not have a port that directly receives data packets from any guest machine or provides data packets to any guest machine. Instead, in some such embodiments, data associated with the guest machine is routed to a port proxy module executed on the same host computer, and the other module has a service plane port. In some embodiments, the port proxy module can indirectly connect more than one guest machine on the same host to the service plane (i.e., can be used as a port proxy module for more than one guest machine on the same host).

[0050] In some embodiments, a guest machine is any machine other than a service machine or node. A guest machine can be a tenant's machine in a multi-tenant data center, but it does not have to be. In some embodiments, a guest machine is a guest virtual machine or a guest container. In some embodiments, a service node is a service virtual machine, a service container, or a service device. In some embodiments, a service node performs middleware service operations, such as a firewall, an intrusion detection system, an intrusion prevention system, a load balancer, an encryptor, a message monitor, a message collector, or any number of other middleware services. Therefore, in some embodiments, a service as used in this document is any type of middleware service operation.

[0051] In addition, as used in this document, datagram refers to a set of bits in a specific format sent across a network. Those of ordinary skill in the art will recognize that the term datagram is used in this document to refer to various formatted bit sets sent across a network. The formatting of these bits can be specified by a standardized protocol or a non-standardized protocol. Examples of datagrams that follow standardized protocols include Ethernet frames, IP datagrams, TCP segments, UDP datagrams, etc. In addition, as used in this document, mentioning L2, L3, L4, and L7 layers (or layer 2, layer 3, layer 4, layer 7) refers to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer model, respectively.

[0052] Figure 1An example of a separate guest plane and service plane is illustrated, which in some embodiments is implemented by two logical forwarding elements (LFEs) 130 and 132. As shown, two guest machines 102 and 104 and three service machines 106, 108, and 110 and three software forwarding elements 120, 122, and 124 are executed on three host computers 112, 114, and 116. In this example, the guest machines and service machines are guest virtual machines (GVMs) and service virtual machines (SVMs), but in other embodiments, these machines can be other types of machines, such as containers.

[0053] In addition, in this example, each logical forwarding element is a distributed forwarding element implemented by configuring multiple software forwarding elements (SFEs) on multiple host computers. To this end, in some embodiments, each SFE or a module associated with the SFE is configured to encapsulate the data message of the LFE with an overlay network header containing a virtual network identifier (VNI) associated with the overlay network. Therefore, in the following discussion, the LFE is referred to as an overlay network structure spanning multiple host computers.

[0054] In some embodiments, the LFE also spans configured hardware forwarding elements (e.g., rack switches). In some embodiments, each LFE is a logical switch implemented by configuring multiple software switches (referred to as virtual switches or vswitches) or related modules on multiple host computers. In other embodiments, the LFE can be other types of forwarding elements (e.g., logical routers), or any combination of forwarding elements (e.g., logical switches and / or logical routers) that form a logical network or part thereof. There are currently many examples of LFEs, logical switches, logical routers, and logical networks, including those provided by VMware's NSX network and service virtualization platform.

[0055] As shown, LFE 130 defines a guest forwarding plane connecting GVMs 102 and 104 to forward data packets between these GVMs. In some embodiments, the LFE is a logical switch connected to a logical router, which connects the GVMs directly or through a logical gateway to a network outside the logical network of the logical switch. In some embodiments, LFE 130 is implemented by configuring software switches 120 and 122 and / or their related modules (e.g., related port / VNIC filter modules) on host computers 112 and 114 to implement a first distributed logical switch.

[0056] Figure 1The other figures discussed below show the source GVM and the destination GVM on the same logical network and connected to the same LFE. One of ordinary skill will appreciate that the service operations of some embodiments do not require the source machine and the destination machine to be connected to the same LFE, or even to be in the same network or the same data center. These service operations are performed on data packets leaving the source machine's network or entering the source machine's network. These figures depict the source machine and the destination machine as being connected to the same LFE to emphasize that the service plane 132 is implemented by a logical network that is independent of the logical network that forwards data packets associated with the guest machine.

[0057] LFE 132 defines a service forwarding plane that connects SVMs 106, 108, and 110 to forward data packets associated with the GVM through a service path that includes the SVM. In some embodiments, LFE 132 is also a logical switch that is implemented by configuring software switches 120, 122, and 124 and / or their associated modules on host computers 112, 114, and 116 to implement a second distributed logical switch. Instead of configuring the same set of SFEs to implement both a guest forwarding plane and a service forwarding plane (i.e., a guest LFE and a service LFE), other embodiments configure a set of SFEs on a set of host computers to implement a guest forwarding plane, and configure another set of SFEs on the set of host computers to implement a service forwarding plane. For example, in some embodiments, each host computer executes a guest software switch and a service software switch, and these two switches and / or their associated modules can be configured to implement a guest logical switch and a service logical switch.

[0058] In some embodiments, the software switches 120, 122, and 124 and / or their associated modules may be configured to implement multiple guest forwarding planes (e.g., guest LFEs) and multiple service forwarding planes (e.g., service LFEs) for multiple groups of machines. For example, for a multi-tenant data center, some such embodiments define a guest LFE and a service LFE for each tenant for which at least one service chain needs to be implemented. For each group of related machines (e.g., for each tenant's machine), some embodiments define two virtual network identifiers (VNIs) to configure a common set of software forwarding elements (e.g., software switches) to implement two different forwarding planes, namely, a guest forwarding plane and a service forwarding plane. These two VNIs are referred to below as a guest VNI (GVNI) and a service VNI (SVNI). In Figure 1 , guest LFE ports 150 and 152 are associated with the GVNI, while service LFE ports 154, 156, and 158 are associated with the SVNI, as shown.

[0059] In some embodiments, the service plane 132 is also implemented by inserting modules in the input / output (IO) chain of the ingress and egress data paths of the GVM to and from the SFE 120 or 122. In this embodiment, the service plane 132 can identify a data packet sent from or received for the GVM, forward the data packet to a collection of SVMs to perform a service chain on the data packet, and then return the data packet to the data path of the GVM so that the data packet can travel along its data path to the software switch or GVM (i.e., so that the data packet can be processed according to the destination network address specified by the source GVM). Such a GVM is referred to as the source GVM below because the data packet being processed by the service node is a data packet identified on the egress path or ingress path of the GVM. In some embodiments, the egress / ingress IO chain of the GVM is implemented as a collection of hooks (function calls) in the GVM's VNIC (virtual network interface card) 180 or an SFE port associated with the GVM's VNIC (e.g., an SFE port that communicates with the GVM's VNIC).

[0060] Before providing examples of IO chain components that implement some embodiments of the service plane, Figure 2 The following example is illustrated: a data packet 202 from GVM 102 to GVM 104 is redirected along service plane 132 so that the data packet can be processed by SVMs 108 and 110 that perform a chain of two service operations. As shown, service LFE 132 first forwards the data packet to SVM 108, then forwards the data packet to SVM 110, and then returns the data packet to the egress path of GVM 102 so that the data packet can be processed based on the destination network address specified by the source GVM 102.

[0061] In some embodiments, the service LFE forwards the data packet between hosts 112, 114, and 116 by using an overlay encapsulation header storing the SVNI for the service LFE. In addition, when the service LFE is a service logical switch, the service forwarding plane in some embodiments uses a MAC address associated with the SVM (e.g., a MAC address of the SVM VNIC) to forward the data packet between ports of the service logical switch. In some embodiments, the MAC forwarding also uses a service plane MAC address associated with the source GVM, even if the GVM is not directly connected to the service plane but is connected to the service plane through a port proxy as further described below.

[0062] Once data packet 202 returns to the egress path of GVM 102, guest LFE 130 forwards the data packet to its destination (e.g., as specified by the destination network address in the header of the data packet), namely GVM 104. In some embodiments, guest LFE 130 forwards the data packet between hosts 112 and 114 by using an overlay encapsulation header that stores the GVNI for the guest LFE. In addition, when the guest LFE is a logical switch, the guest forwarding plane in some embodiments forwards the data packet using the guest plane MAC address associated with GVMs 102 and 104 (e.g., by using the guest plane MAC address of GVM 104 to forward the data packet to the guest forwarding port 152 associated with this GVM). Although Figure 2 The service plane captures the data packet passing through the egress path of the GVM, but in some embodiments, the service plane may also capture the data packet when the data packet passes through the ingress path of the GVM before the data packet reaches the VNIC of the GVM.

[0063] In some embodiments, a chain of service operations is referred to as a service chain. In some embodiments, a service chain can be implemented with one or more sets of service nodes (e.g., service machines or devices), where each set of service nodes defines a service path. Thus, in some embodiments, a service chain can be implemented by each of the one or more service paths. In some embodiments, each service path includes one or more service nodes for executing a set of one or more services of the service chain and a specific order through these nodes.

[0064] Figure 3An object diagram is presented that illustrates the relationship between a service chain 302 and a set of one or more service paths 304 that implement the service chain. Each service chain has a service chain (SC) identifier 306, and each service path has a service path identifier (SPI) 308. Each service path is associated with a set of m service nodes, which are identified with respect to a service instance endpoint 310 as shown. In some embodiments, a service instance endpoint is a logical location in a network, and traffic at a service instance endpoint can go to or come from a service node connected to a service plane. In some embodiments, a service instance endpoint is an LFE port (e.g., an SFE port) associated with a service node (e.g., a VNIC of an SVM). In these or other embodiments, a service instance endpoint can be associated with two LFE ports for a service node, as further described below for an embodiment using GRE encapsulation. In addition, in some embodiments, a service endpoint can be addressed by a MAC address associated with an LFE port or a MAC address associated with an SVM VNIC associated with these LFE ports (e.g., communicating with these LFE ports).

[0065] In some embodiments, each service chain 302 is defined by a reference to one or more service profiles 312, each of which is associated with a service operation in the chain. As described below, in some embodiments, a service node (1) receives a mapping of a service chain identifier to a service profile that it must implement from a service manager, and (2) receives the service chain identifier using a datagram and maps it to the service profile to determine the service operation that it must perform. In some embodiments, the received mapping is based not only on a service chain identifier (SCI), but also on a service index value (which specifies the position of the service node in the service path) and a direction through the service chain (which specifies the order in which the sequence of services specified by the service chain is executed). In some embodiments, the service profile describes the service operations that the service node must perform. In some embodiments, the service profile can identify a set of rules for the service node to check.

[0066] Furthermore, in some embodiments, service insertion rules 314 are defined by referencing service chain identifiers 306 for service insertion modules associated with the GVM. Such service insertion modules use these service insertion rules 314 to identify service chains for processing data packets associated with the source GVM. As described above, the data packets are hereinafter referred to as being from the source GVM because the data packets processed by the service chain are identified on either an egress path from the GVM or an ingress path to the GVM.

[0067] As further described below, a service insertion (SI) rule associates a flow identifier with a service chain identifier. In other words, some embodiments attempt to match the flow attributes of a data packet with the flow identifier of a service insertion rule (hereinafter referred to as the rule identifier of the SI rule) to identify a matching service insertion rule (i.e., a rule having a set of flow identifiers that match the flow attributes of the data packet) and assign the specified service chain of the matching rule as the service chain of the data packet. A specified flow identifier (e.g., a flow identifier defined by reference to a five-tuple identifier) ​​can identify a specified data packet flow, while a more general flow identifier (e.g., a flow identifier defined by reference to less than a five-tuple) can identify a set of several different data packet flows that match the more general flow identifier. Therefore, a matching data packet flow is any set of data packets having a common set of attributes that match the rule identifier of the service insertion rule.

[0068] As further described below, other embodiments use context attributes associated with data message flows to associate data messages with service insertion rules. A variety of techniques for capturing and using context attributes to perform forwarding and service operations are described in U.S. Patent Application 15 / 650,251, which is incorporated herein. Any of these techniques can be used in conjunction with the embodiments described herein.

[0069] In some embodiments, the next hop forwarding rule 316 is defined by referencing the SPI value 308 and the service instance endpoint 310. Specifically, in some embodiments, a service path is selected for a service chain that has been identified for a data packet. At each hop, these embodiments use a forwarding rule 314 to identify the next service instance endpoint based on the SPI value of the service path together with a current service index (SI) value that identifies the position of the hop in the service path. In other words, each forwarding rule in some embodiments has a set of matching criteria defined with respect to the SPI / SI values, and specifies the network address of the next hop service instance endpoint associated with these SPI / SI values. In order to optimize the next hop lookup for the first hop, some embodiments provide the next hop network address with the SPI to the service insertion module of the source GVM as part of the service path selection process.

[0070] Figure 4An example of a service chain and its associated service path is illustrated. As shown, in some embodiments, each service chain 405 is defined as a sequential list of service profiles 410, in which each profile is associated with a different middleware service (e.g., firewall, load balancer, intrusion detector, data packet monitor, etc.). In addition, in this example, each of the M profiles can be implemented by an SVM in a cluster m of VMs. As shown, different clusters of different profiles can have different numbers of SVMs. In addition, in some embodiments, a service profile is implemented by a service node (i.e., a cluster of several service nodes is not required to implement the service profile).

[0071] Because multiple SVMs in a cluster can provide a particular service, some embodiments define multiple service paths for a given service chain through multiple different combinations of SVMs, with each combination using one SVM from each cluster. Figure 4 In the example of FIG. 4 , there are N service paths associated with service chain 405 that a data packet originating from GVM 402 traverses on its way to GVM 404. Each service path is identified by a different set of dashed lines in the figure.

[0072] Specifically, the first service path passes through the first SVM 1,1 of the cluster of the first service profile to implement the first service of the forward service chain 405, passes through the first SVM 2,1 of the cluster of the second service profile to implement the second service of the forward service chain 405, and passes through the third SVM M,3 of the cluster of the Mth service profile to implement the Mth service of the forward service chain 405. The second service path passes through the second SVM 1,2 of the cluster of the first service profile to implement the first service of the forward service chain 405, passes through the first SVM 2,1 of the cluster of the second service profile to implement the second service of the forward service chain 405, and passes through the first SVM M,1 of the cluster of the Mth service profile to implement the Mth service of the forward service chain 405.

[0073] The third service path passes through the third SVM 1,3 of the cluster of the first service profile to implement the first service of the forward service chain 405, passes through the second SVM 2,2 of the cluster of the second service profile to implement the second service of the forward service chain 405, and passes through the second SVM M,2 of the cluster of the Mth service profile to implement the Mth service of the forward service chain 405. The Nth service path passes through the third SVM 1,3 of the cluster of the first service profile to implement the first service of the forward service chain 405, passes through the second SVM 2,2 of the cluster of the second service profile to implement the second service of the forward service chain 405, and passes through the fourth SVM M,4 of the cluster of the Mth service profile to implement the Mth service of the forward service chain 405. As illustrated in the example, for a given service operation, different service paths can use the same SVM. However, regardless of the service path traversed by a given data packet, the same set of service operations are performed in the same order for the paths associated with the same service chain and the same service direction.

[0074] In some embodiments, service chaining must be performed in a forward direction for data packets from a first GVM to a second GVM, and then service chaining must be performed in a reverse direction for data packets from the second GVM to the first GVM. In some such embodiments, the service plane selects a service path for a forward direction and a service path for a reverse direction when it processes a first data packet in a flow from the first GVM to the second GVM. Furthermore, in some of these embodiments, the forward and reverse service paths are implemented by the same set of service nodes in reverse order.

[0075] Figure 5 Pictured Figure 4 404. The forward service path is used to perform M services on the data packets from GVM 402 to GVM 404, while the reverse service path is used to perform M services on the data packets from GVM 404 to GVM 402. In addition, these services are reverse, Figure 5 The service path in the execution service profile M to 1, and Figure 4 The service paths in the execution service profiles 1 to M.

[0076] In addition, Figure 4 and Figure 5In the example of , each reverse service path has a corresponding forward service path that is implemented by the exact same set of SVMs, but in reverse order, as indicated by the service path legend and similar dashed lines in these figures. For example, the forward second service path passes through SVM 1,2 of the first service associated with the first profile, SVM 2,1 of the second service associated with the second profile, and SVM M,1 of the Mth service associated with the Mth service profile, while the associated reverse second service path passes through SVM M,1 of the first service associated with the Mth service profile, SVM 2,1 of the second service associated with the second profile, and SVM 1,2 of the second service associated with the first profile.

[0077] In some embodiments, the same service node is used for both the forward and reverse paths because at least one of the service nodes (e.g., firewall SVM) implementing one of the service profiles needs to see data traffic in both directions between two data endpoints (e.g., two GVMs). In other embodiments, the same service node need not be used for both directions of data packet flow between two data endpoints, as long as the same set of service operations are performed in reverse order.

[0078] Figure 6 The example of the IO chain component that realizes the service plane in some embodiments is illustrated.As shown in the figure, the service plane 132 is implemented by two sets of software switches 120, 122 and 124 executed on the host computer and modules 610, 612, 614, 620, 624, 626 and 628 on these computers.The service plane implemented in this example and some other examples illustrated in some subsequent figures are overlay logic L2 service planes.Ordinary technicians will recognize that other embodiments are implemented by other types of service planes, such as overlay L3 service planes, or overlay networks with multiple L2 logical switches and one or more logical L3 routers.

[0079] exist Figure 6In the embodiment of the present invention, software switches 120, 122 and 124 and modules 610, 612, 614, 620, 624, 626 and 628 implement two different layers of the service plane, namely, service insertion layer 602 and service transport layer 604. Service insertion layer 602 (1) identifies a service chain for a data packet, (2) selects a service path for performing service operations for the service chain, (3) identifies the next hop service node at each hop in the selected service path (including identification of the source host computer to which the data packet should be returned after completing the service chain), and (4) specifies service metadata (SMD) header attributes for the data packet for the service path. In some embodiments, the SMD attributes include network service header (NSH) attributes according to RFC (Request for Comments) 8300 of the IETF (Internet Engineering Task Force).

[0080] On the other hand, the service transport layer 604 formulates a service overlay encapsulation header and encapsulates a data message with the header, so that the data message can be transmitted between service jumps. In some embodiments, the service transport layer 604 modifies the SMD header to generate a service overlay encapsulation header. For example, in some of these embodiments, the overlay encapsulation header is a Geneve header, wherein the SMD attribute is stored in the TLV (type, length, value) part of the Geneve header. In other embodiments, the service transport layer 604 adds the service overlay encapsulation header to the SMD header that is first used to encapsulate the data message. In addition, when passing between two hops executed on the same host computer (for example, between two service nodes), in some embodiments, the service transport layer in several embodiments described below does not encapsulate the data message with the overlay encapsulation header. In other embodiments, even when passing between two hops on the same host computer, the service transport layer encapsulates the data message with the overlay encapsulation header.

[0081] In some embodiments, the service insertion (SI) layer 602 includes an SI pre-processor 610 and an SI post-processor 612 for which one or more service chains are defined in each of the two IO chains 650 and 652 of the GVM (i.e., the egress IO chain 650 and the ingress IO chain 652). The SI layer 602 also includes a service agent 614 for each service node connected to the service plane (e.g., for each SVM having a VNIC paired with a service plane LFE port). The service transport (ST) layer 604 includes an STL port agent 620 on each host computer, which has one or more possible source GVMs for which one or more service chains are defined. The ST layer 604 also has (1) an STL caller 624 in each IO chain of each source GVM, (2) an STL module 626 in the IO chain of each SVM, and (3) one or more encapsulation processors 628.

[0082] For a data message passing through an ingress or egress data path of a GVM, an SI preprocessor 610 on the data path performs several operations. The preprocessor identifies a service chain for the data message and selects a service path for the identified service chain. The preprocessor also identifies the network address of the first hop service node in the selected service path and specifies the SMD attribute of the data message. In some embodiments, the SMD attribute includes a service chain identifier (SCI), SPI and SI values, and the direction of the service operation for processing the service chain (e.g., forward or reverse). In some embodiments, the SPI value identifies the service path, and the SI value specifies the number of service nodes.

[0083] After the SI preprocessor completes its operation, the STL caller 624 in the same data path calls the STL port agent 620 to relay the SMD attributes identified by the preprocessor and the network address of the first hop, so that the port agent can forward the SMD attributes to the first hop through the service plane. The port agent formats the data message to forward it to the first service node. In some embodiments, the formatting includes replacing the original source MAC address and destination MAC address in the data message with the service plane MAC address associated with the source GYM 102 and the MAC address of the first hop service node. The formatting also stores a set of attributes for data messages that should be processed by other service transport layer modules (e.g., other STL modules, etc.) on the same host computer. These data message attributes include SMD attributes and the original source MAC address and destination MAC address.

[0084] The STL port agent 620 passes the formatted datagram along with its stored attributes to the software switch 120. Based on the destination MAC address of the formatted datagram (i.e., the first-hop MAC address), the software switch delivers the datagram to the switch port associated with the first-hop SVM. When the first hop is on the same host computer as the port agent 620, the datagram is provided to the STL module 626 in the ingress IO chain of the service node of the first hop on the same host computer. When the first hop is not on the same host computer, the datagram is encapsulated with an encapsulation header and forwarded to the next hop, as further described below.

[0085] The STL module 626 of each hop reformats the data message by replacing the service plane source MAC address and the service plane destination MAC address (i.e., the MAC address of its service node) with the original source MAC address and destination MAC address of the data message. The module then passes the reformatted data message along with its accompanying SMD attributes to the service agent 614 of its hop. The service agent is in the IO chain of the ingress data path of the GVM. In order to prevent Figure 6The illustration in is overcomplicated with unnecessary detail, and unlike the ingress and egress paths 650 and 652 of GVM 102, the ingress and egress paths of each SVM in this example are combined in the figure.

[0086] The service agent 614 encapsulates the received data message with an encapsulated NSH header storing the SMD attribute of the data message, and provides the encapsulated data message to the service node when its service node can support the NSH header. When the service node is an SVM, the service agent in some embodiments provides the data message and its NSH header to the VNIC of the SVM through the VNIC injection process, as further described below. When the service node cannot process the NSH header, the service agent 614 stores the SMD attribute in a traditional (legacy) QinQ encapsulation header or a GRE encapsulation header, and then passes the encapsulated data message to the VNIC of the SVM. These headers will be further described below.

[0087] In some embodiments, the service agent 614 for each service hop separates the service node for that hop from the service transport layer. This separation improves the security of the SVM and the service transport layer. The separation also allows the service agent to ensure that the data packets provided to its SVM are properly formatted, which is particularly important for traditional SVMs that do not support the newer NSH format.

[0088] In some embodiments, service agent 614 also performs activity (liveness) detection signaling with its service node to ensure that the service node is operable. In some embodiments, the service agent sends a data message with activity value to its service node at least once in each cycle time period. For this reason, the service agent sets and resets a timer to ensure that it has sent the activity signal of each time period to its service node. Each activity value is accompanied by an activity sequence number to allow the service agent to track the activity response provided by the SVM. When the service node replies the activity signal each time, it provides the same activity value to the service agent in the response data message in some embodiments, or provides its corresponding value in the response data message in other embodiments. In addition, for each activity response data message, the service node provides the same sequence number in some embodiments, or provides an incremental version of the sequence number provided by the service agent in other embodiments.

[0089] As further described below, the service agent of some embodiments carries some activity detection signaling of the service agent on each data packet that it passes from the service forwarding plane to its service node. Each time the service agent sends an activity signal to its service node, its activity timer is reset. Each time the service node processes a data packet, the processed data packet is returned to the service node along with a responsive activity value and an associated sequence number (incremented in some embodiments, or not incremented in other embodiments, as described above).

[0090] In some embodiments, when a service node does not respond to the service agent's activity signal within a specific time (e.g., within 0.3 seconds), the service agent registers an activity detection failure. After registering two consecutive activity detection failures, in some embodiments, the service agent notifies the local control plane (LCP) module executed on its host that the SVM has failed, so that the LCP can notify the central control plane (CCP) server. In response to such a notification, the CCP removes the SVM and the service path where the SVM is located from the forwarding and path selection rules of the data plane, and if necessary, generates additional service paths for the associated service chains of the failed SVM. In addition, in some embodiments, the service agent sends an in-band data packet back to the source GVM to program its classifier not to select the service path where the failed service node is located.

[0091] In some embodiments, the service agent also performs flow programming at the request of its service node. In some embodiments, the flow programming involves modifying how the IO chain of the source GVM selects a service chain, a service path, and / or forwards a datagram flow along a service path. In other embodiments, the flow programming involves other modifications to how the datagram flow is processed by the service plane. Flow programming will be further described below.

[0092] Upon receiving a datagram and its SMD attribute (in an encapsulating NSH header or some other encapsulating header), the SVM performs its service operation. In some embodiments, the SVM uses a mapping record received from its service manager to map the SCI, SI, and direction values ​​in the SMD attribute to a service profile, which is then mapped to one of its rule sets, and then checked to identify one or more service rules for processing. In some embodiments, each service rule has a rule identifier defined with respect to datagram attributes (e.g., a five-tuple attribute, i.e., source and destination IP addresses, source and destination port addresses, and protocols). The SVM in some embodiments compares the identifier of the rule with the attributes of the datagram to identify a matching rule. When identifying one or more matching rules, the SVM in some embodiments performs the action specified by the highest priority matching rule. For example, a firewall SVM can specify that a datagram should be allowed to pass, should be discarded, and / or should be redirected.

[0093] Once the SVM has completed its service operation, the SVM forwards the data message along its egress data path. The service agent in the IO chain of the egress data path then captures this data message, and for this data message, identifies the network address of the next hop in the service path. For this reason, in some embodiments, the service agent decrements the SI value, and then uses the decremented value together with the SPI value in the stored attribute set of the data message to identify the complete match forwarding rule for identifying the next hop network address. In some embodiments, the SVM can decrement the SI value. For this situation, the service agent in some embodiments can be configured to not decrement the SI value when its corresponding SVM decrements the SI value.

[0094] In either configuration, the service agent identifies the next hop network address by using the appropriate SPI / SI value to identify the next hop forwarding rule applicable to the data message. When the service node of the agent is on multiple service paths, the forwarding rule storage device of the agent stores multiple fully matched forwarding rules, which can specify different next hop network addresses for different SPI / SI values ​​associated with different service paths. Assuming that the decremented SI value is not zero, the next hop in the service path is another service node. Therefore, in some embodiments, the agent provides the MAC address of the next hop to the associated STL module 626 of the agent in the egress data path of the SVM. This module then reformats the data message by the following operations: specifying the MAC address of the SVM and the MAC address of the next hop as the source MAC address and the destination MAC address, and storing the original source MAC address and the destination MAC address of the data message in the stored set of attributes stored for the data message. The STL module 626 then forwards the data message along the egress path, and the data message arrives at the software switch through the egress path, and then the software switch must forward the data message and its stored attributes to the next hop service node.

[0095] When the next hop is on the same host computer, the software switch passes the datagram and its attributes to the port connected to the STL module of the service node of the next hop, as described above. On the other hand, when the next hop service node is on another host computer, the software switch provides the datagram to the uplink port connected to the VTEP (VXLAN tunnel endpoint), which communicates with the VTEP on the other host computer through the overlay network tunnel. The encapsulation processor 628 then captures the datagram along the exit path of the port, defines an encapsulation overlay header for the datagram, and encapsulates the datagram with the overlay header. In some embodiments, the overlay header is a single header that stores both the SMD and STL attributes. For example, in some embodiments, the overlay header is a Geneve header that stores the SMD and STL attributes in one or more TLVs.

[0096] As described above, the SMD attributes in some embodiments include an SCI value, an SPI value, an SI value, and a service direction. In addition, in some embodiments, the STL attributes include the original L2 source MAC address of the source GVM, the original L2 destination MAC address, the data message direction, and the service plane source MAC address. In some embodiments, the service direction and the service plane source MAC address are already part of the SMD attributes. In some embodiments, the service transport layer requires these attributes and each processed data message in order to recreate the original data message and later return the data message to the original host at the end of the service path to restart along its data path.

[0097] When the encapsulated datagram is received at the host computer of the next hop, the datagram is captured by the encapsulation processor 628 of the downlink port of the software switch, which is connected to the VTEP that received the datagram from the VTEP of the previous hop. The encapsulation processor removes the encapsulation header from the datagram and stores the STL and SMD attributes as a set of attributes of the datagram. The encapsulation processor then passes the decapsulated message to the downlink port, which then passes the message to the software switch for forwarding to the switch port of the next hop. From there, the datagram is processed by the STL module and the service agent before reaching the service node, as described above.

[0098] When the service agent determines that the SI value of decreasing is zero, the service agent matches the SI value of decreasing and the embedded SPI value with the rule that the service plane MAC address of the source GVM is identified as by the guiding service agent by guiding the ...

[0099] The STL module then passes the formatted datagram and its attributes along the egress path, through which the datagram arrives at its associated software switch port. The software switch then passes this message to its uplink port. The encapsulation processor 628 of the port then captures the datagram, defines an encapsulation overlay header for the datagram, and encapsulates the datagram with the overlay header. As described above, this overlay header is a Geneve header that stores the SMD and STL attributes in one or more TLVs. This encapsulated datagram then traverses the overlay network to the host computer of the source GVM, where the datagram is decapsulated by the encapsulation processor of the downlink port and then provided to the software switch, which then forwards it to the port agent.

[0100] Once the port agent 620 receives the decapsulated datagram, it identifies the GVM associated with this datagram from the original source MAC address, which is now part of the stored attributes of the decapsulated datagram. In some embodiments, the port agent has a record of mapping the original source MAC address and service direction in the SMD attributes of the received data to the GVM on its host (e.g., mapped to the software switch port associated with the guest forwarding plane and the GVM on its host). The port agent then formats the datagram to include its original SMAC and DMAC, and provides the datagram back to the IO chain of the source GVM. The SI post-processor 612 in the IO chain then processes the datagram and then returns the datagram to the egress data path of the GVM. The operation of the post-processor will be further described below.

[0101] One of ordinary skill will appreciate that the service insertion layer and service transport layer in other embodiments are implemented differently than the exemplary implementation described above. For example, instead of using an L2 overlay (L2 transport layer) that relies on MAC addresses to traverse different service hops, other embodiments use an L3 overlay (L3 transport layer) that uses L3 and / or L4 network addresses to identify consecutive service hops. Furthermore, the service insertion module and / or service transport module described above may be configured to operate differently.

[0102] Now refer to Figure 7-Figure 19 A more detailed example describing the operation of the service insertion layer and the service transport layer. Figure 7 The process 700 performed by the SI preprocessor 610 and the STL caller 624 of some embodiments is illustrated. Figure 8 The data flow example shown describes the process. The process 700 begins when the SI pre-processor 610 is invoked to analyze a data packet sent along an ingress path or an egress data path of a GVM.

[0103] As shown, process 700 initially determines (at 705) whether preprocessor 610 has pre-selected a service chain and a service path for a flow of data packets and stored SMD attributes for the selected service chains and paths. In some embodiments, process 700 makes this determination by using attributes of the data packets (e.g., its five-tuple attributes) to attempt to identify records of the flow of packets in a connection tracker storing records of flow of packets for which service chains and paths have been pre-selected, and SMD attributes have been pre-stored in the connection tracker records for these chains and paths.

[0104] Figure 8 Pre-processor 610 is illustrated receiving data packet 802 along the egress data path of GVM 102 . Figure 8Also shown is a pre-processor that initially checks the connection tracking storage 804 to try to find a connection record with a flow identifier (e.g., a five-tuple identifier) ​​that matches the set of attributes (e.g., a five-tuple attribute) of the received datagram. In this example, the pre-processor 610 cannot find such a connection record because the received datagram is the first datagram of its flow.

[0105] When process 700 determines (at 705) that connection storage device 804 has a connection record that matches the received data message, the process retrieves (at 710) SMD attributes from the record or from another record referenced by the matching connection record. The SMD attributes in some embodiments include SCI, SPI, SI, and direction values. The process transitions from 710 to 740, which will be described below.

[0106] On the other hand, when the process 700 determines (at 705) that the connection storage device 804 does not have a connection record that matches the received data message, the process performs (at 715) a classification operation that attempts to match the data message with the SI rule storage device (at Figure 8 In some embodiments, the SI rule storage device 806 stores service insertion rules 822 with rule identifiers defined with respect to one or more data packet flow attributes (e.g., one or more of the five-tuple attributes or portions thereof). Each service rule also specifies an SCI that identifies a service chain applicable to a data packet flow that matches the rule identifier of the service rule.

[0107] At 720, the process determines whether the classification operation matches the attribute of the data message with a rule identifier of a service insertion rule that requires the execution of a service chain on the data message. When the classification operation does not identify a service insertion rule that requires the execution of a service chain on the data message, the process 700 ends. In some embodiments, when the attribute of the data message does not match any higher priority SI rule, the SI rule storage device 806 has a default low priority rule that matches any data message, and the default low priority rule specifies that a service chain is not defined for the flow of the data message. In some embodiments, when it is not necessary to perform a service operation on the data message flow, a service chain is not defined for the data message flow.

[0108] On the other hand, when the classification operation matches the attribute of the data packet with a rule identifier of a service insertion rule requiring execution of a service chain on the data packet, the process 700 performs (725) a path selection operation to select a service path of the service chain specified by the service insertion rule identified at 715. Figure 8As shown, the pre-processor 610 performs a path selection operation by examining a path storage table 808 that identifies one or more service paths for each service chain identifier.

[0109] Each service path is specified with respect to its SPI value. When multiple service paths are specified for a service chain, the path storage device 808 stores a set of selection metrics 820 for selecting an SPI from the available SPIs for each service chain. Different embodiments use different selection metrics. For example, some embodiments use a selection metric that costs the service path based on the number of hosts on which the service nodes of the service path execute. In other embodiments, these selection metrics are weight values ​​that allow the preprocessor to select an SPI for the service chain in a load balancing manner indicated by these weight values. For example, in some embodiments, these weight values ​​are generated by the central control plane based on the load on each service node in the service node in the service path and / or based on other costs (such as the number of hosts traversed by the service path, etc.).

[0110] In some of these embodiments, the preprocessor keeps a record of previous selections it has made for a particular service chain, and selects subsequent service paths based on these previous selections. For example, for four service paths, the weight values ​​may be 1, 2, 2, 1, which specifies that over six consecutive SPI selections for the service chain, the first SPI should be selected once, then the second and third SPIs should each be selected twice, and the fourth SPI should be selected once. The next SPI selection for this service chain will then select the first SPI, since the selection mechanism is round robin.

[0111] In other embodiments, the weight value is associated with a range of values ​​(e.g., a range of hash values), and a number is randomly or deterministically generated for each datagram flow to map the datagram flow to the range of values ​​and thereby to its associated SPI. In other embodiments, the host LCP selects a service path for each service chain identifier from a pool of available service paths, so only one SPI is stored for each SCI in the path table 808. The LCP in these embodiments selects a service path for each service chain based on cost (e.g., the number of hosts traversed by each service path and / or the load on the service nodes of the service path).

[0112] After identifying the service path for the identified service chain, process 700 next identifies (at 730) the network address of the first hop of the selected service path. In some embodiments, the MAC address of the hop is stored in the same record as the SPI of the selected path. Therefore, in these embodiments, the MAC address is retrieved from the path selection storage device 808 using the selected SPI. In other embodiments, the preprocessor retrieves the MAC address of the first hop from the full match forwarding table 810, which stores the next hop network address of the associated SPI / SI value pair, such as Figure 8 In some embodiments, the initial SI values ​​of the service chain are stored in the SI rules of the SI rule storage device 806 , while in other embodiments, these initial SI values ​​are stored in the path table 808 together with the SPI value.

[0113] At 735, process 700 specifies SMD attributes of the data message and associates these attributes with the data message. As described above, in some embodiments, the SMD attributes include SCI, SPI, SI, and direction values. Because the direction through the service chain depends on the service path, the service direction of the service path is stored in the path table 808 together with the SPI value. In addition, as described below, in some embodiments, the service chain must be performed in the forward direction for the data message from the first GVM to the second GVM, and then in the reverse direction for the data message from the second GVM to the first GVM. For such a service chain, the preprocessor 610 selects both the service path for the forward direction and the service path for the reverse direction when it processes the first data message in the flow from the first GVM to the second GVM.

[0114] After the SI preprocessor completes its operations, the STL caller 624 in the same data path calls (at 740) the STL port agent 620 to relay the SMD attributes identified by the preprocessor and the network address of the first hop, so that the port agent can forward the SMD attributes to the first hop through the service plane. Figure 9-Figure 19 6 and other modules in the service insertion layer and the service transport layer. These figures describe an example of processing a data packet from GVM 102 through a service path that includes SVM 106, then SVM 108, and then SVM 110.

[0115] In these figures, each GVM is a computer of a tenant in a multi-tenant data center and is connected to a software switch via a switch port associated with the tenant's guest VNI (GVNI). In addition, in these figures, each SVM is a service machine for processing GVM message traffic and is connected to the software switch via a switch port associated with the tenant's service VNI (SVNI). As described above and further described below, some embodiments use GVNI to perform guest logical forwarding operations for tenants (i.e., establish guest logical forwarding elements, such as logical switches or routers, or guest logical networks), while using SVNI to perform service logical forwarding operations for tenants (i.e., establish service logical forwarding elements, such as logical switches or routers, or service logical networks).

[0116] In some embodiments, these logical network identifiers (i.e., GVNI and SVNI) are generated by the management or control plane for the tenant. The management or control plane of some embodiments generates different GVNI and SVNI for different tenants so that no two tenants have the same GVNI or SVNI. In some embodiments, each SVM is dedicated to one tenant, while in other embodiments, one SVM can be used by multiple tenants. In the case of multi-tenancy, each SVM can be connected to different ports of different service planes (e.g., different logical switches) for different tenants.

[0117] like Fig. 9 As shown, port proxy 620 formats the data packet for forwarding to the first service node by replacing the original source MAC address and destination MAC address in the data packet with the service plane MAC address associated with source GVM 102 and the MAC address of the first hop service node. This operation is depicted as Fig.10 Operation 1005 in process 1000. The process 1000 is a process started by the port agent 620 or the STL module 626 whenever the SI module (eg, the SI preprocessor 610 or the SI agent 614) completes processing of a data message.

[0118] In this process 1000, the port agent also adds (at 1010) the original source MAC address and destination MAC address of the data packet to the collection of attributes of the data packet that should be processed by other service transport layer modules on the same host computer (e.g., vswitch, other STL modules, encapsulation processors, etc.). Fig. 9 A reformatted datagram 902 and an enhanced attributed set 904 are depicted.

[0119] After reformatting the datagram and enhancing its attribute set, the port agent 620 passes the formatted datagram along with its stored attribute set along its egress path (at 1015), through which the datagram reaches the software switch 120. Based on the destination MAC address of the formatted datagram (e.g., the first-hop MAC address), the software switch determines (at 1020) whether the port of the next hop is local. This is Fig. 9 Thus, the software switch delivers (at 1025) the datagram to the switch port associated with the first-hop SVM 106. The port then sends the datagram along the ingress path of the SVM, where the datagram 902 and its enhanced attribute set 904 are identified by the STL module 626 through a function call of the ingress IO chain of the first-hop SVM, as shown in FIG. Fig. 9 shown.

[0120] The STL module 626 then reformats (at 1030) the data packet by replacing the service plane MAC address and the first-hop MAC address of the GVM (i.e., the MAC address of the SVM 106) with the original source MAC address and destination MAC address of the data packet it retrieved from the enhanced attribute set 904. Upon retrieving the original SMAC and DMAC addresses, the STL module 626 modifies the attribute set of the data packet. Fig. 9 906 and the modified attributed set 908. The STL module then passes this reformatted datagram and its accompanying SMD attributes along the ingress path of the SVM, where the datagram is subsequently processed by the ingress service agent 614 of that hop.

[0121] Fig.11 The process 1100 that the service agent 614 performs each time it receives a data packet traversing along the ingress path of a service node in some embodiments is illustrated. As shown, the service agent initially makes (at 1105) a copy of the data packet, if necessary. For example, in some embodiments, a service node only needs to receive a copy of the data packet to perform its operation. An example of such a service node is a monitoring SVM that needs to obtain a copy of the data packet for its packet monitoring operation or mirroring operation.

[0122] In these embodiments, the service agent copies the data message and performs the remaining operations 1110-1125 on the copy while passing the original data message to the next service hop or back to the source GVM. In order to forward the original data message to the next service hop or back to the GVM, the service agent must perform a next hop lookup based on the SPI / SI value and then provide the next hop address (e.g., the address of the next service hop or the service plane MAC of the source GVM) to the STL module for forwarding. These lookup and forwarding operations are similar to those described below with reference to Figure 15-17 The operations described.

[0123] Next, at 1110, the service agent sets an activity attribute in the stored SMD attribute set for the data message (in some embodiments, the data message may be a copy of the data message at this point). The activity attribute is a value that directs the service node to provide a response activity value (the same value or a related value) to the data message after it has processed the data message. For the activity attribute, the service agent also provides a sequence number to the response activity value, which the service node must return, or increment and return, as described above.

[0124] At 1115, if necessary, the service agent formats the data message to convert it into a form that can be processed by the service node. For example, when the service node does not know the current next-hop MAC set as the destination MAC of the data message, the service agent changes the destination MAC of the message to a destination MAC associated with the service node.

[0125] After formatting the data message to sanitize it for forwarding to the service node, the service agent 614 encapsulates (at 1120) the data message with one of three encapsulation headers that it may be configured to use and passes (at 1125) the encapsulated message along the ingress path of the service node so that the message can be forwarded to the service node. Fig. 9 An encapsulated data packet 920 is illustrated being passed from a service proxy to an SVM 106 with a native NSH encapsulation header. As shown, the encapsulation header 922 includes a service chain identifier, a service index, a service chain direction, and an activity signal.

[0126] Fig.12 Three encapsulation headers of some embodiments are illustrated, namely (1) a native NSH encapsulation header 1205 for a service node that supports NSH, (2) a GRE encapsulation header 1210 for a legacy service node that does not support NSH, and (3) a QinQ encapsulation header 1215 for a legacy service node that does not support NSH. The native NSH header is referred to below. Fig.21 and Fig. 22 The format described below is used to store storage metadata. Figure 25-26 The GRE header format is further described. In both GRE and QinQ formats, a portion of the service metadata is stored in the GRE and QinQ header fields, but the service metadata cannot be stored as richly as in the native NSH header. The QinQ header is used for simple traditional service nodes that do not require a lot of service metadata, for example, only a service chain identifier and service direction, or a service chain identifier and service index. The service metadata is stored in the VLAN header field of the QinQ header.

[0127] In addition to the three different types of encapsulation headers 1205, 1210, and 1215, Fig.12 Also illustrated is a vmxnet3 paravirtualized NIC 1240 of the SVM of some embodiments. As shown, the NIC can provide encapsulated data packets to a polling mode driver 1202 of a DPDK driver 1204 of the SVM, or an interrupt mode driver 1204. Specifically, depending on the driver used in the SVM, the vmxnet3 paravirtualized NIC can be configured to operate in different operating modes. The polling mode driver 1202 can be viewed as a backend for a DPDK (Data Plane Development Kit) driver 1206. The polling mode driver periodically polls the VNIC to obtain data packets to be retrieved, while the VNIC generates an interrupt to enable the interrupt-based driver 1204 to retrieve the data packets.

[0128] The polling mode driver passes the data packet to the DPDK driver 1206, which then passes the data packet to the message processing module in the user space when the stream is initially received. On the other hand, the interrupt-based driver 1204 provides the data packet to the message processing module 1212 in the kernel or user space. The message processing module of the SVM then decapsulates the encapsulated data packet and performs the service operation of the SVM. In some embodiments, different SVMs perform different service operations based on the SCI, SI, and service direction values ​​they receive with the data packet.

[0129] Fig.13An exemplary process 1300 is illustrated that in some embodiments, the SVM performs each time when it receives a data message to be processed from a service agent. In other embodiments, the SVM can perform its operation using SCI, SI, and service direction values ​​differently. As shown, process 100 initially (at 1305) removes the encapsulation header and retrieves SCI, SI, direction, and activity parameters from the encapsulation header. Then, the process uses (at 1310) the mapping record it receives from its service manager to map SCI, SI, and direction values ​​to a service profile, then (at 1315) maps the service profile to one of its rule sets, and then checks to identify (at 1320) one or more service rules for processing.

[0130] Fig.14 A first mapping table 1405 of the SVM is shown. As shown, each record in the table maps the SCI, SI, and direction values ​​to a service profile. The figure also shows a second mapping table 1410 of the SVM, and the table maps the service profile to a rule set identifier that identifies several rules in the service rule table 1415. Fig.14 As shown, the service insertion manager in some embodiments provides records of the first table 1405 (e.g., the SI network manager provides these records to the service manager of the SVM, and then the service manager provides these records to the SVM), and the service manager of the SVM provides these records for the second table 1410 and the third table 1415. In some embodiments, the two service managers are two different management planes managed by two different entities (e.g., a data center administrator and a third-party administrator, or a tenant administrator and a data center administrator).

[0131] In some embodiments, each service rule 1420 in the service rule table 145 has a rule identifier defined with respect to a data packet attribute (e.g., a five-tuple attribute). The SVM compares (at 1320) the rule identifier with the attribute of the data packet to identify a matching rule. Upon identifying one or more matching rules, the SVM in some embodiments performs (at 1325) the action specified by the highest priority matching rule. For example, the firewall SVM may specify that the data packet should be allowed to pass, should be dropped, and / or should be redirected.

[0132] Once the SVM completes its service operation, assuming the service operation does not result in the discarding of the data packet, the SVM encapsulates (at 1330) the data packet with an encapsulation header. The encapsulation header has the same format as the data packet received by the SVM (e.g., an NSH header, a GRE header, or a QinQ header). In the encapsulation header, the SVM in some embodiments sets (1) an activity value to respond to the activity value of the service agent and (2) an appropriate sequence number for the activity value (e.g., an unadjusted or incremented sequence number).

[0133] In some embodiments, some service nodes are configured to decrement the SI value they receive, while other service nodes are not configured to decrement the SI value. If the service node is configured to decrement the SI value, the service node inserts the decremented SI value into the encapsulation header after decrementing the SI value at 1330. The SVM in some embodiments also sets the SMD attributes (SCI, SI, and service direction) in the encapsulation header, while in other embodiments, the service agent in the egress path retrieves these values ​​from an earlier record created by the service agent before passing the data packet to the SVM.

[0134] In some embodiments, the SVM may also set (one or more) flow programming attributes in the encapsulation header to guide the service agent to modify the service processing of the flow of the data message. The flow programming will be further described below. After encapsulating the data message, the SVM forwards the data message along its egress path. Fig.15 An example is illustrated in which the SVM 106 returns an encapsulated data packet 1502 having SMD and liveness attributes in its encapsulation header 1504 .

[0135] Fig.16 The process 1600 that the service agent 614 performs each time it receives a data packet traversing along the egress path of its service node is illustrated in some embodiments. As shown, the service agent in some embodiments initially removes (at 1605) the encapsulation header from the data packet, removes the SMD attributes from the header, and stores these attributes in the attribute set it creates for the data packet. In some embodiments, the service agent retrieves (at 1605) some or all of the SMD attributes of the data packet (e.g., the service plane MAC address of the source GVM, the SPI value) from a previous record created by the service agent before passing the data packet along the ingress path to the service node. Fig.15 An example of a set of attributes 1506 created by the service agent 614 for a decapsulated datagram 1507 is illustrated.

[0136] Next, at 1610, the process resets the activity timer it maintains (e.g., a timer that expires every 0.25 seconds) to note the activity value it received from the service node, which indicates that the node is still usable. Using this activity value, the service agent receives a sequence number from the service node, which the process verifies to ensure that it is the next activity value that needs to be received.

[0137] At 1615, the process determines whether the SVM specifies any stream programming attribute(s) that require the service agent to direct the SI post-processor 612 of the source GVM to perform stream programming by sending an in-band datagram to the post-processor 612. In some embodiments, the service agent sends the in-band stream programming control signal with another datagram generated by the service agent to be sent back to the source GVM and to be intercepted by the post-processor 612 of the source GVM.

[0138] When the source GVM receives a datagram with a stream programming control signal, its postprocessor can uniquely identify the datagram stream to which it is applied by using a stream identifier unique to the stream. As further described below, the stream identifier is derived based in part on the unique identifier of the source GVM. The unique stream identifier also allows other service plane modules (e.g., service nodes, service agents, and STL modules) to uniquely identify each datagram stream. In some embodiments, the unique stream identifier is a part of the SMD attribute that is passed between service jumps of the service path and passed back to the source GVM.

[0139] However, in some embodiments, the service agent sends an in-band flow programming control signal along with the current data packet it is processing. In some of these embodiments, the service agent does this only when its associated service node is the last hop service node of the service path, while in other embodiments, the service agent does this even when its service node is not the last hop service node. When the service node of the service agent is not the last hop service node of the service path, the service agent embeds the flow programming in the SMD attribute of the data packet, which in some embodiments is ultimately forwarded to the SI post-processor of the source GVM as part of the data packet encapsulation header when performing the last hop service. Even in this case, the last hop service agent in other embodiments sends the flow programming signal as a separate message.

[0140] The following will refer to Fig. 20Flow programming signals are further described. In addition, as further described below, when the service agent detects that its service node has failed, the service agent also sends a flow programming signal back to the source GVM so that the classifier at the source GVM can select another service path for the current data packet flow and other data packet flows. In this case, the service agent will also notify the LCP on its host computer, so that the LCP can notify the CCP, and the CCP can in turn modify the service path specified for the service chain that uses the failed service node.

[0141] At 1620, process 1600 determines whether its service node specifies that the data packet should be discarded. If so, the process discards the data packet and then ends. Otherwise, assuming that the data packet should not be discarded and should continue along its service path, in some embodiments, the service agent decrements (at 1625) the SI value if the service node has not already decremented the SI value, and then uses (at 1630) the decremented value together with the SPI value in the stored attribute set of the data packet to identify a fully matched forwarding rule for identifying the next-hop network address. When the agent's service node is on multiple service paths, the agent's forwarding rule storage device stores multiple fully matched forwarding rules, which can specify different next-hop network addresses for different SPI / SI values.

[0142] When the decremented SI value is zero, in some embodiments, the service agent matches the decremented SI value and the embedded SPI value to a rule that directs the service agent to identify the next hop as the service plane MAC address of the source GVM. In some embodiments, the rule does not provide a MAC address, but rather references the service plane MAC address as part of the SMD attribute set stored for the data packet. In some embodiments, the instruction to return the data packet to the service plane MAC address of the source GVM when the SI value is zero is not specified by a forwarding entry of the forwarding table, but is hard-coded into the logic of the service.

[0143] At 1630, the service agent stores the next hop network address (eg, MAC address) in a set of attributes stored for the data packet. Fig.15 An example of service proxy 614 is illustrated, storing the next hop MAC address associated with the next service node in attribute set 1506 of decapsulated data packet 1507. After identifying the next hop network address, the service proxy returns (at 1635) the data packet to the egress path of its service node, and process 1600 ends.

[0144] Once the service agent returns the datagram to the egress path of the service node, the STL module 626 receives the datagram and starts Fig.10The STL module 626 performs the first three operations 1005-1015 of the process each time it receives a data packet from the service insertion layer. Specifically, the STL module replaces the original source MAC address and destination MAC address in the data packet with the service plane MAC address of the current service hop and the next service hop (i.e., Fig.15 The datagram is formatted (at 1005) using the hop1mac address and the hop2mac address in the illustrated example) to be forwarded to the next hop service node.

[0145] At 1010, the STL module also adds the original source MAC address and destination MAC address of the data packet to the set of attributes of the data packet that should be processed by other service transport layer modules (eg, vswitch, encapsulation processor, etc.) on the same host computer. Fig.15 1508 and the enhanced attributed set 1510 are depicted in FIG. After reformatting the data packet and enhancing its attribute set, the STL module 626 passes the formatted data packet along an egress path (at 1015) through which the data packet reaches the software switch 120.

[0146] Based on the destination MAC address of the formatted datagram (i.e., the next hop MAC address), the software switch determines (at 1020) that the port of the next hop is not local. Therefore, the software switch provides (at 1035) the datagram to uplink port 1550 connected to VTEP1, which communicates with VTEP2 on host 114 through an overlay network tunnel, as shown in FIG. Fig.15 As shown in the example of . As shown, the STL encapsulation processor 628 receives the data message along the egress path of the uplink port (at 1040) (e.g., referred to as one of the hooks specified for the uplink port), defines an encapsulation overlay header 1540 for the data message, and encapsulates the data message with the overlay header.

[0147] In some embodiments, the overlay header is a Geneve header that stores SMD and STL attributes in one or more of its TLVs. As described above, the SMD attributes in some embodiments include an SCI value, an SPI value, an SI value, and a service direction. In addition, in some embodiments, the STL attributes include an original L2 source MAC address and an original L2 destination MAC address. Fig.15 The following figure shows an example of this encapsulation header. Fig.28 It is further described.

[0148] When the encapsulated data packet is received at the host computer 114 at the next hop, the data packet is captured by the STL encapsulation processor 628 (e.g., defined as a hook for the downlink port 1552) connected to the downlink port 1552 of the VTEP, which is connected to the VTEP of the previous hop through the overlay network tunnel. Fig.17 The process 1700 is illustrated as being initiated by the encapsulation processor 628 on the next-hop computer that receives an encapsulated data packet that needs to be processed by the SVM executing on its computer.

[0149] As shown, the encapsulation processor removes the encapsulation header from the data message (at 1705) and stores the STL and SMD attributes as a set of associated attributes of the data message (at 1705). The encapsulation processor then passes the decapsulated message to the downlink port (at 1710), which then passes it to the software switch for forwarding (at 1715) to its port connected to the next-hop SVM (i.e., the port associated with the destination MAC address). The port then passes the data message 1508 and the attribute set 1510 to the ingress path of the next-hop SVM, as shown. Fig.15 An example of SVM 108 is shown.

[0150] The STL module 626 on this ingress path then reformats the data packet by replacing the previous hop and current hop service plane MAC addresses (i.e., hop1mac and hop2mac) with the original source MAC address and destination MAC address of the data packet retrieved from the data packet attribute set (at 1720). Upon retrieving the original SMAC and DMAC addresses, the STL module 626 modifies the attribute set of the data packet. Fig.15 The reformatted datagram 1530 and the modified attributed set 1532 are depicted in FIG. The STL module then passes this reformatted datagram and its accompanying SMD attributes along the ingress path of the SVM, whereupon the datagram is processed by the ingress service agent 614 of that hop.

[0151] The operation of the service agent is as mentioned above. Fig. 9 and Fig.11 described. Fig.15A service agent of SVM 108 on host 114 is shown passing an encapsulated data packet to the SVM. The encapsulation header of the data packet is supported by SVM 108 and stores SCI, SI, service direction, and activity values. In some embodiments, SVMs that are part of the same service path support different encapsulation headers. In some of these embodiments, service agents along the service path may encapsulate the data packet with different encapsulation headers before passing the data packet to its associated SVM. For example, in one case, a first-hop service agent passes a data packet with an NSH encapsulation header to SVM 106, while a second-hop service agent passes a data packet with a QinQ encapsulation header to SVM 108.

[0152] Once SVM 108 performs its service operation on the data packet (e.g., Fig.13 1300), such as Fig.18 As shown, the SVM sends the processed data message along its egress data path. As shown, the service agent then identifies the MAC address of the next service hop and adds this MAC address to the stored attribute set for the data message. At this point, the next hop is the third service hop, which corresponds to SVM 110. The agent identifies the MAC by decrementing the SI value (when SVM108 does not decrement the SI value), and then uses the embedded SPI value and the decremented SI value to find the forwarding rule that provides the next hop MAC address. The STL module in the egress path then replaces the original SMAC and DMAC in the data message with the current hop and next hop MAC addresses (i.e., Fig.18 hop2mac and hop3mac in the example), store the original SMAC and DMAC in the stored attribute set of the data message, and then pass the data message along the egress path, through which the data message is received by the software switch 122.

[0153] The software switch then determines that the next hop is associated with its uplink port 1552, and therefore passes the datagram to that port. Fig.18 As shown, the encapsulation processor 628 on the egress path of the port (e.g., designated as a hook on the egress path) then encapsulates the data packet with a Geneve header that stores the SMD and STL attributes in one or more TLVs and specifies that the data packet passes from VTEP2 associated with this port to VTEP3 associated with port 1554 of host 116.

[0154] The STL encapsulation processor 628 in the ingress path of port 1554 then removes the encapsulation header from the data packet and stores the STL and SMD attributes as the associated attribute set of the data packet. The STL encapsulation processor 628 then passes the decapsulated packet to port 1554, which then passes the decapsulated packet to the software switch 124 for forwarding to its port connected to the next-hop SVM 110 (i.e., to its port associated with the service plane DMAC). This port then passes the data packet and the attribute set to the ingress path of this SVM, as shown in FIG. Fig.18 shown.

[0155] The STL module 626 in the ingress path replaces the previous and current hop service plane MAC addresses (i.e., hop2mac and hop3mac) with the original source MAC address and destination MAC address of the data message retrieved from the data message attribute set. The STL module 626 also modifies the attribute set of the data message by removing the original SMAC and DMAC addresses, and then passes the reformatted data message and its accompanying SMD attributes along the ingress path of the SVM to be processed by the ingress service agent 614 of the hop. The service agent passes the encapsulated data message with an encapsulation header supported by the SVM 110 and storing the SCI, SI, service direction, and activity values ​​to the SVM 110.

[0156] Once SVM 110 performs its service operation on the data packet (e.g., Fig.13 1300), such as Fig.19 As shown, the SVM sends the processed data message along its egress data path. The service agent decrements the SI value (assuming that SVM 110 has not already done so). In this example, the decremented SI value is now zero. In some embodiments, the service agent then matches the SI value and the SPI value with the rule identifier of the forwarding rule, which specifies that it should select the service plane MAC (spmac) of the source GVM as the next-hop MAC address. In other embodiments, the service agent's hard-coded logic guides it to identify the service plane MAC of the source GVM as the next-hop MAC. In either case, the service agent adds the service plane MAC of the source GVM to the attribute set of the data message.

[0157] The STL module then replaces the original SMAC and DMAC in the datagram with the third hop MAC address and the service plane MAC of the source GVM, stores the original SMAC and DMAC in the stored attribute set of the datagram, and then passes the datagram to its software switch 124. The software switch then determines that the next hop is associated with its port 1554, so it passes the datagram to that port. Fig.19As shown, the encapsulation processor 628 on the egress path of the port then encapsulates the data packet with a Geneve header that stores the SMD and STL attributes in one or more TLVs and specifies that the data packet is to pass from the associated VTEP3 of the port to the VTEP1 associated with the port 1550 of the host 112.

[0158] The STL encapsulation processor 628 in the ingress path of port 1550 then removes the encapsulation header from the data packet and stores the STL and SMD attributes as an associated set of attributes of the data packet. The encapsulation processor then passes the decapsulated packet to port 1550, which then passes the packet to the software switch 120 to forward to its port connected to port proxy 620. The port then passes the data packet and the attribute set to port proxy 620, as shown in FIG. Fig.19 shown.

[0159] Port agent 620 then replaces the previous and current hop service plane MAC address (i.e., hop3mac and spmac) with the original source MAC address and destination MAC address of the data message retrieved from the data message attribute set. Port agent 620 also modifies the attribute set of the data message to remove the original SMAC and DMAC, and then passes the reformatted data message together with its accompanying SMD attribute back to the STL caller 624 that first calls it. In some embodiments, the port agent uses the connection record that it creates when the STL caller initially calls it to identify the STL caller for callback. In other embodiments, the port agent uses a mapping table that maps each service plane MAC to the STL caller of the GVM. In some embodiments, the mapping table has a record that associates the service plane MAC and service direction with the guest forwarding plane port identifier associated with the GVM.

[0160] Once called, the STL caller passes the datagram along the egress path of the GVM 102 , where it will then be forwarded to the SI post-processor 612 . Fig. 20The process 2000 performed by the SI post-processor 612 in some embodiments is illustrated. The post-processor performs the process 2000 each time it receives a data message passed to it along the IO chain of the GYM. As shown in the figure, in some embodiments, the post-processor 612 initially determines (at 2005) whether it needs to check the received data message for SI post-processing. This is because as a module along the IO chain of the GVM, the post-processor will be called for all data message streams passed along this IO chain, and some of these data messages may not match the SI rules that require service insertion operations to be performed on these data messages. In some embodiments, the process 2000 determines (at 2005) whether it needs to process the data message by determining whether the data message has associated service metadata. If not, the process transitions to 2020, which will be described below.

[0161] When the SI post-processor 612 determines that a datagram needs to be processed, the process determines (at 2010) whether the SMD metadata associated with the datagram specifies a stream programming tag that requires the post-processor to perform stream programming. In some embodiments, such a stream programming tag will be specified by the service node in the SMD attribute of the datagram to change the service path processing at the source GVM, or by the service agent for the same reason when a failure of its service node is detected. When the stream programming tag does not specify any stream programming, the process transitions to 2020, which will be described below.

[0162] Otherwise, when the flow programming tag specifies a flow programming operation, process 2000 performs the operation and then transitions to 2020. In some embodiments, the flow programming operation requires modifying the connection record in connection tracking storage 804 to specify the desired operation and / or SMD attributes (e.g., allow, drop, etc.) for the flow of data packets. The post-processor writes to connection tracker 804 in Fig.19 As described above and further described below, the SMD metadata for the processed data packets includes a flow identifier that uniquely identifies the flow of the data packets by being derived at least in part from the unique service plane identifier of the source GVM. In some embodiments, the post-processor 612 uses the flow identifier to match the flow of the data packets in the connection tracker.

[0163] In some embodiments, the flow programming label can specify the following actions: (1) NONE: when no action is required (which results in no flow programming operation being performed), (2) DROP: when the data packets of the flow should not be forwarded further along the service chain but should be discarded at the source GVM, (3) ACCEPT: when the data packets of the flow should not be forwarded further along the service chain but should be accepted at the source GVM. In some embodiments, the flow programming label can also specify DROP_MESSAGE. DROP_MESSAGE is used when the service node needs to communicate with the agent (for example, to respond to a ping request) and wants to discard user data packets (if any), even if flow programming at the source is not required.

[0164] In some embodiments, additional actions may be used for the service agent to internally communicate the failure of its SVM. In some embodiments, the action will direct the SI post-processor to select another service path (e.g., another SPI) for the flow of data packets. By setting appropriate metadata fields in some embodiments, the action is carried in-band with the user data packet in some embodiments. For example, as further described below, the service agent communicates with the post-processor of the source GVM via the OAM (operation, administration, and maintenance) metadata of the NSH attributes via in-band data packet traffic on the data plane. Given that design flow programming actions are subject to signaling delays and are prone to loss, the SVM or service agent may still see data packets belonging to a flow that is expected to be dropped, accepted, or redirected at the source for some time after the flow programming action is communicated to the agent. In this case, the service plane should continue to set actions to drop, allow, or redirect at the source.

[0165] After completing the flow programming operation, the process 2000 transitions to 2020. When the process determines (at 2005) that SI post-processing does not need to be performed on the data packet or determines that flow programming does not need to be performed on the data packet, the process also transitions to 2020. At 2020, the process 2000 allows the data packet to pass through the egress path of the GVM 102 and then ends.

[0166] References Figure 8 , Fig. 9 , Fig.15 , Fig.18 and Fig.19 The described example shows service plane operations performed on data packets identified along the egress path of the source GVM. These service plane operations (by reference to Figure 7 , Figure 10-Figure 14 , Fig.16 , Fig.17 and Fig. 20The SI pre-processor 610 and SI post-processor 612 on the ingress path are opposite to those on the egress path in order to perform these ingress side operations. Specifically, Figure 6 As shown, the pre-processor 610 receives a data packet entering the ingress path of the GVM from a software switch port associated with the VNIC of the GVM, and the post-processor 612 transmits the processed data packet along the ingress IO chain to the VNIC of the GVM.

[0167] However, the service insertion operations and service transfer operations for ingress side processing are similar to the egress side processing of data packets to and from a particular GVM. In some cases, this GVM exchanges data packets with another GVM. Figure 4 and Figure 5 As described, the service plane may be directed to perform the same service chain on the data packets in each direction but in reverse order. In this case, the service node of the service path on the ingress side performs a series of service operations on the data packets sent by another GVM to a specific GVM in the first direction of the service chain, while the service node of the service path on the egress side performs the same series of service operations in the opposite second direction through the service chain. In addition, as described above, in some embodiments, the two sets of service nodes for the forward direction and the reverse direction include the same service nodes.

[0168] Now refer to Fig.21 , Fig. 22 and Figure 25-28 To describe the header format used in some embodiments. Fig.21 The NSH header 2100 is illustrated, and some service agents in some embodiments use the NSH header 2100 to encapsulate the data message before providing it to the service node associated with it. In some of these embodiments, the service node returns the processed data message encapsulated with such an NSH header. In some embodiments, the NSH header is also used by the service plane module of the host computer to forward the double-encapsulated data message to other host computers, wherein the first encapsulation header is the NSH header and the second encapsulation header is the service transmission header. However, in other embodiments, the service insertion attribute and the service transmission attribute are placed in an encapsulation header, as described below. In addition, as described above and further described below, the service agents and service nodes in some embodiments do not use the NSH header to encapsulate the data messages they exchange.

[0169] As shown, all fields of the first 8 bytes of the NSH header are used in accordance with RFC 8300. In some embodiments, the header includes a fixed-length metadata (MD) content header 2110. In some embodiments, the header also includes (1) an MD type, which is set to 1, (2) a next protocol value, which is 3 to indicate Ethernet communication, and (3) a length value, which is 6 because the MD content header 2110 has a fixed length. In addition, in some embodiments, the SPI field 2122 and the SI field 2124 are populated with the service path identifier of the selected path and the current service index value, which is the initial SI value (i.e., the initial service hop count) when defined by the pre-processor 610 of the source GVM.

[0170] In some embodiments, the service insertion module does not store or cache metadata except in the NSH header carried with the data message. In this model, the service node retains metadata fields that it does not intend to change. In some embodiments, certain metadata fields are used as a communication mechanism for data plane mediated signaling between the service agent / node and the service module of the source GVM. In some embodiments, the data message metadata is encoded on the line in the NSH fixed length context header 2110. In some embodiments, the fixed size header provides a total of sixteen bytes of information. In some embodiments, each service insertion deployment can freely define its own MD content format.

[0171] Fig. 22 The diagram shows an example of metadata content stored in an MD content header 2110 in some embodiments to send service metadata to consecutive service jumps, to a service node, and / or to a service agent. As shown, the header has sixteen bytes including many fields. A field 2202 includes an F bit, which is used to distinguish the content type in the MD content header, such as service metadata content, stream programming content, etc. In some embodiments, the F bit of the service metadata content is b00. Another field 2204 stores a P bit, which can be set to 1 to force a service node to respond to a data message. In some embodiments, the response must carry an NSH header containing a sequence number identical to the sequence number of the request, wherein the P bit is also set to 1.

[0172] The source node identifier (ID) field 2206 explicitly identifies the data computing node (e.g., GVM) as the source or destination of the data message to the service plane. In some embodiments, the source node ID includes the service plane MAC address of the source data computing node (DCN) for which the data message is inserted into the service plane. The MD content header also includes a sequence number 2208, which is an opaque 6-bit value used to identify the data message for activity detection. This value is usually zero unless the service agent fills it in before forwarding the data message to its service node as part of its activity detection.

[0173] The MD content header also includes a tenant ID 2212, which uniquely identifies the tenant for a multi-tenant data center. In some embodiments, the tenant ID is a VNI associated with the tenant. The MD content header 2200 also includes a flow ID 2215 and a flow ID validity bit 2222. In some embodiments, when the rest of the flow ID (also called a flow label) is present, the flow ID validity bit is set to 1. The flow ID 2215 is a unique identifier for each flow and source DCN (i.e., each flow and source node ID 2206). In some embodiments, the flow ID is set by a classifier of the source DCN (e.g., an SI preprocessor 610 that performs a classification operation).

[0174] In some embodiments, the flow ID may be discarded when a data packet traverses a service that is not in native mode (i.e., the service is not aware of the service plane). In this case, the flow ID will be discarded when there are not enough bits in the compatibility mode header to carry the flow ID, as described below. The flow ID may also be discarded when the native service (i.e., the service plane is aware of the service node) modifies the data packet in a way that makes the flow ID meaningless, such as when the service encrypts traffic from multiple flows entering a single IPsec tunnel. In this case, it would be meaningless to retain the flow label of the inner data packet. In some embodiments, in this case, the service node sets the A bit to zero.

[0175] The MD content header 2200 also includes an action field 2230 for flow programming of the service agent. In some embodiments, the action specifies the action that the post-processor 612 of the source DCN should perform on the flow. For flow programming, the action field must be non-zero in some embodiments. In addition, for flow programming, the F bit 2202 is also set to 10 and the P bit 2204 is set to 0 at the agent and ignored by the classifier, and the flow validity bit 2222 and the flow label 2215 must be valid.

[0176] The following is an exemplary set of values ​​for the action field 2230, but a person of ordinary skill will recognize that other values ​​are specified in other embodiments. A value of 0 for the action bit indicates that no flow programming action is specified. A value of 1 indicates that all packets of the flow of the data packet should be discarded at the source, and further data packets of the flow should not be forwarded to the service plane. Instead, the data packet should be discarded at the source after classification.

[0177] In some embodiments, a value of 2 in the action field indicates that the datagram should be accepted at the source and other datagrams of the same flow should not be forwarded to the service function. Instead, the service function should be skipped and the next service in the chain should be called directly. A value of 3 in the action field indicates that only this datagram should be discarded, without indicating the action to be taken for other datagrams of the same flow. In some embodiments, this action is used when a service node communicates with a service agent (e.g., in response to a ping request) and wants to discard a datagram, even if flow programming should not occur.

[0178] The MD content header 2200 also includes a direction field 2214, which indicates the direction of the data packet from the source DCN to the network perspective (for example, from the DCN to the network is the egress direction, and from the network to the DCN is the ingress direction). When the direction is not unknown, a value of 0 in the direction field indicates no direction or an unknown direction. In some embodiments, a value of 1 indicates that the data packet is moving in the ingress direction (i.e., the data packet is being processed for the source DCN as the destination of the data packet), for example, the data packet is on the way from the VTEP to its corresponding DCN. In some embodiments, a value of 2 indicates an egress direction (for example, the data packet is being processed for the source DCN as the source of the data packet).

[0179] In some embodiments, a value of 3 indicates that the datagram is only in transit and applies to both ingress and egress. When used to define a rule, this means that the rule should match datagrams in one direction or in some embodiments in any direction. From a service perspective, in some embodiments, a value of 3 in the direction field indicates that the traffic is forwarded to the service plane by a transit device that is neither the source of the traffic nor the sink of the traffic. In some embodiments, the transit indication is used for traffic that is transited by a router.

[0180] The MD content header 2200 also includes a service chain ID 2216, which specifies the service chain along which the data message should flow. Some embodiments do not embed the SCI in the NSH header, but only store the SPI value. However, other embodiments store the SCI in field 2216 because many SPIs can correspond to the same service chain and the SPI is not continuous. In other words, some embodiments embed the service chain ID because the SCI provides a more stable identifier for the service node to identify the service rules that match the data messages it processes.

[0181] In some embodiments, other metadata content formats are used internally by the service plane and are not exposed to the service node in order to perform data plane signaling between the service post-processor and the service agent of the source DCN. In some of these embodiments, when other metadata content formats are used, the OAM bit ( Fig.21 The 0 bit 2170 in the NSH is set and does not carry user payload (or, if required by the NSH, this bit is ignored at the destination). In some embodiments, in this case the NSH Next Protocol field is set to 0.

[0182] In some embodiments, the service plane is unaware that the service node receives only a subset of the metadata, depending on the type of non-NSH header that the service agent uses to communicate with the service node. As described above, when the service node cannot process the NSH header, the service node in some embodiments can receive service metadata in a GRE header or a QinQ header. The GRE header and the QinQ header are referred to as compatibility mode headers below because the GRE header and the QinQ header are headers supported by some existing service nodes. This compatibility mode encapsulation header is required in some embodiments to distinguish data packet flows subject to different service processing and to isolate flows with conflicting L3 addresses (in the case where a single service node performs services on data packets of multiple networks (such as multiple tenant networks)).

[0183] In some embodiments, a service node in GRE compatibility mode is connected to its service agent via two VNICs and is configured to be in bump-in-the-wire mode. In addition, in some embodiments, the VNIC is a vmxnet3 device whose MAC address remains unchanged and the MTU size for it is set to a fixed size (e.g., 2048 bytes). One VNIC of the service node is defined as an unprotected side for receiving egress side traffic of a source DCN and providing ingress side traffic of the source DCN, while the other VNIC is defined as a protected side for receiving ingress side traffic of the source DCN and providing egress side traffic of the source DCN. In some embodiments, this information is communicated to the service manager or service node via OVF (Open Virtual Format) parameters, where OVF is a file format that supports the exchange of virtual devices across products and platforms.

[0184] Even if two VNICs are present to support wire plug-in mode, some embodiments use only one service proxy instance for each pair of compatibility mode VNICs and only one endpoint on the service plane to refer to a pair of interfaces. Fig.23 and Fig.24 The diagram shows an example of service agent 2305 forwarding data packets using encapsulated GRE headers to the SVM 2310 egress and ingress sides of GVM 2315. To this end, the service agent creates several virtual GRE tunnel endpoints 2320 for the protected and unprotected VNICs of SVM 2310.

[0185] Each protected virtual tunnel endpoint has a corresponding unprotected virtual tunnel endpoint. Each virtual tunnel endpoint is associated with a virtual IP address, a virtual MAC address, and a GRE parameter. The service agent encapsulates the data message with a GRE header to pass between the corresponding endpoint pairs through a service node, wherein the node operates in a line plug-in mode that does not modify the GRE header. As further described below, the service agent embeds service metadata into the GRE header to provide the service node with the service metadata required for processing the data message. In addition, in some embodiments, different tunnel endpoint pairs are used for different streams.

[0186] In some embodiments, the service insertion platform supports the GRE encapsulation defined in RFC 2784 and the key extension defined in RFC 2890. In some embodiments, the GRE tunnel uses an IPv4 address and the GRE protocol type is set to transparent Ethernet bridging according to RFC 1701. In GRE compatibility mode, the service insertion layer (e.g., service agent) generates a tuple (e.g., source IP, destination IP, GRE key) for each flow. In some embodiments, the process is deterministic and based on the content of the SMD header, which can then be stripped and replaced with an IP and GRE stack. In some embodiments, the IP address generated by the process is virtual and is not configured on any network entity other than the service agent and its associated SVM, and as a result, the scope of these IP addresses is limited to the local link between the service agent and its service node.

[0187] Generate an IP address pair and a GRE key so that metadata is carried along with the datagram even when the service node does not support GRE. In some embodiments, both the service node and the service agent consume the metadata. In addition, in some embodiments, the service node is expected to retain the outer header as is without modification. In some embodiments, each flow is consistently encapsulated in the same GRE tunnel and there is no IP address conflict within the tunnel. In addition, datagrams that differ only in their direction (entrance relative to exit) are encapsulated using the same GRE key and swapping the source IP and destination IP, and pass through the GRE tunnel endpoint in the correct direction (protected end to unprotected end, or unprotected end to protected end).

[0188] In some embodiments, the IP source / destination address and GRE key may be checked by the service node as needed to perform appropriate data packet processing. Fig.25 and Fig.26 Describes how to encode service metadata in the GRE encapsulation header in place of the source and destination IP addresses and the GRE key field. Fig.25 The GRE header format for storing service data for the egress direction (e.g., from the GVM to the switch) in some embodiments is illustrated. Fig.26 The GRE header format that stores service data for the ingress direction (eg, from a software switch to a source GVM) in some embodiments is illustrated.

[0189] In these figures, all fields are in network byte order. In some embodiments, the path ID is generated together with the service path and has a global value for each service. Fig.25 and Fig.26As shown, in some embodiments, the IP address fields are reversed for the egress and ingress side datagrams. As in native mode, the service plane in GRE compatibility mode can modify or generate any traffic as long as the traffic has a valid encapsulation when it reaches the service proxy. In some embodiments, this means reusing one of the IP and GRE stacks received by the service node for the relevant flow.

[0190] In some embodiments, when entering the first GRE compatibility mode service, flow label information along the service chain is discarded and not restored downstream. This can prevent subsequent services from being able to declare flow actions. Therefore, flow programming is not provided to the service node in the GRE compatibility mode of some embodiments. In addition, in some embodiments, activity detection is supported by passing BFD (Bidirectional Forwarding Detection) packets between trusted interfaces and untrusted interfaces. In some embodiments, these data packets are injected by the service agent from the trusted side and the untrusted side. The service node can identify this traffic because the traffic is not encapsulated in GRE. In some embodiments, the service node is expected to forward this traffic (actually any non-GRE encapsulated traffic) without modification by bridging the traffic to the other side of the virtual wire. In addition, in some embodiments, if an instance of BFD is not available, the data packets can be hard-coded.

[0191] Due to space limitations in some embodiments, certain header fields are encoded as aggregated versions. In some embodiments, the service chain tag, SPI, and SI are aggregated in a single 4-bit field. In some embodiments, each compatibility mode service node can therefore exist in up to 16 service chain hops. Each time a service exists within a service chain, a service path ID is consumed. If a service exists on multiple chains, multiple service path IDs are consumed. In addition, each time a service appears in both directions of a service chain, two service path IDs are consumed.

[0192] In some embodiments, locally generated traffic is supported in compatibility mode as long as the relevant outer header stack (up to and including GRE) is used. In some embodiments, no modifications to the outer header stack are allowed except that (1) the outer Ethernet destination address is optionally replaced with a broadcast, (2) the IP total size field and IP checksum are updated, and (3) the GRE checksum is ignored but the GRE key must be present.

[0193] Fig. 27 and Fig.28An example of an encapsulation header used in some embodiments to send a data packet from one VTEP associated with at least one service node (e.g., from a host computer) to another VTEP associated with another service node (e.g., to another host computer) is illustrated. Both examples are Geneve encapsulation headers and carry service metadata (e.g., SMD metadata) in one or more Geneve TLVs. The Geneve header supports logical L2 overlay transport and has a variable TLV space for carrying service metadata. Therefore, different service insertion platforms can specify different amounts of service metadata to be carried between consecutive hops.

[0194] Fig. 27 The diagram illustrates the use of two Geneve encapsulation headers, an outer Geneve header 2705 for carrying service transport layer data and an inner Geneve header 2710 for carrying service insertion layer metadata. As shown, the service metadata is stored in an SMD TLV 2715. In some embodiments, the TLV 2715 has Fig.21 Therefore, the TLV stores the service metadata in the fixed length header 2110 as described above, and stores the SPI and SI values ​​in the SPI field 2122 and SI field 2124 of the header 2100.

[0195] For efficiency, some embodiments combine these two headers into Fig.28 single Geneve header 2805. To this end, these embodiments replace the original source MAC address and destination MAC address of the data packet with the service plane MAC of the current hop and the next hop, and store the original source MAC and destination MAC together with the service direction of the source GVM, the service plane MAC and other SMD metadata (such as service chain identifier, SPI value, SI value, flow programming value, tenant label, flow label, etc.) in a new GeneveTLV. In some embodiments, this new Geneve TLV has a 24-byte SMD metadata field, and 12 bytes for storing STL data, such as the original source MAC address and destination MAC address. In some embodiments, the 12 bytes of STL data precede the 24 bytes of SMD metadata, and in some embodiments the 24 bytes of SMD metadata include Fig.21 and Fig. 22 Metadata shown in the figure.

[0196] As shown in the figure, Fig. 27 and Fig.28In both embodiments, the Geneve encapsulation header stores the SVNI of the service plane, which allows multiple service planes to be defined. For example, as described above, some embodiments use different SVNIs to define different service planes for different entities or tenants in a multi-entity or multi-tenant data center. Different service planes for different entities or tenants can be associated with the same or different QoS and / or SLA guarantees for the data packet type of the entity or tenant. Other embodiments use multiple SVNIs for different service planes for the same entity or tenant (for example, different service planes associated with different QoS and / or SLA guarantees for different data packet types of the same entity or tenant). In addition, both headers store the MAC addresses of the source and destination VTEPs and the UDP and IP source and destination addresses.

[0197] Fig.29 The object data model 2900 of some embodiments is illustrated. In this model, the object shown in the solid line is provided by the user, and the object shown in the dotted line is generated by the service plane manager and the controller. As shown in the figure, these objects include service manager 2902, service 2904, service profile 2906, supplier template 2907, service attachment 2908, service instance 2910, service deployment 2913, service instance runtime (SIR) 2912, instance endpoint 2914, instance runtime port 2916, service chain 2918, service insertion rule 2920, service path 2922 and service path jump 2924.

[0198] In some embodiments, a service manager object 2902 may be created before or after a service object 2904 is created. An administrator or service management system may call a service manager API to create a service manager. A service manager 2902 may be associated with a service at any point in time. In some embodiments, a service manager 2902 includes service manager information, such as a vendor name, a vendor identifier, a restUrl (for callbacks), and authentication / certificate information.

[0199] As described above, the service plane does not require the presence or use of a service manager, as the service nodes can operate in zero-awareness mode (i.e., have zero awareness of the service plane). In some embodiments, the zero-awareness mode allows only basic operations (e.g., redirecting traffic to the SVM of the service). In some such embodiments, integration is not provided to distribute object information (such as service chain information, service profiles, etc.) to the service manager servers. Instead, these servers can poll the network manager to find objects of interest.

[0200] The service object 2904 represents the type of service provided by the service node. The service object has a transport type attribute that specifies the mechanism (e.g., NSH, GRE, QinQ, etc.) of the service object for receiving service metadata. Each service object also has a status attribute returned by the service manager (which can be enabled or disabled), and a reference to the service manager, which can be used to expose a REST API endpoint to transmit events and perform API calls. The service object also includes a reference to the OVA / OVF attribute used to deploy the service instance.

[0201] The vendor template object 2907 includes one or more service profile objects 2906. In some embodiments, the service manager can register the vendor template, and the service profile can be defined on a per-service basis and based on the vendor template with possibly dedicated parameters. A service chain can be defined by reference to one or more service profiles. In some embodiments, the service profile is not assigned a label and is not explicitly identified on the line. In order to determine which function to apply to the traffic, the service node performs a lookup (e.g., based on the service chain identifier, service index, and service direction, as described above) to identify the applicable service profile. Whenever a service chain is created or modified, the mapping of this lookup is provided to the service manager by the management plane.

[0202] In some embodiments, the service profile object 2906 includes: (1) a vendor template attribute to identify its associated vendor template, (2) one or more custom attributes when the template exposes configurable values ​​through the service profile, and (3) action attributes, such as forwarding actions or copy and redirect, which direct the service agent to forward the received data packet to its service node, or forward a copy of the received data packet to its service node, while forwarding the received data packet to the next service hop, or when the service node of the service agent is the last hop, forward it back to the original source GVM.

[0203] The service attachment object 2908 represents the service plane (i.e., is a representation of the service plane from the perspective of a user, such as a network administrator of a tenant in a multi-tenant data center, or a network administrator in a private data center). The service attachment object is an abstract concept that supports any number of different implementations of the service plane (e.g., logical L2 overlay, logical L3 overlay, logical network overlay, etc.). In some embodiments, each endpoint (on a SIR or GVM) that communicates through the service plane specifies a service attachment. The service attachment is a communication domain. Therefore, services or GVMs outside of the service attachment may not be able to communicate with each other.

[0204] In some embodiments, service attachments can be used to create multiple service planes with hard isolation between them. A service attachment has the following attributes: (1) a logical identifier (e.g., the SVNI of a logical switch) that identifies the logical network or logical forwarding element that carries the service attachment traffic, (2) the type of the service attachment (e.g., L2 attachment, L3 attachment, etc.), and (3) an "applied_to" identifier that specifies the scope of the service attachment (e.g., transport node 0 and transport node 1 for north-south operations and a cluster or set of hosts for east-west operations). In some embodiments, a control plane (e.g., a central control plane) converts the service attachment representation received from the management plane to a specific LFE or logical network deployment based on parameters specified by a network administrator (e.g., a data center administrator for a private cloud or public cloud, or a network virtualization provider in a public cloud).

[0205] Service instance object 2910 represents the actual deployment instance of the service. Therefore, each such object is associated with a service object 2904 by specifying a service deployment object 2913 between the service object 2904 and the service instance object 2910. The deployed service instance can be an independent service node (e.g., an independent SVM), or can be a high availability (HA) service node cluster. In some embodiments, service deployment object 2913 describes the service instance type, such as independent or HA. As described below, in some embodiments, the API of the service deployment object can be used to deploy several service instances of a service.

[0206] A service instance runtime (SIR) object 2912 represents an actual runtime service node operating in a standalone mode, or an actual runtime service node of an HA cluster. In some embodiments, a service instance object includes the following attributes: (1) a deployment mode attribute that specifies whether the service instance is operating in a standalone mode, an active / standby mode, or an active / active model, (2) a state attribute that specifies whether the instance is enabled or disabled, and (3) a "deployed_to" attribute that includes a reference to a service attachment identifier in the case of north-south operation.

[0207] In some embodiments, SVM provisioning is initiated manually. To this end, in some embodiments, the management plane provides APIs for the following operations: (1) creating a service instance of an existing service, (2) deleting a service instance, (3) evolving a service instance that is already configured as a high availability cluster by adding additional SIRs, and (4) scaling down a service instance by removing one of its SIRs. When creating a service instance of an existing service, in some embodiments the service instance can be created based on a template contained in the service. The caller can choose between a standalone instance or an HA cluster, in which case all VMs in the HA cluster are provisioned. Similarly, in some embodiments, an API for service instance deployment allows multiple service instances (e.g., for an HA cluster) to be deployed with only a single API call.

[0208] In some embodiments, an API that creates one or more SVMs specifies one or more logical locations (e.g., clusters, hosts, resource pools) where the SVMs should be placed. In some embodiments, the management plane attempts to place SVMs belonging to the same service instance on different hosts whenever possible. Anti-affinity rules can also be appropriately configured to maintain the distribution of SVMs across migration events (such as VMotion events supported by VMware's dynamic resource scheduler). Similarly, the management plane can configure affinity rules with specific hosts (or host groups) when available, or a user who provisions a service instance can explicitly select a host or cluster.

[0209] As described above, the service instance runtime object 2912 represents the actual SVM that runs on the host to implement the service. The SIR is part of the service instance. Each SIR can have one or more traffic interfaces that are entirely dedicated to service plane traffic. In some embodiments, each SIR runs at least one service proxy instance to handle the data plane signaling and data packet format conversion of the SIR as needed. In some embodiments, when a service instance is deployed, a SIR is created for each SVM associated with the service instance. The network manager also creates instance endpoints for each service instance in the east-west service insertion. In some embodiments, each SIR object 2912 has the following attributes: (1) a state attribute, which is active for SVMs that can handle traffic and inactive for all other SVMs, regardless of the reason, and (2) a runtime state, which specifies whether the data plane activity detection detects that the SIR is rising or falling.

[0210] The instance runtime interface 2916 is a per-endpoint version of the service instance endpoint 2914. In some embodiments, the instance runtime interface 2916 is used to identify an interface that can be a SIR or GVM for source or sink service plane traffic. In east-west service insertion, in some embodiments, the lifecycle of the instance runtime interface is tied to the lifecycle of the service instance runtime. In some embodiments, no user action is required to configure the instance runtime interface.

[0211] In some embodiments, the instance runtime interface 2916 has the following attributes: an endpoint identifier, a type, a reference to a service attachment, and a location. The endpoint identifier is the data plane identifier of the SIR VNIC. The endpoint identifier is generated when the SIR or GVM registers with the service transport layer and can be a MAC address or a portion of a MAC address. The type attribute can be shared or dedicated. The SIR VNIC is dedicated, which means that only service plane traffic can reach the SIR VNIC; while the GVM VNIC is shared, which means that the GVM VNIC will receive and transmit both service plane and regular traffic. The service attachment reference is a reference to a service attachment that implements a service plane for transmitting and receiving service plane traffic. In some embodiments, the reference is to the SVNI of the service plane. The location attribute in some embodiments specifies the location of the instance runtime interface, which is the UUID of the host where the instance runtime interface is currently located.

[0212] In some embodiments, the user defines service chain objects 2918 based on an ordered list of service profiles 2906. In some embodiments, each service chain conceptually provides separate paths for forward and reverse service directions, but if only one direction is provided at creation time, the other direction is automatically generated by reversing the service profile order. Either direction (or even both) of a service chain can be empty, meaning that no service will handle traffic in that direction. In some embodiments, the data plane will perform a lookup even for an empty service chain.

[0213] A service chain is an abstract concept. A service chain does not point to a specific set of service nodes. Instead, a network controller that is part of the service plane platform automatically generates a service path pointing to a sequence of service nodes of the service chain and directs packets / flows along the generated service path. In some embodiments, a service chain is identified in the management plane or control plane by its UUID (unique identifier of the service chain). A service node is provided with the meaning of the service chain ID through the management plane API, which is received through its service manager. Fig.14 One such example is described.

[0214] In some embodiments, service chain labels can be used to identify service chains in the data plane because UUIDs are too long to be carried in the encapsulation header. In some embodiments, the service chain ID is an unsigned integer similar to the rule ID. Each data packet redirected to the service carries the service chain label of the service chain that the data packet is traversing. When creating or modifying a service chain, the management plane notifies the UUID to the service chain label mapping. There is a 1-to-1 mapping relationship between the service chain label and the service chain UUID, and a single service chain can have 0 to multiple service path indexes.

[0215] In addition to the service chain ID, the service chain in some embodiments has the following attributes: (1) references to all calculated service paths, (2) failure policies, and (3) references to service profiles. References to calculated service paths are described above. When the service path selected for the service chain cannot be traversed, the failure policy is applied. In some embodiments, the failure policy can be PASS (forward traffic) and FAIL (discard traffic). The reference to the service profile of the service chain can include an egress list of service profiles that egress traffic (e.g., data packets traveling from a GVM to a switch) must traverse, and an ingress list of service profiles that ingress traffic (e.g., data packets traveling from a switch to a GVM) must traverse. In some embodiments, the ingress list is initialized by default to the inverse of the egress list.

[0216] Different techniques may be used in some embodiments to define the service paths of a service chain. For example, in some embodiments, a service chain may have an associated load balancing policy, which may be one of the following policies. The load balancing policy is responsible for load balancing traffic across different service paths of the service chain. Under the ANY policy, the service framework is free to redirect traffic to any service path regardless of any load balancing considerations or flow locking. Another policy is the LOCAL policy, which specifies that a local service instance (e.g., an SVM executing on the same host computer as the source GVM) is preferred over a remote service instance (e.g., an SVM executing on another host computer or an external service device).

[0217] Some embodiments generate scores for service paths based on how many SIRs are local, and select the highest score regardless of load. Another strategy is a cluster strategy, which specifies that service instances implemented by VMs co-located on the same host are preferred, whether that host is local or a different host. The round-robin strategy directs hitting all active service paths with equal probability or based on a probability specified by a set of weight values.

[0218] SI rule object 2920 associates a set of data packet attributes with a service chain represented by service chain object 2918. A service chain is implemented by one or more service paths, each defined by a service path object 2922. Each service path has one or more service hops, represented by one or more service path hop objects 2924, where each hop is associated with an instance runtime interface 2916. Each service hop also references an associated service profile, an associated service path, and, in some embodiments, a next hop SIR endpoint identifier.

[0219] In some embodiments, the service path object has several attributes, some of which can be updated by the management or control plane when the underlying conditions change. These properties include the service path index, state (e.g., enabled or disabled), management mode (e.g., enabled or disabled) used when the service path must be manually disabled (e.g., for debugging reasons), host cross count (indicates the number of times a data packet traversing the service path passes through a host), location count (indicates how many of the SIRs along the path are located on the local host), a list of backup service paths, service length paths, reverse paths (listing the same set of SIRs in reverse order), and a maintenance mode identifier (in some embodiments, this indicator is a bit that indicates true if any hop in the service path is in maintenance mode).

[0220] The host transit count is an integer and indicates the number of times a data packet traversing the service path must be sent out of the PNIC. In some embodiments, the local or central control plane uses this metric to determine the preferred path when there are multiple available alternatives. This value is populated by the management plane or control plane and is the same for each host using the service path. In some embodiments, the location count is not initialized by the management plane or control plane, but is calculated by the local control plane when the service path is created or updated. Each LCP may calculate a different number. When there are multiple available alternatives, this value is used by the local control plane to identify the preferred path. The service path length is a parameter used by the service plane to set the initial service index.

[0221] In some embodiments, the list of backup service paths is a pointer to a sorted list of all service paths for the same service chain. The list lists all possible alternatives to be tried when a particular SIR along the path drops. This list may contain all possible permutations of service paths for the SVMs in each HA cluster that the service path passes through. In some embodiments, the list will not contain SIRs belonging to different HA clusters.

[0222] In some embodiments, a service path is disabled when at least one service hop is inactive. This condition is temporary and is triggered by a service activity detection failure. A service path may be disabled in this manner at any time. In some embodiments, a service path is also disabled when at least one service hop has no matching SIR. A service hop enters this state when the SIR referenced by the service hop disappears, but the service path still exists in the object model.

[0223] The service plane must be able to uniquely identify each SPI. In some embodiments, a UUID generated by the control plane is sent for each service path. Due to data message header limitations in the service plane, in some embodiments, a large ID is not sent with each data message. In some embodiments, the control plane generates a small unique ID for each service path when generating a UUID, and in these embodiments, the ID is sent with each data message.

[0224] Fig.30 Conceptually illustrates several operations that the network manager and controller perform in some embodiments to define rules for service insertion, next service hop forwarding, and service processing. As shown, these operations are performed by a service registrar 3004, a service chain creator 3006, a service rule creator 3008, a service path generator 3010, a service plane rule generator 3012, and a rule distributor 3014. In some embodiments, each of these operators can be implemented by one or more modules of a network manager or controller and / or can be implemented by one or more independent servers.

[0225] Through a service partner interface 3002 (e.g., an API set or a partner user interface (UI) portal), a service registrar 3004 receives provider templates 3005 specifying services performed by different service partners. These templates define partner services with respect to one or more service descriptors (including service profiles). The registrar 3004 stores the service profiles in a profile storage 3007 for use by a service chain creator 3006 to define service chains.

[0226] Specifically, through the user interface 3018 (e.g., an API set or a UI portal), the service chain creator 3006 receives one or more service chain definitions from a network administrator (e.g., a data center administrator, a tenant administrator, etc.). In some embodiments, each service chain definition associates a service chain identifier that identifies a service chain with an ordered sequence of one or more service profiles. Each service profile in a defined service chain is associated with a service operation that needs to be performed by a service node. The service chain creator 3006 stores the definition of each service chain in a service chain storage device 3020.

[0227] The service rule creator 3008 receives one or more service insertion rules from a network administrator (e.g., a data center administrator, a tenant administrator, etc.) through a user interface 3018 (e.g., an API set or a UI portal). In some embodiments, each service insertion rule associates a set of data packet flow attributes with a service chain identifier. In some embodiments, the flow attributes are flow header attributes, such as L2 attributes or L3 / L4 attributes (e.g., five-tuple attributes). In these or other embodiments, the flow attributes are context attributes (e.g., AppID, process ID, active directory ID, etc.). A variety of techniques for capturing and using context attributes to perform forwarding and service operations are described in U.S. patent application Ser. No. 15 / 650,251, which is incorporated herein. Any of these techniques may be used in conjunction with the embodiments described herein.

[0228] The service rule creator 3008 generates one or more service insertion rules and stores these rules in the SI rule storage device 3022. In some embodiments, each service insertion rule has a rule identifier and a service chain identifier. The rule identifier in some embodiments can be defined with respect to a flow identifier (e.g., header attribute, context attribute, etc.) that identifies the (one or more) data packet flows to which the SI rule is applicable. On the other hand, the service chain identifier of each SI rule identifies the service chain that the service plane must execute for any data packet flow that matches the rule identifier of the SI rule.

[0229] For each service chain that is part of a service rule, the service path generator 3012 generates one or more service paths, each path identifying one or more service instance endpoints for one or more service nodes to perform the service operation specified by the chain's sequence of service profiles. In some embodiments, the process of generating service paths for a service chain takes into account one or more criteria, such as (1) the data packet processing load on the service nodes (e.g., SVMs) that are candidate service nodes for the service path, (2) the number of host computers that the data packets of the flow pass through as it traverses each candidate service path, etc.

[0230] The generation of these service paths is further described in U.S. Patent Application 16 / 282,802, which is incorporated herein by reference. As described in that patent application, some embodiments identify a service path to be used for a particular GVM on a particular host based on one or more metrics, such as a host transit count (indicating the number of times a data packet traversing the service path passes through the host), a location count (indicating how many of the SIRs along the path are located on the local host), etc. Other embodiments identify service paths (i.e., select service nodes for a service path) based on other metrics (e.g., financial and licensing metrics).

[0231] The service path generator 3012 stores the identity of the generated service path in a service path storage 3024. In some embodiments, the storage associates each service chain identifier with one or more service path identifiers and provides, for each service path (i.e., each SPI), a list of service instance endpoints that define the service path. Some embodiments store the service path definitions in one data storage while storing the association between the service chain and its service path in another data storage.

[0232] Then, the service rule generator 3010 generates rules for service insertion, next service hop forwarding, and service processing based on the rules stored in the storage devices 3020, 3022, and 3024, and stores these rules in the rule storage devices 3026, 3028, and 3030, and the rule distributor 3014 can retrieve these rules from these storage devices and distribute them to the SI preprocessor, service agent, and service node. In some embodiments, the distributor 3014 also distributes the path definition from the service path storage device 3024. In some embodiments, the path definition includes the first hop network address (e.g., MAC address) of the first hop along each path. In some embodiments, the service rule generator 3010 and / or the rule distributor 3014 specifies and distributes different service path sets to different host computers for the same service chain because different service path sets are optimal or preferred for different host computers.

[0233] In some embodiments, the SI classification rules stored in the rule storage 3026 associate a flow identifier with a service chain identifier. Therefore, in some embodiments, the rule generator 3010 retrieves these rules from the storage 3022 and stores them in the classification rule storage 3026. In some embodiments, the rule distributor 3014 retrieves the classification rules directly from the SI rule storage 3022. For these embodiments, the description of the SI classification rule storage 3026 is more of a conceptual illustration to highlight the three types of distributed rules and next-hop forwarding rules and service node rules.

[0234] In some embodiments, the service rule generator 3010 generates a next-hop forwarding rule for each hop service agent of each service path of each service chain. As described above, in some embodiments, the forwarding table of each service agent has a forwarding rule for the next-hop network address of each service path where the associated service node of the identification agent is located. Each such forwarding rule maps the current SPI / SI value to the next-hop network address. The service rule generator 3010 generates these rules. For embodiments in which the SI preprocessor must find the first-hop network address, the service rule generator also generates a first-hop lookup rule for the SI preprocessor.

[0235] In addition, in some embodiments, the service rule generator 3010 generates service rules for the service node, which map the service chain identifier, service index value and service direction to the service profile of the service node. To this end, the service rule generator uses the service chain and service path definitions from the storage devices 3020 and 3024, and the service profile definitions from the service profile storage device 3007. In some embodiments, the rule distributor forwards the service node rules to the service node through the service manager of the service node (when such a service manager exists). In some embodiments, the service profile definitions are also distributed to the host computers (e.g., to their LCPs) by the distributor 3014, so that these host computers (e.g., LCPs) can use these service profiles to configure their service agents, for example, to configure the service agent to forward the received data message to its service node, or to copy the received data message and forward the copy to its service node, while forwarding the original received data message to its next service node hop or back to its source GVM when it is the last hop.

[0236] In some embodiments, the management and control plane dynamically modifies the service path of the service chain based on the status of the service nodes of the service path and the data packet processing load on these service nodes. Fig.31 How to dynamically modify service paths in some embodiments is illustrated. In these embodiments, the central control plane 3100 works with the local control plane 3110 on the host computer 3120 to define service paths for service chains and modify these service paths. In some embodiments, CCP 3100 is a server cluster (e.g., three servers) that provides control plane operations for defining configurations based on service rules specified by a network administrator through a cluster of management servers that provide management operations.

[0237] As shown, the CCP has a status updater 3102 that receives service node status data from a status publisher 3103 on a host computer 3120. As described above, each time a service agent determines that its associated service node has failed (e.g., each time a service node fails to respond to the service agent's activity signal twice in a row), the service agent notifies its host's LCP 3110. The LCP then has its status publisher 3103 notify the CCP's status updater 3102 of the service node's failure.

[0238] The state updater 3102 relays any service node failures to the service path generator 3012, which in some embodiments is part of the CCP along with the SP rule generator 3010 and the statistics collector 3104. Each time a service node fails, the service path generator removes its previously defined service paths that use that service node from the service path storage 3024. For each removed service path, the service path generator 3012 deletes or deactivates the SPI value of the removed path for the service chain identifier of the corresponding service chain.

[0239] In some embodiments, each removed service path is removed (e.g., deleted or deactivated) from the records of all hosts that previously received forwarding rules or path definitions for that service path. In some embodiments, the CCP (e.g., service path generator 3010 or rule distributor 3014) directs the hosts to remove the service path from the forwarding and path definition rules in their forwarding rule storage 3128 and path definition storage 808. In some embodiments, the LCP of the failed service node removes the service path from its forwarding and path definition rules, while in other embodiments, even the LCP waits for instructions from the CCP to do so.

[0240] In some embodiments, each host 3120 also has a statistics publisher 3105 that publishes data packet load statistics generated by the service agent for its service nodes. Each time the service agent receives a data packet that has been processed by its service node, in some embodiments, the service agent increments the statistics (e.g., data packet count, byte count, etc.) it maintains for its service node in the statistics storage 3107. In some embodiments, the statistics publisher 3105 retrieves the collected statistics from the storage 3107 periodically or on demand and forwards these statistics to the statistics collector 3104 of the CCP. In some embodiments, the statistics collector 3104 receives the statistics received from the service node by the service manager of the service node (via the management plane).

[0241] Statistics collector 3104 relays the collected statistics to service path generator 3012. As described above, in some embodiments, the service path generator defines a service path through a service node based in part on a data packet load on the service node. For example, when the data packet load on a service node exceeds a threshold, in some embodiments, the service path generator performs one or more actions to reduce the load on the service node. For example, in some embodiments, the service path generator stops adding service nodes to any new service paths that it may define. In these or other embodiments, the service path generator also directs distributor 3014 to remove service paths that use the service node from some or all hosts.

[0242] In conjunction or alternatively, the service path generator directs a CCP module (e.g., distributor 3014) to direct the LCP of one or more host computers to adjust selection criteria 820 for selecting service paths generated by the LCP in order to control how the SI preprocessor performs its path selection. In other embodiments, the service path generator or another CCP module aggregates load statistics for each service node and distributes the aggregated load along with its associated SPI value to the host LCP so that the LCP can analyze these statistics and adjust the path selection criteria it generates. In some embodiments, each LCP uses or has a path evaluator 3115 to generate path selection criteria to evaluate and select paths based on service node statistics and / or based on other criteria (such as the number of hosts traversed by each service path).

[0243] In some embodiments, the servers implementing the management plane, control plane, service manager are in the same data center as the host computers on which the guest machines and service machines and modules (e.g., GVM, SVM, service agent, port agent, STL module, SFE, etc.) are executed. In these embodiments, the management plane servers, control plane servers, service manager, and host computer modules (e.g., LCP, SVM, GVM, hypervisor modules, etc.) communicate with each other through the shared network infrastructure (e.g., switches, routers, wired and wireless links, etc.) of the data center.

[0244] In other embodiments, the management plane servers, control plane servers, service managers and / or host computers operate in different data centers (e.g., an enterprise private data center and a public cloud data center). In some such embodiments, the management plane servers, control plane servers, service managers and / or host computer modules (e.g., LCP, SVM, GVM, hypervisor modules, etc.) communicate with each other through a network infrastructure external to their respective data centers. In addition, some such embodiments implement the service transport layer as a distributed logical L3 router and / or network across multiple data centers (e.g., multiple private data centers, multiple public data centers, multiple private / public data centers).

[0245] Fig.32Illustrated is a process 3200 that some embodiments perform to define a service plane and its associated service nodes for tenants in a multi-tenant data center. This process presents only an exemplary sequence of operations and is not meant to convey any required order of operations. As shown, the process initially specifies (at 3205) a service attachment for establishing a service plane. The service attachment construction is agnostic to the implementation of the service plane. In some embodiments, the service attachment is implemented as a logical switch, but, as described above, in other embodiments, the service attachment is implemented differently (e.g., a logical router, a logical network, etc.).

[0246] In some embodiments, the service plane is used to separate service processing of data traffic of one tenant from service processing of data traffic of other tenants. In these or other embodiments, different service planes are used to provide different QoS or SLA guarantees for different types of traffic. For example, some embodiments use different service planes to provide different QoS or SLA guarantees for traffic between different data computing endpoints of different tenants, or to provide different QoS or SLA guarantees for different types of content carried by different data packet flows of the same tenant or different tenants.

[0247] After creating the service attachment, the process creates (at 3210) service instances of the services to be provided by the service plane. For each deployed service instance, the process specifies whether the service instance should be provided by a high availability cluster or by a standalone service node. The process also provides a service attachment identifier that identifies the service attachment associated with the service instance. The process also provides a deployment specification and an instance deployment configuration.

[0248] Next, at 3215, the process deploys each service instance runtime for each service instance created at 3210. For each service instance runtime, an instance endpoint must be created on the service attachment. When the service attachment is a logical switch, the instance endpoint created is the logical port of the logical switch. In some embodiments, when the SVM (used as a service instance runtime) is attached to the logical switch, the logical switch port is automatically created. In some embodiments, the service instance endpoint is created by the management plane each time a service instance is deployed. In addition, in some embodiments, the service instance and service instance runtime of a service can be deployed by calling a service deployment object API. As just mentioned, using this single API greatly alleviates the need to repeatedly call an API multiple times to deploy multiple service instances and service instance runtimes.

[0249] At 3220, the process creates one or more service chains. Each service chain is created as an ordered list of service profiles. Each service chain has a forward processing direction and a reverse processing direction. For each service chain, a failure policy is defined as described above. In addition, as described above, in some embodiments, the load balancing criteria for each service chain is defined to one of the following types: any, local, service cluster, or round-robin. Finally, at 3225, service rule segments are defined for the tenant, and one or more service rules are defined in these segments. Each service rule associates a set of data packet flow attributes with a service chain identifier to specify a service chain that must be executed on data packets that match the specified set of flow attributes.

[0250] Some embodiments provide novel methods for migrating machines associated with a service plane between hosts operating in one or more data centers. For example, some embodiments migrate a service machine (e.g., SVM) or a guest machine (e.g., GVM) associated with a service plane by: (1) collecting service plane data about the machine at a source host, (2) transmitting the data to a target host, (3) deploying the machine on the target host, and (4) removing the machine from the source host.

[0251] Instead of assigning a new service plane network address to the machine at its new location, the machine may be addressed by the same service plane network address at the destination host as at the source host. For example, in some embodiments, a guest machine addressable by a service plane MAC (SPMAC) address may be addressed by the same SPMAC after migrating to a different host. Similarly, in some embodiments, when a service machine associated with a particular next-hop MAC address is migrated, the service machine may be addressed by the same next-hop MAC address both before and after the migration. In some such embodiments, there is no need to update the next-hop forwarding rules that reference the migrated service machine based on the change in the location of the service machine after the machine migration. Therefore, next-hop forwarding may be independent of the migration.

[0252] Instead, in some embodiments, the service plane itself is responsible for tracking the location of the migrated machines. In some embodiments, the service plane (e.g., a set of controllers that configure the service plane) maintains and updates a mapping of service plane MAC addresses to VTEPs to track machine locations and ensure that packets addressed to the MAC addresses are forwarded to the correct tunnel endpoint for delivery to the machine.

[0253] In some embodiments, the migrated machine is associated with locally maintained data that must be maintained with the machine migration. For example, in some embodiments, a preprocessor for the IO chain of the guest machine identifies flows and provides service metadata (SMD) for service plane forwarding based on dynamic state maintained at the host of the guest machine (e.g., connection tracking). In addition, in some embodiments, the host stores static data specific to the guest machine, such as SI rules applicable to the guest machine and service paths for the service chains identified in the SI rules. In some embodiments, some or all of this information is not maintained elsewhere in the service insertion platform.

[0254] Now refer to Figure 33-Figure 35 A more detailed example of the operation of guest machine migration is described below. Fig.33 The process 3300 performed by the source host of the migrated GVM and the process 3305 performed by the destination host of the migrated GVM are illustrated. Figure 34A-Figure 34C The process is described using the GVM migration example shown in . It should be noted that although Figure 34A-Figure 34C Only one IO chain per GVM VNIC is illustrated, but a GVM VNIC typically has two IO chains, one for ingress from the switch to the VNIC and another for egress from the VNIC to the switch. In some embodiments, the egress IO chain includes the same modules as the ingress IO chain, but is placed in reverse order between the VNIC and the switch. Fig.34A The source host 112 and the destination host 114 are illustrated prior to migration of a migrating GVM 3402. The process 3300 begins when the source host 112 determines that the GVM 3402 needs to be migrated to the destination host 114.

[0255] As shown, process 3300 initially (at 3310) collects and saves GVM data about a migration GVM 3402. In some embodiments, the GVM data includes GVM deployment data for deploying or instantiating an instance of GVM 3402 on a destination host 114. In addition, in some embodiments, the GVM data includes dynamic state data and static rule data used by a service insertion module to assign service metadata to data packets associated with the GVM. Host 112 transmits (at 3320) the collected GVM data 3490. The GVM data includes a collection of GVM deployment data for deploying the migration GVM 3402. In this example, GVM data 3490 also includes connection tracking data collected from connection tracking storage 804 (including dynamic state data, such as a mapping of service metadata (SMD) to flow identifiers), service insertion rules associated with GVM 3042 from SI rule storage 806, and service paths associated with GVM 3402 for service chains identified by SI rules from path table 808. In some embodiments, the GVM data is sent via a virtual machine mobility management service (e.g., Enterprise Mobility Management by VMware, Inc.) Furthermore, in some embodiments, when transmitting the GVM data, a TLV header is defined for each of the GVM deployment data, connection data, SI rules, and service paths.

[0256] In some embodiments, the service path selection metric and next hop address are collected from the path table and transmitted with the GVM data. In other embodiments, the first hop MAC address of the service path is stored in a separate forwarding table (e.g., Figure 8 In some embodiments, the service paths and service path selection metrics are not collected and transmitted with the GVM data. Some such embodiments instead store each service path distributed from the CCP at each host or only store the service paths distributed by the CCP to that particular host at each particular host.

[0257] After the GVM data is transferred from the source host 112, the process 3305 starts and receives (at 3325) the GVM data at the destination host 114. In some embodiments, the GVM data alerts the destination host of the machine migration. The process deploys (at 3335) the migration GVM 3402 on the destination host 114 using the GVM data. Fig.34BThe source host 112 and the destination host 114 are illustrated after the GVM 3402 has been deployed on the host 114. At 3335, the process also deploys the port 3452 and the ingress and egress IO chains for the newly deployed GVM 3402. The GVM 3402 is connected to these IO chains and through these IO chains to the port 3452 of the switch 122 on the host 114. In this example, the deployed ingress IO chain includes the SI preprocessor 3411, the STL invoker 3425, and the SI postprocessor 3413. In some embodiments, the port proxy 3420 stores the mapping of the GVM 3402 to its new location. In some embodiments, the port proxy 3420 maps the GVM SPMAC address to the STL invoker 3425. In addition, in some embodiments, the GVM 3402 has the same SPMAC at the source host 112 and the destination host 114.

[0258] Once the GVM 3402 and its IO chain have been deployed, the process 3305 begins restoring the GVM 3402 dynamic state and static rule data from the received GVM data 3490. In some embodiments, the SI preprocessor 3411 cannot process the flow of the migrated GVM 3402 until at least a portion of the GVM 3402 state and rule data has been restored on the destination host 114 and can be accessed by the SI preprocessor. At 3345, the process restores the connection data for the GVM 3402, storing the received connection data in the connection tracking storage 3404. In some embodiments, the SI preprocessor 3411 can immediately begin processing the existing data flow, where the connection tracking storage contains sufficient service metadata (e.g., SPI, SI, direction, first next hop address, etc.) for service plane forwarding. In this case, the preprocessor 3411 can restart flow processing before restoring the service insertion rules and paths of the GVM 3402. In some embodiments, the new data flow is held (e.g., at the SI preprocessor) until the service insertion rules and service path data are restored. In other embodiments, new flows that do not match a service insertion rule are forwarded to their destination.

[0259] At 3355, process 3305 restores the SI rules for GVM 3402 based on the received GVM data 3490. The SI rules are stored in SI rule storage 3406. In this example, the SI rule storage stores the SI rules for GVM 3402 and GVM 3402 in different segments. In other embodiments, each GVM has its own rule storage. In some embodiments, the rules for all GVMs are stored in a single segment of a single storage. After restoring the SI rules for GVM 3402, the process restores (at 3365) the received paths and path data (e.g., selection criteria, first hop MAC address, etc.) in path table 3408. In this example, GVM 104 and GVM 3402 share path table 3408, but in some embodiments, guest machines on the same host may also share a single segment or have their own storage. In some embodiments, once the path table is restored, preprocessor 3411 begins processing new flows. However, in other embodiments, the preprocessor does not begin processing new flows until the control plane updates the path table. In some such cases, the preprocessor continues to maintain data packets belonging to the new data flow.Once the path table has been restored, the process 3305 notifies (at 3365) the control location (eg, CCP) that the GVM deployment and data recovery is complete and ends.

[0260] When host 112 receives confirmation that GVM 3402 has been deployed on host 114 and that the state and rules of GVM 3402 have been restored, process 3300 initiates destruction of GVM 3402 from host 112 at 3330. In some embodiments, source host 112 receives confirmation from the CCP, while in other embodiments, confirmation is received from destination host 114 itself. In some embodiments, GVM 3402 is first disconnected from its IO chain at host 114 and then removed.

[0261] At 3330, host 112 also destroys the GVM's IO chain and port 3450 through which GVM 3402 is connected to vswitch 120, including SI preprocessor 3410, STL caller 3424, and SI postprocessor 3412. In some embodiments, the IO chain is disconnected from port 3450 before being removed from host 112. In some embodiments, process 3300 removes data specific to the migrated GVM 3402 from host 112. In this case, host 112 removes GVM 3402 data from connection tracking storage 804, SI rule storage 806, and path table 808. Port proxy 620 also removes the mapping of GVM 3402 to its SPMAC. Once GVM 3402, its IO chain, and its associated data have been removed from host 112, process 3300 notifies (at 3340) the control plane (eg, CCP) that GVM 3402 has been disconnected from source host 112 and ends. Fig.34C The diagram shows host 112 and host 114 after the migration of GVM 3402 is completed.

[0262] In some embodiments, the control plane is responsible for detecting machine migrations and updating the service plane accordingly. Fig.35 Further describes the control plane operations performed after a GVM migration, Fig.35 The process 3500 performed by the control plane in some embodiments is illustrated. Figure 34A-Figure 34C Describe the process.

[0263] As shown, process 3500 begins when the control plane determines (at 3510) that GVM 3402 has migrated from source host 112 to destination host 114. In some embodiments, the control plane determines that the guest machine has migrated hosts based on one or more notifications received at the CCP. In some embodiments, the control plane receives a first notification from the destination host that the guest machine has been deployed and a second notification from the source host that the guest machine has been disconnected. In some embodiments, the first notification is sent by the LCP of the destination host and the second notification is sent by the LCP of the source host.

[0264] Once the control plane determines that GVM 3402 has migrated, the process (at 3520) publishes the new location of the migrated GVM. In some embodiments, the control plane publishes a mapping of the GVM SPMAC (which is the same after migration as before) to the VTEP through which the GVM can be reached. In some embodiments, the VTEP is used to return data flows to their source GVM IO chain after the data flow has completed the service path. In some embodiments, the VTEP is a VTEP for a port proxy on the new host that inserts and plugs data flows associated with the GVM from and to the service plane (e.g., port proxy 3420).

[0265] In some embodiments, the mapping of VTEP to SPMAC is published to a global forwarding table maintained by the service plane and replaces the mapping of SPMAC to the VTEP of the source host in the table. In some embodiments, the mapping is published from the CCP to the LCP at the host of each forwarding element of the service plane.

[0266] In some embodiments, the new location of the migrated GVM is also learned by the service plane through source learning based on the data flow inserted into the service plane from the IO chain of the GVM. In some such embodiments, the inserted traffic is attached with a VTEP source label that identifies the source VTEP through which the traffic is forwarded from the GVM. The VTEP source label is carried along each hop through the service plane. In some embodiments, the VTEP source label is added to the service metadata of the data packet of the data flow. In some such embodiments, the data packet is checked at each hop for the SPMAC and VTEP labels. The SPMAC and VTEP of the source label are retrieved from the service metadata of the data packet, and the mapping of SPMAC to VTEP is learned. In some embodiments, this mapping can be used to return the data packet at the end of the service path to the IO chain of the source GVM, from which the data packet was inserted into the service plane.

[0267] If necessary, the control plane generates (at 3530) a new service path for the destination host 114 of the migrated GVM. In some embodiments, a service path is selected for a service chain based at least in part on the location of the SVM of the service path relative to the host that selected the service path (e.g., where the datagram is inserted into the service plane for the service path to be executed). Therefore, the service path generated for the source host 112 of the GVM migration may not be optimal for the destination host 114. In some embodiments, the new service path is the best service path selected based on the location of the destination host relative to the SVM of the service. The new service path is delivered to the destination host of the GVM and stored in the path table. In some embodiments, the new service path is generated at the CCP and distributed to the destination host by the LCP of the destination host. Some embodiments deliver only a single service path for each service chain to each host. Other embodiments distribute multiple service paths for each service chain for each host. In some such embodiments, the LCP of each host selects one service path for each service chain from multiple received service paths to store based on a selection criterion. In other such embodiments, each received service path is stored and the LCP generates a selection metric for the received service path. In some such embodiments, a service path is selected for a service chain on a flow-by-flow basis using a service path selection metric, for example based on a weighted round-robin selection.

[0268] In some embodiments, service path selection is independent of the location of the SVMs of the service path relative to the GVM host. For example, service path selection may depend on the location of the SVMs relative to each other, rather than relative to the location of the GVM's host. In this case, migration of the GVM will not affect service path selection. In this case, generation of a new service path will not be necessary after the GVM migration and may be skipped. In some embodiments, service rules and / or service paths are not migrated during machine migration, but rather the CCP distributes the old service rules and / or service paths of the GVM to the destination host (e.g., host 114).

[0269] At 3540, if necessary, the control plane calculates a new service path selection metric for the service path of the path table 3408 at the destination host 114. In some embodiments, the service path selection metric is always calculated for new service paths received from the central control plane. Further, in some embodiments, the service path is migrated without a service path selection metric, and the service path selection metric is calculated for the received service path. When the service path selection metric is based at least in part on the location of the service path SVM relative to the GVM host, the migrated service path selection metric may no longer be accurate, and in some embodiments a new service path selection metric is calculated to replace the migrated service path selection metric. In some embodiments, when the migrated GVM is the first GVM on the host, the host does not store any service paths prior to the migration. In some such embodiments, all service paths stored on the destination host after the migration are service paths migrated from the source host. In other embodiments, the service paths are not migrated with the GVM at all.

[0270] In some embodiments, the service path selection metric is independent of the location of the GVM itself, but rather is dependent on the location of the host generating the selection metric. In some such instances, it is not necessary to calculate the service path selection metric for a service path that is already stored in a path table on the destination host. In such embodiments, the service path selection metric may still be calculated due to other changes in the service plane. In other embodiments, when the service path selection metric is independent of the location of the GVM host, calculating the service path selection metric for any service path that already has a selection metric is unnecessary and can be skipped. Once any necessary service path selection metrics are calculated, process 3500 ends.

[0271] Some embodiments also provide novel methods for migrating service machines (e.g., SVMs) between hosts operating in one or more data centers. For example, when a first host computer hosting one or more service machines needs to be shut down for maintenance, the service machines running on the first host computer can be migrated to a second host computer. In some cases, service machines can be migrated between host machines to balance workloads, simplify permissions or firewall implementations, or provide improved service paths.

[0272] In some such embodiments, the SVM executes in a shared memory of a host computer (e.g., shared memory in a VMware ESX host). In some such embodiments, to perform a service operation on a file, the SVM locates the storage location of the file in the shared memory and opens the file in kernel space before performing the service operation. In some such embodiments, the SVM can only perform service operations on files stored in the same ESX. In some such embodiments, migration of the SVM between hosts requires migration of the filter module (e.g., dvFilter) associated with the VM. In other such embodiments, the SVM is instead a MAC addressable entity that does not run in shared memory. Reference will now be made to Figure 36-Figure 39 A more detailed example describing a MAC-addressable service machine migration operation.

[0273] Fig.36 The process 3600 performed at the source host 112 of the migrated SVM 3706 and the process 3605 performed at the destination host 114 of the migrated SVM are illustrated. Figure 37A-Figure 37C The process is described using the SVM migration example illustrated in . Fig.37A The host 112 and the host 114 are illustrated prior to migration of the SVM 3706. The process 3600 begins when the source host 112 determines that the SVM 3706 needs to be migrated to the destination host 114.

[0274] As shown, process 3600 initially saves (at 3610) SVM data regarding SVM 3706. In some embodiments, the SVM data includes data for deploying SVM 3706 on host 114. In some embodiments, the SVM data also includes dynamic state SVM data and static rule SVM data used by SVM 3706 to identify data packet flows and perform services on the data packet flows.

[0275] In some embodiments, the SVM rule data includes a mapping of service metadata (e.g., SCI, SI, and destination) to a set of service rules for performing services on a data packet flow. In addition, in some embodiments, the SVM state data includes a mapping of a flow identifier to a service operation performed on a data packet flow. In some embodiments, the state data includes a mapping of a decision based on a lookup of a data flow identifier in a service rule set to a data flow identifier. In some embodiments, the SVM data also includes data for configuring an SI agent (e.g., SI agent 3715) for the SVM IO chain and any state data used by the SI agent. In some embodiments, the SI agent data includes a set of next-hop forwarding rules for the service path to which the migrated SVM belongs. In addition, in some embodiments, the SI agent state data includes metrics and statistics maintained at the SI agent, such as traffic records.

[0276] The source host 1112 transmits (at 3620) the saved SVM data 3790 to the destination host 114. In some embodiments, the SVM data 3790 is sent via a virtual machine mobility management service (e.g., Enterprise Mobility Management by VMware, Inc.). In some embodiments, all SVM data, including any SVM rule data, SVM state data, and SI agent data, is moved with the migrated SVM 3706. In other embodiments, some or all of the SVM rule data, SVM state data, and SI agent data are migrated separately via a virtual machine migration framework.

[0277] After the SVM data 3790 is transferred from the source host 112, the process 3605 begins and receives (at 3625) the SVM data at the destination host 114. In some embodiments, the SVM data alerts the destination host of the machine migration. The process deploys (at 3635) the SVM 3706 on the destination host 114 using the SVM data 3790. Fig.37B Host 112 and host 114 are shown after SVM 3706 has been deployed on host 2. In some embodiments, migrated SVM 3706 is deployed with some or all of the migrated SVM state and SVM rules using migrated SVM data 3790. In other embodiments, some or all of the SVM state and rule data is restored after deployment, e.g., restoring separately migrated SVM data (e.g., SI agent state).

[0278] At 3635, process 3605 also deploys port 3755 and ingress and egress IO chains. The process connects SVM 3706 to the IO chain and connects SVM 3706 to port 3755 on switch 122 at destination host 114 through the IO chain. In this example, the deployed IO chain includes SI agent 3715 and STL module 3727. In some embodiments, the migrated next-hop forwarding rules are consumed during the deployment of SI agent 3715, and the SI agent is deployed with the configured next-hop forwarding rules. In other embodiments, the next-hop forwarding rules are stored globally by the service plane and provided to SI agent 3715. In still other embodiments, the next-hop forwarding rules are provided by the CCP after deployment. In some embodiments, during the deployment of the IO chain, the next-hop address of the migrated SVM 3706 is saved by the service plane. In addition, in some embodiments, the migrated SVM can be addressed by the same next-hop address at the destination host as at the source host. In this example, the service plane stores a mapping of SVM 3706 to a VTEP through which SVM 3706 can be reached at destination host 114 via port 3755, which is the same next-hop MAC address at host 114 as assigned to SVM 3706 at host 112.

[0279] In some embodiments, once the SVM 3706 and its IO chain have been deployed, the process 3605 restores (at 3645) the SVM 3706 dynamic state and static rule data from the received SVM data 3790. In some such embodiments, the restored state can be used to process data packets of previously processed flows before restoring the static rules. In some embodiments, when the migrated SVM is the first SVM on the destination host, the destination host does not have any next-hop forwarding rules or VTEP to MAC address mappings. In some such embodiments, the process 3605 restores the next-hop forwarding rules and / or VTEP to MAC mappings from the migrated SVM data 3790. In some embodiments, when the CCP is inactive or otherwise unable to distribute service plane information, the process 3605 also restores the next-hop forwarding rules and / or VTEP to MAC address mappings. In other embodiments, all migrated SVM data is instead consumed when the migrated SVM is deployed. In this case, restoring the SVM state after deployment is unnecessary and can be skipped. Once any necessary state and / or rules are restored, the process 3605 ends.

[0280] When host 112 receives confirmation that SVM 3706 has been deployed and configured on host 114, process 3600 initiates destruction of SVM 3706 from host 112 (at 3630). In some embodiments, the source host receives confirmation from the CCP, while in other embodiments, confirmation is received from the destination host itself. In some embodiments, SVM 3706 is first disconnected from its IO chain and then removed.

[0281] At 3630, the process also destroys the IO chain and port 3754 of SVM 3706, through which SVM 3706 is connected to vswitch 120, including SI agent 3714 and STL module 3726. In some embodiments, the IO chain is disconnected from port 3754 before being removed from host 112. In some embodiments, the source host removes data specific to the migrated SVM. In some embodiments, the service plane also removes the mapping of the migrated SVM 3706 to its location at host 112. Once SVM 3705, the IO chain of SVM 3706, and its associated data have been removed from source host 112, process 3600 ends. Fig.37C Host 112 and host 114 are illustrated after the migration of SVM 3706 has been completed.

[0282] In some embodiments, the control plane is responsible for detecting SVM migration and updating the service plane and forwarding plane accordingly. Fig.38 Further describing the control plane operations performed after SVM migration, Fig.38 A process 3800 is shown that is performed by the control plane in some embodiments. Figure 37A-Figure 37C Describe the process.

[0283] As shown, process 3800 begins when the control plane determines (at 3810) that SVM 3706 has migrated from source host 112 to destination host 114. In some embodiments, the control plane determines that the SVM has migrated hosts based on one or more notifications received at the CCP. In some embodiments, the control plane receives a first notification from destination host 114 that SVM 3706 has been deployed and a second notification from source host 112 that SVM 3706 has been detached. In some embodiments, the first notification is sent by the LCP of the destination host and the second notification is sent by the LCP of the source host.

[0284] Once the control plane determines that SVM 3706 has migrated, the process (at 3820) publishes the new location of the migrated SVM 3706 to configure the service plane to forward packets addressed to SVM 3706 to destination host 114 instead of source host 112. In some embodiments, the control plane publishes a mapping of the SVM next-hop MAC address (which is the same after migration as before) to the VTEP through which the SVM can now be reached. In some embodiments, the mapping is published to a global forwarding table maintained at the service plane, where the mapping replaces the previous mapping of the SVM to the VTEP associated with the source host (e.g., 112). In some such embodiments, the mapping is published from the CCP to the LCP at the host of each forwarding element implementing the service plane. In this example, encapsulation modules 628 and 3428 are configured to encapsulate packets addressed to the MAC address of SVM 3706 with the VTEP of host 114 instead of the VTEP of host 112.

[0285] If necessary, the control plane generates (at 3830) new service paths based on the migration of SVM 3706. In some embodiments, the service path selection metric depends at least in part on the relative positions of the service nodes in the service path with respect to each other. In some such embodiments, once an SVM has been migrated to a different host, the service path including the SVM is no longer optimized. In some embodiments, a new service path is generated to replace a service path including a migrated SVM 3706, where SVM 3706 is replaced by a different SVM performing the same service. In some embodiments, the migration of the SVM allows for the generation of new service paths including the migrated SVM that are optimal compared to previous service paths of the service chain.

[0286] At 3840, the process distributes the service path. In some embodiments, the service path is sent in the form of a next hop forwarding rule. In some embodiments, the control plane (e.g., CCP) distributes any service paths that include the migrated SVM 3706 to the destination host 114. In other embodiments, the next hop forwarding rules are maintained globally in the service plane, and the destination host 114 will already have the next hop forwarding rules for the service path that includes the migrated SVM 3706. In some embodiments, the received service path is stored by a service agent (e.g., SI agent 3715) of a service node of the service path.

[0287] In some embodiments, service paths are generated at the CCP and distributed to hosts via the LCP of each host. In some embodiments, the CCP distributes the next-hop forwarding rules for the new service path to each host of the SVM hosting the service plane, while in other embodiments, the next-hop forwarding rules for each specific service path are distributed to the hosts spanned by the specific service path.

[0288] In some embodiments, the newly generated service path is distributed to at least a subset of hosts executing the GVM associated with the service chain for which the service path was generated. In some embodiments, each host receives one service path for each service chain, while in other embodiments, each host receives multiple service paths for each service chain. In some embodiments, a host receives the service path for the service chain associated with its GVM, while in other embodiments, each host receives each distributed service path. In some embodiments, the received service paths are stored in the service path table of its respective host. In addition, in some embodiments, the LCP of each GVM host selects a service path for each service chain to store.

[0289] At 3850, the control plane calculates new service path selection metrics for service paths of certain path tables. In some embodiments, a new service path selection metric is calculated for each service path including the migrated SVM 3706. In some embodiments, when a new service path is generated, a service path selection metric is generated for each new service path. In some embodiments, the service path selection metric is calculated for each host's path table by the host's LCP. Once the service path selection metric is calculated, process 3800 ends. When only one service path is stored for each service chain, in some embodiments, the service path selection metric is not stored with the service path.

[0290] Some embodiments also provide novel methods for migrating service machines between disk provisionings of the same host computer. For example, a user or administrator may wish to migrate an SVM from a private disk provisioning to a shared disk provisioning, or vice versa. In some embodiments, a first disk provisioning and a second disk provisioning are associated with separate data stores (e.g., two VMware NFS data stores) on the same host computer. In some embodiments, the disk provisionings are first and second provisionings of the same disk storage device. In other embodiments, the disk provisionings are separate first and second disk storage devices. In some such embodiments, the SVM may be migrated between hardware disk storage devices of the same host computer to balance the workload therebetween. The process for migrating an SVM between disk provisionings follows Fig.36 However, in the case of intra-host SVM migration, process 3600 is performed by the source disk provisioning to perform the migration, and process 3605 is performed by the destination disk provisioning. Fig.39The process 3600 begins when it is determined that an SVM 3906 on an NFS datastore 3910 on a first disk provisioning of a host 112 needs to be migrated to an NFS datastore 3920 on a second disk provisioning also on the host 112. 3901 illustrates the host 112 before the migration.

[0291] As shown, process 3600 initially saves (at 3610) SVM data about SVM 3906. Source NFS 3910 transmits (at 3620) the saved SVM data 3990 to destination NFS 3920. In some embodiments, SVM data 3990 includes at least a subset of SVM data (e.g., SVM data 3790) to be collected during SVM migration between host computers. In some embodiments, SVM data 3990 is sent via a virtual machine mobility management service (e.g., Enterprise Mobility Management by VMware, Inc.) on host 112. After SVM data 3990 is transmitted from source NFS 3910, process 3605 begins and receives (at 3625) SVM data 3990 at destination NFS 3920. In some embodiments, SVM data 3990 alerts NFS 3920 of machine migration.

[0292] Process 3605 uses SVM data 3990 to deploy (at 3635) SVM 3906 on destination NFS 3920. In some embodiments, SVM 3906 is deployed with some or all of the migrated SVM state and rules using migrated SVM data 3990. In other embodiments, some or all of the SVM state and rule data is restored after deployment instead. At 3635, the process also deploys port 3955 and ingress IO chain and egress IO chain. SVM 3906 is connected to the IO chain and, through the IO chain, to port 3955 of switch 120. In this example, the deployed IO chain includes SI agent 3915 and STL module 3927.

[0293] At 3902, two instances of the IO chain of the SVM are executed on the host 112. In some embodiments, the IO chains of the two instances of SVM 3906 can be addressed by the same VNIC MAC address. In addition, in some embodiments, port 3954 and port 3955 have different port IDs. In some embodiments, as part of the deployment (at 3635), the process saves a mapping from a handle to the SI agent 3915 deployed on NFS 3920 to the MAC address and port 3955 port ID (e.g., in a port table or handle table).

[0294] In some embodiments, once the SVM 3906 and its IO chain have been deployed on the NFS 3920, the process 3605 restores (at 3645) the SVM 3906 dynamic state and static rule data at the NFS 3920 from the received SVM data 3990. In other embodiments, all migrated SVM data is consumed when the SVM 3906 is deployed instead, and restoring the SVM state and rules after deployment is unnecessary and can be skipped. Once any necessary state and / or rules are restored, the process 3605 ends.

[0295] When NFS 3910 receives confirmation that SVM 3906 has been deployed and configured on NFS 3920, process 3600 will initiate the operation of destroying SVM 3906 (at 3630) from NFS 3920. In some embodiments, the source NFS receives the confirmation from the CCP, while in other embodiments, the confirmation is received from the LCP or the destination NFS itself. In some embodiments, SVM 3906 is first disconnected from its IO chain and then removed from NFS 3920.

[0296] At 3630, the process also destroys the IO chain and port 3954 of SVM 3906 at NFS 3910 through which SVM 3906 is connected to vswitch 120 from NFS 3910, including SI agent 3914 and STL module 3926. In some embodiments, the IO chain is disconnected from port 3954 before being removed from NFS 3910. In some embodiments, the source NFS 3910 removes data specific to the migrated SVM 3906. In some embodiments, the service plane also removes its mapping of SVM 3906 to port 3954. Once SVM 3906, the SVM's IO chain, and its associated data have been removed from NFS 3910, process 3600 ends. 3903 illustrates NFS 3910 and NFS 3920 after the intra-host migration of SVM 3906 has been completed.

[0297] As described above, in some embodiments, there is an interval during intra-host migration in which the IO chain of the SVM is executed in the first disk provisioning and the second disk provisioning on the same host with the same MAC address. In some embodiments, this occurs after the IO chain of the SVM has been deployed in the second disk provisioning (at 3635) but before the IO chain of the SVM has been removed from the first disk provisioning (at 3630). For example, at 3902, the SI agent 3914 at NFS 3910 and the SI agent 3915 at NFS 3920 are both executed on host 112. In some embodiments, this can cause problems in communication between the control plane and the SI agent. For example, in some embodiments, when LCP 3930 needs to send a message to SI agent 3915, LCP looks up the handle of SI agent 3915 based on the MAC address of SVM 3906. However, when two SI agents with the same MAC address are running on the same host, a lookup based only on the MAC address will identify two SI agent handles. The MAC address lookup therefore returns a handle to SI agent 3914 and a handle to SI agent 3915.

[0298] In some embodiments, a second lookup is performed using the port ID of the desired SI agent to select the correct handle. In this example, to identify the handle of SI agent 3915, LCP 3930 performs a second lookup using the port ID of port 3955. This second lookup returns the handle of SI agent 3915. In other embodiments, an initial lookup is performed using both the MAC address and the port ID of the SI agent instead to identify the correct handle in one search. In some embodiments, the LCP uses the handle to send a configuration message to configure the SI agent.

[0299] In some embodiments, activity thread 3940 sends a message to SVM 3906 agent upon determining that its activity timer for SVM 3906 has expired to send a message to SVM 3906. To ensure that the activity timer message is sent to SI agent 3915 instead of 3914, activity thread 3940 also performs a first lookup using the MAC address and a second lookup using the port ID, or a single lookup using both the MAC address and the port ID, to identify the handle of SI agent 3915. In addition, in some embodiments, the service plane automatically stores the association of the port ID with the MAC address and uses both the associated port ID and the MAC address of the destination machine to identify the IO chain of the SVM during data plane forwarding.

[0300] Many of the above features and applications are implemented as software processes designated as sets of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of a processor, or other processing units), they cause the (one or more) processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or via wired connections.

[0301] In this specification, the term "software" is intended to include firmware residing in a read-only memory or an application stored in a magnetic storage device, and can be read into a memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that implement the software inventions described herein together are within the scope of the present invention. In some embodiments, when the software program is installed to run on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.

[0302] Fig.40 A computer system 4000 is conceptually illustrated with which some embodiments of the present invention are implemented. The computer system 4000 can be used to implement any of the above-described hosts, controllers, and managers. Thus, the computer system 4000 can be used to perform any of the above-described processes. The computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 4000 includes a bus 4005, (one or more) processing units 4010, a system memory 4025, a read-only memory 4030, a permanent storage device 4035, an input device 4040, and an output device 4045.

[0303] Bus 4005 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system 4000. For example, bus 4005 communicatively connects processing unit(s) 4010 to read-only memory 4030, system memory 4025, and permanent storage 4035.

[0304] The processing unit(s) 4010 retrieves instructions to be executed and data to be processed from these different memory units in order to perform the process of the present invention. In different embodiments, the processing unit(s) may be a single processor or a multi-core processor. Read-only memory (ROM) 4030 stores static data and instructions required by the processing unit(s) 4010 and other modules of the computer system. On the other hand, permanent storage device 4035 is a read-write memory device. The device is a non-volatile memory unit that stores instructions and data even when the computer system 4000 is turned off. Some embodiments of the present invention use a mass storage device (e.g., a disk or optical disk and its corresponding disk drive) as a permanent storage device 4035.

[0305] Other embodiments use removable storage devices (e.g., flash drives, etc.) as permanent storage devices. Like permanent storage device 4035, system memory 4025 is a read-write memory device. However, unlike storage device 4035, system memory is a volatile read-write memory, such as random access memory. System memory stores some instructions and data that the processor needs at runtime. In some embodiments, the processes of the present invention are stored in system memory 4025, permanent storage device 4035, and / or read-only memory 4030. From these various memory units, (one or more) processing units 4010 retrieve instructions to be executed and data to be processed to perform the processes of some embodiments.

[0306] The bus 4005 is also connected to an input device 4040 and an output device 4045. The input device enables a user to communicate information and select commands to the computer system. The input device 4040 includes an alphanumeric keyboard and a pointing device (also called a "cursor control device"). The output device 4045 displays images generated by the computer system. The output device includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include devices that function as both input devices and output devices, such as a touch screen.

[0307] Finally, if Fig.40 As shown, bus 4005 also couples computer system 4000 to a network 4065 via a network adapter (not shown). In this manner, the computer may be part of a network of computers (e.g., a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet). Any or all components of computer system 4000 may be used in conjunction with the present invention.

[0308] Some embodiments include electronic components such as microprocessors, storage devices, and memories that store computer program instructions on machine-readable or computer-readable media (or computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact disk (CD-ROM), recordable compact disk (CD-R), rewritable compact disk (CD-RW), read-only digital versatile disk (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Blu-ray players, and / or hard drives. Disks, ultra-density optical disks, and any other optical or magnetic media. Computer-readable media can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer codes include machine code, such as produced by a compiler, and files including high-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0309] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0310] As used in this specification, the terms "computer", "server", "processor" and "memory" refer to electronic devices or other technical devices. These terms do not include people or groups of people. For the purpose of this specification, the terms display or display means display on an electronic device. The terms "computer-readable medium" and "machine-readable medium" used in this specification are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient or temporary signals.

[0311] Although the present invention has been described with reference to many specific details, it will be appreciated by those skilled in the art that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. For example, several figures conceptually illustrate the process. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process may be implemented using several sub-processes, or as part of a larger macro process.

[0312] Although the service insertion rules in several of the above examples provide service chain identifiers, some of the inventions described herein may be implemented by having the service insertion rules provide service identifiers (e.g., SPIs) for different services specified by the service insertion rules. Similarly, several of the above embodiments perform distributed service routing that relies on identifying the next service hop at each service hop by performing an exact match based on the SPI / SI values. However, some of the inventions described herein may be implemented by having the service insertion preprocessor embed all service hop identifiers (e.g., service hop MAC addresses) as a service attribute set of a data packet and / or in an encapsulation service header of a data packet.

[0313] In addition, some embodiments decrement the SI value in a different manner than the above method (e.g., at a different time). In addition, instead of performing a next hop lookup based only on the SPI and SI values, some embodiments perform this lookup based on the SPI, SI, and service direction values, because these embodiments use a common SPI value for both the forward and reverse directions of data packets flowing between the two machines.

[0314] The above methods are used in some embodiments to express path information in a single tenant environment. Therefore, one of ordinary skill will appreciate that some embodiments of the present invention are equally applicable to a single tenant data center. Conversely, in some embodiments, when an entity (e.g., a company) is a tenant in multiple different data centers of different providers, the above methods are used to carry path information across different data centers of different data center providers. In these embodiments, the tenant identifier embedded in the tunnel header must be unique across data centers or must be converted when passing from one data center to the next. Therefore, one of ordinary skill in the art will understand that the present invention is not limited by the foregoing illustrative details, but is defined by the appended claims.

Claims

1. A method of performing services for data packets associated with a machine executing on a host computer, wherein the method is implemented in a data center using guest machines serving as source machines and destination machines for data packet flows and service machines serving as at least a subset of service nodes, the method comprising: On the host computer: configuring a first distributed forwarding element to forward data packets sent by the machine based on a network address specified by the machine, wherein the first distributed forwarding element: (1) defines a guest forwarding plane to forward data packets based on the network address specified by the machine, and (2) includes ports for receiving data packets from guest machines connected to the guest forwarding plane and providing data packets to guest machines connected to the guest forwarding plane; and Before the first distributed forwarding element forwards the data packet based on the network address specified by the machine, configure a second distributed forwarding element to forward the data packet sent by the machine to a set of one or more service nodes, wherein the second distributed forwarding element: (1) defines a service forwarding plane for forwarding the data packet to the service node before forwarding the data packet based on the network address specified by the machine, and (2) includes ports for providing data packets to service machines connected to the service forwarding plane and receiving data packets from service machines connected to the service forwarding plane, wherein each distributed forwarding element is implemented by at least one software forwarding element executed on the host computer and at least one other software forwarding element executed on at least one other host computer, wherein the service machine is separated from the guest forwarding plane by not defining a port for the service machine on the first distributed forwarding element, and the guest machine is separated from the service forwarding plane by not defining a port for each guest machine on the second distributed forwarding element, and The separation improves the security of the guest machine and the server machine by ensuring that the server machine cannot forward data packets directly to the guest machine and the guest machine cannot forward data packets directly to the server machine.

2. The method of claim 1, wherein: The first distributed forwarding element and the second distributed forwarding element are forwarding elements of the same type.

3. The method of claim 2, wherein: Each distributed forwarding element is a distributed software switch, and each software forwarding element is a software switch.

4. The method of claim 1, wherein: A software forwarding element on the host computer is configured to implement both the first distributed forwarding element and the second distributed forwarding element.

5. The method of claim 1, wherein: A first software forwarding element and a second software forwarding element on the host computer are configured to implement the first distributed forwarding element and the second distributed forwarding element, respectively.

6. The method of claim 1, wherein The first distributed forwarding element has a port for receiving a data packet from the machine, The second distributed forwarding element does not have a port for receiving data packets from the machine, but has a specific port for receiving data packets from a specific port agent, which is executed on the host computer to receive data packets sent by the machine and forward the data packets to the specific port.

7. The method of claim 6, wherein: A port proxy serves as an interface between a plurality of machines executing on the host computer and the second distributed forwarding element.

8. The method of claim 1, wherein: The second distributed forwarding element includes a service agent for each service node, the service agent executing on the host computer to perform service operations on data packets sent by the machine, and the service agent is used to format data packets provided to the service node associated with the service agent.

9. A machine-readable medium storing a program, which, when executed by at least one processing unit, implements the method according to any one of claims 1 to 8.

10. An electronic device comprising: A collection of processing units; as well as A machine-readable medium storing a program which, when executed by at least one of the processing units, implements the method according to any one of claims 1 to 8.

11. An electronic system comprising means for implementing the method according to any one of claims 1-8.

12. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images