Service operation linking method, device, system, and readable storage medium

Abstract

The present disclosure relates to service operation linking methods, devices, systems, and readable storage media. For a multi-tenant environment, some embodiments of the present invention provide a novel method for forwarding tenant traffic through a set of service machines to perform a set of service operations on the tenant traffic. In some embodiments, the method performs a classification operation on the tenant's data message stream to identify a set of service operations to be performed on the data message stream. For some data message streams, the classification operation selects an identified set of service operations from several candidate service operation sets, which are feasible service operation sets for similar data message streams of the tenant. In some embodiments, the classification operation is based on a set of attributes associated with the data message stream (e.g., a five-tuple identifier, i.e., a protocol and a source port and a destination port and a source IP address and a destination IP address).

Description

[0001] This application is a divisional application based on the patent application with application number 201880057484.X, application date October 23, 2018, and invention name “Service operation link method and computer program”. Background Art

[0002] Historically, middlebox services have been hardware devices implemented at one or more points in the network topology of an enterprise or data center. With the advent of software-defined networking (SDN) and network virtualization, traditional hardware devices cannot take advantage of the flexibility and control provided by SDN and network virtualization. Therefore, in recent years, some people have proposed various methods for providing middlebox services on the host. Some people also propose accessing middlebox services in different clouds (e.g., different data centers of different cloud providers). However, current service chaining solutions are not as robust as they should be to take advantage of today's various heterogeneous middlebox products. Summary of the invention

[0003] For a multi-tenant environment, some embodiments of the present invention provide a novel method for forwarding tenant traffic through a set of service machines to perform a set of service operations on the tenant traffic. In some embodiments, the service machine can be an independent service device or computer, and / or a service machine (e.g., a virtual machine, a container, etc.) executed on a host computer together with other service machines and / or tenant machines. Moreover, in some embodiments, one or more service machines are middlebox service machines that perform middlebox service operations (such as load balancing operations, firewall operations, intrusion detection operations, intrusion prevention operations, encryption / decryption operations, etc.).

[0004] In some embodiments, the method performs classification operations on the data message stream of the tenant so as to identify the service operation set to be performed on the data message stream. For some data message streams, the classification operation selects the identified service operation set from several candidate service operation sets, and these candidate service sets are feasible service operation sets for similar data message streams of the tenant. In some embodiments, the classification operation is based on the attribute set associated with the data message stream (for example, five tuple identifiers, that is, protocol and source port and destination port and source IP address and destination IP address). In some embodiments, the attribute set only includes the 2nd-4th layer header value of the data message stream (for example, five tuple identifiers, that is, protocol and source port and destination port and source IP address and destination IP address). However, in other embodiments, the attribute set includes other context attributes related to the data message stream, such as the traffic type of the data message stream (that is, the type of content carried in the data message stream), QoS level, layer 7 parameters, process identifier, user identifier, group identifier, etc.

[0005] In some embodiments, the service operations in the identified set of service operations must be performed according to a specific order. In order to express the sequential nature of these service operations, the identified set of service operations is referred to as an identified service operation chain below. After identifying the service chain operation of the data message stream, in some embodiments, the method embeds the identified service chain in the tunnel header of the data message used to encapsulate the stream. Similarly, in some embodiments, the method embeds a tenant identifier (e.g., embeds a VNI, a virtual network identifier) ​​in the tunnel header to specify that the data message is associated with a specific tenant (e.g., is sent from a specific tenant's machine). The method sends these encapsulated messages along the tunnel to a first service machine, which performs the first service operation in the identified service chain.

[0006] In some embodiments, the method identifies a service chain by identifying and embedding a set of network addresses (e.g., destination IP addresses) of a set of service machines that will perform service operations in the chain, and embeds the chain in a tunnel header. In some embodiments, the method embeds in the tunnel header a service operation descriptor (e.g., a tag) for each service machine identified in the tunnel header to explain the type of service operation performed by the service machine. In other embodiments, no such descriptor is embedded in the tunnel header. Moreover, in some embodiments, the method embeds in the tunnel header a service index value that identifies one of the embedded network addresses as the network address of the "current" service operation. As further described below, in some embodiments, the index is used to identify the next service machine for performing the next service operation in the service chain.

[0007] In some embodiments, a service machine addressed by a network address embedded in a tunnel is communicatively connected to service nodes connected to each other through a set of tunnels. In different embodiments, a service node can be a network element of different types. For example, in some embodiments, a service node can be (1) a host computer on which a service machine executes, (2) an independent forwarding element connected to a service machine, or (3) a service device that performs both service node functions (to connect to other service nodes via tunnels) and service machine functions (to perform one or more service operations). Similarly, in some embodiments, a service node can be connected to two or more service machines in an identified service chain.

[0008] Thus, after identifying a service chain for a data message and encapsulating the data message with a tunnel header containing the network addresses of a set of service machines that perform the service operations in the identified service chain, in some embodiments, the method passes the encapsulated data message along the tunnel to a service node that is communicatively connected to a first service machine that performs a first service operation in the identified service chain on the data message. In these embodiments, the embedded network addresses of the service machines define a service function path (SFP) in the tunnel header. The SFP, together with the service nodes (the service machines connected to the SFP) and the tunnels connecting these service nodes, define a service function chain (SFC) for the data message flow.

[0009] At each service node along the SFP, including the service node connected to the first service machine, the service node examines the tunnel header and determines that the received data message is addressed to a service machine that is communicatively connected to the service node. In some embodiments, the service node makes this determination by extracting a service index from the tunnel, using the index to retrieve the network address of the current service machine that must perform the service operation, and then determining that the service machine is the service machine connected to the service node.

[0010] Once the service node determines that the received data message is addressed to one of its associated service machines, the service node then removes the tunnel header (i.e., decapsulates the received data message), stores the information in the header (e.g., SFP and service index) in a connection storage device for later reuse, and provides the data message to the identified connection service machine for processing. As further described below, in some embodiments, the stored information includes information for the service node to reassemble the tunnel header in the event that the tunnel header needs to be recapsulated in order to forward the data message to another service node after it has been processed by the service machine.

[0011] Once the service machine performs its service operation on the data message, it returns the data message to the service node. In some cases, the service machine may instruct the service node to discard the data message based on its service operation. In some embodiments, the service node can be connected to two or more service machines that perform two or more consecutive service operations in a service chain. In this case, the service node provides the data message (in its current decapsulated state) to the next service machine in the service chain (connected to the service node) after receiving the data message from the previous service machine in the chain (assuming that the previous service chain did not discard or instruct the service node to discard the data message). In some embodiments, the service node determines that the next service machine is also connected to it after receiving the data message from the previous service machine connected to it. In other embodiments, the service node makes this determination before passing the data message to any service machine connected to it (for example, when it receives a data message through a tunnel, it identifies the next N service machines in the service chain to which it is connected when it receives the data message).

[0012] When a service node receives a data message from a connected service machine and the service machine for the next service operation in the service chain is not connected to the service node, the service node resolves the network address of the next service machine on the SFP list (stored in its connection storage for the data message) to the underlying tunnel that terminates at the service node connected to the next service machine. In some embodiments, the current service node identifies the network address of the next service machine by using the service index embedded in the tunnel header of the data message and now stored in the connection storage of the current service node.

[0013] After resolving the network address of the next service machine to another underlying tunnel, the current service node sends the re-encapsulated data message to the next service node along the underlying tunnel. The tunnel header of the re-encapsulated data message includes the SFP list contained in the original tunnel header received by the current service node. In some embodiments, the current service node modifies the SFP list to remove any network addresses of any service machines that are connected to the current service node and perform service operations in the service chain on the data message. In other embodiments, the current service node does not remove the network addresses of any of its associated service machines that perform service operations on the data message, but only adjusts the service index in the tunnel header to identify the network address of the next service machine that must perform the next service operation in the service chain.

[0014] To formulate a tunnel header for the re-encapsulated data message, in some embodiments, the current service node retrieves information stored in the tunnel header of the received data message from its connected storage device and updates the information (e.g., updates the SFP list or adjusts the service index). In some embodiments, the service node decrements the service index as the service machine network addresses are identified in the embedded SFP list in reverse order, with the address of the first service machine appearing last in the list and the address of the last service machine appearing first in the list.

[0015] In some cases, the last service node is communicatively connected to the destination machine of the data message (e.g., the last SVM and the destination GVM execute on the same host computer) without having to traverse an intermediate routing / switching fabric. In these embodiments, the last service node provides the decapsulated data message to the destination machine. In other cases, the last service node must traverse an intermediate routing / switching fabric to reach the destination of the message. In these cases, the last service node, after receiving the processed data message from the last service machine, forwards the data message to its destination through the intermediate routing fabric. In some of these embodiments, the last service node forwards the data message to its destination through another tunnel.

[0016] Some embodiments use the Generic Network Virtualization Encapsulation (Geneve) tunneling protocol to carry service function path information (e.g., service index and service machine network address list) and tenant identifiers. In some embodiments, the SFP information is embedded in a new Geneve Service Function List Option TLV (type, length, value) for use between service nodes (referred to as Network Virtualization Edge, NVE) that perform service forwarding operations in the same Layer 3 Network Virtualization Overlay (NVO3) domain.

[0017] In addition to embedding the SFP information and tenant identifier in the data message tunnel header, the method of some embodiments also captures and embeds context metadata in these tunnel headers so that some or all service machines along the SFP can process to perform their service operations. For example, the method can embed the data message traffic type, and the service machine can base its service operation on the traffic type (for example, perform its load balancing operation or firewall operation based on the traffic type). Other examples of context metadata include generated QoS levels, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.

[0018] In addition to basing its service operations on metadata embedded in a received data message's tunnel header, in some embodiments, a service machine along the SFP may also generate metadata and provide the metadata to its associated service node for embedding in a tunnel header that the service node uses to re-encapsulate the data message before sending the data message along the SFP. For example, the service machine may be a DPI machine that identifies the traffic type and provides the traffic type for embedding in a data message tunnel header for use by subsequent service machines in performing their operations. Alternatively, the service machine may provide or modify a provided QoS level for embedding in a data message tunnel header for use by subsequent service machines in performing their operations.

[0019] In some embodiments, after receiving the metadata from the service machine, the service node associated with the service machine may have to perform a new classification operation to verify the remaining service operations and / or remaining service machines in the SFP list, or re-assign these remaining operations and / or service machines. In some embodiments, the service node cannot perform such a classification operation, and therefore forwards the data message and metadata (e.g., a tunneled encapsulated data message with metadata) to another service node to perform the classification operation. After verifying or re-assigning the rest of the SFP list, the service node (or the service node associated with the service machine or another service node performing the classification operation, if any) forwards the data message in the tunnel to the service node associated with the next service machine in the SFP list.

[0020] In some embodiments, the above methods are used to express service chains in a single-tenant environment. Thus, one of ordinary skill in the art will recognize that some embodiments of the present invention are equally applicable to single-tenant data centers. Conversely, in some embodiments, when an entity (e.g., a company) is a tenant in multiple different data centers of different providers, the above methods are used to perform service chain specifications across different data centers of different data center providers. In these embodiments, the tenant identifiers embedded in the tunnel header must be unique across data centers or must be translated as they traverse from one data center to another.

[0021] The foregoing summary is intended to serve as a brief introduction to some embodiments of the present invention. This is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The following detailed description and the drawings mentioned in the detailed description will further describe the embodiments described in the summary of the invention as well as other embodiments. Therefore, in order to understand all embodiments described by this document, a comprehensive review of the summary of the invention, the detailed description, the drawings and the claims is required. Moreover, the claimed subject matter is not limited by the illustrative details in the summary of the invention, the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are set forth in the following figures.

[0023] Figure 1 An example of specifying and using a service operation chain according to some embodiments of the invention is illustrated.

[0024] Figure 2 The new Geneve tunnel header of some embodiments is illustrated.

[0025] Figure 3 The new Geneve base header of some embodiments is illustrated.

[0026] Figure 4 A new SFL (Service Capabilities List) options TLV header of the Geneve tunnel header of some embodiments is shown.

[0027] Figure 5 The SFL Flags field of the new SFL Options TLV header of some embodiments is illustrated.

[0028] Figure 6 The HMAC (Hash Message Authentication Code) sub-TLV of the new SFL Options TLV of some embodiments is illustrated.

[0029] Figure 7 The new Geneve NSH (Network Services Header) metadata options TLV of some embodiments is illustrated.

[0030] Figure 8 The processing performed by a portal service node in some embodiments is conceptually illustrated.

[0031] Fig. 9 The diagram illustrates the processing performed by a service node when it receives a Geneve encapsulated data message.

[0032] Fig.10 Illustrated is a host computer used in some embodiments to execute the virtual machines, service machines, and service nodes of some embodiments.

[0033] Fig.11 An example of how service nodes may be managed in some embodiments is illustrated.

[0034] Fig.12 and Fig.13 An example is illustrated for forwarding and processing metadata related to service operations performed by a service machine in some embodiments.

[0035] Fig.14A computer system for implementing some embodiments of the invention is conceptually illustrated. DETAILED DESCRIPTION

[0036] In the following detailed description of the present invention, many details, examples and embodiments of the present invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the present invention is not limited to the embodiments set forth, and the present invention can be practiced without some of the specific details and examples discussed.

[0037] For a multi-tenant environment, some embodiments of the present invention provide a novel method for specifying a set of service operations that a set of service machines must perform on a tenant's data message stream. In some embodiments, the service machine can be an independent service device or computer, and / or a service machine (e.g., a virtual machine, a container, etc.) executed on a host computer together with other service machines and / or tenant machines. Moreover, in some embodiments, one or more service machines are middlebox service machines that perform middlebox service operations (such as load balancing operations, firewall operations, intrusion detection operations, intrusion prevention operations, encryption / decryption operations, etc.).

[0038] In some embodiments, the method performs a classification operation on the tenant's data message stream in order to identify a sequence of service operations to be performed on the data message stream (hereinafter also referred to as a service operation chain). After identifying the sequence, in some embodiments, the method embeds the identified sequence in a tunnel header, which is used to encapsulate the data message of the stream. In some embodiments, the method identifies the service chain by identifying and embedding a set of network addresses (e.g., a destination IP address) of a set of service machines that will perform the service operations in the chain, and embeds the chain in a tunnel header. In these tunnel headers, the method also embeds a tenant identifier (e.g., embeds a VNI, a virtual network identifier) ​​to specify that the data message is associated with a specific tenant (e.g., is sent from a specific tenant's machine). The method then sends these encapsulated messages along the tunnel to a service machine, which performs the first service operation in the identified service chain.

[0039] In some embodiments, a service machine addressed by a network address embedded in a tunnel is communicatively connected to service nodes connected to each other through a set of tunnels. In different embodiments, a service node can be a network element of different types. For example, in some embodiments, a service node can be (1) a host computer on which a service machine executes, (2) an independent forwarding element connected to a service machine, or (3) a service device that performs both service node functions (to connect to other service nodes via tunnels) and service machine functions (to perform one or more service operations). Similarly, in some embodiments, a service node can be connected to two or more service machines in an identified service chain.

[0040] Thus, after identifying a service chain for a data message and encapsulating the data message with a tunnel header that includes the network addresses of a set of service machines that perform service operations in the identified service chain, in some embodiments, the method forwards the encapsulated data message along a tunnel to a service node that is communicatively connected to a first service machine that performs a first service operation in the identified service chain on the data message. In some embodiments, an outer portion of the tunnel header of the encapsulated message identifies a network address (e.g., an IP address) of a service node of the first service machine (or a virtual tunnel endpoint (VTEP) of the service node of the first service machine) as a destination network address (e.g., a destination IP address) of the encapsulated data message.

[0041] As used in this document, a data message refers to a collection of bits in a specific format that is sent across a network. One of ordinary skill in the art will recognize that the term "data message" is used in this document to refer to a collection of bits in various formats that can be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Similarly, as used in this document, references to L2, L3, L4, and L7 layers (or Layer 2, Layer 3, Layer 4, Layer 7) are references to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open Systems Interconnection) layer model, respectively.

[0042] Figure 1 An example of specifying and using a service operation chain according to some embodiments of the present invention is illustrated. In this example, five service machines 130-138 perform five service operations on a data message 100 sent from a guest virtual machine (GVM) 102 executing on one host 104 in a multi-tenant data center to another GVM 106 executing on another host 108 in the same data center. The GVM belongs to a tenant associated with a specific virtual network identifier VNI in the data center.

[0043] Four of these service machines 130-136 are service virtual machines (SVMs) executed on three hosts 112-116 in the data center, and two of these SVMs, SVMs 132 and 134, are executed on one host 114. The fifth service machine is a service device 138. All of these service machines 130-138 perform middlebox service operations. Figure 1 In the example shown, data message 100 is encapsulated with tunnel headers 140-146 and forwarded to service machines 130-138 by service nodes 120-126 and 138 connected to each other by tunnels 150-156. Service device 138 acts as its own service node.

[0044] Like most tunnel headers, each of tunnel headers 140-146 has an outer portion (not shown) that identifies the IP addresses of the source and destination endpoints of the tunnel as the source IP address and destination IP address of data message 100 encapsulated by the tunnel header. In some embodiments, the source and destination tunnel endpoints are VTEPs associated with the two service node endpoints of the tunnel. For example, the outer portion of tunnel header 140 identifies the IP address of the VTEP of ingress service node 120 as the source IP address of the encapsulated data message, while identifying the IP address of the VTEP of service node 122 as the destination IP address of the encapsulated data message.

[0045] Each of the tunnel headers 140-146 includes the tenant's VNI and a service function path (SFP) list that identifies the remainder of the service chain at each service node along the service path. As shown, in some embodiments, the SFP list is represented by the IP address of the service machine 130-138. The SFP list, along with the service nodes (the service machines connected to the SFPs) and the tunnels connecting these service nodes, defines the service function chain (SFC) of the data message flow.

[0046] exist Figure 1 In the embodiment, service node 120 is an ingress service node that receives data messages from source GVM 102. As described further below, in some embodiments, the service node is formed by a software forwarding element (SFE) and a service coordinator executing on source host computer 104. The SFE intercepts data messages sent by source GVM 102 and forwards them to the service coordinator, which then performs classification operations on the data messages to identify a set of service operations to be performed on the data message flow.

[0047] For some data message streams, the classification operation selects an identified set of service operations from several candidate sets of service operations, which are feasible sets of service operations for similar data message streams of the tenant. In some embodiments, the classification operation is based on a set of attributes associated with the data message stream. In some embodiments, the set of attributes only includes layer 2-4 header values ​​of the data message stream. However, in other embodiments, the set of attributes includes contextual attributes related to the data message stream, such as the traffic type of the data message stream (i.e., the type of content carried in the data message stream), QoS level, layer 7 parameters, process identifier, user identifier, group identifier, etc.

[0048] In some embodiments, after identifying the service chain of the intercepted data message, the ingress service node 120 embeds the identified service chain into the tunnel header 140 it uses to encapsulate the data message 100. As shown, in some embodiments, the service node identifies and embeds the service chain based on the IP addresses of the service machines 130-138 that will perform the service operations in the chain. In addition to the IP addresses of these service machines, the ingress service node 120 embeds a tenant identifier (in this case, the VNI of a specific tenant) in the tunnel header to specify that the data message 100 is associated with a specific tenant (e.g., is sent from a GVM of a specific tenant).

[0049] In some embodiments, the ingress service node 120 embeds other attributes (e.g., other network addresses) in the tunnel header 140. Also, in some embodiments, the ingress service node 120 embeds a service operation descriptor (e.g., a label) for each service machine in the tunnel header 140 to explain the type of service operation performed by the service machine. In other embodiments, no such descriptor is embedded in the tunnel header. Also, in some embodiments, the ingress service node 120 embeds a service index value in the tunnel header that identifies one of the embedded network addresses as the network address of the "current" service operation. As further described below, in some embodiments, subsequent service nodes use this index to identify the next service machine for performing the next service operation in the service chain.

[0050] After identifying the service chain for data message 100 and encapsulating the data message with tunnel header 140, ingress service node 120 passes the encapsulated data message along tunnel 150 to service node 122 communicatively connected to first service machine 130, which performs a first service operation in the identified service chain on the data message. As described above, in some embodiments, the outer portion of tunnel header 140 identifies the IP address of the VTEP of ingress service node 120 as the source IP address of the encapsulated data message, while identifying the IP address of the VTEP of service node 122 as the destination IP address of the encapsulated data message.

[0051] At each service node 122-126 along the SFP, including service node 138, the service node examines the tunnel header and determines that the received data message is addressed to a service machine that is communicatively connected to the service node. In some embodiments, the service node makes this determination by extracting a service index from the tunnel, using the index to retrieve the network address of the current service machine that must perform the service operation, and then determining that the service machine is the service machine that is connected to the service node.

[0052] Once the service node determines that the received data message is addressed to one of its associated service machines, the service node removes the tunnel header (i.e., decapsulates the received data message), stores the information in the header (e.g., SFP and service index) in a connection storage device for later reuse, and provides the data message to the identified connection service machine for processing. As further described below, in some embodiments, the stored information includes information for the service node to reassemble the tunnel header in the event that the tunnel header needs to be recapsulated in order to forward the data message to another service node after it has been processed by the service machine.

[0053] Once the service machine performs its service operation on the data message, it returns the data message to the service node. In some cases, the service machine may instruct the service node to discard the data message based on its service operation. In some embodiments, a service node (e.g., service node 124) may be connected to two or more service machines (e.g., service machines 132 and 134) that perform two or more consecutive service operations in a service chain. In this case, the service node (e.g., node 124) provides the data message (in its current decapsulated state) to the next service machine in the service chain (e.g., service machine 134) after receiving the data message from the previous service machine in the chain (e.g., service machine 132) (assuming that the previous service chain did not discard or instruct the service node to discard the data message). In some embodiments, the service node determines that the next service machine is also connected to it after receiving the data message from the previous service machine connected to it. In other embodiments, the service node makes this determination before passing the data message to any service machine connected to it (e.g., when it receives a data message through a tunnel, it identifies the next N service machines in the service chain that are connected to it when it receives the data message).

[0054] When a service node receives a data message from a connected service machine and the service machine for the next service operation in the service chain is not connected to the service node, the service node resolves the network address of the next service machine on the SFP list (stored in its connection storage for the data message) to the underlying tunnel that terminates at the service node connected to the next service machine. In some embodiments, the current service node identifies the network address of the next service machine by using the service index embedded in the tunnel header of the data message and now stored in the connection storage of the current service node.

[0055] After resolving the network address of the next service machine to another underlying tunnel, the current service node sends the re-encapsulated data message to another service node along the underlying tunnel. Similarly, the outer portion of the tunnel header identifies the IP addresses of the source and destination service nodes of the tunnel as the source and destination IP addresses of the encapsulated data message. The tunnel header also includes the SFP contained in the original tunnel header received by the current service node minus any network address of any service machine connected to the current service node and performing a service operation in the service chain on the data message. Therefore, the SFP list 162 inserted by the service node 122 in the tunnel header 142 does not include the IP address of the service machine 130. Similarly, the SFP list 164 inserted by the service node 124 in the tunnel header 144 does not include the IP addresses of the previous service machines 130, 132 and 134, and the SFP list 166 inserted by the service node 138 in the tunnel header 146 does not include the IP addresses of the service machines 130, 132, 134 and 138.

[0056] In other embodiments, the service node does not remove the network address of its associated service machine before sending out the re-encapsulated data message. As further described below, in these embodiments, the current service node merely adjusts the service index in the tunnel header to identify the network address of the next service machine that must perform the next service operation in the service chain. To formulate the tunnel header for the re-encapsulated data message, in some embodiments, the current service node retrieves information stored in the tunnel header of the received data message from its connected storage device and updates the information (e.g., updates the SFP list or adjusts the service index). In some embodiments, the service node decrements the service index as the service machine network addresses are identified in reverse order in the embedded SFP list, with the address of the first service machine appearing last in the list and the address of the last service machine appearing first in the list.

[0057] In some cases, the last service node is communicatively connected to the destination machine of the data message (e.g., the last SVM and the destination GVM execute on the same host computer) without having to traverse an intermediate routing / switching fabric. In these embodiments, the last service node provides the decapsulated data message to the destination machine. In other cases, the last service node must traverse an intermediate routing / switching fabric to reach the destination of the message. Figure 1After receiving the processed data message 100 from the last service machine 136, the service node forwards the data message along the tunnel 158 to its destination through the intermediate routing architecture. To send the data message along the tunnel, the service node encapsulates the data message with a tunnel header 148, which includes the tenant's VNI but does not include the SFP list because all service operations have been performed on the data message.

[0058] Some embodiments use the Generic Network Virtualization Encapsulation (Geneve) tunneling protocol to carry service function path information (eg, service index and service machine network address list) and tenant identifiers.

[0059] In some embodiments, SFP information is embedded in a new Geneve Service Capability List Option TLV (Type, Length, Value) for use between service nodes (referred to as Network Virtualization Edge, NVE) that perform service forwarding operations in the same Layer 3 Network Virtualization Overlay (NVO3) domain.

[0060] In addition to embedding the SFP information and tenant identifier in the data message tunnel header, the service node of some embodiments also captures and embeds context metadata in these tunnel headers so that some or all service machines along the SFP can process to perform their service operations. For example, the service node can embed the traffic type of the data message, and the service machine can base its service operation on the traffic type (for example, perform its load balancing operation or firewall operation based on the traffic type). Other examples of context metadata include generated QoS levels, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.

[0061] In addition to basing its service operations on metadata embedded in a received data message's tunnel header, in some embodiments, a service machine along the SFP may also generate metadata and provide the metadata to its associated service node for embedding in a tunnel header that the service node uses to re-encapsulate the data message before sending the data message along the SFP. For example, the service machine may be a DPI machine that identifies the traffic type and provides the traffic type for embedding in a data message tunnel header for use by subsequent service machines in performing their operations. Alternatively, the service machine may provide or modify a provided QoS level for embedding in a data message tunnel header for use by subsequent service machines in performing their operations.

[0062] In some embodiments, the metadata returned by the service machine to the service node includes an SFP list identifier that the service node uses to identify the SFP list associated with the processed data message. In some of these embodiments, the service machine has the SFP list identifier because when the service node provides the data message to the service machine, it also provides the SFP list identifier to the service machine.

[0063] In some embodiments, after receiving the metadata from the service machine, the service node associated with the service machine may have to perform a new classification operation to verify the remaining service operations and / or remaining service machines in the SFP list, or re-assign these remaining operations and / or service machines. In some embodiments, the service node cannot perform such a classification operation, and therefore forwards the data message and metadata (e.g., a tunneled encapsulated data message with metadata) to another service node to perform the classification operation. After verifying or re-assigning the rest of the SFP list, the service node (or the service node associated with the service machine or another service node performing the classification operation, if any) forwards the data message in the tunnel to the service node associated with the next service machine in the SFP list.

[0064] Figure 2 The Geneve tunnel header 200 of some embodiments is illustrated. As shown, the tunnel header includes an outer header 205, a protocol field 210, a Geneve base header 215, an SFL (Service Function List) option TLV 220, and a metadata option TLV 225. As described above, the outer header 205 includes network addresses in the underlying network that allow the encapsulated data message to traverse the underlying network and reach the tunnel destination endpoint (e.g., the source and destination IP addresses of the two tunnel endpoints, where the designation of the source and destination depends on the direction of the message flow). The protocol field 210 specifies the UDP protocol and the properties associated with the protocol.

[0065] As reference Figure 3 As further described, the Geneve base header 215 is 64 bits wide and stores several tunnel parameters, including a tenant identifier and an option length. The Geneve base header is followed by zero or more options in TLV format. Figure 2 In the example shown, the options include a new SFL option TLV and a new metadata option TLV. Figure 4 As further described, the SFL Option TLV includes a list of SFPs and a service index. The Metadata Option TLV stores metadata associated with the data message in the NSH metadata format. Figure 7Describe the metadata option TLV. Other embodiments do not use the metadata option TLV, but instead deliver metadata associated with the data message in a L2 VLAN header or a Q-in-Q header or another encapsulation known to both the service node and the service machine.

[0066] Figure 3 The Geneve base header 215 of some embodiments is illustrated. As shown, in some embodiments, this header is 64 bits wide and stores several tunnel parameters. Its version field 305 is a 2-bit value that specifies the current Geneve tunnel version in use. A tunnel endpoint that receives a message with an unknown version will discard the message. Non-endpoint devices that process Geneve packets with unknown version numbers will treat these packets as UDP packets with unknown payloads.

[0067] The Option Length field 310 specifies the length of the Option field, expressed in multiples of four bytes, excluding the eight-byte Fixed Tunnel Header. This makes the total size of the Geneve header a minimum of 8 bytes and a maximum of 260 bytes. The start of the Payload Header can be found using the offset from the end of the Basic Geneve Header.

[0068] The O bit 315 specifies whether the data message is an OAM frame containing a control message instead of a data payload. The endpoint does not forward the payload of this message, and the transit device does not attempt to interpret or process it. Since control messages are infrequent, the endpoint typically directs these messages to a high-priority control queue. The transit device does not change forwarding behavior, such as ECMP link selection, based on this bit.

[0069] The C bit 320 is a critical option field which, when set, indicates the presence of options. One or more options have this critical bit set. If this bit is set, the tunnel endpoint will parse the option list to interpret all critical options. After this bit is set, if the endpoint does not recognize the option, they must discard the encapsulated message. On devices that do not support option parsing, frames will be discarded based on the "C" bit in the base header. If this bit is not set, the tunnel endpoint can strip all options using the "Option Length (Opt Len)" and forward the decapsulated frame. Transit devices will not discard or modify packets based on this bit.

[0070] The first set of reserved bits 325 are 6 bits that must be zero when transmitting and are ignored when receiving. The protocol type bits 330 are 16 bits that indicate the type of protocol data unit that appears after the Geneve header. The VNI bits 335 represent a 24-bit identifier for a unique element of the virtual network. In many cases, this may represent an L2 segment; however, the control plane defines the forwarding semantics of the decapsulated packets. The VNI may be used as part of an ECMP forwarding decision when load balancing across CPUs, or as a mechanism to distinguish overlapping address spaces contained in encapsulated packets. The second set of reserved bits are 8 bits that must be zero when sending and are ignored when receiving.

[0071] The Geneve base header is followed by zero or more TLV formatted options. Each option consists of a four-byte option header and a variable amount of option data interpreted according to the option type. Figure 4 The SFL Option TLV header 220 of some embodiments is shown. In this example, the SFL Option TLV is 8 bytes. The Option TLV 220 contains the parameters required to specify the SFP list and service index. As shown, the Option Header 405 includes a 16-bit Option Class, which is a namespace for a Type field 410 that indicates the format of the data contained in the option. The Type field is assigned by IANA when it creates the "Geneve Option Class" registry to assign identifiers to organizations, technologies, and vendors interested in creating types of options.

[0072] The three R bits 415 are option control flags that are reserved for future use. These must be zero when transmitting and are ignored when receiving. The length field includes 5 bits. The length of the option is expressed in multiples of four bytes, but does not include the option header. The total length of each option can be between 4 and 128 bytes ([1 to 32]*4). In some embodiments, a data message in which the total length of all options is not equal to the option length in the base header is invalid and is discarded if received by an endpoint.

[0073] As shown, the variable option portion 475 of the SFL option TLV header 220 includes a four-byte header field 425, an SFP list 450, and an optional subtype TLV 455. The SFP list provides a 32-bit or 128-bit address of a service machine in a service chain embedded in the Geneve tunnel header. The header field 425 includes a version field 430, an SFL flag field 435, a reserved bit 440, and an SFP index field 445.

[0074] The version field 430 is a 5-bit value that specifies the current SFL Option TLV version in use. A tunnel endpoint that receives a message with an unknown version will discard the message. The reserved bit 400 should not be set on transmission and will be ignored on reception. The SFP index contains the service index in the SFP list. In some embodiments, the service node decrements the service index after one of the node's service machines has processed the data message. This is because the SFP list is encoded starting from the last hop of the path, that is, the first element of the list (SF List [0]) contains the last service function of the path, and the last element of the SF list (SF Category [n]) contains the first service function in the path.

[0075] The SFL flags field is in Figure 5 435. The first value 505 in this field is the HMAC flag, while the other values ​​510 are not used. When set, the first value 505 specifies the presence of an HMAC (Hash Message Authentication Code) sub-TLV, and encodes it as the last sub-TLV 455. The HMAC sub-TLV carries parameters for a cryptographic hash function and a secret cryptographic key. In some embodiments, HMAC is used to verify data integrity and message authentication simultaneously.

[0076] In some embodiments, only service nodes that are the destination of Geneve tunnel packets will check the SFP list defined in the SFL Options TLV of the tunnel header. In deployments using Geneve SFL Options TLV, only service nodes within a single NVO3 management domain are trusted nodes enabled to view these TLVs. Service nodes will ignore Geneve SFL lists created by external parties based on information from a network virtualization authority or some other trusted control plane information.

[0077] To prevent non-participating service nodes from using the Geneve SFL Option TLV, some embodiments use an optional security sub-TLV based on a keyed hash message authentication code (HMAC) in the SFL Option TLV. The HMAC optional sub-TLV is located at the end of the Geneve Service Capability List Option TLV. The purpose of the HMAC optional sub-TLV is to verify the validity, integrity, and authorization of the Geneve SFL Option TLV itself.

[0078] The HMAC sub-TLV shall contain (1) a 32-bit wide HMAC key ID, and (2) a 256-bit wide HMAC field. The HMAC field is the output of an HMAC calculation using the pre-shared key identified by the HMAC key ID, and the output of a text formed by concatenating together (1) the source IPv4 / IPv6 generic tunnel address, (2) version and flags data, (3) the HMAC key ID, and (4) all addresses in the SFP list. The HMAC key ID field serves as an index to the correct combination of pre-shared key and hash algorithm, and an expected value of 0 indicates that the HMAC field is not present. In some embodiments, HMAC selection of the hash algorithm and pre-shared key management follows the procedures described in Draft-ietf-6man-segment-routing-header.

[0079] Figure 6 The HMAC sub-TLV 455 for some embodiments of the present invention is illustrated. This sub-TLV contains HMAC information. The type field 605 will be assigned by IANA. The length field 610 is eight bits to represent 38 octets of HMAC information. The reserved bit 615 is 2 octets wide. These bits should not be set when transmitting and should be ignored when receiving. The HMAC key ID 620 is 4 octets wide, and the HMAC parameter field 625 is 32 octets wide. When the HMAC sub-TLV is present, the H flag is set and the HMAC sub-TLV is encoded as the last sub-TLV. When the H flag is set, the service node that checks the Geneve SFP list option TLV must find the HMAC sub-TLV in the last 38 octets of the option TLV.

[0080] Figure 7 The new Geneve NSH metadata option TLV 225 of some embodiments is illustrated. As shown, the TLV 225 has a 32-bit header that includes Figure 4 The first 32 bits of the SFL Option TLV 220 are the same as the first 32 bits of the Option, Type, Reserved, and Length fields 405, 410, 415, and 420. The Type field 410 in the NSH Metadata Option TLV header specifies one of the two metadata types of the NSH metadata format: MD Type 1 or MD Type 2. Figure 7 As shown, the TLV 225 also includes fixed or variable size option data 725. As shown, the option data is either a fixed 16-byte value for MD Type 1 metadata or a variable size for MD Type 2 metadata.

[0081] Figure 8A process 800 is conceptually illustrated as being performed by an ingress service node (e.g., node 120) of some embodiments. The process identifies an SFP list for a data message and embeds the SFP list in a Geneve tunnel header, uses the tunnel header to encapsulate the data message, and then sends the encapsulated message along a tunnel to a service node that is communicatively connected to a first service machine. In some embodiments, the ingress service node is formed by an SFE (e.g., a software switch) and a service coordinator executing on a source host computer (e.g., computer 104) and one or more GVMs (e.g., GVM 102). In some embodiments, the GVM is communicatively connected to the SFE, which forwards data messages to and from the GVM.

[0082] As shown, process 800 initially receives a data message from a tenant GVM (e.g., GVM 102) executing on its host (e.g., 104). In some embodiments, process 800 receives the data message when the SFE intercepts the data message and forwards it to the service coordinator. Next, based on a set of attributes associated with the data message, the process (e.g., service coordinator) performs (at 810) a classification operation on the data message to identify a set of service operations to be performed on the data message flow.

[0083] For some data message flows, the classification operation selects an identified set of service operations from a number of candidate sets of service operations that are feasible sets of service operations for similar data message flows of the tenant. Likewise, in different embodiments, the classification operation is based on different sets of data message flow attributes. In some embodiments, the set includes only layer 2-4 header values ​​of the data message flow, while in other embodiments, the attribute set includes contextual attributes related to the data message flow, such as the traffic type of the data message flow (i.e., the type of content carried in the data message flow), QoS level, layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.

[0084] For example, in some embodiments, the processing performs the classification operation by comparing one or more attributes of the data message (e.g., a 5-tuple identifier of the data message and / or associated metadata) to rule identifiers of several service rules stored in a rule store. In addition to its rule identifier, each rule also specifies a set of service actions, which in some embodiments are specified as IP addresses of service machines for performing service actions. As described above, in some embodiments, each IP address can be designated as a VIP that specifies a service cluster of two or more service machines that perform the same service operation. In some embodiments, the service node performs a load balancing operation to convert the VIP of each service machine cluster into a DIP for the data message stream being processed. Reference will be made to the following. Fig.10 A host architecture for capturing and using context attributes is further described.

[0085] After identifying (at 810) the service chain of the intercepted data message, the process 800 embeds (at 815) the identified service chain in the SFL Options TLV 220 of the Geneve tunnel header 200 that will be used to encapsulate the received data message. As described above, the SFL Options TLV stores the service chain according to the IP addresses of the service machines that are to perform the service operations in the chain. Also, as further described above, the SFL Options TLV stores these IP addresses in reverse order, with the address of the first service machine appearing last in the list and the address of the last service machine appearing first in the list.

[0086] In the SFL Options TLV, the process also stores (at 815) a service index value 445. The process sets this value to identify the last network address in the SFP list, which is the address of the first service machine. The service index value is used to identify the embedded network address of the "current" service operation, and in some embodiments, subsequent service nodes use this index to identify the next service machine for performing the next service operation in the service chain. In some embodiments, the process 800 embeds a service operation descriptor (e.g., a tag) with each service machine address in the SFL Options TLV to explain the type of service operation performed by the service machine. In other embodiments, no such descriptor is embedded in the tunnel header. If the Geneve tunnel needs to have its HMAC parameters set, the process also defines (at 815) an HMAC sub-TLV and sets its parameters.

[0087] At 815, the process 800 also embeds the VNI of the tenant associated with the source GVM that sent the received data message into the base header 215 of the Geneve tunnel header. After embedding the VNI, the process 800 identifies (at 820) one or more metadata attributes associated with the received data message and stores the identified metadata in the NSH metadata option TLV 225. In different embodiments, the process 800 identifies and embeds different metadata. Examples of such metadata include data message traffic type, QoS level, layer 7 parameters, process identifier, user identifier, group identifier, etc. Reference will be made to Fig.10 Further describes the host architecture for capturing and embedding metadata attributes.

[0088] The process embeds the metadata into the Geneve tunnel header because one or more services along the SFP may process the metadata in order to perform their service operations. In some cases, the service machines along the identified SFP do not use any metadata associated with the data message to perform their operations on the data message. In these cases, the process does not embed any metadata in the Geneve tunnel header, and therefore does not include the NSH Metadata Option TLV in the header.

[0089] Next, at 825, the process configures the rest of the Geneve tunnel header. For example, the process stores the L2 and L3 network addresses of its VTEP (e.g., the VTEP associated with its SFE) and the VTEP of the service node associated with the first service machine as source and destination network addresses in the outer portion header 205 of the tunnel header 200. At 825, based on the above discussion and the tunnel deployment configuration used, the process defines the UDP protocol in the Geneve tunnel header and sets any other fields that need to be defined.

[0090] After completing the tunnel header configuration (at 825), the process passes (at 830) the encapsulated data message along the Geneve tunnel to a service node (e.g., node 122) associated with a first service machine (e.g., service machine 130) that performs a first service operation in the identified service chain on the data message. To send the data message, in some embodiments, the process resolves the IP address of the first service machine to a tunnel that connects the ingress service node to the service node of the first service machine. After 825, the process ends.

[0091] In some cases, the ingress service node executes on a host computer that executes the source machine (e.g., source VM or container) for the data message and the first service machine or machines that will perform the first service operation or the first few service operations on the data message. In these cases, the ingress service node first passes the data message to the service machine or machines, then encapsulates the data message with a tunnel header and forwards it to the next service machine in the chain.

[0092] Fig. 9The process 900 that a service node performs upon receiving a Geneve-encapsulated data message is illustrated. In some embodiments, the service node is executed on a host computer together with one or more service machines (e.g., SVMs or service containers) and one or more tenant machines (e.g., GVMs). As shown, the process examines (at 905) a tunnel header and determines that the received data message is addressed to a service machine that is communicatively connected to the service node. In some embodiments, the service node makes this determination by extracting a service index from the tunnel, using the index to retrieve the network address of the current service machine that must perform the service operation, and then determining that the service machine is a service machine connected to the service node.

[0093] Once the process 900 determines (at 905) that the received data message is addressed to one of its associated service machines, the process 900 removes the tunnel header (i.e., decapsulates the received data message) (at 910) and stores the information in the header (e.g., SFP, service index, metadata, etc.) in a connected storage device for later reuse. At 915, in some embodiments, the process 900 provides the data message to the identified, connected service machine for processing and removes the IP address of the service machine from the received SFP list. In other embodiments, after providing the data message to the identified, connected service machine for processing, the process 900 does not remove the IP address of the service machine, but adjusts the current service index (e.g., in embodiments where the IP addresses of the service machines are defined in reverse order, the service index is decremented by 1).

[0094] When the extracted tunnel header contains metadata for the data message, and the service machine requires some or all of the metadata to perform its service operation, the process provides (at 915) the required metadata to the service machine. In some embodiments, the process provides the data message and / or metadata by passing one or more identifiers to the service machine that identify locations in memory where the data message and / or metadata are stored.

[0095] Once the service machine performs its service operation on the data message, the process receives (at 920) the processed data message at the service node. In some embodiments, the service machine returns metadata along with the processed data message. In some embodiments, the service machine provides the data message and / or metadata by passing one or more identifiers to the process 900, the one or more identifiers identifying the location in memory where the processed data message and / or metadata is stored. The returned metadata can be metadata generated by the service machine when processing the data message or data message stream, or metadata captured from a received data message or data message stream (e.g., through deep packet inspection).

[0096] In some embodiments, the metadata returned (at 920) by the service machine to the service node includes an SFP list identifier that the service node uses to identify the SFP list associated with the processed data message. In some of these embodiments, the service machine has the SFP list identifier because when the service node provided (at 915) the data message to the service machine, it also provided the SFP list identifier to the service machine.

[0097] To capture or generate metadata, in some embodiments, the service machine may need to process several data messages that are part of the same data message stream as the data message for which process 900 was performed. For example, the service machine may have to perform a soft connection termination to examine the contents of the data messages in the stream. Thus, in these embodiments, the service node may provide multiple data messages that are part of the same stream before it receives the processing results of the earlier provided data messages.

[0098] In some cases, the service machine may instruct the service node to discard the data message based on its service operation. Assuming that the service machine does not instruct the service node to discard the received data message, the process 900 determines (at 925) whether it should perform a classification operation to identify a new service chain for the data message based on the metadata returned (at 920) by the service machine. This operation is performed because in some embodiments, the service machine can return metadata that provides additional information about the data message being processed, and this additional information may require redefining the service chain for the data message.

[0099] In some embodiments, the determination (at 925) is based on whether the service machine returns a certain type of metadata when it returns the processed data message. When the process determines (at 925) that it must perform a classification operation, it performs (at 930) a classification operation to identify a new SFP list. In some embodiments, the process performs the classification operation by comparing one or more attributes of the data message (e.g., a 5-tuple identifier of the data message and the associated metadata) with the rule identifiers of several service rules stored in the rule store. In addition to its rule identifier, each rule also specifies a set of service actions, which in some embodiments are specified as the IP addresses of the service machines for performing the service actions. As described above, in some embodiments, each IP address can be specified as a VIP that specifies a service cluster of two or more service machines that perform the same service operation. In some embodiments, the service node performs a load balancing operation to convert the VIP of each service machine cluster into a DIP for the processed data message stream.

[0100] At 930, the process defines a Geneve tunnel header for the data message and embeds the SFP list it generates (at 930) in the tunnel header along with the VNI of the source tenant machine, the service index of the SFP list, any required metadata associated with the data message (possibly including metadata returned by the service machine at 920), and other Geneve tunnel attributes. Next, at 935, the process encapsulates the processed data message with the tunnel header defined at 930 and sends the encapsulated data message along the Geneve tunnel to the next service node associated with the first service machine in the reassigned SFP list. To send the data message, in some embodiments, the process resolves the IP address of the first service machine to a tunnel that connects the current service node to the service node of the first service machine. The outer portion of the tunnel header identifies the source IP address and the destination IP address as the IP addresses of the VTEPs of the current service node and the next service node. After 935, the process 900 ends.

[0101] When the process determines (at 925) that it does not need to perform a classification operation, the process determines (at 940) whether the SFP list for the data message is now empty. An empty SFP list would indicate that all service operations have been specified for the received data message. As described above, in some embodiments (at 915), the process 900 removes the IP address of the last service machine from the SFP list or adjusts the service index before providing (at 915) the data message to the service machine. In embodiments where the service machine IP address is not removed from the SFP list but the service index value is adjusted, the process makes this determination (at 940) by determining whether the service index value indicates that all service machines identified in the SFP list have processed the data message. For example, in some embodiments, the service index is decremented each time a data message is delivered to another service machine, and the process determines that the SFP list is empty when the service index or some other service index value associated with the empty SFP list reaches zero.

[0102] When the process determines (at 940) that the SFP list is empty, the process determines (at 945) whether the destination of the data message is on the same host computer (i.e., determines whether a machine having a network address (e.g., an L2-L4 address) specified in the header of the data message is executing on the same host computer). When the destination is on the same host computer, the process delivers (at 950) the processed data message to the destination computer and then ends. When the destination is not on the same host computer, the process (at 955) uses the destination IP address to identify a tunnel for forwarding the data message to its destination, then encapsulates the data message with a header for the tunnel, and then sends the data message along the tunnel to its destination (e.g., sending message 100 to SFE 128 of host 108 using tunnel 158 to deliver the data message to GVM 106). The outer portion of the tunnel header identifies the source IP address and the destination IP address as the IP addresses of the current service node and the destination machine (when it is a standalone machine) or the VTEP of the host computer on which the destination machine is executing. After 955, the process ends.

[0103] When the process determines (at 940) that the SFP list is not empty (i.e., determines that the list or service index still identifies at least one IP address of at least one other service machine that must perform at least one other service operation), the process determines (at 960) whether the service machine associated with the next IP address is executing on the same host computer. In some embodiments, the process identifies the network address of the next service machine by using the service index embedded in the tunnel header of the data message and now stored in the connection store of the current service node after being decremented at 915.

[0104] Process 900 performs a check at 950 because, in some embodiments, a service node (e.g., service node 124) may be connected to two or more service machines (e.g., service machines 132 and 134) that perform two or more consecutive services in a service chain. In this case, the process returns to 915 (e.g., service node 124) to provide the data message (in its current decapsulated state) to the next service machine in the service chain (e.g., service machine 134) after receiving the data message from the previous service machine in the chain (e.g., service machine 132) (assuming that the previous service chain did not drop, or did not instruct the service node to drop, the data message). In some embodiments, the process determines (at 960) that the next service machine is connected to it after receiving the data message from the previous service machine connected to it. In other embodiments, the process makes this determination (at 905) before passing the data message to any service machine connected to it (e.g., when it receives the data message through a tunnel, it identifies the next N service machines in the service chain to which it is connected when it receives the data message).

[0105] When the process determines (at 960) that the data message will no longer be processed next by another service machine executing on the same host computer, the process resolves (at 965) the network address of the next service machine on the SFP list (stored in its connected storage device for the data message) to an underlying tunnel that terminates at a service node connected to the next service machine. After resolving the network address of the next service machine to another underlying tunnel, the process sends (at 965) the re-encapsulated data message along the underlying tunnel to the next service node. The outer portion of the tunnel header identifies the source IP address and the destination IP address as the IP addresses of the VTEPs of the current service node and the next service node.

[0106] The tunnel header also includes the SFP list included in the original tunnel header of the current service node with an adjusted service index (e.g., with a decremented service index), or minus any network addresses of any service machines associated with the current service node that performs a service operation in the service chain on the data message (e.g., SFP list 162 inserted into tunnel header 142 by service node 122 does not include the IP address of service machine 130). To formulate the tunnel header for the re-encapsulated data message (at 935 or 965), in some embodiments, process 900 retrieves information stored in the tunnel header of the received data message from its attached storage device and updates the information if necessary (e.g., updates metadata based on metadata received from the service machine(s)).

[0107] For some service operations (e.g., firewall operations, load balancing operations) of some service machines, the service node stores the results of the service machine's operations in a cache storage device after processing a first data message so that the service node does not have to re-execute 900 for subsequent data messages that are part of the same stream as the first data message. When the service node receives another data message that is part of the same stream as the first data message, the process checks the cache storage device to determine whether it has already processed an earlier data message of the same stream. If so, the service node retrieves the results of the service machine's operations for the earlier data message from the cache storage device and uses the results for the data message it is processing.

[0108] Fig.10 The host computer 1000 for executing the GVM, service machine and service node of some embodiments is illustrated in some embodiments. The host computer executes attribute-based services with rich context in the data center. The host computer 1000 includes several GVMs 1005, SVMs 1032, software forwarding elements 1010, context engines 1050, service engines 1030, context-based service rule storage devices 1040, context attribute storage devices 1045, attribute mapping storage devices 1023, connection state cache storage devices 1025, MUX (multiplexer) 1027 and context engine policy storage devices 1043. The service engine 1030 includes a service coordinator 1022, an encryption engine 1024, a load balancer 1026, a firewall engine 1028 and one or more other service engines (not shown).

[0109] exist Fig.10 In some embodiments, the context engine 1050, the software forwarding element 1010, the service engine 1030, the context-based service rule storage 1040, the connection state cache storage 1025, the context engine policy storage 1043, and the MUX 1027 operate in the kernel space of the hypervisor, while the GVM 1005 operates in the user space of the hypervisor. In other embodiments, one or more service engines are user space modules (e.g., service VMs).

[0110] In some embodiments, GVMs 1005 are used as data endpoints in a data center. Examples of such machines include web servers, application servers, database servers, etc. In some cases, all VMs belong to one entity, such as an enterprise operating the host. In other cases, the host 1000 operates in a multi-tenant environment (e.g., in a multi-tenant data center), and different GVMs 1005 may belong to one tenant or multiple tenants.

[0111] Each GVM 1005 includes a GI agent 1048 that communicates with a context engine 1050 to provide a set of context attributes to the engine and receives instructions and queries from the engine. This communication between the context engine 1050 and the GI agent 1048 is relayed through the MUX 1027. An example of such a multiplexer is the multiplexer used by the Endpoint Security (EPSec) platform of the ESX hypervisor of VMware. In some embodiments, the attributes collected by the context engine 1050 from the GI agent 1048 include a rich set of parameters (e.g., layer 7 parameters, process identifiers, user identifiers, group identifiers, etc.). U.S. patent application 15 / 650,251 filed on July 14, 2017 further describes the capture and use of these context attributes by the GI agent 1048. U.S. patent application 15 / 650,251 is incorporated herein by reference.

[0112] SVM 132 performs service operations on data messages (including data messages forwarded by service coordinator 1022), as described further below. As shown, in some embodiments, each VM 1005 and 1032 includes a virtual network interface card (VNIC) 1055. Each VNIC is responsible for exchanging messages between its VM and SFE 1010. Each VNIC is connected to a specific port 1060 of SFE 1010. SFE 1010 is also connected to a physical network interface card (NIC) of the host (not shown). In some embodiments, the VNIC is a software abstraction created by the hypervisor of one or more physical NICs (PNICs) of the host.

[0113] In some embodiments, SFE 1010 maintains a single port 1060 for each VNIC of each VM. SFE 1010 is connected to the host PNIC (through a NIC driver (not shown)) to send outgoing messages and receive incoming messages. In some embodiments, SFE 1010 is defined to include port 1065, which is connected to the driver of the PNIC to send messages to and receive messages from the PNIC. SFE 1010 performs message processing operations to forward messages received on one of its ports to another of its ports. For example, in some embodiments, the SFE attempts to match the message to a flow-based rule using data in the message (e.g., data in the message header) and performs the action specified by the matching rule upon finding a match (e.g., passing the message to one of its ports 1060 or 1065, the latter indicating that the message is provided to the destination VM or PNIC).

[0114] In some embodiments, SFE 1010 is a software switch, while in other embodiments, it is a software router or a combined software switch / router. In some embodiments, SFE 1010 implements one or more logical forwarding elements (e.g., logical switches or logical routers) where the SFEs are executed on other hosts in a multi-host environment. In some embodiments, the logical forwarding elements can span multiple hosts to connect VMs that are executed on different hosts but belong to one logical network.

[0115] Different logical forwarding elements can be defined to specify different logical networks for different users, and each logical forwarding element can be defined by multiple software forwarding elements on multiple hosts. Each logical forwarding element isolates the traffic of a VM of one logical network from the VM of another logical network served by another logical forwarding element. Logical forwarding elements can be connected to VMs executed on the same host and / or different hosts. In some embodiments, the SFE extracts a logical network identifier (e.g., VNI) and a MAC address from a data message. The SFE in these embodiments uses the extracted VNI to identify a logical port group, and then uses the MAC address to identify a port within the port group.

[0116] Software switches (e.g., a hypervisor's software switch) are sometimes called virtual switches because they operate in software and they provide VMs with shared access to the host's PNIC(s). However, in this document, software switches are referred to as physical switches because they are items in the physical world. This term also distinguishes software switches from logical switches, which are an abstraction of the type of connectivity provided by a software switch. There are various mechanisms for creating logical switches from software switches. VXLAN provides a way to create such logical switches. The VXLAN standard is described in Mahalingam, Mallik; Dutt, Dinesh G.; et al. (2013-05-08), VXLAN: A Framework for Overlaying Virthalized Layer 10 Networks over Layer 3 Networks, IETF.

[0117] In some embodiments, the ports of SFE 1010 include one or more function calls to one or more modules that implement special input / output (I / O) operations on incoming and outgoing messages received at the ports. Examples of I / O operations implemented by port 1060 include ARP broadcast suppression operations and DHCP broadcast suppression operations, as described in U.S. Patent Application 9,548,965. In some embodiments of the present invention, other I / O operations (such as firewall operations, load balancing operations, network address translation operations, etc.) can be implemented in this way. By implementing a stack of such function calls, in some embodiments, the port can implement a series of I / O operations on incoming and / or outgoing messages. Moreover, in some embodiments, other modules in the data path (such as VNIC 1055, port 1065, etc.) implement I / O function call operations instead of port 1060 or in conjunction with it.

[0118] In some embodiments, one or more function calls of SFE port 1060 may be one or more service engines 1030 that process context-based service rules in context-based service rule storage 1040. In some embodiments, each service engine 1030 has its own context-based service rule storage 1040, attribute map storage 1023, and connection state cache storage 1025. Fig.10 Only one context-based service rule storage 1040, attribute mapping storage 1023, and connection state cache storage 1025 for all service engines are presented to avoid obscuring the representation in the figure with unnecessary detail. Moreover, in some embodiments, each VM can have its own service engine instance (e.g., its own instance of encryption engine 1024, load balancer 1026, and firewall engine 1028). In other embodiments, one service engine can provide data message flows for multiple VMs on a host (e.g., VMs for the same logical network).

[0119] In order to perform its service operation on the data message stream, in some embodiments, the service engine 1030 attempts to match the flow identifier (e.g., a five-tuple identifier) ​​and / or the associated context attribute set of the flow to the rule identifier. Its service rules are in its context-based service rule storage device 1040. Specifically, in order for the service engine 1030 to perform its service inspection operation on the data message stream, the SFE port 1060 of the service engine is called to supply the attribute set of the message received by the port. In some embodiments, the attribute set is a message identifier (such as a traditional five-tuple identifier). In some embodiments, one or more of the identifier values ​​can be a logical value defined for a logical network (for example, it can be an IP address defined in a logical address space). In other embodiments, all identifier values ​​are defined in the physical domain. In other embodiments, some of the identifier values ​​are defined in the logical domain, while other identifier values ​​are defined in the physical domain.

[0120] Then, in some embodiments, the service engine uses the attribute set of the received message (e.g., the five-tuple identifier of the message) to identify the set of context attributes that the service engine has stored for this flow in the attribute mapping storage device 1023. As mentioned above, in some embodiments, the context engine 1050 supplies the context attributes for the new flow (i.e., the new network connection event) to the service engine 1030 together with the flow identifier (e.g., the five-tuple identifier). The context engine policy storage device 143 contains rules that control the operation of the context engine 110. In some embodiments, these policies instruct the context engine to generate rules for the service engine or instruct the service engine to generate rules. The service engine 130 in these embodiments stores the context attributes they receive from the context engine in the attribute mapping storage device 223.

[0121] In some embodiments, the service engine 1030 stores the set of context attributes for each new flow in an attribute mapping storage device together with an identifier (e.g., a five-tuple identifier) ​​for the flow. In this way, the service engine can identify the set of context attributes for each new flow it receives from the SFE port 1060 by searching its attribute mapping storage device 1023 for a context record with a matching flow identifier. The context record with a matching flow identifier includes the set of context attributes for the flow. Similarly, to identify the set of context attributes for a processing event, in some embodiments, the service engine searches its attribute mapping storage device 1023 for a context record with a matching processing identifier.

[0122] In some embodiments, some or all of the service engines extract a set of context attributes for the new flow from the context engine. For example, in some embodiments, the service engine provides the five-tuple identifier of the new flow it receives from SFE port 1060 to context engine 1050. The engine 1050 then checks its attribute storage device 1045 to identify the set of attributes stored for the five-tuple identifier and then provides the set of attributes (or a subset of the set of attributes obtained by filtering the identified attribute set of the service engine) to the service engine.

[0123] Some embodiments implement the extraction model by encoding a set of attributes for a new message flow using a service token. When notified of a new network connection event, in some embodiments context engine 1050 (1) collects a set of context attributes for the new event, (2) filters this set to discard attributes that are not relevant to performing one or more services on the flow, (3) stores the remaining filtered attribute subset in attribute storage 1045 along with the service token, and (4) provides the service token to GI agent 1048. GI agent 1048 then causes this token to be transmitted to the service engine(s) in-band (e.g., in the header of a data message sent by the agent's VM to a destination) or out-of-band (i.e., separate from the data message sent from the agent's VM to a destination).

[0124] When the service engine obtains a new flow through SFE port 1060, it provides the service token for this flow to the context engine, which uses this service token to identify the context attributes to be provided to the service engine in its attribute store 1045. In embodiments where the SFE port does not provide this service token to the service engine, the service engine must first identify the service token by searching its data store using the identifier of the flow before supplying the service token to the context engine.

[0125] After identifying the set of contextual attributes of the data message flow, in some embodiments, the service engine 1030 performs its service operations based on the service rules stored in the context-based service rule storage device 1040. To perform its service operations, the service engine 1030 matches the received attribute subset with the corresponding attribute set stored for the service rule. In some embodiments, each service rule in the context-based service rule storage device 1040 has a rule identifier and a set of action parameters.

[0126] As described above, in some embodiments, the rule identifier of the service rule can be defined based on one or more context attributes that are not L2-L4 header parameters (e.g., L7 parameters, user identifiers, group identifiers, process names, loaded module identifiers, consumption parameters, etc.). In some embodiments, the rule identifier can also include L2-L4 header parameters. Moreover, in some embodiments, one or more parameters in the rule identifier can be specified based on individual values ​​or wildcard values. Moreover, in some embodiments, the rule identifier can include a set of individual values ​​or a group identifier, such as a security group identifier, a computing configuration identifier, a network configuration identifier, etc.

[0127] To match the received set of attributes to a rule, the service engine compares the received set of attributes to the associated identifier of the service rule stored in the context-based service rule storage 1040. After identifying a matching rule, the service engine 1030 performs a service operation (e.g., a service coordination operation, a firewall operation, a load balancing operation, an encryption operation, other middlebox operations, etc.) based on a set of action parameters of the matching rule (e.g., based on a service action list, allow / drop parameters, load balancing criteria, encryption parameters, etc.).

[0128] In some embodiments, the context-based service rule storage 1040 is defined in a hierarchical manner to ensure that a message rule check will match a higher priority rule before matching a lower priority rule when a subset of attributes of a message matches multiple rules. Furthermore, in some embodiments, the context-based service rule storage 1040 contains a default rule that specifies a default action for any message rule check that cannot recognize any other service rule; in some embodiments, this default rule will be a match for all possible attribute subsets and ensure that the service rule engine will return actions for all received attribute subsets. In some embodiments, the default rule will not specify a service.

[0129] For example, when the messages are part of a flow associated with a communication session between two machines, multiple messages may have the same set of message identifier attributes. Thus, after matching a data message to a service rule in the context-based service rule storage 1040 based on the identified set of context attributes of the message, the service engine of some embodiments stores the service rule (or a reference to the service rule) in the connection state cache storage 1025 so that the service rule can be used later for subsequent data messages of the same flow.

[0130] In some embodiments, the connection state cache storage 1025 stores service rules or references to service rules identified by the service engine 1030 for different sets of message identifiers (e.g., different five-tuple identifiers for identifying different data message flows). In some embodiments, the connection state cache storage 1025 stores each service rule or reference to a service rule with an identifier generated from a matching set of message identifiers (e.g., a five-tuple identifier for a flow).

[0131] Before checking a particular message using the context-based service rule storage 1040, the service rule engine 1030 of some embodiments checks the connection state cache storage 1025 to determine whether the storage has previously identified a service rule for the flow of the message. If not, the service engine 1030 identifies a set of context attributes for the message flow and then checks the context-based service rule storage 1040 for a service rule that matches the identified set of attributes of the message and / or its five-tuple identifier. When the connection state data storage has an entry for the particular message, the service engine performs its service operation based on the set of action parameters for the service rule.

[0132] As described above, in some embodiments, a service node on a host computer is formed by (1) a software forwarding element 1010 associated with a VTEP serving as a Geneve tunnel endpoint and (2) a service coordinator 1022. In some embodiments, for an ingress service node (e.g., service node 120), the service coordinator executes Figure 8 1040. For example, it retrieves metadata captured by context engine 1050 for a particular data message stream. For a data message sent by GVM 1005 on its host computer, service coordinator 1022 identifies the SFP list by matching the message's stream identifier and / or context metadata attributes with the service rules in its rule store 1040.

[0133] In some embodiments, each service rule of the coordinator in the storage device 1040 specifies the service chain according to the list of IP addresses (and in some cases the service label descriptor) for each service operation in the service chain. After matching the attribute set of the message with one of its service rules, the service coordinator 1022 of the ingress node embeds the SFP list of the matching service rule and the other tunnel attributes (e.g., service index, etc.) described above into the Geneve tunnel header.

[0134] Likewise, in some embodiments, the coordinator embeds some or all of the metadata obtained from the context engine 1050 into the NSH metadata options TLV of the Geneve tunnel header. In some embodiments, the service coordinator 1022 causes another of the service engines 1030 to perform tunnel encapsulation. After encapsulating the data message with the Geneve tunnel header 200, the service coordinator returns the data message to the SFE (e.g., provides a handle to the location in memory where the encapsulated data message is stored) so that the SFE can forward the data message along the Geneve tunnel to the service node associated with the first service machine in the specified SFP list.

[0135] In some cases, the first service operation or the first few service operations may have to be performed by one or more SVMs 1032 or service engines 1030 on the same host as the source GVM and the ingress service coordinator. In these cases, the service coordinator directs the data message to the (one or more) SVMs or (one or more) service engines on its host, which then encapsulate the data message with a tunnel header. After all specified service operations have been performed on its host computer, in this case, the service coordinator then encapsulates the processed data message and instructs the SFE to send the data message to the service node associated with the next service operation in the SFP list.

[0136] Moreover, for each data message flow, in some embodiments, the ingress service coordinator performs a load balancing operation to convert each VIP address of a service operation in the rule storage device 1040 that specifies a matching service rule (i.e., a rule that matches the identifier and / or context metadata of the flow) to a DIP address of a specific service machine. The ingress service coordinator performs this operation so that it can embed the DIP address of the actual service machine in the tunnel header instead of the VIP address of the service machine cluster. In some embodiments, the service coordinator does not perform load balancing itself, but uses the load balancing engine 1026. Alternatively, in other embodiments, the service coordinator embeds one or more VIP addresses, and one of the service machines in the SFP list (e.g., the first service machine) is the load balancer that converts each VIP address to a DIP address.

[0137] In some embodiments, when the service coordinator is executing an operation of a non-entry service node, the service coordinator executes Fig. 9The operation of process 900 is performed. For example, it determines that the tunnel encapsulated data message has an SFP list that identifies the IP address of the SVM or service engine associated with the coordinator as the address of the next service machine. It decapsulates the Geneve tunnel header from the data message received along the Geneve tunnel, or instructs another of the service engines 1030 to perform the decapsulation. The service coordinator 1022 then stores the decapsulated tunnel information in the tunnel data storage device 1075 for later reuse.

[0138] Once the received data message has been decapsulated and its tunnel information has been stored, the service coordinator passes the data message and, in some cases, its metadata to its associated SVM 1032 or service engine 1030 on its host 1000 to perform the next service operation. If multiple SVMs or service engines must process the data message, the service coordinator sequentially passes the data message to each of its associated service machines that must sequentially process the message. Each time the service coordinator passes a data message to a service machine, the IP address of the machine is removed from the SFP list or the service index is adjusted as described above.

[0139] The service coordinator can obtain metadata from one or more of its service machines. In some embodiments, based on the metadata, the service coordinator can perform a reclassification operation to reassign the SFP list. After its associated (one or more) service machines process the data message, the service coordinator determines whether the SFP list is empty. If so, when the destination of the data message is one of these GVMs, it provides the data message to one of the GVMs on its host 1000. When the destination of the data message is not a GVM on its host computer, the service coordinator encapsulates the data message with a tunnel header (e.g., a Geneve tunnel header) and then provides the encapsulated data message to the SFE to be sent to its destination along the tunnel.

[0140] On the other hand, after processing the data message and determining that the SFP list is not empty, the service coordinator encapsulates the data message with a Geneve tunnel header, in which it embeds the modified SFP list and service index and other tunnel attributes (e.g., metadata, etc.). As described above, and hereinafter by reference Fig.13 Further describing, in some embodiments, the service coordinator can embed metadata captured or generated by the coordinator's associated service machine when the machine processes the data message in the NSH metadata option TLV of the Geneve tunnel header. The service coordinator provides the encapsulated data message to the SFE to be sent along the Geneve tunnel to the service node of the next service machine identified in the SFP list.

[0141] Fig.11An example of how to manage a service coordinator 1022 in some embodiments is illustrated. The figure illustrates multiple hosts 1000 in a data center. As shown, each host includes several service engines 1030, a context engine 1050, a service coordinator 1022, several GVMs 1005, one or more SVMs 1032, and SFEs 1010.

[0142] It also illustrates a controller set 1105 for managing the service coordinator 1022 and the service engine 1030, the GVM 1005, and the SFE 1010. The host and the controller are communicatively connected to each other via a network 1110, which may be a local area network, a wide area network, a network of networks (such as the Internet), etc. The controller set provides a user interface for an administrator to define context-based service rules according to context attributes, and communicates with the host via the network 1110 to provide these policies.

[0143] In some embodiments, the context engine 1050 collects context attributes that are passed to the management server in the controller via the network 1110 so that these context attributes can be used to define policies. In some embodiments, the management server interacts with a discovery engine executed on a host computer 1000 in a data center to obtain and refresh an inventory of all processes and services running on the GVM on the host. Then, in some embodiments, the management plane provides a rule creation interface for allowing administrators to create context-based service rules for service engines, SVMs, and coordination engines 1022.

[0144] Once high-level service policies are defined in the management plane, the management plane provides some or all of these policies directly to management agents (not shown) on the host 1000, and / or indirectly to these agents through a set of configuration controllers. In some embodiments, the management agent publishes the received policies as rules to the context-based service rule storage 1043. In some embodiments, the agent transforms the policies before publishing them to the context-based service rule storage 1043. Moreover, in some embodiments, the context engine 1050 on the host 1000 parses the policies based on the collected context attributes to generate rules for the service engine.

[0145] In some embodiments, different policies are specified for different data message flows from a source GVM based on the different traffic content carried by these flows. For example, one policy may define the use of a low-latency service engine to process an SFP list for a data message flow from a source GVM for a video conference involving company executives, while another policy may define the use of a service engine with a higher latency for an SFP list when a data message flow from the same GVM is related to a sent email.

[0146] Fig.12 and Fig.13 Two examples are illustrated for forwarding and processing metadata related to service operations performed by a service machine in some embodiments. Fig.12 The illustrated example shows the processing of two message flows 1202 and 1204 originating from a VM 1205 on a host 1210 at two different times. Both message flows need to be processed by a firewall service machine and are directed to a firewall machine in one of two firewall service clusters 1230 and 1232 via a load balancing SVM 1220 executing on another host 1225.

[0147] The ingress service node 1212 of the first host 1210 encapsulates the data messages of the two streams with a tunnel header that includes context metadata about the data message streams. The context metadata of the first stream 1202 specifies that the stream carries video conference data of someone in the company's executive staff (Estaff), while the context metadata of the second stream 1204 specifies that the stream carries email data of someone in the human resources (HR) department. In some embodiments, the HR and Estaff information are expressed based on active directory group identifiers. Similarly, in some embodiments, the ingress service node 1212 stores the context metadata in each data message of each stream, while in other embodiments, the ingress service node only stores the context metadata in a subset of the data messages of each stream (e.g., the first message, or the first N messages, or every 100 messages).

[0148] The service node 1222 of the host 1225 decapsulates each encapsulated data message and provides the decapsulated data message along with embedded context metadata attributes indicating the traffic type and the active directory group identifier to the load balancing SVM 1220. Based on the context metadata attributes, the load balancer 1220 determines that the first data message flow must be directed to the low-latency firewall service cluster 1230, while the second data message flow 1204 can be directed to the regular firewall service cluster 1232.

[0149] For each message flow, the load balancer 1220 also selects one firewall service machine 1250 or 1252 in each service cluster 1230 or 1232 and provides an identifier of the selected firewall service machine (e.g., its IP address) to the service node 1222. Based on the provided service machine identifier, the service node 1222 then forwards each data message flow to the firewall machine 1250 or 1252 selected by the load balancer 1220 for each flow.

[0150] Fig.13 The diagram shows Fig.12 A similar example, except that in Fig.13 In some embodiments, ingress service node 1312 does not identify the traffic type, but leaves the traffic type identification to DPI service module 1370 executing on host computer 1375 in conjunction with first service node 1372. In some embodiments, DPI service module 1370 identifies the traffic type by inspecting the first data message or first few data messages in each flow using standard DPI techniques.

[0151] After identifying the traffic type, DPI module generates a traffic type identifier and provides the generated metadata to first service node 1372. After receiving the metadata for the data message, first service node 1372 then re-encapsulates the data message with a tunnel header that includes the active directory metadata received from ingress service node 1312 and the traffic type metadata generated by DPI module 1370. First service node 1372 then forwards each encapsulated data message for each flow to service node 1222 of host 1225. Then, Fig.13 The service node 1222 in Fig.12 The service node 1222 processes each data message stream in the same manner.

[0152] Many of the above features and applications are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of a processor, or other processing units), they cause the (one or more) processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0153] In this specification, the term "software" refers to firmware residing in a read-only memory or an application stored in magnetic storage that can be read into a memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that implement the software inventions described herein together is within the scope of the present invention. In some embodiments, when a software program is installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that perform the operations of the software program.

[0154] Fig.14 A computer system 1400 is conceptually shown that implements some embodiments of the present invention. The computer system 1400 can be used to implement any of the above-described hosts, controllers, and managers. Thus, it can be used to perform any of the above-described processes. The computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 1400 includes a bus 1405, (one or more) processing units 1410, a system memory 1425, a read-only memory 1430, a permanent storage device 1435, an input device 1440, and an output device 1445.

[0155] Bus 1405 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system 1400. For example, bus 1405 communicatively connects processing unit(s) 1410 with read-only memory 1430, system memory 1425, and permanent storage device 1435.

[0156] From these various memory units, the processing unit(s) 1410 retrieves instructions to be executed and data to be processed in order to perform the processes of the present invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only memory (ROM) 1430 stores static data and instructions required by the processing unit(s) 1410 and other modules of the computer system. On the other hand, the permanent storage device 1435 is a read and write memory device. This device is a non-volatile storage unit that stores instructions and data even when the computer system 1400 is turned off. Some embodiments of the present invention use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 1435.

[0157] Other embodiments use removable storage devices (such as flash drives, etc.) as permanent storage devices. Like permanent storage device 1435, system memory 1425 is a read and write memory device. However, unlike storage device 1435, system memory is a volatile read and write memory, such as random access memory. System memory stores some instructions and data that the processor needs at run time. In some embodiments, the processing of the present invention is stored in system memory 1425, permanent storage device 1435, and / or read-only memory 1430. From these various memory units, (one or more) processing unit 1410 retrieves instructions to be executed and data to be processed in order to perform the processing of some embodiments.

[0158] The bus 1405 is also connected to input and output devices 1440 and 1445. Input devices enable a user to communicate information and select commands to the computer system. Input devices 1440 include an alphanumeric keyboard and a pointing device (also called a "cursor control device"). Output devices 1445 display images generated by the computer system. Output devices include printers and display devices, such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs). Some embodiments include devices such as touch screens that function as both input and output devices.

[0159] Finally, if Fig.14 As shown, bus 1405 also couples computer system 1400 to a network 1465 via a network adapter (not shown). In this manner, the computer may be part of a network of computers (such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network such as the Internet), and any or all components of computer system 1400 may be used in conjunction with the present invention.

[0160] Some embodiments include electronic components such as microprocessors, storage devices and memories that store computer program instructions in machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), read-only digital versatile disks (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code such as produced by a compiler, and files including higher-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0161] Although the above discussion mainly refers to a microprocessor or multi-core processor that executes software, some embodiments are performed by one or more integrated circuits, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, such an integrated circuit executes instructions stored on the circuit itself.

[0162] As used in this specification, the terms "computer", "server", "processor", and "memory" all refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or being displayed mean displayed on an electronic device. As used in this specification, the terms "computer-readable medium", "multiple computer-readable media" and "machine-readable medium" are completely limited to tangible, physical objects that store information in a form that is readable by a computer. These terms do not include any wireless signals, wired download signals, and any other transient or temporary signals.

[0163] Although the present invention has been described with reference to many specific details, it will be appreciated by those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit of the present invention. For example, several figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a series of continuous operations, and different specific operations may be performed in different embodiments. In addition, the processes may be implemented using several sub-processes, or as part of a larger macro process.

[0164] In some embodiments, the above methods are used to express service chains in a single-tenant environment. Therefore, those of ordinary skill in the art will recognize that some embodiments of the present invention are equally applicable to single-tenant data centers. Conversely, in some embodiments, when an entity (e.g., a company) is a tenant in multiple different data centers of different providers, the above methods are used to perform service chain specifications across different data centers of different data center providers. In these embodiments, the tenant identifiers embedded in the tunnel header must be unique across data centers or must be translated when they traverse from one data center to another. Therefore, those of ordinary skill in the art will understand that the present invention is not limited by the above illustrative details, but is defined by the appended claims.

Claims

1. A method for specifying a service operation on a data message associated with a specific machine of a specific tenant in a multi-tenant network, the method comprising: storing, in a generic network virtualization encapsulation Geneve tunnel header for encapsulating the data message, a tenant identifier identifying a specific tenant and a plurality of service identifiers associated with a plurality of service machines for performing a plurality of service operations on the data message, the plurality of service identifiers being stored in a variable-size option field of the tunnel header, the variable-size option field allowing different numbers of service machines to be specified to perform different numbers of service operations on different data message flows; as well as forwarding the data message encapsulated with the tunnel header along the tunnel to the first service machine to perform a first service operation identified by the plurality of service identifiers, The Geneve tunnel header is placed outside the layer 2 and layer 3 headers of the data message.

2. The method of claim 1, wherein storing the plurality of service identifiers comprises storing a plurality of network addresses of the plurality of service machines in a tunnel header.

3. The method of claim 2, wherein storing the plurality of service identifiers further comprises storing in the tunnel header a service operation descriptor for each service machine identified by the network address stored in the tunnel header to explain the type of service operation performed by the service machine.

4. The method of claim 1, wherein The service identifiers are stored in the tunnel header in reverse order, such that the first service operation is stored last and the last service operation is stored first.

5. The method of claim 1, further comprising storing a service index value in a tunnel header, the service index value identifying one of the stored service identifiers as a next service operation to be performed.

6. The method of claim 1 further comprises selecting a set of at least two service operations from a plurality of candidate service operation sets based on a set of one or more attributes associated with the data message, the plurality of candidate service operation sets being different sets of feasible operations to be performed on the data message.

7. The method of claim 6, wherein selecting comprises: for a first data message flow from a first machine to a second machine, selecting a first set of service operations based on a first type of content carried in the first data message flow; as well as For a second data message flow from the first machine to the second machine, a second service operation set is selected based on a second type of content carried in the second data message flow, the second service operation set including at least one service operation not in the first service operation set.

8. The method of claim 1, wherein the tunnel is connected to a first service node, which is connected to the first service machine without utilizing any intervening hardware routers or hardware switches.

9. The method of claim 8, wherein the first service machine is one of a stand-alone computer, a service module executed on a host computer, and a stand-alone service device.

10. The method of claim 8, wherein the first service node and the first service machine are modules that execute on a host computer along with other machines.

11. The method of claim 8, wherein the first service node removes a tunnel header, provides a data message to the first service machine, receives a processed data message from the first service machine, encapsulates the processed data message using another tunnel header generated based on information obtained from the removed tunnel header, and sends the encapsulated processed data message along another tunnel to another service node connected to a second service machine to perform a first service operation identified by the plurality of service identifiers.

12. The method of claim 1, wherein the service operation is a middlebox service operation.

13. The method of claim 1, wherein the tenant identifier is stored in a Geneve base header, and the multiple service identifiers stored in the variable-size option field include the multiple service identifiers stored in an option TLV (type, length, value) of the Geneve header.

14. A non-transitory machine-readable medium storing a program for specifying a service operation on a data message associated with a specific machine of a specific tenant in a multi-tenant network, the program comprising a set of instructions for: receiving the data message having a first header; storing, in a generic network virtualization encapsulation Geneve tunnel header for encapsulating the data message, a tenant identifier identifying a specific tenant and a plurality of network addresses of a plurality of service machines to perform a plurality of service operations on the data message, wherein the plurality of network addresses are stored in a variable-size option field of the Geneve tunnel header, wherein the variable-size option field allows specifying a different number of service machines to perform a different number of service operations on different data message flows; encapsulating the data message having the first header using the Geneve tunnel header; as well as The data message encapsulated with the Geneve tunnel header is forwarded along a tunnel to a first service machine identified by a first network address among the plurality of network addresses, the first service machine being to perform a first service operation on the data message.

15. The non-transitory machine-readable medium of claim 14, wherein the program further comprises a set of instructions for storing a service index value in the Geneve tunnel header, the service index value identifying one of the stored network addresses as the network address of a service machine to perform a next service operation, the service index being adjusted after each service machine performs its associated service operation.

16. The non-transitory machine-readable medium of claim 14, wherein the tunnel is connected to a first service node connected to the first service machine without utilizing an intervening hardware router or hardware switch.

17. The non-transitory machine-readable medium of claim 16, wherein the first service node and the first service machine are modules that execute on a host computer along with other machines.

18. The non-transitory machine-readable medium of claim 16, wherein the first service node removes the Geneve tunnel header, provides a data message to the first service machine, receives a processed data message from the first service machine, encapsulates the processed data message using another Geneve tunnel header generated based on information obtained from the removed Geneve tunnel header, and sends the encapsulated processed data message along another tunnel to another service node connected to a second service machine to perform a first service operation identified by the plurality of network addresses.

19. A non-transitory machine-readable medium as described in claim 14, wherein the tenant identifier is stored in a Geneve base header, and the multiple network addresses stored in the variable-sized option field include the multiple network addresses stored in an option TLV (type, length, value) of a Geneve tunnel header.

20. An electronic device, comprising: a set of processing units; as well as A machine-readable medium storing a program which, when executed by at least one of the processing units, implements the method according to any one of claims 1 to 13.

21. A system comprising means for implementing the method of any one of claims 1-13.

22. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-13.

Images