Cross-cluster container communication methods and distributed systems

By using cross-data plane container communication methods and leveraging intra-domain virtual addresses and authentication forwarding channels, the problems of high deployment and maintenance costs and difficult identity verification in cross-cluster communication are solved, achieving transparent cross-cluster communication and improving security and reliability.

CN123123431APending Publication Date: 2026-11-17BEIJING WUWEN CORE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611498683.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-23
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In scenarios such as cloud-edge collaboration, distributed training, and robot clusters, cross-cluster container communication suffers from problems such as high deployment and maintenance costs, address changes after container reconstruction, difficulty in reliably verifying the source identity, and inconsistencies between routing and access authorization status, leading to communication difficulties.

Method used

By using cross-data plane container communication methods, network traffic is intercepted by the source container network proxy. Combined with node proxy and relay server, cross-cluster communication that is transparent to business programs is achieved. Intra-domain virtual addresses and authentication forwarding channels are used to reduce the need for underlying network interconnection, and security and reliability are improved through pre-established reverse channels.

Benefits of technology

It enables cross-cluster container communication without modifying the underlying network, reducing deployment and maintenance costs, improving communication security and reliability, and minimizing the impact of container rebuilding on addressing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123431A_ABST
    Figure CN123123431A_ABST
Patent Text Reader

Abstract

This disclosure proposes a cross-cluster container communication method and a distributed system. The method is used for communication between a first container instance in a first cluster and a second container instance in a second cluster, both belonging to the same logical domain. The method includes: a first container network agent intercepting network traffic destined for a virtual address within the domain of the second container instance, and transmitting the virtual address and business data to a node agent via local inter-process communication; the node agent parsing the target location information and sending the target location information and business data to a relay server via an authenticated forwarding channel; and the relay server delivering the business data to the second container network agent via a pre-built reverse channel in the agent component of the second cluster, and then transmitting it to the target business process. This disclosure achieves cross-cluster communication transparent to business programs without requiring underlying cross-cluster network interconnection, and enhances communication security through source identity verification and target-side authorization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of distributed systems, and more particularly to a cross-data plane container communication method and a distributed system. Background Technology

[0002] In scenarios such as cloud-edge collaboration, distributed training, and robot clusters, business workloads can be deployed in containers and distributed across multiple clusters, such as multiple Kubernetes clusters. Different clusters may be located behind their own private networks, Network Address Translation (NAT), or firewalls. The real network addresses of each container instance are usually only reachable within its own cluster, thus creating a need for cross-cluster communication.

[0003] Cross-cluster communication solutions typically require modifications to the underlying container network, the establishment of a cluster-level virtual private network, or the configuration of interoperability routes, resulting in high deployment and maintenance costs. Solutions using application-layer gateways may require application programs to be aware of the target location. Furthermore, they face challenges such as address changes after container rebuilds, difficulties in reliably verifying source identities, and inconsistencies between routing and access authorization states. Summary of the Invention

[0004] In related solutions, it is difficult to establish direct communication connections between container instances located in different network environments. Even if cross-network forwarding is achieved through proxies or relay servers, problems may still arise such as difficulty in reliably verifying the source identity, lack of consistency between target addressing and access authorization, and the inability of the target data plane to accept external proactive connections.

[0005] To address this, this disclosure proposes a cross-dataplane container communication method and a distributed system. The method involves intercepting network traffic destined for the target container instance's virtual address via a source container network proxy, resolving the target location via a node proxy, and sending the business data to a relay server via an authentication forwarding channel. The business data is then forwarded to the target container instance via a pre-established reverse channel in the target data plane. This enables transparent cross-dataplane container communication to business applications without requiring cross-cluster underlying routing interoperability.

[0006] According to a first aspect of the present disclosure, a cross-cluster container communication method is proposed for communication between a first container instance in a first cluster and a second container instance in a second cluster, wherein the first container instance and the second container instance belong to the same logical domain. The method includes: intercepting network traffic destined for a domain virtual address of the second container instance by a first container network proxy corresponding to the first container instance, and transmitting the domain virtual address and service data in the network traffic to a node proxy of the node where the first container instance is located through a local inter-process communication connection; determining the target location information of the second container instance based on the domain virtual address, and forwarding the target location information and the service data to the relay server through an authentication forwarding channel with a relay server; and forwarding the service data to the second container network proxy corresponding to the second container instance through a reverse channel pre-established by the relay server through a proxy component in the second cluster, so that the second container network proxy transmits the service data to the target service process in the second container instance.

[0007] Optionally, the first cluster and the second cluster communicate with the same control plane, and the method further includes: the control plane receiving container instance information belonging to the logical domain reported by the first cluster and the second cluster respectively; allocating virtual addresses to container instances within the logical domain according to the virtual address range of the logical domain; generating a domain routing directory for the logical domain according to the container instance information and the allocated virtual addresses, the domain routing directory including the correspondence between virtual addresses, cluster identifiers, node identifiers and container instance real addresses; and distributing the domain routing directory to the first cluster and the second cluster respectively.

[0008] Optionally, allocating virtual addresses to container instances within the logical domain includes: constructing a stable instance key based on the cluster identifier, local namespace, and instance name to which the container instance belongs; in response to the reconstruction of the container instance, searching for historical virtual address allocation records based on the stable instance key; and, if a corresponding historical virtual address allocation record is found and the corresponding historical virtual address does not conflict, allocating the historical virtual address to the reconstructed container instance.

[0009] Optionally, the first container network agent corresponding to the first container instance intercepts network traffic destined for the second container instance's intra-domain virtual address, and transmits the intra-domain virtual address and the service data in the network traffic to the node agent of the node where the first container instance is located via a local inter-process communication connection. This includes: intercepting the network traffic through a virtual tunnel network interface card (TUN) device created by the first container network agent; performing protocol processing on the intercepted network traffic through a user-space protocol stack in the first container network agent to obtain the intra-domain virtual address and the service data; and transmitting the intra-domain virtual address and the service data to the node agent via the local inter-process communication connection through a local inter-process communication client in the first container network agent.

[0010] Optionally, the method further includes: the node agent reading the peer process credentials of the local inter-process communication connection provided by the operating system to obtain a peer process identifier; querying the correspondence between the process and the container instance based on the peer process identifier to verify that the peer process of the local inter-process communication connection belongs to the first container instance; and querying the correspondence between the container instance and the logical domain based on the identity information of the first container instance to verify that the first container instance belongs to the logical domain, wherein the node agent determining the target location of the second container instance based on the virtual address within the domain includes: the node agent querying the virtual address within the domain in the domain routing directory of the verified logical domain to determine the target location of the second container instance.

[0011] Optionally, the node agent maintains Secure Shell Protocol (SSH) connection pools for different logical domains, each SSH connection pool including one or more physical SSH connections, and each physical SSH connection capable of carrying multiple logical channels; wherein, the authentication forwarding channel is a logical channel established by the node agent on the physical SSH connection selected by the node agent in the SSH connection pool corresponding to the logical domain.

[0012] Optionally, before the relay server forwards the service data through the reverse channel, the method further includes: the relay server verifying the domain authentication credentials provided by the node agent when establishing the authentication forwarding channel, and determining the logical domain to which the first container instance belongs and the first cluster based on the verified domain authentication credentials; the relay server determining the target real address of the second container instance based on the target location information; the relay server reading the domain routing directory of the logical domain corresponding to the second cluster, and querying whether the target real address is recorded in the domain routing directory; and in response to the target real address being recorded in the domain routing directory, allowing the establishment of a forwarding connection to the second container network agent through the reverse channel.

[0013] Optionally, the relay server forwards the business data to the second container network proxy corresponding to the second container instance through a reverse channel pre-established by the proxy component in the second cluster. This includes the relay server sequentially trying the target node-dedicated reverse channel, the general reverse channel of the second cluster, and the reverse channel of any node in the second cluster according to the target location information, in order to forward the business data to the second container network proxy.

[0014] According to a second aspect of the present disclosure, a distributed system is proposed, including a first cluster, a second cluster, a relay server, and a communication subsystem, wherein the communication subsystem is configured to enable the first cluster, the second cluster, and the relay server to collaboratively execute the cross-cluster container communication method as described in the first aspect above.

[0015] According to a third aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in the first aspect above.

[0016] Therefore, this disclosure achieves cross-data plane target addressing through intra-domain virtual addresses, and combines authentication forwarding channels with pre-built reverse channels on the target side to realize cross-data plane container communication transparent to business applications, without modifying the underlying network or configuring cross-cluster interoperability routing. Furthermore, it improves the security of cross-domain forwarding through source process credential verification and target-side secondary authorization; mitigates the impact of container reconstruction on addressing through virtual address recovery; and reduces the overhead of repeated connection establishment and improves communication reliability through domain authentication connection pools and multi-level reverse channel selection. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0018] Figure 1 This example illustrates a scenario of cross-cluster container communication.

[0019] Figure 2 A schematic flowchart of a cross-cluster container communication method according to some embodiments of the present disclosure is shown.

[0020] Figure 3 An example of a cross-cluster container communication path according to this disclosure is shown.

[0021] Figure 4 A schematic diagram illustrating the relationship between a control plane and multiple clusters according to an implementation of this disclosure is shown.

[0022] Figure 5 A schematic diagram of reverse channel selection according to some embodiments of the present disclosure is shown.

[0023] Figure 6 A schematic diagram of the architecture of a distributed system according to some embodiments of the present disclosure is shown.

[0024] Figure 7 A schematic diagram of the functional modules of a communication subsystem according to some embodiments of the present disclosure is shown. Detailed Implementation

[0025] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0026] As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” and “the” are intended to also include expressions such as “one or more,” unless the context explicitly indicates otherwise. The term “comprising” and its variations, as used herein, indicate an open-ended inclusion, i.e., “including but not limited to.” Unless specifically stated otherwise, the term “or” means “and / or.” The term “according to” means “at least in part according to.” The terms “an example embodiment” and “an embodiment” mean “at least one example embodiment.” The term “another embodiment” means “at least one additional embodiment.” The terms “first,” “second,” etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0027] In a distributed system, there can be a control plane and a data plane. The control plane receives and processes management requests for data plane resources, manages resources such as compute nodes and business services within the data plane, and coordinates access to these resources. The data plane runs computing tasks, provides business services, and hosts target resources. It can include one or more clusters, such as edge clusters, robot field clusters, or enterprise private clusters. Business workloads in the data plane can be deployed as containers. A cluster can include multiple Pods, each running one or more container instances. Container instances in different clusters may need to communicate with each other over the network. The control plane can deploy a server to manage data plane resources, and the data plane can deploy an agent component to establish communication connections with the server and access target resources in the data plane based on server requests.

[0028] Figure 1 This illustrates an example of cross-cluster container communication. For example... Figure 1 As shown, the distributed system includes a first cluster X located in Beijing and a second cluster Y located in Shanghai (Beijing and Shanghai are used as examples only). Figure 1 In the implementation shown, the first data plane corresponds to the first cluster X, and the second data plane corresponds to the second cluster Y. Therefore, the first data plane, the first cluster and cluster X, and the second data plane, the second cluster and cluster Y can be used interchangeably in the following text.

[0029] In a distributed training task, for example, a first container instance is deployed as Pod A in a first cluster X, and a second container instance is deployed as Pod B in a second cluster Y. The two container instances need to communicate over the network. The first cluster X and the second cluster Y may be behind different private networks, Network Address Translation (NAT) devices, or firewalls. The real network addresses of Pod A and Pod B are usually only reachable within their respective clusters, and therefore cannot directly use their internal IP addresses for cross-cluster communication. In addition, when Pod A or Pod B is rebuilt, its real network address may also change.

[0030] In relevant solutions, cross-cluster container communication can be achieved by modifying the underlying container network interface (CNI), establishing a cluster-level virtual private network (VPN), configuring cross-cluster interoperability routing, or having the application explicitly access the gateway. However, these methods may require additional network management privileges or stable inbound connection conditions, and may require the application to be aware of the target cluster or real address. For communication methods using proxy forwarding, if relying on the logical domain or identity information declared by the source container itself, there may be a risk of identity forgery; if the target location and access authorization are maintained separately, inconsistencies between routing and authorization states may occur. In addition, establishing authentication connections for each business flow may incur significant handshake overhead, and instability of the reverse connection to the target data plane may also affect the reliability of cross-cluster communication.

[0031] To address the aforementioned problems in cross-cluster container communication, this disclosure enables container instances located in different clusters to communicate without direct interoperability of the underlying network by employing logical domain virtual addressing, source-side proxy forwarding, and pre-built reverse channels on the target side. The cross-cluster container communication method and its specific implementation proposed in this disclosure are described below with reference to the accompanying drawings.

[0032] Figure 2 A schematic flowchart illustrating a cross-cluster container communication method according to some embodiments of the present disclosure is shown. This method can be used for communication between a first container instance in a first cluster and a second container instance in a second cluster. The first and second container instances belong to the same logical domain, which defines the range of container instances capable of communicating with each other through the virtual network of the present disclosure. Under this premise, the first container instance can initiate communication through the virtual address within the domain of the second container instance and forward service data to the second container instance according to the method of the present disclosure.

[0033] In step S210, the first container network agent corresponding to the first container instance intercepts the network traffic destined for the second container instance's domain virtual address, and transmits the business data in the domain virtual address and network traffic to the node agent of the node where the first container instance is located through a local inter-process communication connection.

[0034] The first container network proxy handles network traffic for the first container instance and can be deployed in the same Pod as the first container instance. For example, in Figure 1 In the scenario shown, the first container instance can be a business container in Pod A, and the first container network proxy can be a Sidecar container in Pod A. A one-to-one correspondence is not required between the first container network proxy and the first container instance; for example, multiple business containers in the same Pod can share the corresponding container network proxy.

[0035] A node proxy is a node-side functional entity deployed on the node where the first container instance resides. It receives intra-domain virtual addresses and business data from the container network proxy on that node, determines the source container instance and its logical domain, resolves the target location, and performs subsequent data forwarding. A node proxy can provide these services to multiple container instances on its node.

[0036] The virtual address within the domain here refers to the virtual network address of the second container instance within its logical domain, used for target addressing within the logical domain. The first container network agent can process the intercepted network traffic, obtain the service data to be transmitted, and pass the virtual address within the domain and the service data to the node agent through a local inter-process communication (IPC) connection. Specifically, the first container network agent is responsible for traffic interception and protocol processing on the source side, while the node agent is responsible for source identity verification, target location resolution, and subsequent forwarding. Through the local IPC connection, the first container network agent does not need to maintain cross-cluster routing and authentication forwarding channels itself. Simultaneously, the node agent can obtain IPC peer process credentials provided by the operating system to verify the source container and its logical domain.

[0037] In step S220, the node agent determines the target location information of the second container instance based on the virtual address within the domain, and forwards the target location information and business data to the relay server through the authentication forwarding channel between the node agent and the relay server.

[0038] Specifically, the node agent can query the corresponding domain routing directory based on the virtual address within the domain to determine the target location information of the second container instance. Here, the domain routing directory can be a directory that records the correspondence between the virtual addresses within the domain and the actual deployment locations of each container instance within the same logical domain, and can include information such as the virtual address within the domain, the cluster it belongs to, the node it resides on, and its real network address. The target location information determined in this way can be used to indicate the actual deployment location of the second container instance, and may include, for example, information such as the cluster where the second container instance resides, the node it resides on, and its real network address.

[0039] After determining the target location information, the node agent can send the target location information and business data to the relay server through an authenticated forwarding channel. The authenticated forwarding channel can be a communication channel established between the node agent and the relay server. Besides carrying the target location information and business data, it can also verify the logical domain identity of the source side through an authentication mechanism, allowing the relay server to perform subsequent forwarding. In one implementation, the authenticated forwarding channel can be implemented using a Secure Shell (SSH) connection and its carried logical channel. The node agent can reuse an established authenticated forwarding channel or establish a new one as needed.

[0040] The relay server is used to relay business data between different clusters. It can receive proxy requests from the source side and perform subsequent forwarding based on the target location information.

[0041] In step S230, the relay server forwards the business data to the second container network agent corresponding to the second container instance through the reverse channel pre-established by the proxy component in the second cluster, so that the second container network agent can pass the business data to the target business process in the second container instance.

[0042] Among them, the agent component is a functional entity deployed in the cluster to establish and maintain the communication connection between the cluster and the relay server, and can access the target resources in the cluster according to the access request of the relay server.

[0043] Because the cluster may be behind a Network Address Translation (NAT) device or firewall, the relay server may not be able to directly initiate inbound connections to the cluster. Therefore, the proxy component in the cluster can proactively initiate outbound connections to the relay server, pre-establishing a reverse channel so that the relay server can access the target resources in the cluster through this reverse channel.

[0044] Specifically, a reverse tunnel can be a reverse tunnel connection established by the proxy component actively connecting to the relay server. After the connection is established, the relay server can save the corresponding connection session so that when it needs to access target resources in the cluster where the proxy component resides, it can transmit access requests and business data to the proxy component through the established reverse tunnel connection. After receiving the access request, the proxy component can access the target resources in its cluster according to the target information in the request and forward the data through the corresponding connection. Thus, although the reverse tunnel connection is actively initiated by the proxy component in the cluster, the relay server can use this connection to access the target resources in the cluster without having to re-initiate an inbound connection to the cluster from the outside.

[0045] In step S230, since the second container instance is located in the second cluster, the relay server forwards data through a reverse channel pre-established by the proxy component in the second cluster. Specifically, the relay server can select the reverse channel established by the corresponding proxy component in the second cluster based on the target location information, and forward the business data to the second cluster through the selected reverse channel. After the business data enters the second cluster, it can reach the second container network proxy via the corresponding proxy component and necessary internal cluster forwarding. The second container network proxy can then deliver the business data to the target business process in the second container instance based on the target information.

[0046] Through steps S210 to S230, the business process in the first container instance can transmit business data to the target business process in the second container instance via the first container network proxy, node proxy, relay server, and second container network proxy. This communication can also support the reverse transmission of response data. For example, after receiving business data, the target business process in the second container instance can generate corresponding response data, which is then transmitted back to the first container network proxy via the second container network proxy through the corresponding proxy connection and forwarding channel, and then transmitted to the business process in the first container instance.

[0047] It should be understood that the first cluster and the second cluster represent the source and target clusters in a cross-cluster communication, respectively, and do not limit each cluster to only playing a specific communication role. For example, container instances in the second cluster can also act as communication initiators, accessing container instances in the first cluster using the cross-cluster container communication method described above.

[0048] For ease of understanding, Figure 3 An example of a cross-cluster container communication path according to this disclosure is shown. In cluster X, Pod A contains a first container instance and a corresponding first container network agent, Sidecar_A, and a node agent (NodeServer) is deployed on the node (Node i) where Pod A resides. In cluster Y, Pod B contains a second container instance and a corresponding second container network agent, Sidecar_B, and a proxy component (Agent) for establishing a reverse channel with a relay server is also deployed in cluster Y. For ease of explanation, in this example, the proxy component (Agent) is deployed on the node (Node j) where Pod B resides.

[0049] When a business process in the first container instance accesses a virtual address within the domain of the second container instance, Sidecar_A intercepts the corresponding network traffic (e.g., network packets) and transmits the virtual address and the processed business data to the node agent of the node where Pod A resides via a local IPC connection. The node agent determines the target location information of the second container instance based on the virtual address and sends the target location information and business data to the relay server via an authenticated SSH connection with the relay server and its carried logical channel. The relay server then forwards the business data to the second cluster Y via a reverse channel pre-established by the Agent component in the second cluster Y, and delivers it to Sidecar_B via the corresponding forwarding path. In this example, since the Agent is deployed on Node j, the business data can be forwarded internally to Sidecar_B, and then from Sidecar_B to the target business process in Pod B.

[0050] Therefore, this disclosure achieves cross-cluster target addressing through intra-domain virtual addresses. The container network agent and node agent on the source side work together to process network traffic, and combined with the authentication forwarding channel and the reverse channel pre-established in the target cluster, business data is forwarded to the target business process. Thus, cross-cluster container communication transparent to business programs can be achieved without the need for direct interoperability of the underlying cross-cluster network.

[0051] In the aforementioned cross-cluster container communication process, virtual addresses and domain routing directories within the domain can be managed uniformly by the control plane. The following section combines... Figure 4 This section introduces the process of virtual address allocation in the control plane and the generation and distribution of domain routing directories.

[0052] Figure 4 A schematic diagram illustrating the relationship between a control plane and multiple clusters according to an implementation of this disclosure is shown. Figure 4 As shown, clusters X and Y communicate with the same control plane. The control plane can receive container instance information reported by the two clusters, manage logical domains and their virtual address ranges, allocate virtual addresses to container instances belonging to the corresponding logical domains, and generate and distribute domain routing directories.

[0053] The control plane is primarily responsible for managing logical domains, virtual addresses, and domain routing directories, while the aforementioned relay server is mainly responsible for authentication, authorization, and data forwarding in cross-cluster communication. In some implementations, the relay server can be part of the control plane. In other implementations, the relay server can be deployed separately from the control plane; this disclosure does not impose any restrictions on this.

[0054] Specifically, a logical domain is used to define the range of container instances that can communicate with each other through the virtual network disclosed herein. A logical domain can span multiple clusters, and different container instances within the same cluster can belong to different logical domains, but each container instance belongs to only one logical domain. Container instances within the same logical domain can access each other through virtual addresses within the domain, and the access scope limitation of a logical domain does not affect the original cluster network of the container instances.

[0055] The container instance information reported by each cluster can include the instance's unique identifier, logical domain, cluster identifier, local namespace, instance name, node, real network address, and runtime stage. The control plane can use this information to determine valid container instances and update the instance status as container instances are created, deleted, or rebuilt.

[0056] The control plane can maintain a corresponding virtual address range for each logical domain, such as an address range represented by Classless Inter-Domain Routing (CIDR), and allocate virtual addresses to each container instance within the domain based on this virtual address range. Based on the container instance information and the allocated virtual addresses, the control plane generates a domain routing directory containing the correspondence between virtual addresses, cluster identifiers, node identifiers, and the real addresses of the container instances, and distributes this directory to the clusters associated with that logical domain for subsequent target addressing and target-side authorization.

[0057] In a Kubernetes implementation, the control plane can use Domain custom resources to store the CIDR and address allocation records of logical domains, use Pod image resources to store the Pod locations reported by each cluster, and use DomainPeer custom resources to distribute global Pod directories and corresponding domain SSH credentials to relevant clusters. The Pod image resource is used to record the location of Pods, not container image files. The same logical domain can use its own domain authentication credentials in different clusters.

[0058] In one implementation, to maintain the stability of virtual addresses before and after container instance reconstruction, the control plane can construct a stable instance key based on the cluster identifier, local namespace, and instance name of the container instance. The stable instance key is a composite identifier used to identify the logical identity of the container instance, consisting of attributes that remain unchanged before and after reconstruction, allowing the reconstructed container instance to be matched with its historical virtual address allocation record before reconstruction. The control plane can then use this stable instance key to look up the corresponding historical virtual address allocation record.

[0059] When a container instance is rebuilt, its unique identifier and real network address may change, but the stable instance key mentioned above can remain unchanged. Therefore, the control plane can look up historical virtual address allocation records based on the stable instance key of the rebuilt container instance, and if a corresponding record is found and the historical virtual address does not conflict, the historical virtual address can be reassigned to the rebuilt container instance.

[0060] If no corresponding record is found, or if the historical virtual address has been occupied by another instance, the control plane can allocate a new virtual address to the rebuilt container instance. The control plane can also deduplicate and detect conflicts in historical records, ensuring that the same virtual address is restored at most once in a single address allocation round. Furthermore, the control plane can retain a predetermined number of expired address allocation records to balance virtual address stability with address pool capacity.

[0061] In a Kubernetes implementation, a stable instance key can be constructed using the Agent namespace, local namespace, and Pod name. The address recovery described above only applies to virtual addresses within the domain; it does not require restoring the original real network address of the container instance or any disconnected service connections.

[0062] The following section provides a further detailed example of the implementation of network traffic processing, source identity verification, and authentication forwarding on the source side.

[0063] In one implementation, the first container network agent can process the network traffic sent by the first container instance through the TUN device (Tunnel device, virtual tunnel network card), user-space protocol stack and local inter-process communication client, and pass the virtual address and business data within the domain to the node agent.

[0064] Specifically, the first container network agent can create a TUN device to intercept network packets sent by the first container instance to virtual addresses within the domain. Then, it processes the intercepted network packets using a user-space protocol stack to obtain the target virtual address within the domain and the service data to be transmitted. In one implementation, the user-space protocol stack can use gVisor netstack to handle network traffic from protocols such as TCP (Transmission Control Protocol), UDP (User Datagram Protocol), and ICMP (Internet Control Message Protocol). The service data can be protocol-processed data for the agent to transmit, rather than the raw IP packets.

[0065] For different types of network traffic, appropriate proxy transmission methods can be adopted. For example, TCP traffic can be bidirectionally forwarded via byte streams; for datagrams such as UDP and ICMP, streaming frames containing a length field can be used in the streaming tunnel to preserve datagram boundaries and process message types and error information according to the semantics of the corresponding protocols. The specific encoding format of the above streaming frames can be determined according to the actual implementation.

[0066] Subsequently, the first container network agent transmits the domain virtual address and business data to the node agent via a local IPC client. In a Kubernetes implementation, the first container network agent and the node agent can communicate via Unix Sockets provided in the HostPath.

[0067] In one implementation, the node agent can read the peer process credentials of the local IPC connection provided by the operating system to obtain the peer process identifier (PID), and query the correspondence between the process and the container instance based on the process identifier to verify the identity of the source container instance.

[0068] In a Kubernetes implementation, a node agent can query the Pod UID of the peer process based on the PID of the Unix socket, and then obtain the corresponding container instance information and logical domain information based on the Pod UID. Therefore, the node agent can determine the source container and its logical domain based on the identity of the actual peer process, without relying on the logical domain identity declared in the request. This reduces the risk of malicious processes forging the source container identity or logical domain identifier and using the node agent to access container instances in other logical domains, thus improving the security of cross-cluster communication.

[0069] After completing the source identity verification, the node agent can query the virtual address within the target domain in the domain routing directory of the verified logical domain to determine the target node and real network address of the second container instance in the second cluster.

[0070] In one implementation, a node agent can maintain separate SSH connection pools for different logical domains. Each connection pool includes one or more physical SSH connections, and each physical SSH connection can carry multiple logical channels. The connection pools for different logical domains are independent of each other, and the same logical domain uses its own authentication credentials to maintain the corresponding connection pool in different clusters.

[0071] Specifically, for cross-cluster communication requests, the node agent can determine the corresponding SSH connection pool based on the verified source logical domain, and prioritize selecting the connection with the fewest active channels from the healthy physical connections. A logical channel is then established on the selected connection to carry the current proxy request, serving as the aforementioned authentication forwarding channel. In one implementation, the node agent can establish a physical SSH connection using the SSH Certificate of the corresponding logical domain and carry target location information and business data through a direct-tcpip channel.

[0072] Since a single physical SSH connection can support multiple channels, the number of physical connections does not need to correspond one-to-one with the number of business requests. For example, a connection pool can maintain four physical SSH connections. When there are four concurrent requests, they can be distributed among the four physical connections. When the number of concurrent requests increases to forty, the requests can still be distributed across multiple channels on the existing four physical connections without needing to establish a new physical SSH connection for each request. Maintaining multiple physical connections also prevents all requests from being concentrated on a single physical connection.

[0073] Furthermore, the connection pool can be configured with a maximum number of physical SSH connections. When the request queue is too long and the number of connections has not yet reached the preset limit, the node agent can use the authentication credentials of the corresponding logical domain to establish a new physical SSH connection and add it to the connection pool to participate in subsequent request allocation. In one implementation, the maximum number of physical connections in the connection pool can be set to 8, for example, maintaining 4 to 8 physical connections. The above numbers are configurable parameters and do not constitute a necessary limitation.

[0074] When establishing a new physical connection fails, if healthy old connections still exist in the connection pool, the node agent can choose the connection with the fewest active channels to continue carrying the request. When an existing physical connection fails, it can be excluded from the set of available connections, and subsequent requests can be carried through other healthy connections. It should be noted that whether the business flow established on the failed connection can be recovered depends on the corresponding protocol and business processing mechanism, and lossless migration of the original business flow is not required.

[0075] Furthermore, for multiple requests triggering the same round of reconnection operations simultaneously within the same logical domain, the node agent can employ a single-fly reconnection mechanism, where only one execution unit performs the actual reconnection, while other requests wait for the reconnection result. Retrying after connection failure can utilize exponential backoff and random perturbation to reduce redundant or concentrated retries. The connection pool can also detect the availability status of physical connections through Keepalive and reclaim connections with no active channels that meet the idle reclamation criteria.

[0076] Therefore, the node agent can reduce the overhead of establishing repeated authentication connections, distribute the handling of concurrent requests, and continue to process subsequent requests using other healthy connections when some connections are unavailable by using multiple physical SSH connections and channel reuse within the same logical domain.

[0077] In one implementation, a node proxy can establish a proxy connection for the traffic to be forwarded, and carry a target Uniform Resource Identifier (URI) at the beginning of the proxy connection to pass the original target information to the target side. The target URI can be used to represent information such as the original target protocol, virtual target, and service port.

[0078] After the proxy connection reaches the second container network proxy via the relay server and the reverse channel of the target cluster, the second container network proxy can read the target URI, determine the original target protocol, virtual target, and service port, and execute the corresponding target connection or protocol processing accordingly. For example, for TCP services, the second container network proxy can obtain the original service port based on the target URI and establish a connection to the target service process; for protocols that do not have the concept of a service port, it can perform processing based on the corresponding target information and protocol semantics.

[0079] In one implementation, the node agent can also determine whether the target container instance and the source container instance are located in the same cluster based on the retrieved target location information. If they are in the same cluster, the node agent can directly connect to the container network proxy port corresponding to the real network address of the target container instance through the cluster's internal network and forward business data. If they are in different clusters, the node agent sends the target location information and business data to the relay server through the authentication forwarding channel of the corresponding logical domain.

[0080] In one implementation, before forwarding business data through the reverse channel, the relay server can verify the domain authentication credentials provided by the node agent when establishing the authentication forwarding channel, and determine the logical domain and the first cluster to which the first container instance belongs based on the verified credentials.

[0081] Subsequently, the relay server determines the target real address of the second container instance based on the target location information, and reads the domain routing directory of the same logical domain corresponding to the second cluster to query whether the target real address is recorded in the directory.

[0082] In a Kubernetes implementation, the relay server can read the corresponding DomainPeer with the same name in the target Agent namespace based on the authenticated logical domain, and check whether the target's real address (LocalIP) belongs to the set of target instances recorded by that Domain.

[0083] If the target's actual address is recorded in the corresponding domain routing directory, the relay server allows the establishment of a forwarding connection to the second container network proxy through a reverse channel; otherwise, it can reject the forwarding request. Thus, by combining the authenticated source logical domain and the target-side same-domain directory, secondary authorization on the target side is achieved for cross-cluster forwarding. This ensures that subsequent forwarding is only allowed if the target container instance belongs to the source logical domain, thereby reducing the risk of erroneous forwarding or unauthorized access due to inconsistencies between the target location and the access authorization status, and improving the security of cross-cluster communication.

[0084] During cross-cluster container communication, the relay server can select a pre-established reverse channel in the second cluster for forwarding business data based on the target location information. The following section combines... Figure 5 This paper introduces the hierarchical selection mechanism of the target-side reverse channel.

[0085] Figure 5 A schematic diagram of reverse channel selection according to some embodiments of the present disclosure is shown. For example... Figure 5 As shown, the second cluster Y may include a cluster agent component (cluster agent, i.e., Agent Y) deployed on the cluster control node, and node agent components (node ​​agents, such as Agent Y-node1 and Agent Y-node2) deployed on multiple business nodes. The cluster agent can establish a general reverse channel as a cluster, while each node agent can establish a dedicated reverse channel as its respective node.

[0086] In one implementation, the relay server can determine the target node where the second container instance is located based on the target location information, and try the following reverse channels in sequence, with priority marked as I, II and III in the figure.

[0087] First, try the dedicated reverse channel of the target node. For example, when the second container instance is located on Node1, the relay server can prioritize the dedicated reverse channel pre-established by the node agent on Node1 (i.e., Agent Y-node1) to send the business data to the target node and then forward it to the second container network agent.

[0088] If the dedicated reverse channel to the target node is unavailable, try the general reverse channel of the second cluster (i.e., Agent Y). The relay server can send business data to the second cluster Y through the general reverse channel pre-established by the cluster Agent, and then forward it to Node1 and the second container network agent within the cluster.

[0089] Finally, if the aforementioned two reverse channels are unavailable, try reverse channels from other nodes in the second cluster. For example, the relay server can choose a pre-established reverse channel on Node2's Agent (i.e., Agent Y-node2) to send business data into the second cluster Y, and then Node2's Agent forwards the business data to Node1. This internal cluster forwarding can be achieved through direct communication between Node2 and Node1, or through relay via the cluster control node, depending on the network connectivity within the cluster.

[0090] Therefore, when the dedicated reverse channel of the target node is unavailable, the relay server can continue to attempt forwarding using the cluster's general reverse channel or the reverse channel of other nodes, thereby improving the availability of cross-cluster communication in the event of a reverse connection failure of the target node.

[0091] Through the above implementation methods, this disclosure can establish a logical domain-based virtual network between multiple clusters and realize cross-cluster container communication using authentication forwarding channels and reverse channels. The following describes the collaborative process of the above mechanism from two stages: network establishment and data forwarding.

[0092] During the network establishment phase, the control plane allocates virtual addresses within the same logical domain to container instances belonging to the same logical domain based on the logical domain configuration and the container instance information reported by each cluster. It also generates and distributes the domain routing directory and corresponding authentication information. For container instances that have undergone reconstruction, the control plane can prioritize restoring historical virtual addresses based on the stable instance key. After obtaining the corresponding virtual address configuration, the container network agents in each cluster can create TUN devices and start the user-space protocol stack. Simultaneously, the agent components in each cluster can proactively establish reverse channels to the relay server, enabling the relay server to access target resources in the corresponding cluster via these reverse channels.

[0093] After completing the above preparations, the business processes within the container instance can access other container instances in the same logical domain via virtual addresses within the domain. The source container network agent intercepts the corresponding network traffic, processes it through the user-space protocol stack, and then transmits the target information and business data to the node agent via a local IPC connection. The node agent verifies the source container instance and its logical domain based on the IPC peer process credentials and resolves the target location in the corresponding domain routing directory. For target container instances located in the same cluster, the node agent can directly connect to the target container network agent through the cluster's internal network; for target container instances located in different clusters, the target location information and business data are sent to the relay server through the authenticated SSH connection and logical channel of the corresponding logical domain.

[0094] After receiving a cross-cluster proxy request, the relay server determines the source logical domain based on the verified domain authentication credentials and performs secondary authorization on the target side in conjunction with the corresponding same-domain directory of the target cluster. Once authorization is successful, the relay server selects a pre-established reverse channel for the target cluster based on the target location information and sends the business data to the target container network proxy. The target container network proxy then executes the connection or corresponding protocol processing to the target business process based on the original target protocol, virtual target, and business port information. During this process, if the target node's dedicated reverse channel is unavailable, the relay server can also try the cluster's general reverse channel and other node reverse channels in sequence to complete the forwarding on the target side.

[0095] After communication is established, the source container network agent and the target container network agent can conduct bidirectional data transmission through the corresponding forwarding channels. During the above process, historical virtual address recovery, SSH connection pool expansion, and reverse channel rollback are all performed according to specific conditions and are not required for every communication.

[0096] In some embodiments, this disclosure can also be implemented as a distributed system. Figure 6 A schematic diagram of the architecture of a distributed system according to some embodiments of the present disclosure is shown. For example... Figure 6 As shown, the distributed system 600 may include a first cluster 620, a second cluster 630, a relay server 610, and a communication subsystem 640. The communication subsystem is configured to enable the first cluster, the second cluster, and the relay server to collaboratively execute the aforementioned cross-cluster container communication method.

[0097] Specifically, the first cluster may include a first container instance, a first container network proxy corresponding to the first container instance, and node proxies deployed on the nodes where the first container instance resides, used to perform source-side traffic interception, target addressing, and authentication forwarding. The second cluster may include a second container network proxy corresponding to the second container instance and a proxy component for establishing a reverse channel. The relay server can receive proxy requests sent by the first cluster through the authentication forwarding channel, and forward business data to the second container network proxy through the reverse channel pre-established by the second cluster, thereby delivering it to the target business process in the second container instance.

[0098] It should be noted that the first cluster and the second cluster represent the source cluster and the target cluster in a cross-cluster communication, respectively, and do not limit each cluster to only assume a specific communication role. For example, a container instance in the second cluster can also act as a communication initiator and access a container instance in the first cluster using the aforementioned method. Furthermore, the distributed system 600 may also include a third cluster or more clusters. Container instances belonging to the same logical domain in different clusters can all perform cross-cluster communication using the methods disclosed herein, and multiple clusters can concurrently execute the corresponding communication processes.

[0099] In some implementations, the distributed system 600 may also include a control plane, with the first and second clusters communicating with it respectively. The control plane can manage logical domains, allocate virtual addresses within the domains, generate and distribute domain routing directories, and restore historical virtual addresses when certain conditions are met to support the aforementioned cross-cluster container communication. In some implementations, a relay server can be part of the control plane, where the control plane is responsible for logical domain management, virtual address allocation, and the generation and distribution of domain routing directories, while the relay server is responsible for authentication, authorization, and business data forwarding in cross-cluster communication. In other implementations, the relay server may be deployed separately from the control plane; this disclosure does not impose any restrictions on this.

[0100] The communication subsystem can be a logical division of cross-cluster container communication functions. These functions can be distributed across different components in the first cluster, the second cluster, and the relay server, without requiring them all to be deployed on the same physical device or process. The communication subsystem can achieve the cross-cluster container communication method described above through the coordinated operation of these components.

[0101] To further illustrate the functional composition of the communication subsystem, the following will combine... Figure 7 This document describes an exemplary modular division of the communication subsystem. This modular division is for illustrative purposes only and does not limit the specific deployment location or implementation form of each module.

[0102] Figure 7 A schematic diagram of the functional modules of a communication subsystem according to some embodiments of the present disclosure is shown. For example... Figure 7 As shown, the communication subsystem 740 may include a control plane resource management module 7401, an instance status reporting module 7402, a virtual address allocation module 7403, a domain routing directory distribution module 7404, a container network proxy module 7405, a source identity resolution module 7406, a hierarchical routing module 7407, a domain authentication tunnel module 7408, a relay authorization module 7409, and a reverse channel module 7410.

[0103] Specifically, the control plane resource management module 7401 is used to maintain logical domains, virtual address ranges, address allocation records, container instance locations, and domain authentication credentials. The instance status reporting module 7402 is used to report information such as the identity, logical domain, deployment location, and running status of container instances to the control plane. The virtual address allocation module 7403 is used to allocate virtual addresses to container instances within the domain and can perform historical virtual address recovery, conflict detection, and historical record management based on stable instance keys. The domain routing directory distribution module 7404 is used to generate and distribute the mapping between virtual addresses within the domain and the actual locations of container instances, as well as the corresponding domain authentication information.

[0104] The container network proxy module 7405 is used to intercept container-side network traffic, process user-space protocols, transmit local IPC messages, and process target service connections. The source identity resolution module 7406 is used to determine the source container instance and its logical domain based on the peer process credentials of the local IPC connection. The hierarchical routing module 7407 is used to resolve the target location and select either a direct forwarding path within the same cluster or a cross-cluster authentication relay path based on the target's cluster location.

[0105] The domain authentication tunnel module 7408 maintains SSH connection pools divided by logical domains and performs logical channel multiplexing, physical connection selection, expansion, reconnection, and connection status management. The relay authorization module 7409 verifies the source-side domain authentication credentials and performs secondary authorization on the target side based on the same-domain routing directory corresponding to the target cluster. The reverse channel module 7410 manages the cluster's general reverse channel and node-specific reverse channels, and performs reverse channel selection and necessary intra-cluster forwarding based on the target location information.

[0106] It should be noted that the above modules are logical functional divisions and can be distributed across the control plane, various clusters, and relay servers according to actual deployment needs. They do not need to correspond one-to-one with the deployment entities in the distributed system 600. The specific implementation of each module and its collaboration process can be found in the aforementioned method implementation methods, which will not be repeated here.

[0107] The cross-cluster container communication method and distributed system of this disclosure have been described above in conjunction with the embodiments and accompanying drawings. This disclosure achieves transparent communication between container instances in different clusters through the collaborative processing of logical domain virtual addresses, container network proxies, and node proxies, as well as authentication forwarding channels and pre-established reverse channels in the target cluster. This allows business applications to communicate without needing to be aware of the cross-cluster tunnel, the target cluster, or the real address, and without needing to modify the underlying container network interfaces of each cluster or configure cross-cluster Layer 3 interoperability routing. The proxy components in each cluster can proactively establish outbound reverse channels, thereby adapting to network environments behind NAT devices or firewalls.

[0108] Furthermore, this disclosure determines the source container and its logical domain based on process credentials provided by the operating system, reducing the risk of source identity or logical domain being forged; by performing target addressing and target-side authorization based on the same logical domain directory, it reduces the risk of erroneous forwarding or unauthorized access caused by inconsistencies between routing and authorization states; for targets within the same cluster, direct internal paths can be used to reduce unnecessary hops; by using SSH connection pools and logical channel reuse divided by logical domain, it reduces the number of repeated authentication handshakes and physical connections; by restoring historical virtual addresses using stable instance keys, it mitigates the impact of container reconstruction on distributed task addressing relationships; and it can uniformly support different types of network traffic such as TCP, UDP, and ICMP.

[0109] In one embodiment, this disclosure can also be implemented as a non-transitory machine-readable storage medium storing executable code that, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0110] In one embodiment, this disclosure can also be implemented as a computer program product, including computer program instructions that, when executed by a processor, implement the method described above.

[0111] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. For this purpose, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] Various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A cross-cluster container communication method for communication between a first container instance in a first cluster and a second container instance in a second cluster, wherein the first container instance and the second container instance belong to the same logical domain, the method comprising: The first container network agent corresponding to the first container instance intercepts the network traffic destined for the second container instance's domain virtual address, and transmits the domain virtual address and the business data in the network traffic to the node agent of the node where the first container instance is located through a local inter-process communication connection. The node agent determines the target location information of the second container instance based on the virtual address within the domain, and forwards the target location information and the business data to the relay server through the authentication and forwarding channel between the node agent and the relay server. as well as The relay server forwards the business data to the second container network proxy corresponding to the second container instance through a reverse channel pre-established by the proxy component in the second cluster, so that the second container network proxy can pass the business data to the target business process in the second container instance.

2. The method as described in claim 1, wherein, The first cluster and the second cluster communicate with the same control plane, and the method further includes: The control plane receives container instance information belonging to the logical domain reported by the first cluster and the second cluster, respectively. Based on the virtual address range of the logical domain, allocate virtual addresses to container instances within the logical domain; Based on the container instance information and the allocated virtual address, a domain routing directory for the logical domain is generated. The domain routing directory includes the correspondence between virtual addresses, cluster identifiers, node identifiers, and the real addresses of the container instances; and The domain routing directory is distributed to the first cluster and the second cluster respectively.

3. The method as described in claim 2, wherein, Assigning virtual addresses to container instances within the logical domain includes: Construct a stable instance key based on the cluster identifier, local namespace, and instance name of the container instance; In response to the container instance reconstruction, historical virtual address allocation records are retrieved based on the stable instance key; and If a corresponding historical virtual address allocation record is found and there is no conflict with the corresponding historical virtual address, the historical virtual address is allocated to the rebuilt container instance.

4. The method according to any one of claims 1-3, wherein, The first container network proxy corresponding to the first container instance intercepts network traffic destined for the second container instance's intra-domain virtual address, and transmits the intra-domain virtual address and the service data in the network traffic to the node proxy of the node where the first container instance resides via a local inter-process communication connection, including: The network traffic was intercepted by the virtual tunnel network interface card (TUN) device created through the first container network agent; The intercepted network traffic is processed by the user-space protocol stack in the first container network proxy to obtain the virtual address within the domain and the service data; and The domain virtual address and the business data are transmitted to the node agent via the local inter-process communication client in the first container network agent.

5. The method according to any one of claims 1-3, further comprising: The node agent reads the peer process credentials of the local inter-process communication connection provided by the operating system to obtain the peer process identifier; Based on the peer process identifier, query the correspondence between the process and the container instance to verify that the peer process of the local inter-process communication connection belongs to the first container instance; and Based on the identity information of the first container instance, the correspondence between the container instance and the logical domain is queried to verify that the first container instance belongs to the logical domain. The process of determining the target location of the second container instance by the node agent based on the virtual address within the domain includes: The node agent queries the domain routing directory of the verified logical domain for the virtual address within the domain to determine the target location of the second container instance.

6. The method according to any one of claims 1-3, wherein, The node agent maintains a Secure Shell Protocol (SSH) connection pool for different logical domains. Each SSH connection pool includes one or more physical SSH connections, and each physical SSH connection can carry multiple logical channels. The authentication forwarding channel is a logical channel established by the node agent on the physical SSH connection selected by the SSH connection pool corresponding to the logical domain.

7. The method according to any one of claims 1-3, wherein, Before the relay server forwards the service data through the reverse channel, the method further includes: The relay server verifies the domain authentication credentials provided by the node agent when establishing the authentication forwarding channel, and determines the logical domain to which the first container instance belongs and the first cluster based on the verified domain authentication credentials. The relay server determines the target real address of the second container instance based on the target location information; The relay server reads the domain routing directory of the logical domain corresponding to the second cluster and queries whether the target real address is recorded in the domain routing directory; and In response to the target real address being recorded in the domain routing directory, a forwarding connection to the second container network agent is allowed to be established through the reverse channel.

8. The method according to any one of claims 1-3, wherein, The relay server forwards the business data to the second container network proxy corresponding to the second container instance via a reverse channel pre-established by the proxy component in the second cluster, including: The relay server, based on the target location information, sequentially attempts the target node-dedicated reverse channel of the second cluster, the general reverse channel of the second cluster, and the reverse channel of any node in the second cluster to forward the service data to the second container network proxy.

9. A distributed system, comprising a first cluster, a second cluster, a relay server, and a communication subsystem, wherein, The communication subsystem is configured to enable the first cluster, the second cluster, and the relay server to collaboratively execute the cross-cluster container communication method as described in any one of claims 1-8.

10. A computer program product comprising computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1-8.