GRPC broken line reconnection method
By introducing the etcd service registration and discovery mechanism into the gRPC framework, the gRPC client can automatically discover and reconnect to changes in the server address, solving the problems of connection interruption reconnection delay and manual configuration in existing technologies, and achieving efficient and stable connection management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2026-03-10
AI Technical Summary
The gRPC framework's interrupted connection reconnection mechanism delays the detection of connection anomalies when there is no active data exchange, affecting the high availability and connection stability of real-time applications. Furthermore, when the server address changes, the client configuration needs to be manually updated, which is inefficient and prone to errors.
etcd is introduced as a service registration and discovery middleware. The gRPC server registers its address with etcd, and the gRPC client obtains the address from etcd and subscribes to change notifications. Heartbeat requests are sent periodically, and an exponential backoff strategy is configured for automatic reconnection.
It enables rapid automatic reconnection in the event of network anomalies or changes in server address, improving service availability and stability, enhancing the resilience of distributed systems, and reducing the tediousness and error risk of manual operations.
Smart Images
Figure CN121644628A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer application technology, and in particular to a gRPC disconnection and reconnection method. Background Technology
[0002] gRPC (Google Remote Procedure Call) is a high-performance, open-source, and general-purpose remote procedure call framework designed for both server-side and protocol-side implementations, and built on the HTTP / 2 standard. It introduces bidirectional streaming, flow control, header compression, and multiplexing requests over the same TCP connection. These features not only optimize performance on mobile devices but also improve energy efficiency and reduce memory consumption. With gRPC, clients can directly invoke server-side applications located on another machine, as easily as invoking local objects. Therefore, gRPC simplifies the development process for distributed applications and services.
[0003] Fast reconnection is a common feature in network communication, especially suitable for applications with strict real-time requirements. In the gRPC framework, the reconnection mechanism ensures that the client can quickly re-establish the connection after it is interrupted, thus restoring communication. Although gRPC supports persistent connections and status code-based automatic reconnection by default, without active data exchange, the client may not be able to detect connection anomalies in a timely manner, increasing reconnection latency and impacting the high availability and connection stability of real-time applications. Furthermore, when the server address changes, relying solely on gRPC's built-in automatic reconnection mechanism is insufficient for seamless migration. In this case, it is necessary to manually update the server address in the client configuration and then restart the client to re-establish the connection. This method is inefficient and error-prone, potentially reducing service availability. Summary of the Invention
[0004] Therefore, it is necessary to provide a method for gRPC disconnection and reconnection that can automatically detect connection interruption issues and quickly re-establish the connection.
[0005] To achieve the above objectives, this invention provides a method for rapid reconnection of gRPC after disconnection, comprising the following steps:
[0006] S1, Set up the etcd service registration and discovery environment;
[0007] S2, the gRPC server registers its service address with etcd using a unique registration key;
[0008] S3, the gRPC client obtains the service address of the gRPC server from etcd through the registration key, establishes an initial connection with the gRPC server, and calls the interface of the gRPC server to interact with the gRPC server for data according to business needs;
[0009] S4, the gRPC client periodically retrieves the list of service addresses of the gRPC server from etcd at the first time interval, and subscribes to etcd for notifications of changes in the service address of the gRPC server. When the service address of the gRPC server changes, the connection with the gRPC server is re-established.
[0010] S5, define and implement a heartbeat method for each gRPC service. The gRPC client periodically calls the heartbeat method at a second time interval to send a heartbeat request to the gRPC server. If no heartbeat response is received within the preset heartbeat response time, the connection with the gRPC server is re-established.
[0011] Preferably, the WithDefaultServiceConfig function configures the retry status code RetryableStatusCodes, and configures the maximum number of attempts MaxAttempts, the default backoff time InitialBackoff, the maximum backoff time MaxBackoff, and the backoff time increment multiplier BackoffMultiplier.
[0012] Preferably, in step S4, after the gRPC client periodically obtains the list of service addresses of the gRPC server from etcd at a first time interval, it saves the list of gRPC service addresses to the local cache of the gRPC client.
[0013] Preferably, in step S4, the gRPC client periodically obtains the list of service addresses of the gRPC server from etcd at a first time interval, and subscribes to etcd for service address change notifications of the gRPC server. This is implemented by the resolver in the gRPC client. The resolver compares the new address with the old address, saves the new address to the local cache of the gRPC client, and deletes the old address with the same registration key as the new address from the local cache of the gRPC client.
[0014] Preferably, in steps S4 and S5, the specific method for re-establishing the connection with the gRPC server is as follows:
[0015] Create a new connection instance and re-establish a connection with the gRPC server using the service address of the gRPC server that was successfully connected to in the gRPC client's local cache.
[0016] If the connection fails, poll other service addresses in the local cache of the gRPC client and try to re-establish the connection with the gRPC server in turn.
[0017] If a connection to the gRPC server cannot be successfully established using all service addresses cached in the gRPC client's local cache, a new service address is obtained from etcd, and then a new connection is established with the gRPC server using the new service address.
[0018] Preferably, the specific method for re-establishing the connection with the gRPC server further includes:
[0019] Set a maximum number of retries. For each service address in the gRPC client's local cache, if the connection with the gRPC server cannot be successfully established after the maximum number of retries is exceeded, the service address will be removed from the gRPC client's local cache.
[0020] Preferably, in steps S4 and S5, the re-establishment of the connection with the gRPC server is implemented by the connection manager of the gRPC client.
[0021] Preferably, in step S1, setting up the etcd service registration and discovery environment specifically refers to setting up an etcd cluster.
[0022] Preferably, in step S2, the specific method for registering one's service address with etcd using a unique registration key includes:
[0023] Define a unique registration key for each gRPC service. The registration key is named in a hierarchical structure. Multiple service nodes with the same business function use the same prefix for the registration key.
[0024] The gRPC server registers with etcd using the registration key as the key and the gRPC server IP address and port number as the value.
[0025] Preferably, the specific method for the gRPC server to register its service address with etcd using a unique registration key when starting up further includes:
[0026] The gRPC server sets a TTL during registration and sends a keep-alive request to etcd within the TTL.
[0027] This invention offers the following advantages: The solution introduces etcd as middleware for service registration and discovery. The gRPC server registers its service address with etcd, and the gRPC client obtains the service address from etcd. The gRPC client subscribes to service address change notifications from etcd, and upon detecting a service address change, automatically re-establishes the connection with the gRPC client using the new service address. Furthermore, the gRPC client periodically sends heartbeat requests to the gRPC server; if no heartbeat response is received within a preset time, the connection with the gRPC client is re-established. This solution can promptly detect network anomalies when data transmission is not occurring and automatically reconnect, shortening reconnection latency. It also responds quickly and automatically reconnects when the gRPC server's service address changes, eliminating the need for manual intervention, thus improving service availability, ensuring service continuity and stability, enhancing the resilience of the distributed system, and improving the user experience. Attached Figure Description
[0028] The accompanying drawings, which are provided to further illustrate the invention and constitute a part of this invention, are illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention.
[0029] Figure 1 This is a schematic diagram illustrating a scenario of disconnection and reconnection of the gRPC client, etcd, and gRPC server provided in an embodiment of the present invention.
[0030] Figure 2 This is a schematic diagram illustrating the steps of the gRPC disconnection and reconnection method provided in an embodiment of the present invention.
[0031] Figure 3 This is a schematic diagram illustrating the interaction scenario between multiple gRPC clients and etcd and gRPC servers provided in an embodiment of the present invention. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0033] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. It should be understood that the embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.
[0034] like Figures 1-3As shown, this embodiment of the invention provides a gRPC disconnection and reconnection method, including the following steps:
[0035] S1. Set up the etcd service registration and discovery environment.
[0036] This invention introduces etcd as middleware for service registration and discovery. The gRPC server registers its service address with etcd, and the gRPC client obtains the service address of the gRPC server from etcd. To ensure the reliability of the etcd service, in some embodiments of this invention, step S1, setting up the etcd service registration and discovery environment, specifically refers to setting up an etcd cluster.
[0037] An etcd cluster is a distributed system consisting of multiple nodes that communicate with each other to provide services. Each node stores complete data, and the Raft protocol ensures that the data maintained by each node is consistent. To prevent system unavailability caused by the failure of a single etcd node, an etcd cluster is built to provide service registration and service discovery support, thereby enhancing the system's fault tolerance.
[0038] After the etcd server starts, it listens for requests from etcd clients via a port number. The gRPC server and gRPC clients access the etcd server via IP address and port number.
[0039] The steps to set up an etcd cluster are as follows:
[0040] 1. Install the etcd software on each server;
[0041] 2. Configure etcd nodes: For each etcd node, create a configuration file that includes the following information: node name, listening address and port, a list of addresses of other nodes in the cluster, data storage directory, and a cluster token. The cluster token is used to join the cluster, and each node's cluster token is unique. Select a node in the cluster as the first member, and then initialize that node.
[0042] 3. Start the etcd cluster:
[0043] Start the etcd process on each node, specifying the path to the configuration file in the startup command to ensure that each node can communicate with each other. After the first node starts successfully and is running, add other nodes to the cluster by running a command similar to: `etcdctl member add`. <member-name>https: / / <member-ip> : <member-port>Then use the etcdctl tool to verify the cluster state, ensure that all nodes have successfully joined and are in normal operation. You can run a command similar to the following to get the health status of the cluster: etcdctl cluster-health.
[0044] 4. Configure client access:
[0045] Need to specify multiple etcd node access addresses for gRPC client and gRPC server, if TLS security connection is enabled, you also need to configure appropriate certificate and key information. If you need to take etcd service node offline, you need to follow the step-by-step offline strategy, backup data before offline, and update the etcd access address of the gRPC client and gRPC server after offline.
[0046] In some embodiments of the application, a single-machine version of etcd is built, there is only one etcd node, and etcd runs in the form of a background process. The address of the etcd node is configured in the gRPC client and gRPC server.
[0047] S2, the gRPC server registers its service address through a unique registration key in etcd.
[0048] In some embodiments of the application, in step S2, the gRPC server registers its service address through a unique registration key in etcd when starting, and the specific method includes:
[0049] A unique registration key is defined for each gRPC service. The registration key uses a hierarchical structure, and multiple service nodes with the same business function use the same prefix registration key. The gRPC server registers with etcd using the registration key as the key and the gRPC server IP address and port number as the value.
[0050] The gRPC server sets TTL when registering, and sends a keep-alive request to etcd within TTL (time to live).
[0051] Using the same prefix in a hierarchical structure facilitates classification, viewing, and management. If the registration key is not unique, the service address registered first will be overwritten by the registration address of the same registration key.
[0052] etcd supports using TTL function to automatically delete expired key-value pairs. If the gRPC server does not send a keep-alive request to etcd within TTL, etcd will automatically delete the key-value pair registered by the gRPC server. Therefore, if a gRPC server goes offline, there is no need to actively cancel the registration interface, and etcd will automatically clear the expired registration key-value pair.
[0053] S3, the gRPC client obtains the service address of the gRPC server from the etcd through the registration key, creates an initial connection with the gRPC server, and calls the interface of the gRPC server according to the business needs to interact with the gRPC server.
[0054] The gRPC client includes a parser and a connection manager, the parser is used to obtain the service address of the gRPC server from the etcd, and the connection manager is used to create, maintain and close the gRPC connection. The registration key of the gRPC server is configured in the gRPC client, and the connection manager obtains the service address of the gRPC server from the etcd through the registration key by means of the parser, and then creates a connection instance according to the service address to establish an initial connection with the gRPC server. As shown in Figure 3 The gRPC client is usually integrated into different microservices to establish connections with the etcd and the gRPC server.
[0055] The gRPC development package itself also supports automatic reconnection, and in the embodiment of the application, the automatic reconnection function of the gRPC development package is enabled to obtain the optimal disconnection reconnection effect. Specifically, the gRPC disconnection reconnection method further includes: configuring a retry status code RetryableStatusCodes in a WithDefaultServiceConfig function to trigger the automatic reconnection mechanism of the gRPC bottom layer when the gRPC server returns the corresponding error code, and configuring a maximum attempt number MaxAttempts, a default backoff time InitialBackoff, a maximum backoff time MaxBackoff and a backoff time increase multiplier BackoffMultiplier. Among them, the parameters default backoff time InitialBackoff, maximum backoff time MaxBackoff and backoff time increase multiplier BackoffMultiplier are related configurations of the exponential backoff strategy, and the exponential backoff strategy is introduced to avoid excessive load on the server caused by frequent connection attempts. In the process of reconnection, the waiting time is increased in an exponential manner. Through the above configuration, if the gRPC client appears an exception corresponding to the configuration of RetryableStatusCodes during the connection with the etcd or the gRPC server, the gRPC client will automatically reconnect according to the exponential backoff strategy.
[0056] In the process of gRPC client calling the interface of gRPC server to interact with gRPC server, if the RetryableStatusCodes returned by gRPC server has a problem, the gRPC client will automatically reconnect. If the information returned by gRPC server indicates that the gRPC connection state is abnormal, such as the gRPC connection state becomes non-ready state, the connection manager of gRPC client will recreate the connection instance and reconnect with gRPC server. The above reconnection process is triggered only when gRPC client and gRPC server interact with each other. However, the data interaction between gRPC client and gRPC server is intermittent, and if the network is abnormal when there is no data interaction, it cannot be discovered and processed in time, which brings hidden dangers to the next data transmission and affects the real-time performance of the business.
[0057] The gRPC connection itself is a long connection by default, and in the embodiment of the application, the read and write timeout time of the gRPC connection is configured in the function WithDefaultServiceConfig.
[0058] S4, the gRPC client periodically acquires the service address list of the gRPC server from the etcd at a first time interval, and subscribes to the service address change notification of the gRPC server from the etcd, and when the service address of the gRPC server changes, the connection with the gRPC server is re-established.
[0059] In the embodiment of the application, in the step S4, after the gRPC client acquires the service address list of the gRPC server from the etcd at a first time interval, the gRPC service address list is saved to the local cache of the gRPC client. Because there can be multiple service nodes with the same business function, and the registration keys used by each service node are different, the resolver obtains the service addresses of all service nodes according to the configured multiple registration keys and stores them in the local cache.
[0060] In the embodiment of the application, the first time interval is set according to specific circumstances.
[0061] In the step S4, the gRPC client periodically acquires the service address list of the gRPC server from the etcd at a first time interval, and subscribes to the etcd for a service address change notification of the gRPC server, which is implemented by a parser in the gRPC client. Specifically, the gRPC client subscribes to the service address change notification of the gRPC server through a watch API of the etcd. The parser compares the new address with the old address, saves the new address to the local cache of the gRPC client, deletes the old address of the same registration key as the new address from the local cache of the gRPC client, and the connection manager re-establishes the connection with the gRPC server according to the new address.
[0062] By subscribing to the service address change of the gRPC server to the etcd, the gRPC client can quickly respond when the address of the gRPC server changes, and re-establish the connection with the gRPC server using the new service address, thereby reducing the change cost of the address of the gRPC server. When a new service of micro-service is put into operation, the new version of the gRPC server is usually put into operation on a new server, and the service address of the gRPC server will change. In the prior art, the gRPC client address configured in the gRPC client needs to be manually modified synchronously, and the gRPC client needs to be restarted, so that the gRPC client can re-establish the connection with the gRPC server using the new service address. In a large-scale business, the above manual operation is tedious and prone to errors. However, by using the scheme of the embodiment of the present application, after the service address of the gRPC server changes, the gRPC client can quickly establish the connection with the gRPC server using the new service address, without manual operation, thereby improving the availability of the service.
[0063] S5, a heartbeat method is defined and implemented for each gRPC service, and the gRPC client periodically calls the heartbeat method to send a heartbeat request to the gRPC server at a second time interval, and if no heartbeat reply is received within a preset heartbeat reply time, the connection with the gRPC server is re-established.
[0064] In some embodiments of the present application, the gRPC heartbeat interface is defined using Protocol Buffers, and go language code files are generated. The gRPC server and the gRPC client implement different functions to achieve the purpose of the gRPC client calling the heartbeat method of the gRPC server through the gRPC connection. When the gRPC client calls the heartbeat method, the gRPC server returns a successful response. If no heartbeat response is received within the preset heartbeat response time, it indicates that the network between the gRPC client and the gRPC server may have a problem, and the connection with the gRPC server needs to be re-established. In practical applications, the second time interval and the preset heartbeat response time are set according to specific conditions. In some embodiments of the present application, the second time interval and the preset heartbeat response time are set to 10 seconds.
[0065] By actively performing heartbeat detection on the gRPC client, network abnormalities between the gRPC client and the gRPC server can be discovered in a timely manner, and the disconnection is quickly reconnected, thereby shortening the delay of reconnection.
[0066] In the step S4 and the step S5 in the embodiments of the present application, the connection with the gRPC server is re-established by the connection manager of the gRPC client, and the specific method is as follows:
[0067] A new connection instance is created, and the gRPC server is reconnected using the service address of the last successful connection in the local cache of the gRPC client;
[0068] If the connection fails, the other service addresses in the local cache of the gRPC client are polled, and the connection with the gRPC server is re-established in turn;
[0069] If the connection with the gRPC server cannot be successfully established through all the service addresses in the local cache of the gRPC client, a new service address is obtained from the etcd, and the connection with the gRPC server is re-established through the new service address.
[0070] In some embodiments of the present application, the specific method of re-establishing the connection with the gRPC server further includes:
[0071] The maximum number of retries is set, and for each service address in the local cache of the gRPC client, when the number of attempts exceeds the maximum number of retries and the connection with the gRPC server still cannot be successfully established, the service address is deleted from the local cache of the gRPC client. Timely deleting the service address of the gRPC server that has failed from the local cache of the gRPC client can reduce unnecessary retry costs.
[0072] In this embodiment of the invention, during the process of reconnecting with the gRPC server, the gRPC client sends a connection request to the gRPC server according to the parameters configured in WithDefaultServiceConfig, using exponential backoff.
[0073] This invention offers the following advantages: The solution introduces etcd as middleware for service registration and discovery. The gRPC server registers its service address with etcd, and the gRPC client obtains the service address from etcd. Upon detecting a service address change, the gRPC client automatically re-establishes a connection using the new service address. Furthermore, the gRPC client periodically sends heartbeat requests to the gRPC server. If no heartbeat response is received within a preset time, the connection is re-established. This solution can promptly detect network anomalies when data transmission is not occurring and automatically reconnect, shortening reconnection latency. It also responds quickly and automatically reconnects when the gRPC server's service address changes, eliminating the need for manual intervention. This improves service availability, ensures service continuity and stability, enhances the resilience of the distributed system, and improves user experience.
[0074] Furthermore, by enabling the automatic reconnection mechanism built into the gRPC development package and configuring the reconnection parameters related to exponential backoff, the solution of this invention works in conjunction with the reconnection mechanism built into gRPC to achieve more intelligent reconnection control, quickly and effectively restore gRPC connections, and improve system availability.
[0075] The above are merely specific embodiments of the present invention and should not be construed as limiting the scope of the present invention. Equivalent variations made by those skilled in the art based on this invention, as well as changes well-known to those skilled in the art, should still fall within the scope of the present invention. < / member-ip>
Claims
1. A gRPC disconnection reconnection method, characterized in that, The method comprises the following steps: S1, building an etcd service registration and discovery environment; S2, a gRPC server registers its service address in the etcd through a unique registration key; S3, a gRPC client obtains the service address of the gRPC server from the etcd through the registration key, creates an initial connection with the gRPC server, and calls the interface of the gRPC server according to the business needs to interact with the gRPC server; S4, the gRPC client periodically obtains the service address list of the gRPC server from the etcd at a first time interval, and subscribes to the service address change notification of the gRPC server in the etcd, and when the service address of the gRPC server changes, the connection with the gRPC server is re-established; S5, a heartbeat method is defined and implemented for each gRPC service, and the gRPC client periodically calls the heartbeat method to send a heartbeat request to the gRPC server at a second time interval, and if no heartbeat reply is received within a preset heartbeat reply time, the connection with the gRPC server is re-established.
2. The gRPC disconnection reconnection method of claim 1, wherein, The method further comprises: configuring a retry status code RetryableStatusCodes in a WithDefaultServiceConfig function, and configuring a maximum number of attempts MaxAttempts, a default backoff time InitialBackoff, a maximum backoff time MaxBackoff, and a backoff time increase multiplier BackoffMultiplier.
3. The gRPC disconnection reconnection method of claim 2, wherein, In the step S4, after the gRPC client periodically obtains the service address list of the gRPC server from the etcd at a first time interval, the gRPC service address list is saved to the local cache of the gRPC client.
4. The gRPC broken line reconnection method of claim 3, wherein, In the step S4, the gRPC client periodically obtains the service address list of the gRPC server from the etcd at a first time interval, and subscribes to the service address change notification of the gRPC server in the etcd, which is implemented by a parser in the gRPC client, the parser compares the new address with the old address, saves the new address to the local cache of the gRPC client, and deletes the old address of the same registration key as the new address from the local cache of the gRPC client.
5. The gRPC broken line reconnection method of claim 4, wherein, In the steps S4 and S5, the specific method of re-establishing the connection with the gRPC server comprises: creating a new connection instance, and re-establishing the connection with the gRPC server using the service address of the gRPC server successfully connected last time in the local cache of the gRPC client; if the connection fails, polling other service addresses in the local cache of the gRPC client, and trying to re-establish the connection with the gRPC server in turn; if the connection with the gRPC server cannot be successfully established through all the service addresses in the local cache of the gRPC client, obtaining a new service address from the etcd, and then re-establishing the connection with the gRPC server through the new service address.
6. The gRPC broken line reconnection method of claim 5, wherein, The specific method of re-establishing the connection with the gRPC server further comprises: A maximum number of retries is set, and for each service address in the gRPC client local cache, when the number of attempts exceeds the maximum number of retries and still cannot successfully establish a connection with the gRPC server, the service address is deleted from the gRPC client local cache.
7. The gRPC broken line reconnection method of claim 1, wherein, In the steps S4 and S5, the re-establishment of the connection with the gRPC server is implemented by a connection manager of the gRPC client.
8. The gRPC broken line reconnection method of claim 1, wherein, In the step S1, the building of the etcd service registration and discovery environment specifically refers to building an etcd cluster.
9. The gRPC broken line reconnection method of claim 1, wherein, In the step S2, the specific method of registering the own service address in the etcd through a unique registration key includes: A unique registration key is defined for each gRPC service, and the naming of the registration key adopts a hierarchical structure, and multiple service nodes with the same business function use the registration key with the same prefix; The gRPC server registers with the etcd by taking the registration key as the key and the IP address and port number of the gRPC server as the value.
10. The gRPC broken line reconnection method of claim 9, wherein, When the gRPC server starts, the specific method of registering the own service address in the etcd through a unique registration key further includes: The gRPC server sets a TTL when registering and sends a keep-alive request to the etcd within the TTL.