A method and system for designing and implementing a DRBD network connection congestion handling

Abstract

The application relates to the technical field of data synchronization, in particular to a DRBD network connection congestion processing design and implementation method and system, which comprises the following steps: environment pre-checking; event collection, wherein events are transferred to a disconnection decision maker by event listening and collecting; filtering of concerned resources, wherein the filtered resources are transferred to the disconnection decision maker; filtering of congestion events; congestion decision and processing; reconnection decision and processing; the DRBD network connection congestion processing design and implementation method and system can automatically and quickly detect congestion events under the condition that a DRBD replication link is congested, disconnects the DRBD connection through event driving and necessary decision algorithms, and preferentially guarantees the availability of services. After the replication link is recovered, the DRBD connection is automatically established, and the synchronization of unsynchronized data is performed. The whole process does not need manual participation, greatly reduces the processing and maintenance difficulty of the replication link congestion events, and improves the availability of the DRBD data synchronization scheme.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, specifically to a design and implementation method and system for handling DRBD network connection congestion. Background Technology

[0002] DRBD, short for Distributed Replicated Block Device, is a distributed replicated storage system consisting of a kernel driver module, user-space management tools, and related shell scripts. It is primarily used to build highly available computer clusters. DRBD is a software-based, shared-nothing, replicated storage solution that mirrors the contents of block devices (hard drives, partitions, logical volumes, etc.) between hosts over a network. Its implementation is similar to RAID 1 storage in servers (RAID 1 mirrors identical data onto two disks of the same capacity, achieving data redundancy through disk mirroring).

[0003] In existing technologies, the Distributed Replied Block Device (DRBD) data synchronization solution, based on disk block replication, supports three different replication modes: Protocol A, Protocol B, and Protocol C. Regardless of the mode, congestion on the replication link can lead to the local TCP network send buffer becoming full (especially in wide area network replication scenarios). This can cause I / O blocking on the master node, adversely affecting application writes to disk. In a real-world production environment, reduced service availability due to replication link congestion is unacceptable.

[0004] However, we need an implementation method for DRBD network congestion handling to dynamically detect and process network congestion events, reduce master node disk I / O congestion time, and improve the availability of upper-layer applications. Summary of the Invention

[0005] The purpose of this invention is to provide a design and implementation method and system for handling DRBD network connection congestion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a design and implementation method for handling DRBD network connection congestion, the method comprising the following steps:

[0007] Environmental pre-inspection;

[0008] Event acquisition involves listening to and collecting events, and then handing the monitored events over to the disconnection decision-maker.

[0009] Filtering the resources of interest, the filtered resources are handed over to the disconnection decision maker;

[0010] Filtering the congestion events;

[0011] Congestion decision and processing;

[0012] Reconnection decision and processing.

[0013] Preferably, the specific operation of the environment pre-checking includes:

[0014] Checking whether the user space tool of DRBD exists to meet the subsequent monitoring and collection of DRBD events.

[0015] Preferably, the specific operation of the event collection includes:

[0016] In DRBD, the data plane path is implemented in the kernel module, and its kernel events are obtained through the user space tool drbdsetup, and all event information is obtained by calling the events2 subcommand of the tool, and the relevant events are obtained by parsing the output of the command.

[0017] Preferably, the specific operation of filtering the resources of interest includes:

[0018] The specified resources are monitored and processed through the white list configuration of the resource filter, and the resource name list in the white list or the resource name is matched through a regular way; the non-interested resources are filtered out through the resource filter, and only the resources in the white list are processed subsequently, and the filtered resources are handed over to the disconnection decision maker;

[0019] The specific operation of filtering the congestion events includes:

[0020] The disconnection decision maker filters out the resources in the congestion connection state according to the resources in the white list and the received various events, and the filtered resources have the role of Primary and have at least one connection, and at least one connection is in the congestion state.

[0021] Preferably, the specific operation of the congestion decision and processing includes:

[0022] For the resources in the connection state, when the fluctuant congestion occurs, in order to avoid "mistaken killing", the disconnection decision maker performs calculation for judging whether to disconnect the connection regularly by default for 1 second; if the conditions set in the configuration file are reached, the disconnection operation is triggered, and the defined decision strategy is: if the congestion occurs 3 times within 5 seconds, it will be judged as congestion; when the connection is disconnected, the graceful way is preferred to disconnect, and if the graceful disconnection is executed for 3 times in succession, the forced disconnection is performed; after the connection disconnection is successfully executed, the disconnection resource is put into the observation queue, and the reconnection operation is executed after the opportunity is mature;

[0023] The specific operations of the reconnection decision and processing include:

[0024] For the resource with the role of Primary, if in the non-connected state and in the reconnection observation queue, the reconnection decision maker performs the judgment of whether the resource can resume the connection periodically by default 1 second; if the condition defined in the reconnection strategy set in the configuration file is reached, the reconnection operation is performed; after the reconnection is successful, the resource is removed from the reconnection observation list, and the resource is continuously monitored thereafter to perform the above-mentioned logic.

[0025] A DRBD network connection congestion handling design and implementation system, the system is composed of pre-checking module, acquisition module, filtering module, decision and processing module;

[0026] The pre-checking module is used for environment pre-checking.

[0027] The acquisition module is used for event acquisition, and the monitored events are transferred to the disconnection decision maker through event monitoring and collection.

[0028] The filtering module is used for filtering the resources of interest, and the filtered resources are transferred to the disconnection decision maker; the congestion events are filtered.

[0029] The decision and processing module is used for congestion decision and processing; the reconnection decision and processing.

[0030] Preferably, the pre-checking module checks whether the user space tool of the DRBD exists to meet the subsequent monitoring and collection of the DRBD events.

[0031] Preferably, the acquisition module obtains the kernel events of the data plane path implemented in the kernel module in the DRBD, obtains all the event information by using the events2 subcommand of the user space tool drbdsetup, and obtains the related events by analyzing the output of the command.

[0032] Preferably, the filtering module realizes the monitoring and processing of the specified resources through the white list configuration of the resource filter, specifies the resource name list in the white list, or matches the resource name through the regular way; the resource filter filters out the non-interested resources, and only the resources in the white list are processed subsequently, and the filtered resources are transferred to the disconnection decision maker.

[0033] The disconnection decision maker filters out the resources in the congestion connection state according to the resources in the white list and the received various events, and the filtered resources have the role of Primary, have at least one connection, and at least one connection is in the congestion state.

[0034] Preferably, the decision and processing module, for the resource in the connection state, when fluctuation congestion occurs, in order to avoid "mis-killing", the disconnection decision maker performs calculation for judging whether to disconnect by default 1 second periodically; if the condition set in the configuration file is reached, the disconnection operation is triggered, and the defined decision strategy is: if 3 times of congestion occur within 5 seconds, the congestion will be judged; when the connection is disconnected, the graceful way is preferred to disconnect, if the graceful disconnection is executed for 3 times continuously, the forced disconnection is performed; after the connection disconnection is executed successfully, the disconnected resource is put into the observation queue, and the reconnection operation is executed after the opportunity is mature.

[0035] For the resource with the role of Primary, if it is in the non-connection state and in the reconnection observation queue, the reconnection decision maker will perform the judgment of whether the resource can be restored to the connection by default 1 second periodically; if the condition set in the configuration file is reached, the reconnection operation is executed; after the reconnection is successful, the resource is removed from the reconnection observation list, and the resource continues to be monitored thereafter, and the above-mentioned logic is executed.

[0036] Compared with the prior art, the beneficial effects of the present application are:

[0037] The DRBD network connection congestion disposal design and implementation method and system provided by the present application can automatically and quickly detect congestion events under the condition that the DRBD replication link is congested, trigger the DRBD connection disconnection through event driving and necessary decision algorithm, and preferentially guarantee the availability of the business. After the replication link is restored, the DRBD connection is automatically established, and the unsynchronized data is synchronized. The whole process does not need manual participation, greatly reduces the processing and maintenance difficulty of the replication link congestion event, and improves the availability of the DRBD data synchronization scheme. BRIEF DESCRIPTION OF DRAWINGS

[0038] Fig. 1 The figure shows the position of the DRBD in the Linux I / O stack;

[0039] Fig. 2 The figure shows the DRBD congestion processing flowchart. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme of the present application clear, complete and more clear and explicit, the embodiments of the present application are further described in detail in combination with the drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present application, but not all the embodiments, which are used to explain the embodiments of the present application, and do not limit the embodiments of the present application, all other embodiments obtained by the person skilled in the art without creative labor are within the scope of protection of the present application.

[0041] Embodiment one

[0042] Please refer to Figs. 1-2 The application provides a technical solution: a DRBD network connection congestion handling design and implementation method, the method comprising the following steps:

[0043] 1. Environmental pre-check

[0044] Check if the user space tool of DRBD exists to meet the subsequent monitoring and collection of DRBD events.

[0045] 2. Event collection

[0046] In DRBD, the main data plane path is implemented in the kernel module, and if you want to get its kernel events, you can get them through the user space tool drbdsetup, and calling the events2 subcommand of the tool can get all the event information. By parsing the output of the command, you can get the relevant events. For example:

[0047] root@worker01:~#drbdsetup events2--timestamps--statistics--now

[0048] 2023-01-24T17:50:57.363675+08:00exists resource name:es role:Primarysuspended:no force-io-failures:no write-ordering:flush may_promote:nopromotion_score:10102

[0049] 2023-01-24T17:50:57.363675+08:00exists connection name:es peer-node-id:1conn-name:remote_worker01_7790connection:Connected role:Secondarycongested:no ap-in-flight:0rs-in-flight:0

[0050] 2023-01-24T17:50:57.363675+08:00 exists device name:es volume:0 minor:1 backing_dev: / dev / lvmvg / elastic disk: UpToDate client: no quorum: yes size: 10485404 read: 4465572 written: 27826680 al-writes: 199 bm-writes: 284 upper-pending: 0 lower-pending: 0 al-suspended: no blocked: no

[0051] 2023-01-24T17:50:57.363675+08:00 exists peer-device name:es peer-node-id:1 conn-name:remote_worker01_7790 volume:0 replication: Established peer-disk: UpToDate peer-client: no resync-suspended: no received: 0 sent: 2019012 out-of-sync: 0 pending: 0 unacked: 0

[0052] 2023-01-24T17:50:57.363675+08:00 exists pathname:es peer-node-id:1 conn-name:remote_worker01_7790 local: ipv4:10.58.240.28:7790 peer: ipv4:10.220.117.4:7790 established: yes

[0053] The output of the command in the above example has a total of 5 events, and the output of each event has a specific format, with different meaning fields separated by spaces, including TimeStamp, EventType, Target, and other event fields (Fields) representing event business attributes.

[0054] For example, in the second output in the above example, the event with Target as connection has a field "congested: no", which indicates that the current resource is not in a network congestion state.

[0055] Through the event listening and collection, we will listen to the event to the disconnector decision.

[0056] 3. Filter the concerned resources

[0057] There may be multiple DRBD resources on a server, and the specified resources can be monitored through the white list configuration of the resource filter. The resource name list can be specified in the white list, or the resource name can be matched by regular expression. The YAML format of the resource filter configuration file is as follows:

[0058] resourceFilter:

[0059] #drbd resource white list

[0060] whitelist:

[0061] #resource name list

[0062] nameList:

[0063] -nfs

[0064] #filter by resource name regular expression

[0065] nameRegex:^has.*$

[0066] Through the resource filter, non-concerned resources are filtered out, and only the resources in the white list are processed subsequently. The filtered resources are handed over to the disconnector decision.

[0067] 4. Filter congestion events

[0068] The disconnector decision will filter out resources in the congestion connection state according to the resources in the white list and the received various events. The filtered resources have the role of Primary, have at least one connection, and at least one connection is in the congestion state.

[0069] 5. Congestion decision and processing

[0070] For resources in the connection state, when there is fluctuating congestion, in order to avoid "mistakes", the disconnector decision will periodically (default 1 second) perform calculations to determine whether to disconnect. If the conditions set in the configuration file are met, the disconnection operation will be triggered. In the following configuration, the decision strategy is defined as: if there are 3 times (not continuous) of congestion within 5 seconds, it will be judged as congestion.

[0071] disconnector:

[0072] #effective duration of congestion judgment

[0073] maxCongestDuration: 5s

[0074] # In the range of valid duration, how many times of congestion occurrence will be determined as real congestion maxCongestTimes: 3

[0075] # Whether to enable graceful disconnect (multiple retries non-mandatory disconnect), disable to directly force disconnect

[0076] enableGracefulDisconnect: true

[0077] When disconnecting, the graceful way will be preferred, if the graceful disconnect fails for 3 times, then force disconnect will be performed. After the connection is successfully disconnected, the disconnect resource will be put into the observation queue, and the reconnection operation will be performed when the opportunity is ripe.

[0078] 6. Reconnection decision and processing

[0079] For resources with the role of Primary, if they are in a non-connected state and in the reconnection observation queue, the reconnection decision maker will periodically (default 1 second) execute the judgment whether the resource can resume connection. If the conditions defined in the configuration file reach the reconnection strategy set, the reconnection operation will be performed.

[0080] The following is the configuration of the reconnection decision maker:

[0081] reConnector:

[0082] # Automatic reconnection strategy, byTime = try to reconnect after waiting for a fixed time, byIO = try to reconnect after waiting for the disk io to fall below the specified threshold (recommended)

[0083] strategy:

[0084] byTime:

[0085] # Reconnection waiting time in the reconnection strategy byTime

[0086] reconnectWaitingDuration: 300s

[0087] # byIO:

[0088] # Disk io write rate threshold in the reconnection strategy byIO, KB / s

[0089] # reconnectIOWriteRate: 500

[0090] # Effective duration for determining io fall (if the io write rate is not higher than the set threshold in this period, it is determined that it can be reconnected)

[0091] #maxIOLowRateDuration: 5s

[0092] #io average calculation sampling number, default 10 (i.e. one sample per second, calculate the average of the last 10 seconds of io write rate)

[0093] #simplingCounts: 10

[0094] #minimum waiting time before starting the reconnection monitoring, to avoid frequent repeated disconnection and reconnection

[0095] #minWaitingDuration: 300s

[0096] Currently, two decision algorithms, byTime and byIO, are implemented, and only one of the two can be configured.

[0097] For the time-based reconnection strategy, if the disconnection time of the resource reaches 300s in the above example, the reconnection operation is triggered.

[0098] For the I / O write rate-based reconnection strategy, if the write rate of the disk I / O is not higher than the reconnectIOWriteRate value within the maxIOLowRateDuration time, the reconnection operation is triggered, otherwise the next reconnection decision is waited.

[0099] After successful reconnection, the resource is removed from the reconnection observation list, and the resource is continuously monitored thereafter, and the above logic is executed.

[0100] Embodiment Two

[0101] On the basis of embodiment one, a DRBD network connection congestion handling design and implementation system is proposed, which is composed of a pre-checking module, a collection module, a filtering module, a decision and processing module;

[0102] The pre-checking module is used for environmental pre-checking; whether the user space tool of DRBD exists is checked to meet the subsequent DRBD event listening and collection.

[0103] The collection module is used for event collection, and the listened events are transferred to the disconnection decision maker; in DRBD, the data plane path is implemented in the kernel module, the kernel events are obtained, the events2 subcommand of the user space tool drbdsetup is called to obtain all event information, and the related events are obtained by analyzing the output of the command.

[0104] Filtering module, for filtering the concerned resources, the filtered resources are handed over to the disconnection decision maker; filtering congestion events; monitoring specified resources through the white list configuration of the resource filter, the white list specifies the resource name list, or matches the resource name through the regular way; filtering out non-concerned resources through the resource filter, only the resources in the white list are processed subsequently, the filtered resources are handed over to the disconnection decision maker;

[0105] The disconnection decision maker filters out the resources in the congestion connection state according to the resources in the white list and the received various events, the filtered resources are Primary, have at least one connection, and at least one connection is in the congestion state.

[0106] Decision and processing module, for congestion decision and processing; reconnection decision and processing; for the resources in the connection state, when fluctuant congestion occurs, in order to avoid "mis-killing", the disconnection decision maker regularly executes calculation for judging whether to disconnect by default for 1 second; if the conditions set in the configuration file are reached, the disconnection operation is triggered, the defined decision strategy is: if 3 times of non-continuous congestion occur within 5 seconds, the resources will be judged as congestion; when the connection is disconnected, the graceful way is preferentially adopted to disconnect, if the graceful disconnection is executed for 3 times continuously and fails, forced disconnection is performed; after the connection disconnection is executed successfully, the disconnection resource is put into the observation queue, and the reconnection operation is executed after the opportunity is ripe;

[0107] For the resources with the role of Primary, if in the non-connection state and in the reconnection observation queue, the reconnection decision maker regularly executes the judgment of whether the resources can be restored to the connection by default for 1 second; if the conditions set in the reconnection strategy defined in the configuration file are reached, the reconnection operation is executed; after the reconnection is successful, the resources are removed from the reconnection observation list, and the resources continue to be monitored thereafter, and the above-mentioned logic is executed.

[0108] The event monitoring is mainly established by the DRBD kernel module, the monitored events are transferred to the disconnection decision maker, the concerned network congestion events and the concerned resource (Resource) objects are filtered out, the configured decision algorithm is used to disconnect the related resources. During the resource disconnection process, multiple graceful disconnections are performed first, if the multiple graceful disconnections fail, forced disconnection is used to ensure the successful execution of the disconnection operation. For the disconnected resources, the reconnection decision maker queue is entered, and then the specified algorithm (time-based decision, I / O load-based decision algorithm, etc.) is used to determine whether the reconnection can be performed, if the reconnection condition is reached, the reconnection operation is performed. The whole disconnection and reconnection operation is continuously executed based on the event driving. The implemented program is run in the form of a Linux service (named drbdmonitor.service) to manage the DRBD resources on the current server.

[0109] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, alternatives and variations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for designing and implementing DRBD network connection congestion handling, characterized in that: The method comprises the following steps: environment pre-checking; event collection, through event monitoring and collection, the monitored events are handed over to the disconnection decision maker; filtering the concerned resources, the filtered resources are handed over to the disconnection decision maker; filtering congestion events; congestion decision and processing; reconnection decision and processing; the specific operations of congestion decision and processing include: for the resources in the connected state, when fluctuant congestion occurs, in order to avoid "mis-killing", the disconnection decision maker performs calculation for judging whether to disconnect by default 1 second regularly; if the conditions set in the configuration file are reached, the disconnection operation is triggered, and the defined decision strategy is: if congestion occurs 3 times within 5 seconds, it will be judged as congestion; when the connection is disconnected, the graceful disconnection is preferentially adopted, if the graceful disconnection is executed for 3 times continuously and fails, forced disconnection is performed; after the connection disconnection is successfully executed, the disconnection resource is put into the observation queue, and the reconnection operation is executed when the opportunity is ripe; the specific operations of reconnection decision and processing include: for the resources with the role of Primary, if in the non-connected state and in the reconnection observation queue, the reconnection decision maker will regularly perform judgment on whether the resource can be restored to the connected state by default 1 second; if the conditions set in the configuration file are reached, the reconnection operation is executed; after the reconnection is successful, the resource is removed from the reconnection observation list, and the resource continues to be monitored thereafter, and the above-mentioned logic is executed.

2. The method of claim 1, wherein: the specific operations of environment pre-checking include: checking whether the user space tool of DRBD exists, so as to satisfy the subsequent monitoring and collection of DRBD events.

3. The method of claim 1, wherein: the specific operations of event collection include: in DRBD, the data plane path is implemented in the kernel module, the kernel events are obtained, the events2 subcommand of the user space tool drbdsetup is called to obtain all the event information, and the related events are obtained by analyzing the output of the command.

4. The method of claim 1, wherein the method further comprises: the specific operations of filtering the concerned resources include: the specified resources are monitored and processed through the white list configuration of the resource filter, the resource names in the white list are specified or the resource names are matched in a regular manner; the non-concerned resources are filtered out through the resource filter, and only the resources in the white list are processed subsequently, and the filtered resources are handed over to the disconnection decision maker; the specific operations of filtering congestion events include: the disconnection decision maker filters out the resources in the congested connection state according to the resources in the white list and the received various events, the filtered resources have the role of Primary, have at least one connection, and at least one connection is in the congested state.

5. A DRBD network connection congestion handling design and implementation system for the method of any of claims 1-4, characterized by: The system is composed of a pre-checking module, a collection module, a filtering module and a decision and processing module; the pre-checking module is used for environment pre-checking; the collection module is used for event collection, and the monitored events are handed over to the disconnection decision maker through event monitoring and collection; the filtering module is used for filtering the concerned resources, and the filtered resources are handed over to the disconnection decision maker; and filtering congestion events; the decision and processing module is used for congestion decision and processing; and reconnection decision and processing. The decision and processing module, for the resources in the connection state, when the fluctuation congestion occurs, in order to avoid "mis-killing", the disconnection decision maker performs calculation for judging whether to disconnect regularly by default for 1 second; If the condition set in the configuration file is reached, the disconnection operation is triggered, and the defined decision strategy is: if 3 times of congestion occur within 5 seconds, it will be judged as congestion; When disconnecting, the graceful way is preferred, and if the graceful disconnection is failed for 3 times, the forced disconnection is performed; After the connection disconnection is successfully executed, the disconnected resource is put into the observation queue, and the reconnection operation is performed after the opportunity is mature; For the resources with the role of Primary, if they are in the non-connection state and in the reconnection observation queue, the reconnection decision maker will regularly perform the judgment of whether the resource can be restored to the connection by default for 1 second; If the condition set in the reconnection strategy defined in the configuration file is reached, the reconnection operation is performed; After the reconnection is successfully performed, the resource is removed from the reconnection observation list, and the resource continues to be monitored thereafter, and the above-mentioned logic is executed.

6. The DRBD network connection congestion handling design and implementation system of claim 5, wherein: The pre-checking module checks whether the user space tool of DRBD exists to meet the subsequent monitoring and collection of DRBD events.

7. The DRBD network connection congestion handling design and implementation system of claim 5, wherein: The acquisition module, in DRBD, the data plane path is implemented in the kernel module, and the kernel events are obtained through the user space tool drbdsetup, the events2 subcommand of the tool is called to obtain all event information, and the related events are obtained by analyzing the output of the command.

8. The DRBD network connection congestion handling design and implementation system of claim 5, wherein: The filtering module realizes the monitoring and processing of the specified resources through the white list configuration of the resource filter, and the resource name list in the white list is specified or the resource name is matched through the regular way; The resources not concerned are filtered out through the resource filter, and only the resources in the white list are processed subsequently, and the filtered resources are transferred to the disconnection decision maker; The disconnection decision maker filters out the resources in the congestion connection state according to the resources in the white list and the received various events, and the filtered resources have the role of Primary, have at least one connection, and at least one connection is in the congestion state.

Images