Disaster recovery switching method of message storage node and message middleware system

Abstract

The application provides a disaster recovery switching method of a message storage node and a message middleware system. The method comprises the following steps: a main monitoring management node in a plurality of monitoring management nodes receives heartbeat information of a plurality of message storage nodes according to a preset period; a fault node is determined from the plurality of message storage nodes according to the heartbeat information of the plurality of message storage nodes, and in the case that the role information of the fault node is a master node, the offset of all other message storage nodes in a first message storage node set where the fault node is located is determined; the role information of the fault node is switched to a slave node, and a message storage node with the largest offset is selected from all other message storage nodes to be determined as a target node; and the role information of the target node is switched to a master node. The application at least solves the technical problem that in the related art, in the case that a message storage node fails, only an artificial switching mode can be used, thereby causing low message transmission efficiency.

Description

Technical Field

[0001] This application relates to the field of message processing technology, and more specifically, to a disaster recovery switching method for message storage nodes and a message middleware system. Background Technology

[0002] In the context of today's information and data explosion, there is a great deal of demand for data sharing and exchange among various system applications. Message middleware is a middleware product designed to solve a series of problems such as information interoperability between multiple application systems, information silos, application data loss, unstable data transmission due to poor network environments, application resource isolation, and application system scalability. However, traditional message middleware may cause some data to become unavailable in extreme scenarios, such as physical downtime that cannot be recovered. Some message middleware uses batch disk flushing technology to achieve high performance and high throughput, which may lead to data loss if the data resides in memory for a period of time. If a message producer sends messages to the same queue under a topic to ensure the order of message delivery, and the message storage node of that queue fails, messages cannot continue to be sent to ensure order until the corresponding message storage node recovers. When the broker (message storage node) in the role of master (the primary node in the message storage node) fails and cannot be recovered, manual intervention (or through operation and maintenance tools) is required to switch the slave node in the corresponding message storage node to master. Summary of the Invention

[0003] The main objective of this application is to provide a disaster recovery switching method for message storage nodes and a message middleware system, so as to at least solve the technical problem in related technologies that when a message storage node fails, it can only be switched manually, resulting in low message transmission efficiency.

[0004] To achieve the above objectives, according to one aspect of this application, a disaster recovery switching method for message storage nodes is provided, comprising: receiving heartbeat information from multiple message storage nodes at a preset period through a master monitoring and management node among multiple monitoring and management nodes; determining a faulty node from the multiple message storage nodes based at least on the heartbeat information of the multiple message storage nodes, and, if the role information of the faulty node is a master node, determining the offset of all other message storage nodes in a first set of message storage nodes where the faulty node is located; switching the role information of the faulty node to a slave node, and selecting the message storage node with the largest offset from all other message storage nodes as a target node; and switching the role information of the target node to a master node.

[0005] Optionally, determining the faulty node from the plurality of message storage nodes based at least on the heartbeat information of the plurality of message storage nodes includes: identifying the message storage node that has not sent heartbeat information to the main monitoring and management node within a preset time period as the initial faulty node; and determining the initial faulty node as the faulty node if the configuration information of the initial faulty node is not found in the normal configuration node corresponding to the initial faulty node, wherein the configuration node is used to store the configuration information of the message storage node.

[0006] Optionally, after switching the role information of the target node to the master node, the method further includes: synchronizing the configuration information and role information of all nodes in the first message storage node set to the slave monitoring management node other than the master monitoring management node, and switching the replication mode of the first message storage node set from synchronous replication mode to asynchronous replication mode.

[0007] Optionally, after switching the role information of the target node to the master node, the method further includes: detecting whether the faulty node has recovered; if the faulty node has recovered, detecting whether the difference value between the data stored in the master node and the slave node in the first message storage node set is less than a preset threshold; if the difference value is less than the preset threshold, switching the replication mode of the first message storage node set from asynchronous replication mode to synchronous replication mode.

[0008] Optionally, if the failure of the faulty node is caused by a split-brain event in the message middleware system, the method further includes: switching the role information of the monitoring and management node; switching the role information of the message storage node; and after the network failure of the message middleware system is recovered, if there is a conflict between the actual role information of the faulty node and the role information of the faulty node stored in the memory of the main monitoring and management node, switching the actual role information of the faulty node to the role information of the faulty node stored in the memory of the main monitoring and management node.

[0009] Optionally, the role information switching of the monitoring management node includes: determining the number of monitoring management nodes in each of the multiple monitoring management node sub-clusters formed after the split-brain event in the message middleware system; identifying all monitoring management nodes in the monitoring management node sub-clusters whose number of monitoring management nodes does not exceed a preset number as slave monitoring management nodes; and identifying one monitoring management node as the master monitoring management node in the monitoring management node sub-clusters whose number of monitoring management nodes exceeds the preset number.

[0010] Optionally, switching the role information of the message storage node includes: if, among all message storage nodes whose role information is that of a master node, there exists a message storage node in the target monitoring and management node sub-cluster that is a master node, then determine the message storage node in the other monitoring and management node sub-cluster as the target fault node, wherein the target monitoring and management node sub-cluster is the monitoring and management node sub-cluster where the master monitoring and management node is located; switch the role information of the target fault node to that of a slave node; and store the switched role information of the target fault node in the memory of the master monitoring and management node.

[0011] Optionally, after switching the actual role information of the faulty node to the role information of the faulty node stored in the memory of the main monitoring and management node, the method further includes: in the case that there is a set of message storage nodes without a master node in the multiple monitoring and management node sub-clusters, determining the message storage node with the largest offset in the set of message storage nodes without a master node as the master node.

[0012] According to another aspect of this application, a message middleware system is provided, comprising: a monitoring and management node cluster and a message storage node cluster; the monitoring and management node cluster is configured to receive heartbeat information from multiple message storage nodes through a master monitoring and management node among multiple monitoring and management nodes at a preset period; determine a faulty node from among the multiple message storage nodes based at least on the heartbeat information of the multiple message storage nodes, and, if the role information of the faulty node is a master node, determine the offset of all other message storage nodes in a first set of message storage nodes where the faulty node is located; switch the role information of the faulty node to a slave node, and select the message storage node with the largest offset from among all other message storage nodes as the target node; the message storage node cluster is configured to store messages and switch the role information of the target node to a master node.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute the above-described disaster recovery switching method for message storage nodes.

[0014] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a disaster recovery switching method for performing the aforementioned message storage node.

[0015] By applying the technical solution of this application, the system employs a method where a master monitoring and management node among multiple monitoring and management nodes receives heartbeat information from multiple message storage nodes at a preset period. At least based on the heartbeat information of the multiple message storage nodes, a faulty node is identified from among the multiple message storage nodes. If the faulty node's role information is that of a master node, the offsets of all other message storage nodes in the first set of message storage nodes containing the faulty node are determined. The role information of the faulty node is switched to that of a slave node, and the message storage node with the largest offset is selected from all other message storage nodes as the target node. The role information of the target node is then switched to that of a master node. This method, through periodically receiving heartbeat information from message storage nodes to identify faulty nodes, and then selecting a new master node by comparing the offsets of slave nodes after identifying the faulty node, achieves the purpose of automatically detecting faulty nodes in message storage nodes and completing the switching of node role information. This solves the technical problem in related technologies where, in the event of a message storage node failure, manual switching is the only method, resulting in low message transmission efficiency. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a disaster recovery switching method for a message storage node according to an embodiment of this application is shown.

[0018] Figure 2 A flowchart illustrating a disaster recovery switching method for a message storage node according to an embodiment of this application is shown.

[0019] Figure 3 A schematic diagram illustrating the relationship between a message storage node and a monitoring and management node according to an embodiment of this application is shown.

[0020] Figure 4 A schematic diagram illustrating the relationship between an optional message storage node and a configuration node according to an embodiment of this application is shown;

[0021] Figure 5 A schematic diagram illustrating the relationship between another optional message storage node and configuration node according to an embodiment of this application is shown;

[0022] Figure 6 A schematic flowchart of a disaster recovery switching method for another message storage section according to an embodiment of this application is shown;

[0023] Figure 7A schematic flowchart of an optional post-split-brain monitoring and management node role switching method according to an embodiment of this application is shown.

[0024] Figure 8 The diagram illustrates three possible scenarios for a post-split-brain monitoring and management node cluster according to an embodiment of this application.

[0025] Figure 9 A schematic flowchart of an optional post-split-brain message storage node role switching method according to an embodiment of this application is shown.

[0026] Figure 10 A schematic flowchart of an optional message middleware system role conflict handling method after split-brain recovery, according to an embodiment of this application, is shown.

[0027] Figure 11 A schematic diagram illustrating an optional message storage node role information conflict scenario according to an embodiment of this application is shown.

[0028] Figure 12 A schematic diagram of an optional message middleware system architecture according to an embodiment of this application is shown;

[0029] Figure 13 A schematic diagram of another optional message middleware system architecture according to an embodiment of this application is shown;

[0030] Figure 14 A schematic diagram of a disaster recovery switching device for an optional message storage node according to an embodiment of this application is shown. Detailed Implementation

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0035] The Write-Ahead Logging (WAL) log is a database transaction log that records changes made to the database. The WAL log records changes to a log file before data is written to disk, ensuring data durability and consistency.

[0036] As described in the background section, in related technologies, when a message storage node fails, it is necessary to wait for the corresponding message storage node to recover. When the broker (message storage node) in the master (master node) role fails and cannot be recovered, manual intervention (or through operation and maintenance tools) is required to switch the slave node in the corresponding message storage node to the master node, resulting in low message transmission efficiency. To solve the technical problem of low message transmission efficiency, the embodiments of this application provide a disaster recovery switching method for message storage nodes.

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0038] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a disaster recovery switching method for a message storage node according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] This embodiment provides a disaster recovery switching method for a message storage node running on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0041] Figure 2 This is a flowchart of a disaster recovery switching method for a message storage node according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0042] Step S201: The main monitoring and management node among multiple monitoring and management nodes receives heartbeat information from multiple message storage nodes according to a preset cycle;

[0043] Step S202: Determine the faulty node from the multiple message storage nodes based at least on the heartbeat information of the multiple message storage nodes, and determine the offset of all other message storage nodes in the first message storage node set where the faulty node is located, provided that the role information of the faulty node is the master node.

[0044] Step S203: Switch the role information of the faulty node to a slave node, and select the message storage node with the largest offset from all other message storage nodes as the target node;

[0045] Step S204: Switch the role information of the target node to the master node.

[0046] This embodiment identifies faulty nodes by periodically receiving heartbeat information from message storage nodes (brokers), and selects a new master node by comparing node offsets after identifying the faulty node, and completes the switch. This achieves the purpose of automatically detecting faulty nodes in message storage nodes and completing the switch of node role information, solving the technical problem in related technologies where message transmission efficiency is low because manual switching is the only way to handle faulty message storage nodes.

[0047] Figure 3 This illustrates a relationship between a message storage node and a monitoring and management node, such as... Figure 3 As shown, the monitoring and management nodes adopt a master-slave mode, with only the master monitoring and management node providing services to the outside world at any given time.

[0048] It should be noted that the number of multiple monitoring and management nodes is odd, and at least 3. All message storage nodes will report heartbeat information to the upper level. The heartbeat information includes the configuration information and role information of the message storage node, as well as the maximum offset information of the WAL log of the stored message (the offset of the message storage node). "slave" indicates that the node's identity information is a slave node.

[0049] After receiving the heartbeat information from the message storage node, the primary monitoring and management node stores it in memory. If it detects changes in the configuration or role information of the message storage node, it will synchronize the corresponding changes to all secondary monitoring and management nodes.

[0050] It should also be noted that the first message storage node set contains at least one master node and one slave node. Only the master node provides services to the outside world, while the slave node is only used for data backup. A message storage node set (broker group) can have multiple slave message storage nodes.

[0051] It is understandable that the master node includes the master message storage node; and the slave node includes the slave message storage node.

[0052] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the disaster recovery switching method for the message storage node of this application will be described in detail below with reference to specific embodiments.

[0053] In some embodiments of this application, determining a faulty node from the plurality of message storage nodes based at least on the heartbeat information of the plurality of message storage nodes can be achieved in the following manner: determining the message storage node that has not sent heartbeat information to the main monitoring and management node within a preset time period as the initial faulty node; determining the initial faulty node as the faulty node if the configuration information of the initial faulty node is not found in the normal configuration node corresponding to the initial faulty node, wherein the configuration node is used to store the configuration information of the message storage node.

[0054] In practical application scenarios, the main monitoring and management node periodically receives heartbeat information from message storage nodes. If the main monitoring and management node does not receive heartbeat information from the second message storage node among multiple message storage nodes within a set time interval (preset period), it determines the second message storage node as the initial fault node and queries the configuration node for information about the initial fault node. If at least one normal configuration node is present at the current time and none of the normal configuration nodes contain information about the initial fault node, the initial fault node is determined as a fault node and fault handling is performed. Otherwise, the operation of periodically receiving heartbeat information from message storage nodes is repeated.

[0055] It should be noted that the initial fault node is only identified as a fault node and the fault is handled when the information of the initial fault node is not found in all normal configured nodes.

[0056] Figure 4 and Figure 5 This diagram illustrates the relationship between two types of message storage nodes and configuration nodes, as shown below. Figure 4 and Figure 5 As shown, a message storage node broker may simultaneously register or report heartbeat information to two or more configuration nodes. The reported information includes the topic configuration information on the broker and the broker's configuration information, such as... Figure 4 In Broker Group 1, the Broker1master node is connected to both configuration node A and configuration node B.

[0057] Specifically, after receiving a heartbeat reported by the broker, if the corresponding configuration already exists, the configuration node will delete the original information and store the latest reported information in the configuration node's memory. When the broker exits, it will send a deregistration request to the configuration node to deregister its information on the configuration node. If the broker fails and does not send a heartbeat to the configuration node for a preset period of time, the configuration node will delete the corresponding broker's information.

[0058] After switching the role information of the target node to the master node, the configuration information and role information of all nodes in the first message storage node set are synchronized to the slave monitoring management node other than the master monitoring management node, and the replication mode of the first message storage node set is switched from synchronous replication mode to asynchronous replication mode.

[0059] It should be noted that when the faulty node is a slave node, there is no need to switch the slave node's role information. The replication mode of the first message storage node set (message storage node group) where the faulty node is located is directly changed from synchronous replication mode to asynchronous replication mode.

[0060] Optionally, after switching the role information of the target node to the master node, it is detected whether the faulty node has recovered; if the faulty node has recovered, it is detected whether the difference value between the data stored in the master node and the slave node in the first message storage node set is less than a preset threshold; if the difference value is less than the preset threshold, the replication mode of the first message storage node set is switched from asynchronous replication mode to synchronous replication mode.

[0061] Figure 6 This paper illustrates a disaster recovery switching method after a message storage node failure, such as... Figure 6 As shown, it includes:

[0062] The first step is for the main monitoring and management node to periodically receive heartbeat information from the broker;

[0063] The second step is to proceed to the next step if the main monitoring and management node does not receive a heartbeat message from a broker within the set time interval; otherwise, repeat the previous step.

[0064] The third step is to identify the corresponding broker failure and query the information of the failure node from the configuration nodes;

[0065] Fourth step: If there is at least one normal configuration node and no information about the faulty broker is found on any of the normal configuration nodes, proceed to the next step; otherwise, repeat the operation of periodically receiving broker heartbeat information in the first step above. In particular, after determining that the broker is faulty, fault handling will only be performed if the broker's information is not found on any of the normal configuration nodes.

[0066] Fifth step: If the faulty broker is the master, perform a master-slave switchover; otherwise, proceed to the next step.

[0067] Specifically, the master-slave failover method for a broker in the master role is as follows: The master monitoring and management node first waits for all other broker nodes in the group where the failed broker is located to report heartbeat information (the heartbeat information includes the maximum offset information of the WAL log of stored messages), then selects the node with the "latest" data among the slave nodes (the broker node with the largest WAL log offset) to switch the broker master-slave role to the master, and finally updates the new configuration and role information of the broker group in memory and synchronizes it to other backup monitoring and management nodes;

[0068] Step 6: Change the replication mode of the faulty broker group to asynchronous replication mode;

[0069] Specifically, if the broker in the Master role fails, a master-slave switch is performed and the broker group is changed from synchronous replication mode to asynchronous replication mode; if the broker in the slave role fails, only the broker group is changed from synchronous replication mode to asynchronous replication mode.

[0070] Step 7: Check whether the faulty node or the broker that was switched from master to slave after the failure has recovered. If it has recovered, proceed to the next step; otherwise, continue to check whether the faulty broker has recovered.

[0071] Step 8: Determine whether the data difference between the master and slave nodes of the broker group containing the faulty node is less than a certain threshold. If so, change the broker group from asynchronous replication mode to synchronous replication mode. Otherwise, continue to check whether the data difference between the master and slave nodes of the broker group is less than the threshold.

[0072] In cases where the failure of the faulty node is caused by a split-brain event in the message middleware system, this application also provides a method for role switching after a split-brain event in the message middleware system, including: switching the role information of the monitoring and management node; switching the role information of the message storage node; and, after the network failure of the message middleware system is recovered, if there is a conflict between the actual role information of the faulty node and the role information of the faulty node stored in the memory of the main monitoring and management node, switching the actual role information of the faulty node to the role information of the faulty node stored in the memory of the main monitoring and management node.

[0073] It should be noted that a split-brain scenario in a message middleware system refers to a phenomenon in a distributed system where communication between message nodes becomes inconsistent or breaks due to network latency, node failure, or other reasons.

[0074] The specific method for switching the role information of the monitoring management node is as follows: determine the number of monitoring management nodes in each of the multiple monitoring management node sub-clusters formed after the split-brain event in the message middleware system; determine all monitoring management nodes in the monitoring management node sub-clusters whose number of monitoring management nodes does not exceed a preset number as slave monitoring management nodes; and determine one monitoring management node as the master monitoring management node in the monitoring management node sub-clusters whose number of monitoring management nodes exceeds the preset number.

[0075] In one optional approach, the preset number is half the total number of monitoring and management nodes.

[0076] Specifically, such as Figure 7 As shown, it includes:

[0077] The first step is to calculate the number of monitoring and management nodes in each sub-cluster of the split-brain system.

[0078] The second step is that if there is no sub-cluster with more than half of the monitoring and management nodes after the split-brain, there will be no master monitoring and management node after the switch. When there is no master monitoring and management node, all brokers will maintain their original roles and end the current processing flow; otherwise, proceed to the next step.

[0079] The third step is to determine whether there is a master monitoring and management node in the sub-cluster that has more than half of the monitoring and management nodes. If not, a master monitoring and management node is elected in the sub-cluster.

[0080] It should be noted that after a split-brain event, there can be at most one monitoring management node sub-cluster with more than half of the monitoring management nodes, or there may be none. If there is no sub-cluster with more than half of the monitoring management nodes after a split-brain event, there will be no primary monitoring management node after the switchover. When there is no primary monitoring management node, all brokers will maintain their original roles. When the original primary monitoring management node is not found in the monitoring management node sub-cluster with more than half of the monitoring management nodes after a split-brain event, the primary monitoring management node will be switched over.

[0081] It is understandable that the monitoring management node cluster consists of multiple monitoring management nodes in the message middleware system. The role information of the main monitoring management node is master, and the role information of the slave monitoring management node is slave.

[0082] Figure 8 The diagram illustrates three possible scenarios for the monitoring and management node cluster after a split-brain event, such as... Figure 8 As shown, in split-brain scenario 1, after the split, there is no sub-cluster with more than half of the monitoring and management nodes, and after the switchover, all monitoring and management nodes become slave monitoring and management nodes; in split-brain scenario 2, after the split, there is a sub-cluster with more than half of the monitoring and management nodes, and there is only one master monitoring and management node in the sub-cluster with more than half of the monitoring and management nodes; in split-brain scenario 3, after the split, there is no master monitoring and management node in the sub-cluster with more than half of the monitoring and management nodes, and there is one master monitoring and management node in the sub-cluster with less than half of the monitoring and management nodes. After the switchover, the role information of the monitoring and management nodes in the sub-cluster with less than half of the monitoring and management nodes is changed to slave monitoring and management nodes, and a monitoring and management node is selected from the sub-cluster with more than half of the monitoring and management nodes as the master monitoring and management node.

[0083] In one optional approach, switching the role information of the message storage node includes: if, among all message storage nodes whose role information is that of a master node, there exists a message storage node in the target monitoring and management node sub-cluster that is a master node, then determine the message storage node in the other monitoring and management node sub-cluster as the target fault node, wherein the target monitoring and management node sub-cluster is the monitoring and management node sub-cluster where the master monitoring and management node is located; switch the role information of the target fault node to that of a slave node; and store the switched role information of the target fault node in the memory of the master monitoring and management node.

[0084] Figure 9 A method for role switching of message storage nodes after a split-brain event is shown, such as... Figure 9 As shown, it includes:

[0085] The first step, after the brain split, is to follow Figure 7 The Chinese method completes the switching operation of the monitoring and management node;

[0086] The second step is to terminate the current process if all broker nodes with master roles are in the sub-cluster where the current master monitoring and management node is located; otherwise, proceed to the next step.

[0087] The third step is for the main monitoring and management node to determine the failure of broker nodes in other sub-clusters and switch the role of the broker in this sub-cluster according to the method in Example 2 above.

[0088] The fourth step is for the main monitoring and management node to update the configuration information and roles of all brokers after the switch to its local memory and synchronize them to all other monitoring and management nodes in the sub-cluster.

[0089] It should be noted that there are two necessary conditions for the switch of broker roles after brain split: (1) After brain split, there is still a master monitoring and management node; (2) The sub-cluster where the master monitoring and management node is located after brain split does not contain all the broker nodes with master roles in the broker groups. That is to say, some master role broker nodes are not in the sub-cluster where the master monitoring and management node is located.

[0090] After recovering from a split-brain failure, the role information of the message storage node can be checked by the main monitoring and management node. The system can detect whether the role information of the message storage node conflicts with the role information stored in the memory of the main monitoring and management node. If a conflict is detected, a role change command is sent to the message storage node with the incorrect role in other sub-clusters according to the broker role information stored in the main monitoring and management node.

[0091] After switching the actual role information of the faulty node to the role information of the faulty node stored in the memory of the main monitoring and management node, if there is a set of message storage nodes without a master node in the multiple monitoring and management node sub-clusters, the message storage node with the largest offset in the set of message storage nodes without a master node is determined as the master node.

[0092] Specifically, such as Figure 10As shown, after recovering from a split-brain failure, the process also includes detecting the recovered faulty nodes and handling nodes with conflicting role information. Specifically: First, after network recovery following the split-brain failure, the main monitoring and management node detects that brokers in other sub-clusters have recovered from failures and checks whether the roles of these brokers conflict with the role information stored in the main monitoring and management node's memory. Second, if there are no broker nodes with conflicting roles, the current processing flow ends directly; otherwise, proceed to the next step. Third, according to the broker role information stored in the main monitoring and management node, a role change command is sent to the brokers in other sub-clusters with incorrect roles. Fourth, the main monitoring and management node determines whether there is a masterless broker group in the current cluster. If so, a new master is elected for the masterless broker group, and a role change command is sent to the corresponding broker to change its role.

[0093] It should be noted that this method mainly addresses the role conflicts of brokers after network recovery when dual-master (two master roles within the same broker) occurs after a split-brain attack. When dual-master occurs, one master is retained, and the other is switched to slave, mainly based on the role information of all brokers stored in the local memory of the master monitoring and management node. The dual masters after a split-brain attack are distributed in different sub-clusters and may provide services to the outside world. When one master is switched to slave, it is necessary to roll back the redundant data added after the split-brain attack.

[0094] Figure 11 This diagram illustrates a scenario where there is a conflict of information regarding the roles of message storage nodes after a split-brain situation. Figure 11 As shown, the cluster before the brain split contained one master monitoring and management node, two slave monitoring and management nodes, and four message storage nodes. After the brain split, it was divided into cluster 1 and cluster 2. The Broker2slave node in cluster 1 was identified as a slave node. After the brain split was restored, the Broker2slave node was transformed into a Broker2master node, and the Broker2master node was identified as a master node. This conflicted with the identity information of the Broker2master node in cluster 2.

[0095] This application also provides a message middleware system, such as... Figure 12As shown, it includes: a monitoring and management node cluster 30 and a message storage node cluster 32; the monitoring and management node cluster 30 is used to receive heartbeat information from multiple message storage nodes through a master monitoring and management node among multiple monitoring and management nodes according to a preset period; to determine a faulty node from the multiple message storage nodes based at least on the heartbeat information of the multiple message storage nodes, and, if the role information of the faulty node is a master node, to determine the offset of all other message storage nodes in the first message storage node set where the faulty node is located; to switch the role information of the faulty node to a slave node, and to select the message storage node with the largest offset from all other message storage nodes as the target node; the message storage node cluster 32 is used to store messages and to switch the role information of the target node to a master node.

[0096] It should be noted that the message middleware system of this application embodiment can be used to execute the disaster recovery switching method for message storage nodes provided in this application embodiment. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0097] like Figure 13 As shown, this application also provides another message middleware system, including: a configuration node cluster 40, a message storage node cluster 42, a production node cluster 44, a consumer node cluster 46, and a monitoring and management node cluster 46.

[0098] The configuration node cluster 40 is a nearly stateless node that can be deployed in a cluster. There is no synchronization information between nodes. It mainly provides functions such as broker (message storage node) registration and topic (topic) routing management.

[0099] The message storage node cluster 42 provides core functions such as message storage, production, consumption, master-slave synchronization, and data flushing.

[0100] The production node cluster of 44 is responsible for message production and is completely stateless.

[0101] The consumer node cluster 46 is responsible for message consumption and is completely stateless;

[0102] The monitoring and management node cluster 46 monitors and manages message storage nodes, and also provides functions such as automatic master-slave failover.

[0103] Figure 14 This is a schematic diagram of an optional message storage node disaster recovery switching device according to an embodiment of this application, as shown below. Figure 14As shown, it includes: a receiving module 50, used to receive heartbeat information from multiple message storage nodes through a master monitoring and management node among multiple monitoring and management nodes according to a preset period; a first determining module 52, used to determine a faulty node from the multiple message storage nodes based at least on the heartbeat information of the multiple message storage nodes, and, if the role information of the faulty node is a master node, determine the offset of all other message storage nodes in the first message storage node set where the faulty node is located; a second determining module 54, used to switch the role information of the faulty node to a slave node, and select the message storage node with the largest offset from all other message storage nodes as the target node; and a switching module 56, used to switch the role information of the target node to a master node.

[0104] The first determining module 52 includes: a determining submodule, which is used to determine the message storage node among the plurality of message storage nodes that has not sent heartbeat information to the main monitoring and management node within a preset time period as an initial fault node; and to determine the initial fault node as the fault node if the configuration information of the initial fault node is not found in the normal configuration node corresponding to the initial fault node, wherein the configuration node is used to store the configuration information of the message storage node.

[0105] The switching module 56 includes: a synchronization submodule, a detection submodule, and a switching submodule. The synchronization submodule is used to synchronize the configuration information and role information of all nodes in the first message storage node set to the slave monitoring management node other than the master monitoring management node, and to switch the replication mode of the first message storage node set from synchronous replication mode to asynchronous replication mode.

[0106] The detection submodule is used to detect whether the faulty node has recovered; if the faulty node has recovered, it detects whether the difference value between the data stored in the master node and the slave node in the first message storage node set is less than a preset threshold; if the difference value is less than the preset threshold, it switches the replication mode of the first message storage node set from asynchronous replication mode to synchronous replication mode.

[0107] The switching submodule is used to, in the case that the failure of the faulty node is caused by a split-brain in the message middleware system, further include: switching the role information of the monitoring and management node; switching the role information of the message storage node; and, after the network failure of the message middleware system is recovered, if there is a conflict between the actual role information of the faulty node and the role information of the faulty node stored in the memory of the main monitoring and management node, switching the actual role information of the faulty node to the role information of the faulty node stored in the memory of the main monitoring and management node.

[0108] The switching submodule includes: a first switching unit, a second switching unit, and a determining unit. The first switching unit is used to determine the number of monitoring management nodes in each of the multiple monitoring management node sub-clusters formed after the split-brain event occurs in the message middleware system; to determine all monitoring management nodes in the monitoring management node sub-clusters whose number of monitoring management nodes does not exceed a preset number as slave monitoring management nodes; and to determine one monitoring management node as the master monitoring management node in the monitoring management node sub-clusters whose number of monitoring management nodes exceeds the preset number.

[0109] The second switching unit is used to determine, when all message storage nodes with role information as master nodes exist in the target monitoring and management node sub-cluster, that the message storage node with role information as master nodes in other monitoring and management node sub-clusters is the monitoring and management node sub-cluster where the master monitoring and management node is located; to switch the role information of the target fault node to a slave node; and to store the switched role information of the target fault node in the memory of the master monitoring and management node.

[0110] The method further includes, after the actual role information of the faulty node is switched to the role information of the faulty node stored in the memory of the main monitoring and management node, determining the message storage node with the largest offset in the message storage node set without a master node in the multiple monitoring and management node sub-clusters as the master node.

[0111] The disaster recovery switching device for the message storage node includes a processor and a memory. The aforementioned multiple unit modules are all stored as program units in the memory, and the processor executes these program units to achieve the corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0112] The processor contains a kernel that retrieves the corresponding program units from memory. One or more kernels can be configured to periodically receive heartbeat information from message storage nodes to identify faulty nodes. After identifying a faulty node, a new master node is selected by comparing its offset with the faulty node, and the switch is completed. This achieves the goal of automatically detecting faulty nodes in the message storage node and switching node roles, solving the technical problem of low message transmission efficiency caused by the reliance on manual switching when a message storage node fails in related technologies.

[0113] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0114] This invention provides a computer-readable storage medium including a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute a disaster recovery switching method for the message storage node.

[0115] This invention provides an electronic device, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. When the processor executes the program, it performs at least the following steps: receiving heartbeat information from multiple message storage nodes at a preset period through a master monitoring and management node among multiple monitoring and management nodes; determining a faulty node from the multiple message storage nodes based at least on the heartbeat information of the multiple message storage nodes, and, assuming the faulty node's role information is a master node, determining the offset of all other message storage nodes in a first set of message storage nodes where the faulty node is located; switching the faulty node's role information to a slave node, and selecting the message storage node with the largest offset from all other message storage nodes as a target node; and switching the target node's role information to a master node. The device in this application can be a server, PC, PAD, mobile phone, etc.

[0116] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0122] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0123] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0125] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A disaster recovery switching method for a message storage node, characterized in that, include: The main monitoring and management node among multiple monitoring and management nodes receives heartbeat information from multiple message storage nodes according to a preset cycle; At least based on the heartbeat information of the multiple message storage nodes, the faulty node is determined from the multiple message storage nodes, and if the role information of the faulty node is the master node, the offset of all other message storage nodes in the first message storage node set where the faulty node is located is determined. The role information of the faulty node is switched to that of a slave node, and the message storage node with the largest offset is selected from all other message storage nodes and determined as the target node. Switch the role information of the target node to the master node; Switch the replication mode of the first message storage node set from synchronous replication mode to asynchronous replication mode; Detect whether the faulty node has recovered; If the faulty node has been recovered, check whether the difference between the data stored in the master node and the slave node in the first message storage node set is less than a preset threshold. If the difference value is less than the preset threshold, the replication mode of the first message storage node set is switched from asynchronous replication mode to synchronous replication mode.

2. The method according to claim 1, characterized in that, The step of determining the faulty node from the plurality of message storage nodes based at least on the heartbeat information of the plurality of message storage nodes includes: The message storage node that has not sent heartbeat information to the main monitoring and management node within a preset time period among the multiple message storage nodes is identified as the initial fault node; If the configuration information of the initial fault node is not found in the normal configuration node corresponding to the initial fault node, the initial fault node is determined to be the fault node, wherein the configuration node is used to store the configuration information of the message storage node.

3. The method according to claim 2, characterized in that, After switching the role information of the target node to that of the primary node, the method further includes: The configuration and role information of all nodes in the first message storage node set are synchronized to the slave monitoring and management nodes other than the master monitoring and management node.

4. The method according to claim 1, characterized in that, If the failure of the faulty node is caused by a split-brain effect in the message middleware system, the method further includes: Switch the role information of the monitoring and management node; Switch the role information of the message storage node; After the network failure of the message middleware system is recovered, if there is a conflict between the actual role information of the faulty node and the role information of the faulty node stored in the memory of the main monitoring and management node, the actual role information of the faulty node will be switched to the role information of the faulty node stored in the memory of the main monitoring and management node.

5. The method according to claim 4, characterized in that, Switching the role information of the monitoring and management node includes: Determine the number of monitoring and management nodes in each of the multiple monitoring and management node sub-clusters formed after the split-brain event occurs in the message middleware system. All monitoring management nodes in a sub-cluster of monitoring management nodes whose number of monitoring management nodes does not exceed a preset number will be designated as slave monitoring management nodes; In a sub-cluster of monitoring and management nodes where the number of monitoring and management nodes exceeds a preset number, one monitoring and management node is selected as the main monitoring and management node.

6. The method according to claim 4, characterized in that, Switching the role information of the message storage node includes: If, among all message storage nodes for main nodes with role information, there exists one that is located in a sub-cluster of the target monitoring and management node, then the message storage node for main nodes with role information in other sub-clusters of the monitoring and management node is determined as the target fault node. Here, the target sub-cluster of the monitoring and management node is the sub-cluster of the monitoring and management node where the main monitoring and management node is located. Switch the role information of the target faulty node to that of a slave node; The role information of the target fault node after switching is stored in the memory of the main monitoring and management node.

7. The method according to claim 6, characterized in that, After switching the actual role information of the faulty node to the role information of the faulty node stored in the memory of the main monitoring and management node, the method further includes: In the case where there is a set of message storage nodes without a master node in the multiple monitoring and management node sub-clusters, the message storage node with the largest offset in the set of message storage nodes without a master node is determined as the master node.

8. A message middleware system, characterized in that, include: Monitor and manage the node cluster and message storage node cluster; The monitoring and management node cluster is used to receive heartbeat information from multiple message storage nodes through the main monitoring and management node in a preset period. At least based on the heartbeat information of the multiple message storage nodes, the faulty node is determined from the multiple message storage nodes, and if the role information of the faulty node is the master node, the offset of all other message storage nodes in the first message storage node set where the faulty node is located is determined. The role information of the faulty node is switched to that of a slave node, and the message storage node with the largest offset is selected from all other message storage nodes and determined as the target node. The message storage node cluster is used to store messages and switch the role information of the target node to the master node; Switch the replication mode of the first message storage node set from synchronous replication mode to asynchronous replication mode; Detect whether the faulty node has recovered; If the faulty node has been recovered, check whether the difference between the data stored in the master node and the slave node in the first message storage node set is less than a preset threshold. If the difference value is less than the preset threshold, the replication mode of the first message storage node set is switched from asynchronous replication mode to synchronous replication mode.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the disaster recovery switching method for the message storage node as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a disaster recovery switching method for performing a message storage node as described in any one of claims 1 to 7.

Images