Message processing method and device, equipment and medium

Abstract

The invention relates to the field of message processing, in particular to a message processing method and device, equipment and a medium, and the method comprises the steps: recognizing and removing expired messages of a first message queue, messages from a disconnected sender and command messages according to a first preset length threshold value and a second preset length threshold value, therefore, redundant and invalid messages in the message queue are effectively reduced, the obtained second message queue enables the amount of messages needing to be processed by the target monitor node to be reduced by adding a message preprocessing mechanism, so that the ability of the target monitor node to deal with a large number of sudden failures is improved, and the message processing efficiency is improved. Frequent election caused by untimely message processing is reduced, and the stability and impact resistance of the target monitor node are improved.

Description

Technical Field

[0001] The present invention relates to the field of message processing, and in particular, to a message processing method, apparatus, device, and medium. Background Art

[0002] A distributed file system (cluster) includes a large number of OSD services and multiple Monitor services. The network is divided into a front-end network and a back-end network, and the two networks are isolated. The Monitor service uses the front-end network to communicate with services such as OSD. The OSD service uses the front-end network to communicate with the Monitor and receive read / write requests from clients, and uses the back-end network for data synchronization and data reconstruction between the primary replica OSDs. When the core switch of the back-end network is abnormal, it will cause the back-end network to be abnormal, resulting in OSD heartbeat timeout. After the OSD heartbeat timeout, it will report a MOSDFailure message to the Monitor, and the reporting volume will be large in a short time. In addition, after the back-end network is abnormal, it will also cause the data synchronization between the primary replicas of the OSDs to be blocked, resulting in front-end IO blocking. IO blocking, OSD heartbeat timeout, etc. will also cause the OSD to generate abnormal logs and send MLog messages to the Monitor. The MOSDFailure message and the MLog message need to be centralized and processed by the primary Monitor (leader). The primary Monitor (mon.0) needs to process a large number of messages in a short time, resulting in timeout and triggering an election to reselect the leader. During the election process, new messages enter the queue and wait, and the message processing is carried out after the election is completed. Due to the timeout of the primary Monitor (mon.0), an election is triggered, and a new Monitor (mon.1) is elected as the primary Monitor, and a large number of MOSDFailures, etc., are resent to mon.1. When mon.0 finishes processing the MOSDFailure, an election is triggered, and at this time mon.0 becomes the primary Monitor again, and a large number of MOSDFailure messages are resent to mon.0. When mon.1 finishes processing, an election is triggered again.

[0003] The above process occurs repeatedly, that is, mon.0 and mon.1 (or other Monitor nodes) alternately become the primary Monitor, and each replacement is accompanied by the resending and processing of a large number of messages. Then the system will fall into a state of election oscillation, resulting in the system being unable to provide stable services.

[0004] Therefore, how to reduce the risk of election oscillation and improve the stability and performance of the distributed system is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] The object of the present invention is to provide a message processing method, device, equipment and medium, which can reduce the risk of election oscillation and improve the stability and performance of a distributed system.

[0006] In a first aspect, a message processing method is provided, which is applied to a distributed file system. The distributed file system includes multiple electronic devices. The message processing method is executed by an electronic device with a target monitor node deployed among the multiple devices. The message processing method includes:

[0007] Obtain the first length of the first message queue of the target monitor node;

[0008] If the first length is greater than a first preset length threshold, determine expired messages from the first message queue as the first messages;

[0009] Determine the messages sent by a target sender from the first message queue as the second messages, where the target sender is a sender that is disconnected from the target monitor node, and the senders are other monitor nodes and / or client devices;

[0010] When the first length of the first message queue is greater than a second preset length threshold, determine command messages from the first message queue as the third messages, where the second preset length threshold is greater than the first preset length threshold;

[0011] Remove at least one of the first messages, second messages, and third messages from the first message queue to obtain a second message queue; call the target monitor node to process messages according to the second message queue.

[0012] In a preferred example of the present invention, it can be further configured that after calling the target monitor node to process messages according to the second message queue, it further includes:

[0013] When the target monitor node processes the messages in the second message queue, if the difference between the target time and the current time is greater than a preset duration, send a target message to a specified monitor node corresponding to the target monitor node;

[0014] where the target time is the time when the target message was last sent.

[0015] In a preferred example of the present invention, it can be further configured that it further includes:

[0016] Obtain the number of times the target message is sent;

[0017] If the number of times is greater than a preset number threshold, trigger a re-election mechanism to facilitate the re-election of a committee, where the committee includes: a main monitor node and multiple slave monitor nodes.

[0018] In a preferred example, the present invention can be further configured as follows: The calling of the target monitor node to process messages according to the second message queue includes:

[0019] Determine, from the second message queue, a target message that can cause a message processing timeout of the target monitor node, and insert the target message into the head of the second message queue to obtain a third message queue;

[0020] Call the target monitor node to process messages according to the third message queue.

[0021] In a preferred example, the present invention can be further configured as follows: The calling of the target monitor node to process messages according to the second message queue includes:

[0022] Obtain the processing capacity levels of each monitor node in the cluster;

[0023] If the processing capacity level of the target monitor node is greater than a preset level, call the target monitor node to process messages according to the second message queue;

[0024] If the processing capacity level of the target monitor node is not greater than the preset level, obtain the message types of each message in the second message queue, where the message types include self - processing types and non - self - processing types; according to the processing capacity levels of other monitor nodes, allocate the non - self - processing type messages to other monitor nodes; call the target monitor node to process the messages of the self - processing type according to the second message queue, where other monitor nodes are the monitor nodes in each monitor node except the target monitor node.

[0025] In a preferred example, the present invention can be further configured as follows: Allocating the non - self - processing type messages to other monitor nodes according to the processing capacity levels of each monitor node includes:

[0026] Estimate the remaining processing capacity of each monitor node according to the processing capacity levels of each monitor node and the current message processing volume of each monitor node;

[0027] Allocate the non - self - processing type messages to other monitor nodes according to the remaining processing capacity of each monitor node.

[0028] In a preferred example, the present invention can be further configured as follows: Calling the target monitor node to process the messages of the self - processing type according to the second message queue includes:

[0029] Obtain the message nature level, urgency level, and estimated message processing duration of messages of each self - processing type;

[0030] According to the message nature level and urgency level of messages of each self - processing type, determine the initial priority of messages of each self - processing type; if the estimated message processing duration is greater than a preset duration, then increase the initial priority to obtain the priority; if the estimated message processing duration is not greater than the preset duration, then decrease the initial priority to obtain the priority;

[0031] Call the target monitor node to process the messages of the self - processing type in the second message queue according to the priority of messages of each self - processing type.

[0032] In a second aspect, a message processing device is provided, including:

[0033] An acquisition module, configured to acquire the first length of the first message queue of the target monitor node; if the first length is greater than a first preset length threshold, then trigger the determination module;

[0034] A determination module, configured to, if the first length is greater than the first preset length threshold, determine the expired messages from the first message queue as the first messages; determine the messages sent by the target sender from the first message queue as the second messages, where the target sender is the sender that is disconnected from the target monitor node, and the senders are other monitor nodes and / or client devices; when the first length of the first message queue is greater than a second preset length threshold, determine the command messages from the first message queue as the third messages, where the second preset length threshold is greater than the first preset length threshold;

[0035] A message processing module, configured to remove at least one of the first messages, second messages, and third messages from the first message queue to obtain a second message queue; a module, configured to call the target monitor node to perform message processing according to the second message queue.

[0036] In a third aspect, an electronic device is provided, including:

[0037] One or more processors;

[0038] A memory;

[0039] One or more applications, where one or more applications are stored in the memory and are configured to be executed by one or more processors, and one or more programs are configured to: perform the operations corresponding to the message processing method shown in any possible implementation manner of the first aspect.

[0040] In a fourth aspect, a computer-readable storage medium is provided. The storage medium stores at least one instruction, at least one segment of program, a code set, or an instruction set. The at least one instruction, the at least one segment of program, the code set, or the instruction set is loaded and executed by a processor to perform the steps of the message processing method shown in any possible implementation manner of the first aspect.

[0041] In a fifth aspect, a computer program product is provided, including a computer program. When the computer program is executed by a processor, it implements the operations corresponding to the message processing method shown in any possible implementation manner of the first aspect.

[0042] In summary, the message processing method provided by the present invention is applied to a distributed file system. The distributed file system includes multiple electronic devices. The message processing method is executed by an electronic device with a target monitor node deployed among the multiple devices. The message processing method has the following beneficial technical effects:

[0043] In this solution, the first length of the first message queue of the target monitor node is obtained; if the first length is greater than the first preset length threshold, expired messages are determined from the first message queue as the first messages; messages sent by a target sender are determined from the first message queue as the second messages, where the target sender is a sender that has disconnected from the target monitor node, and the senders are other monitor nodes and / or client devices; when the first length of the first message queue is greater than the second preset length threshold, command messages are determined from the first message queue as the third messages, where the second preset length threshold is greater than the first preset length threshold; at least one of the first messages, the second messages, and the third messages in the first message queue is removed to obtain a second message queue; the target monitor node is called to process messages based on the second message queue. In the present invention, according to the first preset length threshold and the second preset length threshold, expired messages, messages from disconnected senders, and command messages in the first message queue are identified and removed, thereby effectively reducing redundant and invalid messages in the message queue. The obtained second message queue reduces the amount of messages that the target monitor node needs to process by adding a message preprocessing mechanism, so as to improve the ability of the monitor node to handle a large number of sudden failures, reduce the frequent elections caused by untimely message processing, and improve the stability and shock resistance of the monitor node.

[0044] In addition, the present invention also provides a message processing device, equipment, and medium, all of which have the above beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of a message processing method provided by an embodiment of the present invention;

[0047] Figure 2 It is a schematic flowchart of a pre-timeout check mechanism provided by an embodiment of the present invention;

[0048] Figure 3 It is a schematic structural diagram of a message processing device provided by an embodiment of the present invention;

[0049] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0050] This specific embodiment is only an explanation of the present invention, and it is not a limitation of the present invention. After reading this specification, those skilled in the art can make modifications without creative contributions to this embodiment as needed, but as long as they are within the scope of the present invention, they are protected by the patent law.

[0051] It should be noted that in the optional embodiments of the present invention, for relevant data such as object information, when the embodiments of the present invention are applied to specific products or technologies, object permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions. That is to say, if the embodiments of the present invention involve data related to objects, they need to be obtained under the authorization and consent of the objects, the authorization and consent of relevant departments, and compliance with relevant laws, regulations, and standards of relevant countries and regions. In the embodiments, if personal information is involved, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained, and the embodiments also need to be implemented under the authorization and consent of the object.

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0053] In addition, the term "and / or" in this text is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this text generally represents an "or" relationship between the front and back associated objects unless otherwise specified.

[0054] To better understand and illustrate the solutions of the embodiments of the present invention, some technical terms involved in the embodiments of the present invention are briefly described below.

[0055] OSD, Object-based Storage Device, object storage device;

[0056] PG, Placement Group placement group, which is a carrier for placing objects;

[0057] Mon, Monitor monitor daemon process of the cluster;

[0058] Storage pool. The storage pool is a logical partition in a distributed file system and is a collection of PGs;

[0059] Object object, that is, the smallest unit for data storage, distributed in each PG, and a large number of objects are stored in one PG;

[0060] Cluster, short for distributed file system;

[0061] Node, the physical host of a distributed file system;

[0062] High water level, the used capacity of the hard disk of the OSD exceeds 85%;

[0063] Cli, command-line interface command line interface;

[0064] Data reconstruction. After a cluster fails, the data redundancy will decrease, and the data will be restored on a new OSD. The process of data restoration is called data reconstruction. In addition, when the cluster is expanded online, the data will be allocated to the newly expanded nodes, that is, part of the data will be migrated to the OSDs of the new nodes, and this process of data migration is also called data reconstruction.

[0065] The Monitor module is the core module of the distributed file system. Hereinafter, the distributed file system is referred to as the cluster for short. The main function of the Monitor module is to manage the cluster and monitor various states of the cluster. The Monitor is responsible for managing various Maps of the cluster, such as OSDMap (managed by OSDMonitor), PGMap (managed by PGMonior), MDSMap (managed by MDSMonitor), etc. In addition, the Monitor module is also responsible for responding to query and setting commands of the cluster. When the Monitor module is abnormal, the cluster status cannot be viewed. Generally, multiple different nodes in the cluster are selected to deploy the Monitor service, and the number of deployed Monitors is generally 3, 5, or 7. Through the election mechanism, 1 main Monitor is selected among the Monitor services. The main Monitor is called the leader; other Monitors are called peons. 1 leader and multiple peons form the Monitor committee, and only when more than half of the Monitors participate in the election can the committee be formed. Otherwise, the Monitor cannot elect a committee, and only when the committee exists can the service be provided. When the Monitor is in the election state, the service cannot be provided either. Events that trigger the election: the start, exit, and timeout of the Monitor service. The members of the Monitor committee also have different responsibilities. The Monitor in the peon role can only handle query requests, such as query commands and subscription requests for obtaining OSDMap, etc. The Monitor in the leader state can handle both query requests and update requests, such as setting commands and requests for changing OSDMap. When the Monitor in the peon role receives an update request, it will also forward it to the main Monitor (leader). Therefore, when there are too many update requests, the main Monitor (leader) will become a bottleneck.

[0066] In practical applications, a 120-node cluster is deployed, which includes 2,880 OSD services and 7 Monitor services. Additionally, the network is divided into a front-end network and a back-end network, and these two networks are isolated. The Monitor service uses the front-end network to communicate with services such as OSD. The OSD service uses the front-end network to communicate with the Monitor and receive read / write requests from clients, and uses the back-end network for data synchronization and data reconstruction between primary replica OSDs. When the core switch of the back-end network fails, it will cause the back-end network to be abnormal, resulting in OSD heartbeat timeouts. After the OSD heartbeat times out, it will report a MOSDFailure message to the Monitor, and the reporting volume in a short period will reach 28,800. Additionally, after the back-end network is abnormal, it will cause data synchronization between OSD primary replicas to be blocked, resulting in front-end IO blocking. IO blocking, OSD heartbeat timeouts, etc. will also cause the OSD to generate abnormal logs and send MLog messages to the Monitor. The MOSDFailure message and the MLog message need to be centralized and processed by the primary Monitor (leader). The primary Monitor (mon.0) needs to process a large number of messages in a short period, resulting in timeouts and triggering elections. During the election process, messages enter the queue and wait, and are processed after the election is completed. Due to the timeout of the primary Monitor (mon.0), an election is triggered, and a new Monitor (mon.1) becomes the primary Monitor. A large number of MOSDFailure, etc., are resent to mon.1. When mon.0 finishes processing the MOSDFailure, an election is triggered, and at this time, mon.0 becomes the primary Monitor again, and a large number of MOSDFailure messages are resent to mon.0. When mon.1 finishes processing, an election is triggered again. This will repeat continuously, causing the Monitor to keep oscillating and keep electing, and unable to provide services.

[0067] Based on the above problems, the embodiment of the present invention proposes an optimization method for the processing ability of the Monitor in a distributed file system, which solves the problem that a large number of messages are sent to the Monitor centrally in the case of a large number of node exceptions generated in a short period in a large-scale cluster, resulting in frequent elections of the Monitor.

[0068] Specifically, the embodiment of the present invention provides a message processing method, which is applied to a distributed file system. The distributed file system includes multiple electronic devices, and the message processing method is executed by an electronic device deployed with a target monitor node among the multiple devices, as Figure 2 shown. This method for the client device includes:

[0069] S101. Obtain the first length of the first message queue of the target monitor node;

[0070] Among them, the target monitor node is any monitor node in the committee, which can be the main monitor node or the slave monitor node. The first message queue refers to the queue used to store the messages to be processed by the target monitor node. The first length represents the number of messages currently stored in the message queue.

[0071] S102. If the first length is greater than the first preset length threshold, determine the expired messages from the first message queue as the first messages.

[0072] Among them, the first preset length threshold is a preset value, and the range of the first preset length threshold can be 500 - 1000, which can be customized by the user, such as 500, 650, 750, 800, 900, 1000, etc.

[0073] It can be understood that in a distributed system, each node may maintain a message queue to receive and process incoming messages; if the system is upgraded or a new committee is re-elected, and all messages use a new epoch value after the upgrade or re-election of the new committee, then the messages under the old epoch value are expired messages and need to be discarded and no longer processed. Generally, the new epoch value is greater than the old epoch value.

[0074] In the embodiment of the present invention, when the first length is not greater than the first preset length threshold, continuously monitor the size of the first length. When the first length is greater than the first preset length threshold, traverse the first message queue, compare the epoch value of each message with the current epoch value; the messages with epoch values less than the current epoch value are expired messages and marked as the first messages.

[0075] S103. Determine the messages sent by the target sender from the first message queue as the second messages, where the target sender is the sender disconnected from the target monitor node.

[0076] The sender is other monitor nodes and / or client devices.

[0077] In the embodiment of the present invention, the first message queue can be a queue without the first message mark, or a queue with the first message mark.

[0078] Different client devices / other monitor nodes (senders) may send messages to the target monitor node. If the sender is disconnected from the server due to network problems, program crashes, or has already connected to other monitor nodes, but the messages it sent before disconnection are still received by the server and stored in the first message queue. At this time, the electronic device needs to be able to identify the messages sent by the disconnected client from these messages for subsequent processing, such as discarding.

[0079] Specifically, traverse the messages in the first message queue, and check the sender identifier of each message; compare the sender identifier of each message with the list of senders that are currently disconnected from the target monitor node; if the sender identifier of a certain message is in the list of disconnected senders, determine it as the second message.

[0080] S104. When the first length of the first message queue is greater than the second preset length threshold, determine a command message from the first message queue as the third message, where the second preset length threshold is greater than the first preset length threshold.

[0081] The second preset length threshold is greater than the first preset length threshold, so as to trigger the process of deleting unimportant messages when the length of the message queue further increases and reaches a more serious backlog state.

[0082] Among them, the first preset length threshold is a preset value, and the range of the first preset length threshold can be 2000 - 3000, which can be customized by the user, such as 2000, 2200, 2400, 2600, 2800, 2900, 3000, etc.

[0083] Since most of the command messages are query commands and are periodically called by the management software, when the monitor is under heavy pressure, these unimportant messages need to be discarded to relieve the burden on the monitor.

[0084] S105. Remove at least one of the first message, the second message, and the third message from the first message queue to obtain a second message queue; call the target monitor node to process messages according to the second message queue.

[0085] In the embodiments of the present invention, a message preprocessing mechanism is set up to traverse each message in the message queue one by one and perform preprocessing on each message, so as to remove useless and less important messages to obtain a second message queue, and then call the target monitor node to process messages according to the second message queue.

[0086] In some embodiments, the data in the second message queue can also be calculated for priorities, so as to process each message according to the priorities and be able to give priority to processing some important and time-sensitive messages in a timely manner.

[0087] It can be understood that if the messages to be removed are at least two types of messages, it can be to filter out the first message from the message queue and then filter out the second message from the remaining messages, and so on, until all invalid messages are obtained; it can also be to traverse the message queue every time a message is recognized; it can also be to recognize at least two messages simultaneously during the traversal. The specific method is not limited in the embodiments of the present invention. At the same time, the order of the above three message filtering methods is not limited in this embodiment either, that is, the execution order of S102, S103, and S104 is not limited in this embodiment and can be adjusted according to actual needs.

[0088] It can be seen that in the embodiments of the present invention, according to the first preset length threshold and the second preset length threshold, expired messages, messages from a disconnected sender, and command messages in the first message queue are identified and removed, thereby effectively reducing the redundancy and invalid messages in the message queue. The obtained second message queue reduces the amount of messages that the target monitor node needs to process by adding a message preprocessing mechanism, so as to improve the ability of the monitor to handle a large number of sudden failures, reduce the frequent election situation caused by untimely message processing, and improve the stability and shock resistance of the monitor.

[0089] Conventionally, the main Monitor regularly sends lease messages to the peon, and the peon immediately replies with a lease_ack message after receiving the lease message. If the peon does not send a lease_ack message to the main Monitor within a certain period of time, the main Monitor will determine that the peon has timed out and trigger an election. If the peon does not receive a lease message sent by the main Monitor within a certain period of time, the peon will determine that the main Monitor has timed out and trigger an election. In the case of a large amount of data, it is easy to cause oscillations and continuous elections, and services cannot be provided. Therefore, in a possible implementation manner of the embodiments of the present invention, after calling the target monitor node to process messages according to the second message queue, it further includes: when the target monitor node processes the messages in the second message queue, if the difference between the target time and the current time is greater than the preset duration, send a target message to the specified monitor node corresponding to the target monitor node; where the target time is the time when the target message was last sent; if the target monitor node is the main monitor node, the target message is a lease message; if the target monitor node is a slave monitor node, the target message is a lease_ack message.

[0090] See Figure 2 , Figure 2 FIG. is a schematic flowchart of a pre-timeout check mechanism provided by an embodiment of the present invention. Among them, a timer is started when the Monitor module is initialized, and a check task that is executed every 1 second is inserted. The check task is executed once every second, and the main work is as follows:

[0091] (1) If it is the leader role (primary monitor node), proceed to (2) for further processing; otherwise, proceed to (4) for further processing.

[0092] (2) Check whether the time when the last lease message (the first message) was sent is greater than the preset duration (such as 9 seconds, 10 seconds). If it is greater than the preset duration, proceed to (3) for further processing; otherwise, end and wait for the next timing event to trigger.

[0093] (3) Immediately send a lease message once.

[0094] (4) Check whether it is the peon role (slave monitor node). If it is the peon role, proceed to (5) for further processing; otherwise, end and wait for the next timing event to trigger.

[0095] (5) Check whether the time when the last lease_ack message (the second message) was sent is greater than the preset duration. If it is greater than the preset duration, proceed to (6) for further processing; otherwise, end and wait for the next timing event to trigger.

[0096] (6) Immediately send a lease_ack message once.

[0097] (7) End and wait for the next timing event to trigger.

[0098] In the embodiments of the present invention, adding the pre-timeout check is to give the Monitor more time to process messages when a large number of sudden failures occur in a large-scale cluster, and to avoid interrupting message processing due to timeout.

[0099] Further, a possible implementation manner of the embodiments of the present invention further includes: obtaining the number of times of sending the target message; if the number is greater than the preset number threshold, triggering a re-election mechanism to facilitate re-electing the committee, where the committee includes: a primary monitor node and multiple slave monitor nodes.

[0100] Among them, the preset number threshold can be custom-set by the user, and the embodiments of the present invention do not limit it. If the number of transmissions exceeds the threshold, it means that the message processing is slow, and there may be a failure or a large load. At this time, the re-election mechanism is triggered to select a primary monitor node and multiple slave monitor nodes, so as to maintain the stability and reliability of the monitoring committee.

[0101] A possible implementation of an embodiment of the present invention is to call a target monitor node to process messages according to a second message queue, including: determining, from the second message queue, a target message that can cause a message processing timeout of the target monitor node, and inserting the target message into the head of the second message queue to obtain a third message queue; calling the target monitor node to process messages according to the third message queue.

[0102] Among them, when the target message is a message such as lease, lease_ack, begin, accept, etc. that can cause a Monitor timeout, the message is taken out of the message queue and inserted into a temporary queue. If the temporary queue is not empty, the temporary queue is inserted into the head of the message queue, so as to be able to give priority to processing messages that may cause timeouts, thereby reducing the occurrence of timeouts and improving the stability and efficiency of the system.

[0103] A possible implementation of an embodiment of the present invention is to call a target monitor node to process messages according to a second message queue, including: obtaining the processing ability levels of each monitor node in the cluster; if the processing ability level of the target monitor node is greater than a preset level, calling the target monitor node to process messages according to the second message queue;

[0104] If the processing ability level of the target monitor node is not greater than the preset level, obtain the message types of each message in the second message queue, where the message types include self-processing types and non-self-processing types; allocate non-self-processing type messages to other monitor nodes according to the processing ability levels of other monitor nodes; call the target monitor node to process messages of the self-processing type according to the second message queue, where the other monitor nodes are the monitor nodes other than the target monitor node among each monitor node.

[0105] Among them, the processing ability level refers to the performance level of the Monitor in processing messages, specifically the number of message processes per unit time. Obtain the number of message processes of each monitor node in multiple historical periods before the current moment, and perform weighted calculation according to the number of message processes of each monitor node in multiple historical periods and the weights corresponding to multiple historical periods to obtain the estimated number of message processes corresponding to the current period. Among them, the weight of the historical period closer to the current period is greater. Determine the processing ability level corresponding to the estimated number of message processes according to a preset corresponding relationship, where the corresponding relationship is the corresponding relationship between the preset message processing number range and the level.

[0106] The preset level can be set by users according to actual needs. When the processing capacity level of the target monitor node is greater than the preset level, it indicates that the message processing capacity of the target monitor node is high and it has excellent performance. Therefore, the messages in the second message queue can be directly processed by the target monitor node. When the processing capacity level of the target monitor node is not greater than the preset level, it means that the message processing capacity of the target monitor node is average. At this time, it can be considered to distribute the messages to other monitor nodes to improve the message processing efficiency. Specifically, according to the processing capacity levels and message types of other monitor nodes, a load balancing algorithm can be used to allocate one or more other monitor nodes for messages of non-self-processing types.

[0107] It can be seen that in the embodiments of the present invention, by obtaining the processing capacity levels of each monitor node in the cluster, if the processing capacity level of the target monitor node is not greater than the preset level, the non-self-processing types are allocated according to the processing capacity levels of other monitor nodes, ensuring that messages can be processed efficiently and reasonably, and avoiding message accumulation.

[0108] A possible implementation manner of the embodiments of the present invention is to allocate messages of non-self-processing types to other monitor nodes according to the processing capacity levels of each monitor node, including: estimating the remaining processing capacity of each monitor node according to the processing capacity levels of each monitor node and the current message processing volume of each monitor node; and allocating messages of non-self-processing types to other monitor nodes according to the remaining processing capacity of each monitor node.

[0109] Among them, the remaining processing capacity is the number of messages that can be processed remaining. Each processing capacity level corresponds to a range of the number of messages that can be processed. The intermediate value is selected as the number of messages that can be processed, and the difference is calculated between the number of messages that can be processed and the current message processing volume, and the obtained value is used as the remaining processing capacity.

[0110] In a feasible manner, if only the target monitor node needs other monitor nodes to share messages during the current period, any other monitor node that meets the remaining processing capacity of messages of non-self-processing types can be selected to share the messages; the messages can also be evenly distributed to other monitor nodes. The present invention example does not limit further.

[0111] In another feasible manner, if there are multiple monitors that need to share messages during the current period, all monitor nodes in the cluster that can share messages are determined. And, all the messages that need to be shared are determined, and then the message allocation is performed.

[0112] It can be seen that in the embodiment of the present invention, when allocating messages of non-self-processing types, the processing capacity levels of each monitor node and the current message processing volume are considered, so as to estimate the remaining processing capacity of each monitor node, ensuring that each Monitor can efficiently process messages within its processing capacity and avoiding the occurrence of election situations caused by message processing delays due to overload.

[0113] A possible implementation manner of the embodiment of the present invention is to call the target monitor node to process messages of the self-processing type according to the second message queue, including:

[0114] Obtain the message property level, urgency level, and estimated message processing duration of each message of the self-processing type;

[0115] According to the message property level and urgency level of each message of the self-processing type, determine the initial priority of each message of the self-processing type; if the estimated message processing duration is greater than the preset duration, increase the initial priority to obtain the priority; if the estimated message processing duration is not greater than the preset duration, decrease the initial priority to obtain the priority;

[0116] Call the target monitor node to process the messages of the self-processing type in the second message queue according to the priority of each message of the self-processing type.

[0117] Among them, the message property level is the importance degree representing the basic attributes of the message determined based on the type and source of the message based on the mapping relationship (the corresponding relationship between the type, source, and property level). The urgency level refers to the urgency of the message and is used to represent the urgency of the message to be processed, which can be manually marked by the user. The estimated message processing duration refers to the time estimated by the system for processing the message based on the processing duration of historical messages of the same type.

[0118] Based on IP = α×PL + β×UL, calculate the initial priority; IP is the initial priority; PL is the message property level; UL is the urgency level; α and β are weight coefficients used to adjust the influence degrees of the message property level and urgency level in the priority calculation and can be adjusted according to business requirements.

[0119] If the estimated message processing duration is greater than the preset duration, it means that the message may take a longer time to process. To avoid system congestion or delay, the initial priority of the message can be increased. The specific increase amplitude can be determined according to the degree of exceeding the preset duration. There may be a corresponding relationship between the duration difference and the level change difference. The larger the difference, the greater the increased level.

[0120] If the estimated message processing duration is not greater than the preset duration, it means that the message can be processed within the preset duration. To improve the efficiency of the system in processing other more important or urgent messages, the system can lower the initial priority of the message, and only lower it by one level.

[0121] It can be understood that if there is a target message in the messages of its own processing type that can cause the message processing of the target monitor node to time out, its priority is directly determined to be the highest and processed preferentially.

[0122] It can be seen that in the embodiments of the present invention, the initial priority is determined according to the nature level and urgency level of the message. If the estimated processing duration is greater than the preset duration, the initial priority is increased; if it is not greater than the preset duration, the initial priority is decreased; by preferentially processing those messages that require longer processing time, system resources can be utilized more effectively, reducing the delay and waiting time of message processing, and improving the response speed and user experience of the entire system.

[0123] Based on any of the above embodiments, the present invention designs a pre-timeout check and message preprocessing mechanism in the Monitor module, which greatly improves the ability of the Monitor to handle a large number of sudden failures, avoids the problem of cluster unavailability caused by frequent elections, and can eliminate the problem of message resending, backlog and inability to self-heal caused by frequent elections of the Monitor.

[0124] Specifically, setting up a pre-timeout check mechanism can give the Monitor more time to process messages when a large number of sudden failures occur in a large-scale cluster, avoiding message processing being interrupted due to timeout.

[0125] Set up a message preprocessing mechanism. When the Monitor module is initialized, a timer is started, and a processing task that is executed every 1 second is inserted. The processing task is executed once every second, and the main work is as follows:

[0126] (1) Check whether the length of the Monitor message queue (MOSDFailure message, MLog message) is greater than 1000. If it is greater than 1000, enter (2) to continue processing; if it is not greater than 10, exit and wait for the next timing event to trigger.

[0127] (2) Traverse each message in the message queue one by one, and perform preprocessing on each message. The following is the processing process when traversing each message.

[0128] (3) If the epoch of the Monitor (a new epoch is generated in each election) when the message is sent is smaller than the current epoch, delete the message from the message queue and discard the message. This is because in the original processing flow, such messages are also discarded. Discarding during preprocessing will not waste the processing time of the Monitor main thread.

[0129] (4) If the network connection of the message sender has been disconnected, delete the message from the message queue and discard it. This is because in the original processing flow, such messages were also discarded. Discarding them during preprocessing will not waste the processing time of the Monitor main thread.

[0130] (5) If the message is a command message and the current length of the message queue exceeds 2000, delete the message from the message queue and discard it. Since most command messages are query commands and are periodically called by the management software, when the Monitor is under high pressure, these unimportant messages need to be discarded to relieve the burden on the Monitor.

[0131] (6) When the message types are lease, lease_ack, begin, accept, etc., which will cause the Monitor to time out, take out the message from the message queue and insert it into the temporary queue.

[0132] (7) After the traversal is completed, if the temporary queue is not empty, insert the temporary queue into the head of the message queue.

[0133] The main function of the message preprocessing mechanism is to identify and discard some useless and less important messages in advance. At the same time, it is also necessary to identify some important and time-sensitive messages for priority processing.

[0134] The implementation of the present invention improves the ability of the Monitor to handle a large number of sudden failures, avoids the problem of cluster unavailability caused by frequent elections, and can eliminate the problems of message resending, backlog and inability to self-heal caused by frequent elections of the Monitor; improves the stability and shock resistance of the Monitor; at the same time reduces the operation and maintenance costs, and can quickly recover without manual intervention, and will not cause the cluster to be unavailable for a long time.

[0135] Next, a message processing device provided by an embodiment of the present invention will be introduced. The device described below can be correspondingly referred to the method described above. The message processing device of this embodiment is set in an electronic device. Refer to Figure 3 , Figure 3 is the structural block diagram of the message processing device of one embodiment of the present invention, including:

[0136] An obtaining module 210, configured to obtain the first length of the first message queue of the target monitor node; if the first length is greater than the first preset length threshold, trigger the determining module;

[0137] A determination module 220, configured to determine, if a first length is greater than a first preset length threshold, an expired message from a first message queue as a first message; determine a message sent by a target sender from the first message queue as a second message, where the target sender is a sender that is disconnected from a target monitor node, and the sender is other monitor nodes and / or client devices; when the first length of the first message queue is greater than a second preset length threshold, determine a command message from the first message queue as a third message, where the second preset length threshold is greater than the first preset length threshold;

[0138] A message processing module 230, configured to remove at least one of the first message, the second message, and the third message from the first message queue to obtain a second message queue; and call the target monitor node to process messages according to the second message queue.

[0139] In an implementable manner, it further includes:

[0140] A pre-timeout check module, configured to, when the target monitor node processes messages in the second message queue, if the difference between a target time and the current time is greater than a preset duration, send a target message to a specified monitor node corresponding to the target monitor node; where the target time is the time when the target message was last sent.

[0141] In an implementable manner, the pre-timeout check module is further configured to: obtain the number of times of sending the target message; if the number is greater than a preset number threshold, trigger a re-election mechanism to re-elect a committee, where the committee includes: a primary monitor node and multiple secondary monitor nodes.

[0142] In an implementable manner, the message processing module 230 is further configured to:

[0143] Determine a target message that can cause message processing timeout of the target monitor node from the second message queue, and insert the target message into the head of the second message queue to obtain a third message queue;

[0144] Call the target monitor node to process messages according to the third message queue.

[0145] In an implementable manner, the message processing module 230 is further configured to:

[0146] Obtain the processing capability levels of each monitor node in the cluster;

[0147] If the processing capability level of the target monitor node is greater than a preset level, call the target monitor node to process messages according to the second message queue;

[0148] If the processing capacity level of the target monitor node is not greater than the preset level, obtain the message types of each message in the second message queue. The message types include the self-processing type and the non-self-processing type; allocate the non-self-processing type of messages to other monitor nodes according to the processing capacity levels of other monitor nodes, where the other monitor nodes are the monitor nodes other than the target monitor node among all the monitor nodes; call the target monitor node to process the messages of the self-processing type according to the second message queue.

[0149] In an implementable manner, the message processing module 230 is further configured to:

[0150] Estimate the remaining processing capacity of each monitor node according to the processing capacity levels of each monitor node and the current message processing volume of each monitor node;

[0151] Allocate the non-self-processing type of messages to other monitor nodes according to the remaining processing capacity of each monitor node.

[0152] In an implementable manner, the message processing module 230 is further configured to:

[0153] Obtain the message property level, urgency level, and estimated message processing duration of each message of the self-processing type;

[0154] Determine the initial priority of each message of the self-processing type according to the message property level and urgency level of each message of the self-processing type; if the estimated message processing duration is greater than the preset duration, increase the initial priority to obtain the priority; if the estimated message processing duration is not greater than the preset duration, decrease the initial priority to obtain the priority;

[0155] Call the target monitor node to process the messages of the self-processing type in the second message queue according to the priority of each message of the self-processing type.

[0156] In an embodiment of the present invention, an electronic device is provided, as Figure 4 shown, Figure 4 The electronic device 300 shown includes a processor 301 and a memory 303. Among them, the processor 301 and the memory 303 are connected, such as connected through a bus 302. Optionally, the electronic device 300 may further include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation to the embodiments of the present invention.

[0157] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present invention. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0158] The bus 302 may include a path for transmitting information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a thick line is shown herein, but it does not mean that there is only one bus or one type of bus.

[0159] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or it may also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0160] The memory 303 is used to store the application program code for implementing the solution of the present invention, and is controlled by the processor 301 to execute. The processor 301 is used to execute the application program code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0161] Figure 4 The illustrated electronic device is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

[0162] The embodiments of the present invention provide a computer-readable storage medium, on which a computer program is stored. When the computer program runs on a computer, it enables the computer to execute the corresponding content in the foregoing method embodiments.

[0163] The embodiments of the present invention provide a computer program product, including a computer program. When the computer program is executed by a processor, it implements the corresponding content in the foregoing method embodiments.

[0164] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps is not strictly restricted by order, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0165] The above are only some embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A message processing method, characterized in that: Applied to a distributed file system, the distributed file system includes multiple electronic devices, the message processing method is executed by an electronic device deployed with a target monitor node among the multiple devices, and the message processing method includes: Get the first length of the first message queue of the target monitor node; If the first length is greater than a first preset length threshold, determining an expired message from the first message queue as the first message; Determine a message sent by a target sender from the first message queue as a second message, wherein the target sender is a sender disconnected from the target monitor node, and the sender is another monitor node and / or a client device; When a first length of the first message queue is greater than a second preset length threshold, determining a command message from the first message queue as a third message, wherein the second preset length threshold is greater than the first preset length threshold; Remove at least one of the first message, the second message, and the third message in the first message queue to obtain a second message queue; call the target monitor node to perform message processing according to the second message queue.

2. The message processing method according to claim 1, characterized in that: After calling the target monitor node and performing message processing according to the second message queue, the method further includes: When the target monitor node processes the message of the second message queue, if the difference between the target time and the current time is greater than a preset time length, the target message is sent to a designated monitor node corresponding to the target monitor node; The target time is the time when the target message was sent last time.

3. The message processing method according to claim 2, characterized in that: Also includes: Obtain the number of times the target message is sent; If the number is greater than a preset number threshold, a re-election mechanism is triggered to re-elect the committee, wherein the committee includes: a master monitor node and multiple slave monitor nodes.

4. The message processing method according to claim 2, characterized in that: The calling of the target monitor node and performing message processing according to the second message queue includes: Determine a target message that can cause a target monitor node message processing timeout from the second message queue, and insert the target message into the head of the second message queue to obtain a third message queue; The target monitor node is called to process messages according to the third message queue.

5. The message processing method according to any one of claims 1 to 4, characterized in that: The calling of the target monitor node and performing message processing according to the second message queue includes: Get the processing power level of each monitor node in the cluster; If the processing capability level of the target monitor node is greater than a preset level, calling the target monitor node to process the message according to the second message queue; If the processing capability level of the target monitor node is not greater than the preset level, the message type of each message in the second message queue is obtained, and the message type includes a self-processing type and a non-self-processing type; according to the processing capability levels of other monitor nodes, the messages of the non-self-processing type are allocated to other monitor nodes; the target monitor node is called to process the messages of the self-processing type according to the second message queue, wherein the other monitor nodes are monitor nodes among the monitor nodes except the target monitor node.

6. The message processing method according to claim 5, characterized in that: According to the processing capability level of each monitor node, the messages of the non-self-processing type are allocated to other monitor nodes, including: Estimate the remaining processing capacity of each monitor node based on the processing capacity level of each monitor node and the current message processing volume of each monitor node; According to the remaining processing capacity of each monitor node, the messages of the type not to be processed by the monitor nodes themselves are allocated to other monitor nodes.

7. The message processing method according to claim 5, characterized in that: Calling the target monitor node to process messages of its own processing type according to the second message queue includes: Obtain the message nature level, urgency level, and estimated message processing time for each message of the type being processed; Determine the initial priority of each message of the self-processing type according to the message nature level and urgency level of each message of the self-processing type; if the estimated message processing time is greater than the preset time, increase the initial priority to obtain a priority; if the estimated message processing time is not greater than the preset time, reduce the initial priority to obtain a priority; The target monitor node is called to process the messages of the self-processing type of the second message queue according to the priority of each message of the self-processing type.

8. A message processing device, characterized in that: include: An acquisition module, used for acquiring a first length of a first message queue of a target monitor node; if the first length is greater than a first preset length threshold, triggering a determination module; A determination module, configured to determine an expired message from the first message queue as a first message if the first length is greater than a first preset length threshold; determine a message sent by a target sender from the first message queue as a second message, wherein the target sender is a sender disconnected from the target monitor node, and the sender is another monitor node and / or a client device; when the first length of the first message queue is greater than a second preset length threshold, determine a command message from the first message queue as a third message, wherein the second preset length threshold is greater than the first preset length threshold; A message processing module is used to remove at least one of the first message, the second message and the third message in the first message queue to obtain a second message queue; a module is used to call the target monitor node to perform message processing according to the second message queue.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to: execute the steps of the message processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded by the processor and executes the steps of the message processing method according to any one of claims 1 to 7.

Images