A system message queue processing method, system, computer device and medium

Abstract

The application provides a system message queue processing method, system, computer device and medium, and belongs to the technical field of distributed systems. The method generates a message fingerprint through a composite hash algorithm, dynamically collects system node performance indicators to construct a dynamic hash ring containing virtual nodes, maps the message fingerprint to the hash ring to match a target node and execute message distribution, simultaneously monitors the node health state in real time and redirects the message to a backup node through a secondary hash jump mechanism when the target node is abnormal. The application adopts the above system message queue processing method, system, computer device and medium, solves the problems of message backlog and uneven load, insufficient dynamic scalability, repeated consumption and chaotic order, low fault recovery efficiency and the like in the prior art, and can improve the efficiency, scalability, reliability and order guarantee capability of message queue processing in distributed systems, cloud computing platforms and Internet of Things communication.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed systems, and particularly relates to a system message queue processing method and system, computer equipment and a medium. BACKGROUND

[0002] System message queues are key components in distributed systems, cloud computing platforms, and Internet of Things communications, responsible for efficiently and reliably transmitting messages between various system components. However, existing message queue processing methods have many problems in practical applications:

[0003] Message backlog and uneven load: Traditional polling or random allocation strategies can easily lead to overloading of some nodes while others may be idle, reducing the overall efficiency of the system.

[0004] Lack of dynamic scalability: When nodes are added or removed, traditional methods require the redistribution of all messages, resulting in high system overhead and affecting system availability and performance.

[0005] Repeated consumption and sequence disorder: Lack of unique identifier mapping in message distribution makes it difficult to ensure message sequence consistency, leading to business logic confusion.

[0006] Low fault recovery efficiency: After node downtime, the message redistribution mechanism is not perfect, and the system recovery time is long, affecting business continuity.

[0007] In the prior art, although hash algorithms such as consistent hashing have made some progress in solving the problem of dynamic node changes, there are still the following shortcomings in practical applications:

[0008] Traditional hash modulo method: When nodes change, the hash value mapping is inefficient, resulting in a large number of messages needing to be redistributed.

[0009] Message queue middleware (such as Kafka / RabbitMQ): Relies on partitioning and replication mechanisms, although it improves reliability, but resource consumption is large, difficult to meet the needs of large-scale distributed systems.

[0010] These problems show that there is still room for improvement in existing technologies in terms of system message queue processing, especially in improving message distribution efficiency, enhancing load balancing capability, ensuring message sequence consistency, and improving fault recovery efficiency. SUMMARY

[0011] The application aims to provide a system message queue processing method, system, computer device and medium, solve the problems of message backlog and uneven load, insufficient dynamic expansion, repeated consumption and chaotic order, low fault recovery efficiency and the like in the prior art, and improve the efficiency, expandability, reliability and order guarantee capability of message queue processing in distributed systems, cloud computing platforms and Internet of Things communication.

[0012] To achieve the above-mentioned purpose, the application provides a system message queue processing method, comprising the following steps:

[0013] Step S1, receiving a message to be processed, calculating the message content and business identifier by a composite hash algorithm to generate a message fingerprint;

[0014] Step S2, dynamically collecting real-time performance indicators of system nodes, calculating node weights based on CPU utilization, memory remaining amount and network bandwidth availability, and constructing a dynamic hash ring containing virtual nodes;

[0015] Step S3, mapping the message fingerprint to the dynamic hash ring, matching the target node and executing message distribution;

[0016] Step S4, real-time monitoring of the target node health state, when the target node is abnormal, redirecting the message to the backup node through a secondary hash jump mechanism.

[0017] Preferably, in step S1, the composite hash algorithm is specifically:

[0018] The SHA-256 algorithm is used to perform basic hash calculation on the message content;

[0019] According to the modulo operation result of the business identifier, a 64-bit message fingerprint is generated.

[0020] Preferably, in step S2, the node weight calculation formula is:

[0021]

[0022] Wherein, W represents the node weight, a, b and c all represent the weight coefficient;

[0023] After the nodes are expanded into virtual nodes according to the weight ratio, they are uniformly distributed to the hash ring.

[0024] Preferably, in step S2, the generation method of the virtual node comprises:

[0025] The node weight is converted into the virtual node quantity in proportion, and a node with a weight of N generates 100xN virtual nodes;

[0026] An exponential decay function is used to smooth the mutation of the virtual node quantity;

[0027] Each virtual node is mapped to a ring space through an SHA-1 hash algorithm and is distributed on the ring in a golden section ratio, and the mapping formula is as follows:

[0028]

[0029] wherein, represents a golden section ratio, and the value is 0.618, k [1, 100 x W] represents the kth virtual node, and M represents a standard hash ring size.

[0030] Preferably, in step S3, the method for mapping the message fingerprint to the dynamic hash ring is:

[0031] Message position 1 = SHA256 (service identifier) mod M;

[0032] Wherein, mod represents a modulo operation;

[0033] When matching the target node on the ring, if the first found node is invalid, the secondary hash jump mechanism is directly entered for processing.

[0034] Preferably, in step S4, the secondary hash jump mechanism comprises:

[0035] The message fingerprint of the abnormal node is calculated through a secondary hash calculation and is mapped to the next available node in the hash ring;

[0036]

[0037] The iteration number k' starts from 1 and increases until an available node is found.

[0038] Preferably, it further comprises a message order guarantee mechanism, and specifically:

[0039] The messages of the same service identifier are forced to be allocated to a fixed section of the hash ring;

[0040] And the order consumption is realized through a timestamp sorting queue.

[0041] The application also provides a system message queue processing system, comprising:

[0042] A message distribution module is configured to receive a message to be processed and calculate the message content and service identifier through a composite hash algorithm to generate a message fingerprint;

[0043] A hash ring management module is configured to dynamically collect real-time performance indicators of system nodes, calculate node weights based on CPU utilization, memory remaining amount and network bandwidth availability, and construct a dynamic hash ring comprising virtual nodes;

[0044] A routing decision module is configured to map the message fingerprint to the dynamic hash ring, match a target node and execute message distribution.

[0045] An exception processing module is configured to monitor a node health state in real time, and when a target node is abnormal, redirect a message to a backup node through a secondary hash jump mechanism.

[0046] The application further provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps of the system message queue processing method when executing the computer program.

[0047] The application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the system message queue processing method when executed by a processor.

[0048] Therefore, the system message queue processing method, the system, the computer device and the medium have the following beneficial technical effects:

[0049] (1) Message distribution and load balancing.

[0050] Improve distribution accuracy and efficiency: generate a unique message fingerprint through a composite hash algorithm, accurately identify and distinguish different messages, and provide a basis for accurate message distribution. At the same time, map the message fingerprint to the dynamic hash ring matching target nodes to realize fast and accurate message routing, avoid the problem of partial node overload and partial node idleness caused by traditional polling or random allocation strategy, and improve the overall efficiency of the system.

[0051] Enhance dynamic scalability: dynamically collect system node performance indicators to build a dynamic hash ring. When the number of system nodes increases or decreases, only the newly added or failed virtual nodes need to be adjusted, and an incremental update strategy is used, without the need to redistribute all messages, greatly reducing the amount of data migration when the number of nodes increases or decreases, reducing system overhead, improving system availability and performance, and effectively solving the problem of high overhead in traditional methods when the number of nodes changes.

[0052] (2) Message order and reliability.

[0053] Ensure message order consistency: messages with the same business identifier are allocated to a fixed section of the hash ring, and the order queue is sorted by timestamp to realize sequential consumption, ensuring the order of message processing, avoiding the problem of repeated consumption and order confusion caused by lack of unique identifier mapping, and being of great significance for business scenarios that require strict order processing.

[0054] Improve fault recovery efficiency: Real-time monitoring of node health status, when the target node is abnormal, the message is redirected to the backup node through the secondary hash jump mechanism, avoiding the business interruption caused by the delay of message redistribution after node downtime, reducing the system recovery time and improving the business continuity. If the jump fails for three times in a row, an alarm is triggered and message delivery is suspended to facilitate timely manual intervention and troubleshooting, further ensuring the reliability of the system.

[0055] (3) Hash ring optimization.

[0056] Reduce the probability of hash collision: The virtual node is mapped to the ring space using SHA-1 hash algorithm, and is distributed on the ring according to the golden section ratio. This distribution method can make the virtual node more uniform on the hash ring, reduce the probability of hash collision, and improve the accuracy of message mapping.

[0057] Smooth node weight change: Use exponential decay function to smooth the sudden change of virtual node number, avoid the sharp fluctuation of virtual node number when the node weight changes, ensure the stability of the hash ring, and reduce the influence of node weight change on message distribution. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The figure is the architecture diagram of a system message queue processing system of the present application.

[0059] Figure 2 The figure is the flow chart of dynamic hash ring construction.

[0060] Figure 3 The figure is the timing diagram of message routing and failover. DETAILED DESCRIPTION

[0061] The technical solutions of the present application are further described below through the drawings and examples.

[0062] Unless otherwise defined, the technical terms or scientific terms used in the present application shall have the usual meaning understood by those skilled in the art.

[0063] Example 1

[0064] The present application discloses a system message queue processing method, the specific steps are as follows:

[0065] Step S1, message processing and fingerprint generation.

[0066] Firstly, the system receives the messages to be processed. To ensure the uniqueness and identifiability of each message, a composite hash algorithm is used to calculate the message content and business identification, and finally a message fingerprint is generated. Specifically, the SHA-256 algorithm is used to perform basic hash calculation on the message content, and then a 64-bit message fingerprint is generated based on the modulo operation result of the business identification.

[0067] Step S2, node weight calculation and dynamic hash ring construction.

[0068] As shown in Figure 2 , the system dynamically collects real-time performance indicators of each system node, including CPU utilization, memory remaining capacity, and network bandwidth availability. Based on these indicators, the weight of each node is calculated. The formula for calculating the node weight is:

[0069]

[0070] where W represents the node weight, a, b, and c represent the weight coefficients.

[0071] According to the calculated weight, the physical nodes are proportionally expanded into virtual nodes and evenly distributed on the hash ring to construct a dynamic hash ring. Specifically, a node with a weight of N will generate 100 x N virtual nodes. To smooth the sudden change in the number of virtual nodes, an exponential decay function is used. In addition, each virtual node is mapped to the ring space through the SHA-1 hash algorithm and is distributed on the ring according to the golden section ratio to reduce the probability of hash collision. The mapping formula is as follows:

[0072]

[0073] where represents the golden section ratio, which is 0.618, k ∈ [1, 100 x W] represents the kth virtual node, and M represents the standard hash ring size.

[0074] The update strategy of the dynamic hash ring is:

[0075] When the node performance changes, only the newly added or failed virtual nodes are adjusted, and an incremental update strategy is used.

[0076] The data migration amount is controlled within 5% of the total data, and the consistency hash algorithm is used to ensure the continuity of message routing during the migration process.

[0077] Step S3, message distribution and routing.

[0078] The system maps the generated message fingerprint to the dynamic hash ring to match the target node and perform message distribution. In this way, messages can be quickly and accurately routed to the corresponding nodes for processing.

[0079] The method of mapping the message fingerprint to the dynamic hash ring is:

[0080] Message position 1 = SHA256 (service identification) mod M;

[0081] Wherein, mod represents the modulo operation;

[0082] When matching the target node on the ring, if the first found node is invalid, the secondary hash jump mechanism is directly entered for processing.

[0083] Step S4, node health monitoring and fault recovery.

[0084] Once the target node is found to be abnormal, the system will redirect the message to the backup node through the secondary hash jump mechanism.

[0085] The message fingerprint of the abnormal node is calculated by secondary hash, and is mapped to the next available node in the hash ring;

[0086]

[0087] The iteration number k' starts from 1 and increases until an available node is found.

[0088] As Figure 3 is the message routing and fault transfer timing diagram, which is as follows:

[0089] 1. Send message: the message sender carries the message content and hash Key, such as device ID or User_ID, etc. The system uses hash algorithm (such as SHA-256) to calculate the hash of the message content and service identification, and generates the message fingerprint.

[0090] 2. Calculate the hash ring positioning node: use the message fingerprint to position on the hash ring to find the target node. The hash ring is constructed by the consistent hash algorithm, and the virtual nodes are stored on the node, which are uniformly distributed. The selection of the target node is based on the distribution of the hash ring, which ensures that the message can be quickly and accurately routed.

[0091] 3. Detect whether the message routing is abnormal: the system monitors the health status of the target node in real time. Through the heartbeat detection mechanism (sending a heartbeat packet every 2 seconds), if it is timed out for 3 times in a row, the node is marked as abnormal. The health monitoring module will also evaluate the node state according to the performance indicators such as CPU, memory and network of the node.

[0092] 4. Forward the message to the main node A: if the message routing is normal, the message will be sent to the main node A. After the main node A processes the message, it will return an ACK confirmation message to the sender, indicating that the message has been successfully received and processed.

[0093] 5. Failover: If an abnormal message routing is detected, the system triggers a failover mechanism. At this time, the node status is requested to be checked to confirm whether the node has really failed. If the node failure is confirmed, the system will redirect the message to the backup node B.

[0094] 6. Hash ring excluding node A virtual node: After confirming the failure of node A, its corresponding virtual node is excluded from the hash ring to avoid subsequent messages being routed to the failed node.

[0095] 7. Redirecting the message to the backup node B: The message is re-routed to the backup node B. After receiving the message, the backup node B will process it.

[0096] 8. Data synchronization confirmation: To ensure data consistency, the system performs data synchronization confirmation to ensure that the message is successfully processed on the backup node B. The synchronization delay is controlled within 50 ms to reduce the impact on system performance.

[0097] 9. Return new routing ACK: After the backup node B processes the message, it returns a new ACK confirmation message to the sender, indicating that the message has been successfully processed through the backup node.

[0098] 10. Complete: The message processing flow ends.

[0099] Message order guarantee mechanism.

[0100] In addition to the above steps, the invention also specially adds a message order guarantee mechanism. Messages with the same business identifier are forced to be allocated to a fixed segment of the hash ring and are sequentially consumed through a timestamp-ordered queue. This mechanism ensures the order of message processing, which is particularly important for business scenarios that require strict sequential processing.

[0101] Example Two

[0102] Figure 1 The architecture diagram of the system message queue processing system is shown. The message distribution module is responsible for receiving messages and generating message fingerprints, and then sending the messages to the routing decision module. The routing decision module matches the target node on the hash ring according to the message fingerprint, and constructs a virtual node ring while dynamically adjusting the node weight. The hash ring management module matches the target node on the hash ring according to the message fingerprint, and notifies the exception handling module when the failed node is automatically removed. After receiving the failed node information, the exception handling module will redirect the message back to the routing decision module to ensure that the message is correctly processed and sent to the target node.

[0103] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0104] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0105] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways to obtain the electronic program, and then storing it in the computer memory.

[0106] It is worth noting that the contents not elaborated in detail in the present application are all prior art and are well known to those skilled in the art.

[0107] Therefore, the application adopts the above-mentioned system message queue processing method, system, computer device and medium, solves the problems of message backlog and uneven load, insufficient dynamic expansion, repeated consumption and chaotic order, low fault recovery efficiency and the like in the prior art, and improves the efficiency, expansibility, reliability and order guarantee capability of message queue processing in a distributed system, a cloud computing platform and Internet of Things communication.

[0108] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit them, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements also cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A system message queue processing method, characterized in that, Includes the following steps: Step S1: Receive the message to be processed, and use a composite hash algorithm to calculate the message content and business identifier to generate a message fingerprint; Step S2: Dynamically collect real-time performance indicators of system nodes, calculate node weights based on CPU utilization, remaining memory, and network bandwidth availability, and construct a dynamic hash ring containing virtual nodes. Step S3: Map the message fingerprint to the dynamic hash ring, match the target node, and perform message distribution; Step S4: Monitor the health status of the target node in real time. When the target node is abnormal, redirect the message to the backup node through a two-level hash jump mechanism. In step S1, the composite hash algorithm is specifically as follows: The message content is subjected to a basic hash calculation using the SHA-256 algorithm; Generate a 64-bit message fingerprint based on the modulo operation result of the business identifier; In step S2, the formula for calculating node weights is: ； in, Indicates node weight, , , All represent weighting coefficients; After the nodes are expanded into virtual nodes according to their weight ratios, they are evenly distributed across the hash ring. In step S2, the method for generating virtual nodes includes: The node weights are converted into the number of virtual nodes proportionally, and a node with a weight of N generates 100×N virtual nodes; An exponential decay function is used to smooth out sudden changes in the number of virtual nodes; Each virtual node is mapped to the ring space using the SHA-1 hash algorithm and distributed on the ring according to the golden ratio. The mapping formula is as follows: ； in, This represents the golden ratio, with a value of 0.

618. Indicates the first One virtual node Indicates the standard hash ring size.

2. The system message queue processing method according to claim 1, characterized in that, In step S3, the method for mapping the message fingerprint to the dynamic hash ring is as follows: ； in, This represents the modulo operation; When matching a target node on the ring, if the first node found is invalid, the process directly enters the secondary hash jump mechanism.

3. The system message queue processing method according to claim 1, characterized in that, In step S4, the secondary hash jump mechanism includes: Perform a secondary hash calculation on the message fingerprint of the abnormal node and map it to the next available node in the hash ring; ； Number of iterations Start by incrementing from 1 until a usable node is found.

4. The system message queue processing method according to claim 1, characterized in that, It also includes a message order guarantee mechanism, specifically: Messages with the same service identifier are forcibly assigned to a fixed segment of the hash ring; Then, sequential consumption is achieved by sorting the queue by timestamp.

5. A system message queue processing system, characterized in that, A method for performing the system message queue processing method as described in any one of claims 1-4, comprising: The message distribution module is used to receive messages to be processed and generate message fingerprints by calculating the message content and business identifier using a composite hash algorithm. The hash ring management module is used to dynamically collect real-time performance indicators of system nodes, calculate node weights based on CPU utilization, remaining memory, and network bandwidth availability, and construct a dynamic hash ring containing virtual nodes. The routing decision module is used to map message fingerprints to a dynamic hash ring, match target nodes, and perform message distribution. The exception handling module is used to monitor the health status of nodes in real time. When the target node is abnormal, the message is redirected to the backup node through a two-level hash jump mechanism.

6. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, it implements the steps of the system message queue processing method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of the system message queue processing method according to any one of claims 1-4.

Images