Distributed message queue optimization method and device, electronic equipment and storage medium

Abstract

The invention discloses a distributed message queue optimization method and device, electronic equipment and a storage medium, and particularly relates to the technical field of distributed systems. A load prediction model is utilized to predict a load change trend in a future preset duration and calculate a dynamic load score of each node, partition distribution in a distributed message queue is adjusted according to the dynamic load scores, meanwhile, link state information is monitored, and a message transmission mode is dynamically selected according to a monitoring result, so that the message transmission efficiency is improved. The problems that in the prior art, due to the fact that a static partition strategy cannot adapt to load changes in a self-adaptive mode, the capacity for predicting the load trend is lacked, burst traffic is difficult to deal with, and a fixed message transmission mode cannot adapt to different link states can be solved. The technical effects of realizing system load balancing, improving the adaptability to burst traffic, reducing message transmission delay and optimizing the overall service quality of the system are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed systems, and in particular to a distributed message queue optimization method and device, electronic equipment and a storage medium. BACKGROUND

[0002] As a core component in cloud computing and big data processing architecture, the distributed message queue system is widely used in high-concurrency scenarios such as real-time data stream processing, micro-service communication and edge computing. In related technologies, through the coordinated work of static partitioning strategy, pull-based consumption mechanism and basic monitoring system, mainstream message middleware architectures represented by Kafka are constructed. Specifically, this technical system covers the whole process from message production, partition allocation, consumption pulling to fault recovery, including key links such as Broker node management, Consumer Group coordination, message persistence and replication. With the increasing complexity of business scenarios, the system's demand for dynamic load sensing and cross-layer collaborative scheduling is increasing, and the traditional solution has been difficult to meet the requirements of high flexibility, high availability and differentiated quality of service guarantee. However, in the existing message queue optimization method, the static partitioning strategy is directly used, and there is no adaptive adjustment according to the runtime load changes, which may cause hot partition and low resource utilization, or cause message accumulation and service interruption in burst traffic scenarios. In addition, the existing system usually makes scheduling decisions based on the current state, lacks the ability to predict future load trends, and is difficult to achieve proactive resource allocation. At the same time, there is a lack of unified scheduling mechanism between the Broker layer and the client SDK, resulting in lag in transmission mode switching and inconsistency in policy delivery, thereby affecting the overall throughput and end-to-end delay performance of the system. These technical defects are particularly prominent in complex environments such as node heterogeneity, network fluctuations and mixed deployment of multiple services, limiting the scalability and quality of service guarantee capability of the system. SUMMARY

[0003] The present application provides a distributed message queue optimization method and device, electronic equipment and a storage medium to at least solve the problem of hot partition and low resource utilization of the distributed system in related technologies.

[0004] The present application provides a distributed message queue optimization method, comprising:

[0005] Collecting the running state indicators of each node in the distributed message queue system;

[0006] Inputting the running state indicators into a load prediction model for load prediction, predicting the load change trend in a future preset time period, and calculating the dynamic load score of each node based on the prediction result;

[0007] Adjusting the partition distribution in the distributed message queue according to the dynamic load score.

[0008] monitoring link state information and dynamically selecting a message transmission mode according to a monitoring result.

[0009] Optionally, the running state indicators of each node in the distributed message queue system are collected, including:

[0010] message production rate and consumption rate are obtained at a configurable collection period;

[0011] monitoring resource usage of the node;

[0012] collecting network transmission related running state indicators.

[0013] Optionally, the resource usage includes at least one of central processing unit usage, memory usage, and disk input / output rate.

[0014] Optionally, before inputting the running state indicators into the load prediction model for load prediction, the distributed message queue optimization method further includes:

[0015] inputting historical running state indicators of multiple time periods to train the load prediction model, and the load prediction model is constructed based on a long short-term memory neural network.

[0016] Optionally, the dynamic load score of each node is calculated based on the prediction result, including:

[0017] weighted calculation of multiple running state indicators;

[0018] dynamically adjusting the weight coefficients of each running state indicator through a gradient descent algorithm;

[0019] optimizing the scoring function according to real-time system response, and generating the dynamic load score of each node based on the optimized scoring function.

[0020] Optionally, the partition distribution in the distributed message queue is adjusted according to the dynamic load score, including:

[0021] identifying partitions with load higher than a first threshold or lower than a second threshold;

[0022] adjusting the partition distribution of high priority according to the quality of service level;

[0023] performing partition distribution rebalancing to reduce data migration amount and service interruption time.

[0024] Optionally, link state information is monitored, and a message transmission mode is dynamically selected according to a monitoring result, including:

[0025] calculating a proportion of the current connection number to the maximum connection number;

[0026] measuring an average communication delay and comparing it to a preset threshold value;

[0027] performing mode switching between the push mode and the pull mode according to a comparison result.

[0028] The application further provides a distributed message queue optimization device, comprising:

[0029] a collection unit configured to collect running state indicators of nodes in a distributed message queue system;

[0030] a prediction unit configured to input the running state indicators into a load prediction model to perform load prediction, predict a load change trend in a future preset time length, and calculate dynamic load scores of the nodes based on a prediction result;

[0031] an adjustment unit configured to adjust partition distribution in the distributed message queue according to the dynamic load scores;

[0032] a selection unit configured to monitor link state information and dynamically select a message transmission mode according to a monitoring result.

[0033] Optionally, the collection unit comprises:

[0034] an acquisition module configured to acquire message production rates and consumption rates at a configurable collection period;

[0035] a monitoring module configured to monitor resource usage of the nodes;

[0036] a collection module configured to collect running state indicators related to network transmission.

[0037] Optionally, the resource usage comprises at least one of a central processing unit usage rate, a memory usage rate and a disk input / output rate.

[0038] Optionally, before the running state indicators are input into the load prediction model to perform load prediction, the distributed message queue optimization device further comprises:

[0039] a training unit configured to input historical running state indicators of multiple time periods to train the load prediction model, and the load prediction model is constructed based on a long short-term memory neural network.

[0040] Optionally, the prediction unit comprises:

[0041] a first calculation module configured to perform weighted calculation on multiple running state indicators;

[0042] a first adjustment module configured to dynamically adjust weight coefficients of the running state indicators by using a gradient descent algorithm;

[0043] The generating module is configured to optimize a scoring function according to the real-time system response condition, and generate a dynamic load score of each node based on the optimized scoring function.

[0044] Optionally, the adjusting unit comprises:

[0045] The identifying module is configured to identify a partition whose load is higher than a first threshold or lower than a second threshold.

[0046] The second adjusting module is configured to adjust the distribution of the high-priority partition according to the quality of service level.

[0047] The executing module is configured to perform partition distribution rebalancing with the goal of reducing data migration amount and service interruption time.

[0048] Optionally, the selecting unit comprises:

[0049] The second calculating module is configured to calculate a proportion of the current connection number to the maximum connection number.

[0050] The comparing module is configured to measure the average communication delay and compare it with a preset threshold.

[0051] The switching module is configured to perform mode switching between the push mode and the pull mode according to the comparison result.

[0052] The application further provides an electronic device, comprising a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the above methods.

[0053] The application further provides a computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above methods.

[0054] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of any of the above methods.

[0055] According to the application, the running state indicators of each node in the distributed message queue system can be collected in real time, the load change trend in a future preset time period is predicted by using a load prediction model, the dynamic load score of each node is calculated, the partition distribution in the distributed message queue is adjusted according to the dynamic load score, the link state information is monitored, and the message transmission mode is dynamically selected according to the monitoring result, so that the problems in the prior art, such as the inability to adapt to load changes due to the use of a static partition strategy, the lack of prediction ability for load trends and the inability of a fixed message transmission mode to adapt to different link states, can be solved, and the technical effects of achieving system load balancing, improving the adaptability to burst traffic, reducing message transmission delay and optimizing the overall service quality of the system can be achieved.

[0056] It should be understood that the matters described in this section are not intended to identify key or essential features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings are used to better understand the present application, and do not constitute a limitation on the present disclosure. Among them:

[0058] Figure 1 A flowchart of a distributed message queue optimization method provided by an embodiment of the present application is shown in the figure.

[0059] Figure 2 A structure diagram of a distributed message queue optimization device provided by an embodiment of the present application is shown in the figure.

[0060] Figure 3 A structure diagram of another distributed message queue optimization device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0061] The exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0062] In order to make those skilled in the art better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] In combination with the specific application environment architecture or specific hardware architecture on which the distributed message queue optimization method depends, the specific application environment architecture or specific hardware architecture is described here.

[0064] The embodiments of the present application provide a distributed message queue optimization method, Figure 1 A flowchart of a distributed message queue optimization method provided by an embodiment of the present application is shown in the figure.

[0065] As Figure 1 shown, the method comprises the following steps:

[0066] Step 101, collect the running state indicators of each node in the distributed message queue system.

[0067] Specifically, the "nodes in the distributed message queue system" covers key functional nodes in the system, including Broker nodes responsible for message storage and forwarding, consumer nodes responsible for message processing tasks, etc., to ensure comprehensive monitoring of the core carrier state of the system operation and avoid load monitoring blind spots caused by missing key nodes. The collected "running state indicators" specifically include multi-dimensional key parameters: message production rate (referring to the total amount of messages received by the system from the outside in unit time, directly reflecting the intensity and frequency of message input, which is an important basis for judging the load of the message generation end), message consumption rate (referring to the total amount of messages successfully processed by the consumer node in unit time, reflecting the capacity and efficiency of the message processing end), node CPU / memory / disk utilization (representing the proportion of occupied processor computing resources, memory temporary storage resources, and disk persistent storage resources of the node, directly reflecting the load degree of the node hardware resources, when the utilization rate of a certain type of resource is too high, it will directly affect the efficiency of the node in processing messages), network bandwidth (referring to the rate limit and actual usage of data transmission between the node and other nodes or clients, determining the efficiency of message transmission between nodes, insufficient bandwidth will cause an increase in message transmission delay), and message accumulation amount (referring to the total number of messages currently stored in the queue without being processed by the consumer, which is a key indicator for judging whether the production end and the consumption end are unbalanced, and a high accumulation amount will cause message processing delay and even queue overflow risk). To ensure the accuracy and comprehensiveness of load monitoring, the collection process supports data collection from multiple granularities: collecting from the Broker granularity to grasp the overall running state of a single Broker node and understand the resource load at the node level; collecting from the Partition granularity to accurately locate the load changes of a single message partition and avoid the problem of ignoring local partition overload by only monitoring the overall state of the node; collecting from the Consumer Group granularity to understand the running state of different consumer groups and grasp the load differences of the consumption end. At the same time, the data collection period supports flexible configuration according to the actual running needs of the system, and can achieve millisecond-level update frequency, which can meet the real-time needs of the state in high-concurrency scenarios, timely capture the dynamic changes of the node state, and avoid the load information lag caused by fixed collection period, ensuring that the collected running state indicators can truly reflect the current and recent running conditions of the node.

[0068] Step 102, input the running state indicators into the load prediction model for load prediction, predict the load change trend in the future preset time period, and calculate the dynamic load score of each node based on the prediction result.

[0069] Specifically, the "load prediction model" is specifically a model constructed based on a long short-term memory neural network (LSTM), which has the advantage of processing time series data and can effectively capture the law of the change of the running state index over time. Before use, the model parameters will be trained using historical time series data of the system (including historical data corresponding to the running state index such as message production / consumption rate, node resource utilization rate, network delay, etc. in the past multiple time periods), to ensure that the model can accurately adapt to the running characteristics of the current distributed message queue system. The "future preset time length" can be flexibly configured according to the actual needs of the system, for example, set to 30 seconds, 1 minute, etc. in the future. The basis for configuration mainly combines the timeliness requirements of the business scenario on the load response, such as setting the preset time length to be short in a high-concurrency business scenario, so as to capture the load fluctuation faster, and appropriately prolonging it in a normal scenario to balance the prediction efficiency and resource consumption.

[0070] In the load prediction process, the model will use the running state indicators collected in step 101 (such as production rate, consumption rate, node CPU / memory utilization, network delay, etc. in the past N time periods) as input features, and through the operation of the trained network structure, output the specific prediction results in the future preset time length, including the future message production rate change, consumption capability fluctuation, and the time and value of the load peak point. These prediction results can expose the possible problems of high load or imbalance in the future in advance. Based on the above prediction results, when calculating the dynamic load score of each node, a specific load scoring function will be used: St = a Rprod + b Rcons + g Unode + d Lnet. Where St represents the comprehensive load score of a node at time t, Rprod is the predicted message production rate, Rcons is the predicted message consumption rate, Unode is the predicted node resource utilization rate (taking the weighted average of CPU, memory, and disk utilization), and Lnet is the predicted network delay or bandwidth limitation index; a, b, g, d are weight coefficients of each index, and satisfy a + b + g + d = 1. These weight coefficients will be dynamically optimized through the gradient descent algorithm, and the optimization goal is to minimize the error between the predicted load score and the actual system response delay, to ensure that the score can truly reflect the load level of the node.

[0071] Step 103, adjust the partition distribution in the distributed message queue according to the dynamic load score.

[0072] Specifically, the "dynamic load score" is a quantitative indicator reflecting the real-time and predicted load level of each node, calculated by a specific scoring function in step 102. This score integrates multiple factors such as message production / consumption rate, node resource utilization, network latency, etc., accurately reflecting the load differences between different nodes. The "partition distribution in the distributed message queue" covers both partition replica distribution and partition leader distribution. The former refers to the backup storage of the same message partition on different nodes (to ensure data reliability and service availability), and the latter refers to the node where the main partition responsible for handling message read / write requests is located (directly affecting message processing efficiency). In the specific adjustment process, priority sorting will be performed based on the dynamic load score of each node. If the dynamic load score of a node is significantly higher than the system preset threshold (e.g., due to a sudden increase in predicted message production rate, node CPU / memory utilization approaching the upper limit, resulting in an excessive score), it is determined that the node is in a high load state, and the partitions it carries need to be migrated. At this time, nodes with lower dynamic load scores and sufficient resources (e.g., CPU utilization below 50%, sufficient memory space) will be selected as target nodes, and some non-core partition replicas or leader partitions on the high-load node will be migrated to the target node. During the migration process, the stability of high-priority topics (such as business core transaction topics) will be prioritized, and only non-core or heavily loaded partitions will be adjusted. At the same time, the migration frequency and data volume will be controlled to avoid service interruptions or excessive network bandwidth occupation caused by frequent migration. In addition, after adjustment, the new dynamic load scores of each node will be monitored in real-time to ensure that the adjustment effect meets expectations. If it is found that the load score of a node after adjustment has not decreased to a reasonable range, the partition migration scheme will be further optimized until the load scores of all nodes tend to be balanced.

[0073] Step 104, monitor link state information and dynamically select message transmission mode according to monitoring results.

[0074] Specifically, step 104 "monitoring link state information and dynamically selecting message transmission mode according to the monitoring result" is the key link to adapt the distributed message queue system to link changes and optimize message transmission efficiency. The "link state information" specifically covers three core indicators between the client and the Broker: the current connection number (Ccurrent, which directly reflects the connection pressure of the link, that is, the total number of effective network connections established between the current client and the Broker cluster), the maximum number of connections allowed by the system (Cmax, which is the upper limit of connection bearing determined by the system hardware resources and network configuration, used to judge whether the current connection pressure is close to the threshold), and the average communication delay (Davg, which refers to the average time of multiple times of sending requests from the client to the Broker response, directly reflecting the transmission efficiency and stability of the link). The monitoring process uses a real-time collection mechanism, and the collection period is coordinated with the running state indicator collection period of step 101 to ensure that the link state information can reflect the current link changes in a timely manner, avoiding transmission mode selection deviation caused by information lag.

[0075] The "message transmission mode" in "dynamically selecting message transmission mode" includes push mode and pull mode: push mode is initiated by the Broker to send messages to the client, suitable for low-latency demand scenarios (such as real-time data synchronization business), which can reduce the client waiting time; pull mode is initiated by the client to request Broker to obtain messages, suitable for high-throughput demand scenarios (such as batch data processing business), which can avoid resource overload caused by Broker's active push. The selection logic is based on the link state matrix Mpush: when the proportion of the current connection number to the maximum connection number (Ccurrent / Cmax) exceeds the preset threshold θ (such as 0.7, representing high link connection pressure), and the average communication delay Davg is lower than the preset delay threshold τ (such as 50ms, representing good link transmission efficiency), Mpush = 1, enabling push mode to improve message consumption efficiency; if the above conditions are not met (such as low connection pressure but high delay, or high connection pressure and high delay), Mpush = 0, using pull mode to ensure system stability. The mode switching process is smooth and does not interrupt the current message transmission, ensuring business continuity.

[0076] By the present application, since the running state indicators of each node in the distributed message queue system can be collected in real time, the load prediction model is used to predict the load change trend in the future preset time and calculate the dynamic load score of each node, the partition distribution in the distributed message queue is adjusted according to the dynamic load score, the link state information is monitored and the message transmission mode is dynamically selected according to the monitoring result, therefore, the problems that the static partition strategy cannot adapt to the load change, the prediction ability of the load trend is poor and the fixed message transmission mode cannot adapt to different link states in the prior art can be solved, and the technical effects of realizing system load balancing, improving the adaptability to burst traffic, reducing the message transmission delay and optimizing the overall service quality of the system are achieved.

[0077] In the embodiments of the present disclosure, for the operation of "collecting running state indicators of each node in the distributed message queue system", the specific implementation manners are various, and for clear illustration, the following specific introduction part introduces some exemplary embodiments: obtaining the message production rate and consumption rate with a configurable collection period; monitoring the resource usage of the node; collecting the running state indicators related to network transmission.

[0078] Specifically, when acquiring the message production rate and the consumption rate with a configurable collection period, the configurable collection period can be flexibly set according to a system service scenario, for example, the configurable collection period is set to be millisecond level in a high-concurrency real-time transaction scenario, and the configurable collection period is set to be second level in a common data synchronization scenario, so as to ensure that the load fluctuation can be captured in time and the system resources are not occupied too much by frequent collection. The message production rate refers to the total amount of messages received by the Broker node from the external service end in a unit of time, and the message consumption rate refers to the total amount of messages successfully processed and confirmed by the Consumer Group node in a unit of time. When collection is performed, the two rates are bound to the corresponding nodes, so that the data attribution is clear and the message processing capabilities of different nodes are not confused. When the resource usage of the node is monitored, the node mainly focuses on the Broker node that undertakes the core function, and the monitoring content includes the CPU utilization rate (reflecting the proportion of occupied computing resources of the node, directly affecting the message processing efficiency), the memory utilization rate (reflecting the resource consumption of the node for temporarily storing message data, and insufficient memory will cause the message cache to be limited), and the disk utilization rate (reflecting the resource usage state of the node for persistently storing messages, and full load of the disk will cause the message to be unable to be written). In the collection process, the real-time usage data of each resource is read in real time, and the overall resource utilization rate of the node is calculated according to the preset weight, so as to provide a key basis at the hardware level for subsequent load scoring. When the running state indicators related to network transmission are collected, the network bandwidth (including the actual transmission rate and the bandwidth upper limit, reflecting the data transmission capacity of the link) between nodes and the network delay (referring to the average time consumption of the message from the client to the Broker, reflecting the stability of the link transmission) between the client and the Broker are collected. These indicators are extracted from the node communication link in real time, so as to ensure that the support capacity of the link for message transmission can be truly reflected.

[0079] Further, in the embodiments of the present disclosure, the resource usage includes at least one of a central processing unit usage, a memory usage, and a disk input / output rate.

[0080] Specifically, for the collection of node resource usage, the core monitoring objects include at least one of central processing unit (CPU) usage, memory usage, and disk input / output (I / O) rate, which directly reflect the carrying capacity of node hardware resources to message processing tasks. Among them, "CPU usage" refers to the proportion of time that the node CPU is occupied by message receiving, forwarding, storage and other business tasks. If the CPU usage is too high (such as more than 80%), it will cause message processing thread scheduling delay, directly affecting the message read-write efficiency; "memory usage" is the proportion of space occupied by node memory for caching messages to be processed, running system processes and application programs. When the memory is insufficient, it will cause the message cache hit rate to decrease, forcing the system to frequently read data from the disk, increasing the message processing delay; "disk input / output rate" is the number of bytes per second that the disk completes message data writing (such as persistent storage of messages) and reading (such as loading historical messages). A low disk I / O rate will limit the message persistence speed, especially in high-throughput scenarios, which is prone to cause message accumulation. In the specific collection process, the monitoring Agent deployed on each node will read the system underlying data in real time: for CPU usage, the monitoring Agent obtains the CPU idle time and total running time by calling the operating system kernel interface (such as the / proc / stat file of the Linux system), and calculates the real-time usage; for memory usage, the difference between the total capacity and the free capacity of the memory is read, combined with the cache and buffer space occupation, to obtain the proportion of memory occupied by actual business; for disk I / O rate, the I / O statistical tool (such as iostat) is used to collect disk read-write throughput data. The collection period of the resource usage and the collection period of the running state indicators (such as message production / consumption rate) are kept in coordination (such as both set to once per second), ensuring that the data time dimensions of resource usage and other indicators are consistent, while supporting flexible selection of monitoring indicator combinations according to business needs - for example, in a lightweight message processing scenario, only CPU and memory usage can be monitored; in a high-persistence demand scenario, the disk I / O rate needs to be monitored, achieving precise adaptation of resource monitoring.

[0081] Further, before inputting the running state indicators into the load prediction model for load prediction, the distributed message queue optimization method further includes other specific implementation steps in addition to the foregoing. To clearly present these components, the following detailed description of the related specific embodiments is provided: inputting historical running state indicators of multiple time periods, training the load prediction model, and constructing the load prediction model based on a long short-term memory neural network.

[0082] Specifically, before inputting the running state indicators into the load prediction model for load prediction, the training step for the load prediction model in the distributed message queue optimization method needs to be based on historical running state indicators of multiple time periods, and the load prediction model is constructed based on a long short-term memory neural network (LSTM). Among them, the historical running state indicators of multiple time periods need to cover the indicator data under different running scenarios of the distributed message queue system, specifically including the message production rate, message consumption rate, node CPU usage, memory usage, disk I / O rate, network delay and other indicators collected at fixed time intervals (such as every minute or every 5 minutes) in the past several days to several months, and need to include indicator data of business peak period (such as e-commerce platform promotion period), trough period (such as non-active period in the early morning) and sudden traffic period (such as sudden increase of message volume caused by temporary activities), to ensure the comprehensiveness and representativeness of the data sample; at the same time, the collected historical indicator data needs to be preprocessed, including eliminating abnormal values (such as instantaneous soaring CPU usage) caused by node failure and network interruption, and eliminating the influence of different indicator magnitudes on model training through normalization processing (such as mapping the indicator value to the 0-1 interval) to provide high-quality input data for the model.

[0083] The load prediction model constructed based on the long short-term memory neural network (LSTM) is suitable for the load prediction scenario because LSTM has the advantage of processing time series data - it can capture the long-term dependence of historical indicator data over time (such as periodic fluctuations in load at fixed time periods each day) through the gating mechanism (input gate, forget gate, output gate). In the model training process, the preprocessed historical running state indicators are divided into a training set (70%-80% of the proportion) and a validation set (20%-30% of the proportion) in chronological order, with the indicator sequence of the past N time periods as the model input features and the load trend (such as message production rate change, node resource utilization peak) of the future T time periods as the output label. Through the back propagation algorithm, the model parameters (such as the number of hidden layer neurons, learning rate) are iteratively adjusted, and the error (such as mean square error) between the prediction result and the actual load of the validation set is used as the optimization target until the model prediction accuracy reaches the preset threshold (such as error less than 5%). In addition, the model also needs to support regular retraining, that is, every fixed period (such as every 24 hours), the latest collected historical indicator data is integrated to update the model parameters, ensuring that the model can adapt to the long-term evolution of the system running state (such as performance changes caused by node hardware aging, load characteristics changes caused by business mode adjustment).

[0084] In the embodiments of the present disclosure, for the operation of "calculating the dynamic load score of each node based on the prediction result", the specific implementation manners are various. For clear illustration, the following specific embodiments are introduced: weighted calculation of multiple operation state indicators; dynamic adjustment of the weight coefficients of each operation state indicator through gradient descent algorithm; optimization of the scoring function according to the real-time system response, and generation of the dynamic load score of each node based on the optimized scoring function.

[0085] Specifically, the multi-dimensional indicator weighting, dynamic weight optimization and real-time adaptation of the scoring function are carried out. First, when the multiple operation state indicators are weighted, the core formula is the comprehensive load scoring function St = a Rprod + b Rcons + g Unode + d Lnet, where each parameter is from the output result of the load prediction model: Rprod is the predicted message production rate in the future preset time period, Rcons is the predicted message consumption rate, Unode is the predicted node resource utilization rate (the weighted average of CPU, memory and disk utilization rates), and Lnet is the predicted network delay or bandwidth limitation indicator; a, b, g and d are the weight coefficients of the corresponding indicators, and a + b + g + d = 1 is satisfied. Through this weighting method, the multi-dimensional prediction indicators can be integrated into a single quantitative load score, avoiding the one-sidedness of load evaluation caused by single indicator judgment.

[0086] Secondly, when the weight coefficients of each operation state indicator are dynamically adjusted through the gradient descent algorithm, the core target is to minimize the error between the predicted load score and the actual system response delay. Specifically, the "predicted load score" and "actual system response delay (such as the total time consumption of messages from production to consumption)" in the historical monitoring data are used as supervision samples, and the gradient descent algorithm is used to iteratively update the values of a, b, g and d. For example, when the influence of node resource utilization rate (Unode) on actual response delay significantly increases in a certain period of time, the algorithm will automatically increase the weight of g. Conversely, if the influence of network delay (Lnet) on response delay decreases, the weight of d will be reduced, so that the weight coefficients are always matched with the actual system operation characteristics. Finally, when the scoring function is optimized according to the real-time system response and the dynamic load score is generated, the response data (such as real-time message processing delay and node resource occupation fluctuation) in the actual system operation need to be continuously collected and compared with the load score calculated by the current scoring function. If it is found that the actual response delay of the high-load node output by the scoring function does not increase, or the delay of the low-load node is abnormal, it is determined that there is a deviation between the scoring function and the actual situation. At this time, the parameters of the scoring function will be adjusted based on the deviation data (such as optimizing the weighted calculation method of Unode and supplementing new impact factor weight constraints), and after optimization, the dynamic load score of each node is calculated by using the updated scoring function, so that the scoring result can truly reflect the node load state.

[0087] In the embodiments of the present disclosure, for the operation of "adjusting partition distribution in a distributed message queue according to dynamic load scoring", the specific implementation manners are various, and for clear illustration, the following specific embodiments are introduced: identifying partitions with load higher than a first threshold or lower than a second threshold; adjusting high-priority partition distribution according to quality of service (QoS) level; and performing partition distribution rebalancing with the goal of reducing data migration amount and service interruption time.

[0088] Specifically, the threshold identification, QoS priority adaptation, and migration optimization are developed. First, when identifying partitions with load higher than a first threshold or lower than a second threshold, the "dynamic load score" is a quantitative value calculated based on St=α·Rprod+β·Rcons+γ·Unode+δ·Lnet, the first threshold (e.g. 80 under a full score of 100) is set according to the upper limit of node resource bearing, and when the score of the node where the partition is located exceeds this value, it indicates that the node is overloaded, and the partition on this node needs to be migrated; the second threshold (e.g. 30) corresponds to the idle state of the node resources, and the node with a score lower than this value can be used as a target node for partition migration. In the identification process, the partition's own load (e.g. message accumulation amount, read-write frequency) is associated to avoid local partition misjudgment caused by relying only on node scoring, and to accurately mark "partition to be migrated" and "target receiving node".

[0089] Secondly, when adjusting high-priority partition distribution according to the quality of service (QoS) level, the QoS level is divided according to business importance (high-priority corresponds to core transaction Topic, and low-priority corresponds to log Topic). The stability of high-priority partitions is prioritized during adjustment: if the score of the node where the high-priority partition is located approaches the first threshold, only the low-priority partitions on this node are migrated, and the high-priority partition is migrated to a node with a lower score and more stable resources, and before migration, the partition replica is synchronized to the target node, and after the data is consistent, the Leader is switched, avoiding interruption of core business.

[0090] Finally, when performing rebalancing with the goal of reducing data migration amount and service interruption time, the data migration amount needs to be reduced by prioritizing partitions with less replica amount (e.g. only one replica) or high data overlap with the target node, reducing network transmission pressure; the service interruption time is reduced by using the "start first and stop later" strategy: first start a temporary replica of the partition on the target node and synchronize the data, switch the consumption end flow after the replica is ready, and then offline the original partition, while controlling the rebalancing frequency (e.g. at most once per hour) to avoid frequent operations affecting system stability.

[0091] In the embodiments of the present disclosure, for the operation of "monitoring link state information and dynamically selecting message transmission mode according to the monitoring result", the specific implementation manners are various. For clear illustration, the following specific embodiments are introduced: calculating the proportion of the current connection number to the maximum connection number; measuring the average communication delay and comparing it with a preset threshold; performing mode switching between the push mode and the pull mode according to the comparison result.

[0092] Specifically, the link state core indicators and mode switching logic need to be combined to gradually land. First, when calculating the proportion of the current connection number to the maximum connection number, "the current connection number (Ccurrent)" refers to the total number of effective network connections (including message read-write connections, heartbeat connections, etc.) established between the client and the Broker cluster in the distributed message queue system, which needs to be counted by the connection management module of the Broker end in real time to ensure that no invalid or disconnected connections are counted. "The maximum connection number (Cmax)" is the upper limit of the connection load configured by the system according to hardware resources (such as the number of CPU cores of the Broker node, network bandwidth) and business demand, to avoid link congestion caused by excessive connection number. The proportion calculation method is Ccurrent / Cmax, and it needs to be compared with a preset connection threshold proportion θ (such as 0.7). If the proportion exceeds θ, it means that the link is in a high connection pressure state, and other indicators need to be combined to determine whether to switch the transmission mode. When measuring the average communication delay and comparing it with a preset threshold, "the average communication delay (Davg)" refers to the average time consumed by the client to send a request (such as a message production request or a consumption request) to the Broker and receive a response. The measurement process needs to send low-load test data packets multiple times within a unit of time (such as 10 seconds), and the average time consumed is calculated after excluding abnormal values caused by single network fluctuation to ensure data accuracy. "The preset threshold τ" is a link quality judgment standard configured according to the business delay demand (such as 50 ms). The measured Davg is compared with τ. If Davg is lower than τ, it means that the link transmission efficiency is good and has low delay transmission conditions. Otherwise, it is determined that the link quality is poor and transmission stability needs to be prioritized. When performing mode switching between the push mode and the pull mode according to the comparison result, the judgment logic of the link state matrix Mpush needs to be followed: when "the current connection number proportion exceeds θ" and "Davg is lower than τ", it is determined that the link is in a "high connection pressure but high transmission efficiency" state, and the push mode is switched (the Broker actively sends messages to the client, which is suitable for low delay business scenarios). If neither of the above conditions is met (such as the connection proportion does not exceed θ but Davg is higher than τ, or the connection proportion exceeds θ but Davg is higher than τ), the pull mode is switched (the client actively requests messages from the Broker, which is suitable for high throughput and system stability scenarios). The mode switching process adopts a "smooth transition" mechanism, which first establishes a connection channel for the new transmission mode in the background, and then disconnects the old channel after the data transmission is normal, to avoid message loss or service interruption.

[0093] It should be noted that the embodiments of the present disclosure can include a plurality of steps, which are numbered for the convenience of description, but these numbers are not a limitation on the execution time slot and execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0094] Corresponding to the above-mentioned distributed message queue optimization method, the present disclosure also proposes a distributed message queue optimization device. Since the device embodiments of the present disclosure correspond to the above-mentioned method embodiments, for the details not disclosed in the device embodiments, please refer to the above-mentioned method embodiments, which will not be described in detail in the present disclosure.

[0095] Figure 2 The structure diagram of a distributed message queue optimization device provided by the embodiments of the present disclosure is shown in Figure 2 , which includes:

[0096] The collection unit 21 is configured to collect the running state indicators of each node in the distributed message queue system.

[0097] The prediction unit 22 is configured to input the running state indicators into a load prediction model to perform load prediction, predict the load change trend in a future preset time length, and calculate the dynamic load score of each node based on the prediction result.

[0098] The adjustment unit 23 is configured to adjust the partition distribution in the distributed message queue according to the dynamic load score.

[0099] The selection unit 24 is configured to monitor the link state information and dynamically select the message transmission mode according to the monitoring result.

[0100] Through the present application, since the running state indicators of each node in the distributed message queue system can be collected in real time, the load change trend in a future preset time length is predicted by using a load prediction model and the dynamic load score of each node is calculated, the partition distribution in the distributed message queue is adjusted according to the dynamic load score, and the link state information is monitored and the message transmission mode is dynamically selected according to the monitoring result, therefore, the problems that the static partition strategy cannot adapt to load changes, the lack of prediction ability of load trend makes it difficult to deal with burst traffic, and the fixed message transmission mode cannot adapt to different link states in the prior art can be solved, and the technical effects of achieving system load balancing, improving the adaptability to burst traffic, reducing message transmission delay and optimizing the overall service quality of the system are achieved.

[0101] Further, in a possible implementation manner of the present embodiment, as shown in Figure 3 , the collection unit 21 includes:

[0102] The acquisition module 211 is configured to acquire the message production rate and the consumption rate at a configurable acquisition period.

[0103] The monitoring module 212 is configured to monitor resource usage of the node.

[0104] The collection module 213 is configured to collect network transmission related running state indicators.

[0105] Further, in a possible implementation manner of the embodiment, the resource usage includes at least one of a central processing unit usage rate, a memory usage rate, and a disk input / output rate.

[0106] Further, in a possible implementation manner of the embodiment, as shown in Figure 3 before the running state indicators are input into the load prediction model for load prediction, the distributed message queue optimization apparatus further includes:

[0107] The training unit 25 is configured to input historical running state indicators of a plurality of time periods, and train the load prediction model, the load prediction model being constructed based on a long short-term memory neural network.

[0108] Further, in a possible implementation manner of the embodiment, as shown in Figure 3 the prediction unit 22 includes:

[0109] The first calculation module 221 is configured to perform weighted calculation on the plurality of running state indicators.

[0110] The first adjustment module 222 is configured to dynamically adjust the weight coefficients of the running state indicators by a gradient descent algorithm.

[0111] The generation module 223 is configured to optimize a scoring function according to real-time system response, and generate dynamic load scores of the nodes based on the optimized scoring function.

[0112] Further, in a possible implementation manner of the embodiment, as shown in Figure 3 the adjustment unit 23 includes:

[0113] The identification module 231 is configured to identify a partition whose load is higher than a first threshold or lower than a second threshold.

[0114] The second adjustment module 232 is configured to adjust distribution of the high-priority partition according to a quality of service level.

[0115] The execution module 233 is configured to perform partition distribution rebalancing with a target of reducing data migration amount and service interruption time.

[0116] Further, in a possible implementation manner of the embodiment, as shown in Figure 3 the selection unit 24 includes:

[0117] The second calculation module 241 is configured to calculate a proportion of the current connection number to the maximum connection number.

[0118] The comparison module 242 is configured to measure the average communication delay and compare it with a preset threshold.

[0119] The switching module 243 is configured to perform mode switching between the push mode and the pull mode according to the comparison result.

[0120] It should be noted that the foregoing description of the method embodiments is also applicable to the device of the present embodiment, and the principle is the same, which will not be limited in the present embodiment.

[0121] The description of the features in the embodiment corresponding to the distributed message queue optimization device can be referred to the related description of the embodiment corresponding to the distributed message queue optimization method, which will not be repeated here.

[0122] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above distributed message queue optimization method embodiments.

[0123] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above distributed message queue optimization method embodiments when running.

[0124] In an exemplary embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0125] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above distributed message queue optimization method embodiments.

[0126] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above distributed message queue optimization method embodiments.

[0127] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of embodiments of the present application and that various modifications can be made thereto without departing from the scope of the present application. Accordingly, the appended claims are intended to embrace all such alterations, modifications, and variations of the embodiments described herein that are within the scope of this application, including all the preferred embodiments.

[0128] The above provides a kind of distributed message queue optimization method and device, electronic equipment and storage medium provided in the present application in detail.The principle and implementation of the present application are described in the specific examples in this paper, the above example is only used to help understanding the method of the present application and its core idea.It should be pointed out that, for the ordinary skilled in the art, without departing from the principle of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A distributed message queue optimization method, characterized in that, include: Collect operational status metrics for each node in the distributed message queue system; The operating status indicators are input into the load prediction model to predict the load change trend within a preset time period, and the dynamic load score of each node is calculated based on the prediction results. The partition distribution in the distributed message queue is adjusted based on the dynamic load score; Monitor link status information and dynamically select message transmission mode based on monitoring results.

2. The distributed message queue optimization method according to claim 1, characterized in that, The collected operational status indicators of each node in the distributed message queue system include: Message production and consumption rates are obtained at configurable collection intervals; Monitor the resource usage of the nodes; Collect operational status indicators related to network transmission.

3. The distributed message queue optimization method according to claim 2, characterized in that, The resource usage includes at least one of CPU utilization, memory utilization, and disk I / O rate.

4. The distributed message queue optimization method according to claim 1, characterized in that, Before inputting the operational status indicators into the load forecasting model for load forecasting, the method further includes: Input historical operating status indicators for multiple time periods to train the load prediction model, which is built based on a long short-term memory neural network.

5. The distributed message queue optimization method according to claim 1, characterized in that, The calculation of the dynamic load score for each node based on the prediction results includes: Weighted calculation of multiple operational status indicators; The weight coefficients of various operational status indicators are dynamically adjusted using the gradient descent algorithm. The scoring function is optimized based on the real-time system response, and dynamic load scores for each node are generated based on the optimized scoring function.

6. The distributed message queue optimization method according to claim 1, characterized in that, The step of adjusting the partition distribution in the distributed message queue based on the dynamic load score includes: Identify partitions with loads higher than a first threshold or lower than a second threshold; Adjust the distribution of high-priority partitions based on service quality levels; The goal is to perform partitioned distribution rebalancing to reduce data migration and service downtime.

7. The distributed message queue optimization method according to claim 1, characterized in that, The monitoring link status information, and the dynamic selection of message transmission mode based on the monitoring results, include: Calculate the ratio of the current number of connections to the maximum number of connections; Measure the average communication latency and compare it with a preset threshold; Based on the comparison results, perform a mode switch between push mode and pull mode.

8. A distributed message queue optimization device, characterized in that, include: The data collection unit is used to collect the running status indicators of each node in the distributed message queue system. The prediction unit is used to input the operating status indicators into the load prediction model to predict the load change trend within a preset time period, and calculate the dynamic load score of each node based on the prediction results. An adjustment unit is used to adjust the partition distribution in the distributed message queue according to the dynamic load score; The selection unit is used to monitor link status information and dynamically select the message transmission mode based on the monitoring results.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed, enables the at least one processor to perform the distributed message queue optimization method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the distributed message queue optimization method according to any one of claims 1-7.

Images