Message consistency guarantee method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of computers, and provides a message consistency guarantee method and device, equipment, a storage medium and a product. The method comprises the following steps: acquiring message state information of each stage and real-time performance information of the distributed message queue system; according to the message state information and the real-time performance information of each stage, based on a learning and decision engine of the adaptive dynamic learning system, generating a message configuration strategy of each stage, and based on the message configuration strategy of each stage, performing message transmission; wherein the message configuration strategy aims at minimizing the risk of message loss and the risk of repeated consumption of messages. Through the mode, message inconsistency between a data producer and a data consumer caused by message loss and repeated consumption of messages can be effectively avoided, the effectiveness of the message consistency guarantee method is improved, manual intervention is reduced, the communication process is optimized, and the efficiency of the message consistency guarantee method is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a message consistency assurance method, device, equipment, storage medium and product. Background Art

[0002] In a distributed system, how to ensure message consistency is a core issue. Existing distributed systems have adopted a variety of strategies to ensure message consistency, such as using message queues, distributed locks, and consistency algorithms designed specifically for distributed environments.

[0003] The message queue uses an asynchronous communication mechanism as a buffer layer for message transmission between data producers and data consumers: when a data producer generates a message, the message will first be passed to the message queue; the message queue will then pass the message to the data consumer, or the data consumer will pull the message from the message queue, instead of directly passing the message generated by the data producer to the data consumer. This mechanism ensures that even if the data consumer node is temporarily unavailable, the message generated by the data producer node will not be lost, avoiding inconsistent messages between the data producer and the data consumer due to message loss. The message queue can ensure the consistency and reliability of messages by supporting features such as persistent storage, sequential processing, and transactional delivery.

[0004] Distributed locks prevent conflicts and data inconsistencies by coordinating access to shared resources among multiple nodes. They maintain global consistency of distributed system messages by ensuring that only one node can perform modifications to shared resources at any given time.

[0005] Distributed consensus algorithms, such as the Paxos consensus algorithm and the Raft consensus algorithm, can ensure consistent agreement on decision results within the cluster in complex situations such as network partitions and node failures. These algorithms use a series of complex voting and protocol mechanisms to ensure that even if some nodes cannot communicate or fail, the distributed system can still reach a consistent decision and ensure message consistency.

[0006] However, there are still some significant problems with the existing technology: the asynchronous communication mechanism of the message queue may cause data processing delays, which is not suitable for scenarios with high real-time requirements, and it is difficult to avoid the problems of message loss, slow message processing, and high manual maintenance costs caused by uncertainties in each message transmission link; although distributed locks can prevent data conflicts, they increase the complexity of the system; and when the Paxos consensus algorithm and the Raft consensus algorithm ensure message consistency, the computing and communication costs increase significantly as the cluster scale expands, and the fault tolerance is limited, making it difficult to ensure the normal operation of distributed services. These problems limit the effectiveness and efficiency of existing message consistency guarantee methods. Summary of the invention

[0007] The embodiments of the present application provide a message consistency assurance method, apparatus, device, storage medium and product to solve the technical problems of poor effectiveness and efficiency of existing message consistency assurance methods.

[0008] In a first aspect, an embodiment of the present application provides a method for ensuring message consistency, including: obtaining message status information of each stage and real-time performance information of a distributed message queue system; generating message configuration strategies for each stage based on the message status information and real-time performance information of each stage, based on a learning and decision engine of an adaptive dynamic learning system, and performing message delivery based on the message configuration strategies for each stage; wherein the message configuration strategy aims to minimize the risk of message loss and the risk of repeated message consumption.

[0009] In one embodiment, message status information and real-time performance information are acquired by a data acquisition module of an adaptive dynamic learning system, the message status information includes status information of a message sending stage, and the message configuration strategy includes data producer configuration information; according to the message status information and real-time performance information of each stage, based on the learning and decision engine of the adaptive dynamic learning system, a message configuration strategy of each stage is generated, and based on the message configuration strategy of each stage, message delivery is performed, including: in the message sending stage, the status information and real-time performance information of the message sending stage are input into the learning and decision engine of the adaptive dynamic learning system, and the data producer configuration information output by the learning and decision engine is obtained; based on the data producer configuration information, the data producer is controlled to deliver the to-be-processed message to the message queue cluster.

[0010] In one embodiment, the message status information includes status information of the message queue processing stage, and the message configuration strategy includes Raft algorithm parameters; according to the message status information and real-time performance information of each stage, based on the learning and decision engine of the adaptive dynamic learning system, the message configuration strategy of each stage is generated, and based on the message configuration strategy of each stage, message transmission is performed, including: in the message queue processing stage, after the message queue cluster receives the message to be processed, the status information and real-time performance information of the message queue processing stage are input into the learning and decision engine of the adaptive dynamic learning system, and the Raft algorithm parameters output by the learning and decision engine are obtained; according to the Raft algorithm parameters, based on the Raft consensus algorithm, the target processing node is determined from the message queue cluster; the message to be processed is allocated to the target processing node, and the message is waited for message transmission.

[0011] In one embodiment, according to Raft algorithm parameters, based on Raft consensus algorithm, a target processing node is determined from a message queue cluster, including: according to the Raft algorithm parameters, determining the fault tolerance threshold and the number of consensus participating nodes of the Raft consensus algorithm; based on the number of consensus participating nodes, grouping the message queue cluster to obtain multiple cluster groups; wherein each cluster group includes the same number of consensus participating nodes, and the system fault tolerance probability of the message queue cluster is less than the fault tolerance threshold; determining the leader node of each cluster group; a leader node is elected by all consensus participating nodes in a cluster group through a consistency algorithm; and electing a target processing node from all leader nodes.

[0012] In one embodiment, the message status information includes status information of the message consumption stage, and the message configuration strategy includes data consumer configuration information; according to the message status information and real-time performance information of each stage, based on the learning and decision engine of the adaptive dynamic learning system, the message configuration strategy of each stage is generated, and based on the message configuration strategy of each stage, message transmission is performed, including: in the message consumption stage, the status information and real-time performance information of the message consumption stage are input into the learning and decision engine of the adaptive dynamic learning system, and the data consumer configuration information output by the learning and decision engine is obtained; based on the data consumer configuration information, the data consumer is controlled to obtain the to-be-processed message from the target processing node.

[0013] In one embodiment, the adaptive dynamic learning system includes a storage and historical data analysis module, which is used to store historical data. The learning and decision engine includes an LSTM model and a DQN model; the LSTM model is used to predict the message status information and real-time performance information of each stage based on historical data to obtain the configuration parameter prediction value; the DQN model is used to dynamically adjust the configuration parameter prediction value to obtain the message configuration strategy of each stage.

[0014] In the second aspect, an embodiment of the present application provides a message consistency assurance device, including: an acquisition module, used to obtain message status information of each stage and real-time performance information of a distributed message queue system; a consistency assurance module, used to generate message configuration strategies for each stage based on the message status information and real-time performance information of each stage, based on the learning and decision engine of an adaptive dynamic learning system, and perform message delivery based on the message configuration strategies for each stage; wherein the message configuration strategy aims to minimize the risk of message loss and the risk of repeated message consumption.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the message consistency assurance methods described above.

[0016] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the message consistency assurance methods described above.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned message consistency assurance methods when executed by a processor.

[0018] The message consistency assurance method, apparatus, equipment, storage medium and product provided in the embodiments of the present application generate a message configuration strategy with the goal of minimizing the risk of message loss and the risk of repeated consumption of messages based on the message status information of each stage and the real-time performance information of the distributed message queue system by using the learning and decision engine of the adaptive dynamic learning system, and then transmit the message based on the message configuration strategy, which can effectively avoid the message inconsistency between data producers and data consumers due to message loss and repeated consumption, and improve the effectiveness of the message consistency assurance method; at the same time, the adaptive dynamic learning system is introduced to intervene in each stage of the message transmission process, and the message configuration strategy is automatically generated, which can reduce manual intervention, optimize the communication process, and improve the efficiency of the message consistency assurance method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is one of the flow charts of the message consistency assurance method provided in the embodiment of the present application.

[0021] Figure 2 This is the second flow chart of the message consistency assurance method provided in the embodiment of the present application.

[0022] Figure 3 It is a schematic diagram of the structure of the adaptive dynamic learning system provided in the embodiment of the present application.

[0023] Figure 4 It is a schematic diagram of the processing flow of the adaptive dynamic learning system provided in the embodiment of the present application during the message sending stage.

[0024] Figure 5 It is a schematic diagram of the processing flow of the adaptive dynamic learning system provided in the embodiment of the present application at the message queue processing stage.

[0025] Figure 6 It is a schematic diagram of the processing flow of the adaptive dynamic learning system in the message consumption stage provided in an embodiment of the present application.

[0026] Figure 7 It is a structural diagram of a message consistency assurance device provided in an embodiment of the present application.

[0027] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] See also Figure 1 , Figure 1 This is one of the flow charts of the message consistency guarantee method provided in the embodiment of the present application. In the embodiment of the present application, the message consistency guarantee method is applied to the adaptive dynamic learning system (Adaptive Dynamic Learning System, ADLS), and the message consistency guarantee method includes steps S110 to S120, and each step is as follows: S110: Acquire message status information of each stage and real-time performance information of the distributed message queue system.

[0030] S120: Generate message configuration strategies for each stage based on the learning and decision engine of the adaptive dynamic learning system according to the message status information and real-time performance information of each stage, and perform message delivery based on the message configuration strategies for each stage.

[0031] Among them, the message configuration strategy aims to minimize the risk of message loss and message duplicate consumption.

[0032] Most of the current message consistency guarantee schemes are: the data producer cluster (i.e., upstream service) sends messages to the message queue (MQ) cluster (also known as the MQ message queue cluster), and then the MQ message queue cluster pushes the message to the data consumer (i.e., downstream service), or the data consumer pulls the message from the MQ message queue cluster. However, this method can only ensure the consistency of messages through a single MQ message queue, and cannot guarantee the problems of message loss, slow message processing, and high manual maintenance costs caused by uncertainties in each link. Based on this, this embodiment adopts a distributed message consistency guarantee method based on an adaptive dynamic learning system.

[0033] See also Figure 2 , Figure 2 This is the second flow chart of the message consistency assurance method provided in the embodiment of the present application.

[0034] like Figure 2 As shown, in this embodiment, ADLS (i.e., adaptive dynamic learning system) can monitor the message status (such as message delay) and the performance of the distributed message queue system in real time during the entire message delivery process, dynamically adjust the message processing priority and retry strategy, reduce unnecessary delays and system complexity, and use historical data and real-time performance information of the current system to intelligently select the best message recovery strategy through learning and decision engines to minimize the risk of message loss and duplicate consumption.

[0035] See also Figure 3 , Figure 3 It is a structural diagram of the adaptive dynamic learning system provided in an embodiment of the present application.

[0036] like Figure 3 As shown, ADLS includes three modules: data acquisition module, storage and historical data analysis module, and learning and decision engine.

[0037] Among them, the data collection module can monitor and collect the message status information of each stage and the real-time performance information of the distributed message queue system in real time through the monitoring agent deployed on each node.

[0038] Optionally, the message status information includes message queue entry and exit speed, message backlog, message sending, consumption delay, message trajectory, etc.

[0039] Optionally, the real-time performance information of the distributed message queue system includes CPU usage, memory usage, disk performance, network latency, etc.

[0040] Optionally, the data collection module can collect various indicators through tools such as Prometheus, Telegraf, StatsD, etc., and trigger alarms according to alarm thresholds through tools such as Alertmanager, OpsGenie or PagerDuty integrated in the alarm and notification layer, and notify relevant personnel.

[0041] The storage and historical data analysis module is used to support the learning and decision-making engine. The system needs to store the historical performance information of the distributed message queue system and the historical message processing results of each stage, and analyze these data to generate data of different granularities.

[0042] Optionally, use InfluxDB to store time series data; for event logs, use Elasticsearch for storage.

[0043] The learning and decision-making engine can use the LSTM model based on time series prediction and the DQN (reinforcement learning) model based on the collected historical data to jointly predict the optimal configuration parameters, message processing priority and retry strategy, etc. The model will consider factors such as historical processing efficiency, current system load, message type, etc., dynamically adjust message routing and processing order, and ultimately generate the optimal message configuration strategy for each stage, and perform message delivery based on the message configuration strategy for each stage.

[0044] In this embodiment, the learning and decision engine includes an LSTM model and a DQN model; wherein the LSTM model is used to predict the message status information and real-time performance information of each stage based on historical data to obtain the configuration parameter prediction value; the DQN model is used to dynamically adjust the configuration parameter prediction value to obtain the message configuration strategy of each stage.

[0045] Specifically, the LSTM model includes an input layer, an LSTM layer, and a fully connected layer (i.e., a Dense layer).

[0046] The input layer uses the historical data collected by the storage and historical data analysis modules as input data. For example, if the message status information and system performance indicators of N historical time steps are used as input, the dimension of the input layer will be N times the number of features.

[0047] There can be multiple LSTM layers, and multi-layer LSTM networks can capture long-term dependencies in time series data. The model structure can be optimized by adjusting parameters such as the number of LSTM layers, the number of neurons in each layer, and whether to return sequence output.

[0048] After the LSTM layer, add one or two fully connected layers to further extract features and make predictions. The number of neurons in the last fully connected layer should match the number of configuration parameters of the message configuration policy to be predicted.

[0049] Optionally, an appropriate activation function (such as ReLU activation function) is used between the LSTM layer and the fully connected layer to add nonlinearity and process long-term dependent data to maintain output continuity and improve computational efficiency.

[0050] Optionally, use mean absolute error (MAE) as the loss function for the LSTM model and use the RMSprop optimizer to train the model. Tune the hyperparameters of the LSTM model, such as learning rate, batch size, number of LSTM layers, etc., through techniques such as grid search, random search, or Bayesian optimization. Use the validation set to evaluate the performance of the LSTM model and monitor changes in metrics such as loss and accuracy, and save the best model (i.e., the model with the best performance on the validation set) during training.

[0051] When you need to use the trained LSTM model for prediction, you can load the pre-saved best trained LSTM model while preprocessing the new input data (i.e., the message status information at each stage and the real-time performance information of the distributed message queue system), and use the LSTM model to predict the new input data to obtain the predicted values ​​of the configuration parameters for the next time step or the next few time steps. Applying the predicted configuration parameters to the distributed system and monitoring the system's performance indicators can verify the effectiveness of the prediction, and dynamically adjust the prediction model or strategy based on the actual performance of the system.

[0052] It can be understood that although the configuration parameter prediction values ​​generated by the LSTM model can be directly applied to the distributed message queue system, the configuration parameter prediction values ​​predicted by the LSTM model are not necessarily the optimal parameters, so they can be further optimized through the DQN model.

[0053] Specifically, the DQN model uses the predicted values ​​of configuration parameters generated by the LSTM model as the state input of the model, and combines multiple parameters predicted by the LSTM model into a state vector; define the actions that the DQN model can perform. When the overall performance of the distributed message queue system is lower than the benchmark threshold, the value of a certain configuration parameter can be adjusted or multiple parameters can be combined for adjustment. Define the reward function based on the performance of the distributed message queue system (such as throughput, response time, etc.) (for example, if the system performance improves, a positive reward is given; if the performance decreases, a negative reward is given). The trained DQN model can dynamically adjust the predicted values ​​of the configuration parameters to obtain the optimal message configuration strategy for each stage.

[0054] Optionally, the DQN model includes an input layer (corresponding to the state space), a hidden layer, and an output layer (corresponding to the action space), and uses an experience replay buffer to store the experience data (state, action, reward, and next state) generated by the interaction between the agent and the environment, and randomly samples a batch of experience data from the experience replay buffer to train the DQN model. The training goal of the DQN model is to minimize the gap between the output Q value and the target Q value.

[0055] Specifically, the DQN model can be trained based on historical data: During the training process, the DQN model needs to calculate the target Q value for the purpose of supervised learning.

[0056] Generally, the DQN model can use a target network to calculate the target Q value. This network is the same as the Q network at the beginning, but the parameter update frequency is lower. The calculation formula of the target Q value is: ; Among them, r represents the reward, is the discount factor, Indicates the next state, Indicates the action in the next state, Represents the output of the target network.

[0057] The loss function of the DQN model is the mean square error (MSE) loss function, which is used to measure the gap between the Q value output by the Q network and the target Q value. The parameters of the Q network are optimized by the back propagation algorithm to minimize this gap. In each training iteration, a batch of samples are randomly drawn from the experience replay buffer, the target Q value is calculated, and the loss function is used to update the parameters of the Q network. At the same time, the parameters of the Q network are regularly copied to the target network to maintain the stability of the target network. By continuously interacting with the environment, collecting data, and training the Q network, the performance of the DQN model will gradually improve. This process can continue until the preset number of training rounds is reached or other stopping conditions are met. At the same time, the relationship between exploration and use is weighed during the training process to help the DQN model discover better strategies. This trade-off rule is controlled by adjusting the ε value in the ε-greedy strategy to maximize the reward using known information.

[0058] ADLS coordinates work through three modules to continuously monitor system performance, record adjustment effects, and continuously train and optimize LSTM and DQN models. The LSTM model can regularly predict future network delays and node loads, and based on the prediction results and current system status, the DQN model can dynamically adjust producer configurations, Raft protocol parameters, and consumer configurations to ensure the consistency of distributed messages.

[0059] The message consistency assurance method provided in the embodiment of the present application generates a message configuration strategy with the goal of minimizing the risk of message loss and the risk of message duplicate consumption based on the message status information of each stage and the real-time performance information of the distributed message queue system, using the learning and decision engine of the adaptive dynamic learning system, and then transmits messages based on the message configuration strategy. This can effectively avoid message inconsistencies between data producers and data consumers due to message loss and message duplicate consumption, and improve the effectiveness of the message consistency assurance method. At the same time, the adaptive dynamic learning system is introduced to intervene in each stage of the message transmission process and automatically generate message configuration strategies, which can reduce manual intervention, optimize the communication process, and improve the efficiency of the message consistency assurance method.

[0060] In some embodiments, message status information and real-time performance information are acquired by a data acquisition module of an adaptive dynamic learning system, the message status information includes status information of a message sending stage, and the message configuration strategy includes data producer configuration information; according to the message status information and real-time performance information of each stage, based on the learning and decision engine of the adaptive dynamic learning system, a message configuration strategy of each stage is generated, and based on the message configuration strategy of each stage, message delivery is performed, including: in the message sending stage, the status information and real-time performance information of the message sending stage are input into the learning and decision engine of the adaptive dynamic learning system, and the data producer configuration information output by the learning and decision engine is obtained; based on the data producer configuration information, the data producer is controlled to deliver the message to be processed to the message queue cluster.

[0061] Generally, during the message sending phase, the data producer starts sending pending messages to the consumer.

[0062] Specifically, during the message sending phase, the data collection module collects relevant indicators in real time, namely, the status information and real-time performance information of the message sending phase; the status information and real-time performance information of the sending phase are input into the learning and decision engine of ADLS to obtain the optimal data producer configuration information output by the learning and decision engine, such as the optimal data producer configuration parameters, message processing priority and retry policy data; based on the data producer configuration information, the data producer is controlled to pass the pending messages to the message queue cluster to ensure the consistency of distributed messages.

[0063] See also Figure 4 , Figure 4 It is a schematic diagram of the processing flow of the adaptive dynamic learning system provided in the embodiment of the present application during the message sending stage.

[0064] by Figure 4For example, in the message sending stage, the data producer starts to send pending messages to consumers. The data collection module can collect relevant indicators in real time, and obtain the best configuration parameters, message processing priority, retry strategy and other data producer configuration information through ADLS learning and decision engine processing. Based on the data producer configuration information, the data producer is controlled to pass the pending messages to the message queue cluster to ensure the consistency of distributed messages.

[0065] Optionally, the message status information includes message queue entry and exit speed, message backlog, message sending rate, consumption delay, message trajectory, etc.

[0066] Optionally, the real-time performance information of the distributed message queue system includes CPU usage, memory usage, disk performance, network latency, etc.

[0067] Specifically, the message status information includes the message sending rate of the data producer, the real-time performance information includes the system load and message processing capability of the distributed message queue system, and the data producer configuration information includes the message sending method; when the data acquisition module of ADLS detects that the message sending rate of the data producer is higher than the preset system message carrying benchmark value, the learning and decision engine of ADLS will dynamically adjust the message sending method of the data producer according to the system load and message processing capability of the distributed message queue system.

[0068] For example, non-critical messages can be delayed or scheduled to avoid overloading the system during high-load periods.

[0069] After the message sending method strategy is executed, ADLS can continue to dynamically adjust the message configuration strategy based on the queue length, queuing speed, data consumer backlog messages and system performance indicators of the data producer in the distributed message queue system.

[0070] Optionally, if the queue length of the data producer reaches or approaches the capacity limit, the message configuration strategy of ADLS is to send a back pressure signal to the data producer.

[0071] Optionally, if the enqueue rate exceeds the system processing rate, the ADLS message configuration strategy is to send a back pressure signal to the data producer.

[0072] Optionally, if the number of backlogged messages of the data consumer is higher than a preset threshold, the message configuration strategy of ADLS is to send a back pressure signal to the data producer.

[0073] Optionally, if system performance indicators (such as CPU usage, memory usage, and network bandwidth) reach or approach preset thresholds, the message configuration strategy of ADLS is to send a back-pressure signal to the data producer.

[0074] ADLS uses the above monitoring indicators to provide anti-pressure signals to data producers, slowing down the message production rate and preventing the distributed message queue system from overloading.

[0075] If ADLS detects that messages in the message queue are unevenly distributed, ADLS can use machine learning algorithms to monitor and predict consumer loads and the current load and processing capacity of the queue in real time based on historical data, and intelligently guide data producers to send messages to the most appropriate queue, thereby balancing the load of the entire system.

[0076] In some embodiments, the message status information includes status information of the message queue processing stage, and the message configuration strategy includes Raft algorithm parameters; according to the message status information and real-time performance information of each stage, based on the learning and decision engine of the adaptive dynamic learning system, the message configuration strategy of each stage is generated, and based on the message configuration strategy of each stage, message delivery is performed, including: in the message queue processing stage, after the message queue cluster receives the message to be processed, the status information and real-time performance information of the message queue processing stage are input into the learning and decision engine of the adaptive dynamic learning system, and the Raft algorithm parameters output by the learning and decision engine are obtained; according to the Raft algorithm parameters, based on the Raft consensus algorithm, the target processing node is determined from the message queue cluster; the message to be processed is allocated to the target processing node, and the message is waited for message delivery.

[0077] See also Figure 5 , Figure 5 It is a schematic diagram of the processing flow of the adaptive dynamic learning system provided in the embodiment of the present application at the message queue processing stage.

[0078] During the message sending phase, ADLS can control the data producer to pass the pending messages to the message queue cluster based on the generated data producer configuration information. At this time, the distributed message queue system enters the message queue processing phase.

[0079] like Figure 5 As shown, the message queue cluster includes multiple message queues, each of which is a processing node; in the message queue processing stage, after the message queue cluster receives the message to be processed, it is necessary to allocate a suitable processing node for the message to be processed to adjust system resources and balance the system load. In this embodiment, ADLS can make dynamic adjustments based on indicators such as message queue capacity and performance.

[0080] Specifically, after the message queue cluster receives the message to be processed, the status information and real-time performance information of the message queue processing stage can be input into the learning and decision engine of the adaptive dynamic learning system to obtain the Raft algorithm parameters output by the learning and decision engine; according to the Raft algorithm parameters and based on the Raft consensus algorithm, the target processing node is determined from the message queue cluster; the message to be processed is assigned to the target processing node, and waits for message delivery.

[0081] Specifically, the status information and real-time performance information of the message queue processing stage include node failure conditions and / or node processing message rates; when the data acquisition module of ADLS detects a node failure or a decrease in the node processing message rate, ADLS can automatically generate a message configuration strategy. The message configuration strategy is to perform failover, that is, to quickly migrate the pending messages to healthy nodes, to ensure the continuity and reliability of the pending message processing process.

[0082] The status information and real-time performance information of the message queue processing stage include the load and performance indicators of each message queue in the message queue cluster, and dynamically adjust the message configuration strategy (such as message routing strategy) through the probabilistic fault-tolerant model to reasonably distribute the messages to be processed to the best processing node to achieve load balancing.

[0083] In order to solve the problems existing in the consensus process of the Paxos consensus algorithm and the Raft consensus algorithm, this embodiment proposes a new distributed consensus algorithm to solve the problems of computational overhead and fault tolerance limitations, and to find a balance between the communication efficiency between multiple nodes, the complexity of the algorithm, and the fault tolerance of the system. The following are possible strategies and steps for designing such an algorithm: (1) Consider reducing communication overhead: This embodiment introduces a hierarchical consensus mechanism. By introducing a multi-layer structure, the message queue cluster is divided into multiple cluster groups (or cluster partitions). A leader node is elected within each cluster group through an existing consistency algorithm (such as the Paxos consensus algorithm and the Raft consensus algorithm); then an election is conducted among these leader nodes to elect the target processing node to reach a global consensus. This hierarchical approach can significantly reduce the number of nodes that must participate in each round of communication, thereby reducing communication overhead.

[0084] In this embodiment, the existing Raft consensus algorithm needs to be optimized.

[0085] In the existing Raft consensus algorithm, assuming that there are N nodes in each cluster group, and each cluster group elects a leader node through the Raft consensus algorithm, then within the term T, the probability of successful election of each processing node in the cluster group can be expressed as: ; In the improved Raft consensus algorithm of this embodiment, the probability of successful election of a processing node depends not only on the number of nodes, but also on network delay and vote distribution.

[0086] Based on this, this embodiment improves the above method: Assume that there are N nodes in each cluster group, each node has one vote, and during the election process, each node randomly votes for any node (including itself), so the distribution of votes can be considered random; if each node votes independently and with equal probability, then theoretically, the probability of each node obtaining the leader position is equal, and both are The key to network delay affecting elections is that it may cause inconsistent propagation speeds of ballots between different nodes, thus affecting election results.

[0087] Assuming D represents the network delay, we can introduce a delay factor To measure the impact of network delay on the election; assuming that the network delay D follows a certain probability distribution, such as a normal distribution , where μ is the average network delay, is the variance of network delay. Taking these two factors into consideration, the probability of successful election of the i-th processing node in the cluster group (i.e., the probability of being elected as the leader node) within the term T is It can be expressed as: ; in, Represents a function of the delay factor of node i, which is used to adjust the leader election probability due to network delay. This function can be defined based on the distribution of network delay, and its calculation formula is as follows: ; in, is the average network delay for node i to receive all votes, μ is the average network delay, is the variance of network delay.

[0088] In the improved Raft consensus algorithm of this embodiment, on the basis of the consensus of each cluster group, a fast consensus algorithm based on summary and signature is adopted to form a global consensus.

[0089] For a pending message m, its data summary can be expressed as d=hash(m), where hash represents a hash function; the leader node of each cluster group can sign the summary, and the signature operation can be expressed as s=sign(d, k), where k is the private key, and finally a global consensus verification is performed; other nodes in the cluster group can verify the signature using the public key K, and the verification operation can be expressed as verify(s, d, K).

[0090] Based on the above improved Raft consensus algorithm, the communication overhead is reduced and the throughput of the entire distributed system is improved.

[0091] (2) Consider enhancing the fault tolerance of the (probabilistic fault tolerance model): ADLS can optimize the flexible fault tolerance mechanism of the Raft consensus algorithm, not only considering the static fault tolerance threshold (for example, the fault tolerance threshold is set to half the number of cluster group nodes, that is, N / 2), but also introducing a dynamic adjustment mechanism to adjust the fault tolerance level according to the network status and the actual performance of the node. For example, by real-time monitoring of the response time and reliability of the node, more reliable nodes can be dynamically selected to participate in the key decision-making process.

[0092] Alternatively, a probabilistic model can be introduced into the algorithm to allow it to tolerate more faults with a certain probability, rather than relying on a fixed fault tolerance threshold. This requires a mathematical model to ensure that the system can still operate correctly with a high probability even under a higher failure rate.

[0093] Assuming that the maximum number of faulty nodes that the message queue cluster of the distributed message queue system can tolerate is f, and the total number of nodes is N′, the probability that the system can operate normally under a given number of faults (i.e., the fault tolerance threshold) can be expressed by binomial distribution: ; in, Represents the probability of a single node failing.

[0094] ADLS can dynamically adjust the participation rate and the total number of nodes participating in the consensus according to real-time conditions to optimize .

[0095] ADLS adjusts and optimizes dynamic parameters according to specific application scenarios, system requirements and environmental conditions. Improving the Raft algorithm requires specific analysis of the implementation details of the compression algorithm and batch processing strategy, while the design of the global consensus algorithm needs to consider the actual communication model and signature mechanism. The choice of implementation details and optimization strategies will directly affect the performance and fault tolerance of the algorithm to ensure the consistency of distributed messages.

[0096] In this embodiment, in the message queue processing stage, after the message queue cluster receives the message to be processed, the status information and real-time performance information of the message queue processing stage are input into the learning and decision engine of the adaptive dynamic learning system to obtain the Raft algorithm parameters output by the learning and decision engine, such as the maximum number of faulty nodes f, the total number of nodes participating in the consensus N′, the number of consensus participating nodes of each cluster group (i.e., the number of consensus participating nodes N), etc.; according to the Raft algorithm parameters, the best target processing node is determined from the message queue cluster through the Raft consensus algorithm, and the message to be processed is allocated to the target processing node, waiting for message delivery.

[0097] Specifically, the status information and real-time performance information of the message queue processing stage also include the importance and processing priority of the messages to be processed; ADLS adjusts the storage location and method (such as memory queue or disk storage) of the messages to be processed according to the importance and processing priority of the messages to be processed to optimize performance and resource utilization.

[0098] Specifically, when the pending messages need to be stored in multiple copies, ADLS dynamically selects the most suitable consensus algorithm based on the consistency of the copy messages to ensure the stable operation of the system.

[0099] Specifically, the status information and real-time performance information of the message queue processing stage also include the usage of host resources of each node in the message queue cluster; ADLS uses a greedy algorithm to dynamically allocate and optimize the use of host resources of the message queue cluster based on the usage of host resources of each node in the message queue cluster. When the disk usage reaches a specific threshold, the disk size is dynamically adjusted in combination with the current message persistence space growth rate. When FGC is frequently triggered, the JVM parameters are dynamically adjusted through ADLS, and the host memory size is dynamically adjusted based on the decision engine results. When the CPU usage of the virtual machine exceeds the threshold, the CPU resources are dynamically adjusted when the load is too high to adapt to different load requirements and optimize system performance.

[0100] In addition, ADLS can automatically expand or reduce the capacity of message queue cluster nodes according to real-time load conditions to ensure efficient use of system resources and optimal response speed.

[0101] In some embodiments, according to Raft algorithm parameters, based on the Raft consensus algorithm, a target processing node is determined from a message queue cluster, including: according to the Raft algorithm parameters, determining the fault tolerance threshold and the number of consensus participating nodes of the Raft consensus algorithm; based on the number of consensus participating nodes, grouping the message queue cluster to obtain multiple cluster groups; wherein each cluster group includes the same number of consensus participating nodes, and the system fault tolerance probability of the message queue cluster is less than the fault tolerance threshold; determining the leader node of each cluster group; a leader node is elected by all consensus participating nodes in a cluster group through a consistency algorithm; and electing a target processing node from all leader nodes.

[0102] Specifically, in the message queue processing stage, after the message queue cluster receives the message to be processed, the status information and real-time performance information of the message queue processing stage are input into the learning and decision engine of the adaptive dynamic learning system to obtain the Raft algorithm parameters output by the learning and decision engine, where the Raft algorithm parameters include the maximum number of faulty nodes f, the total number of nodes participating in the consensus N′, and the number of consensus participating nodes in each cluster group (i.e., the number of consensus participating nodes N).

[0103] Furthermore, based on the above Raft algorithm parameters, the fault tolerance threshold of the Raft consensus algorithm is determined. And the number of consensus participating nodes N.

[0104] Furthermore, based on the number of consensus participating nodes, the message queue cluster is grouped to obtain multiple cluster groups.

[0105] Among them, each cluster group includes the same number of consensus participating nodes, and the system fault tolerance probability of the message queue cluster is less than the fault tolerance threshold.

[0106] Specifically, assuming that the total number of nodes participating in the consensus of the message queue cluster is N′, and the number of nodes participating in the consensus is N, the message queue cluster with a total number of nodes N′ is grouped and divided into multiple cluster groups, each of which contains N processing nodes.

[0107] Furthermore, for each cluster group, a leader node of the cluster group is elected from the N processing nodes through a consensus algorithm.

[0108] Furthermore, the target processing node is again elected from all the leading nodes through the consensus algorithm.

[0109] The message consistency guarantee method provided in the embodiment of the present application improves the existing Raft consensus algorithm by introducing a hierarchical consensus mechanism, dividing the message queue cluster into multiple cluster groups, and electing a leader node within each cluster group through the existing consistency algorithm; then an election is conducted among these leader nodes to elect the target processing node to achieve global consensus. This hierarchical method can significantly reduce the number of nodes that must participate in each round of communication, thereby reducing communication overhead.

[0110] In some embodiments, the message status information includes status information of the message consumption stage, and the message configuration strategy includes data consumer configuration information; based on the message status information and real-time performance information of each stage, the message configuration strategy of each stage is generated based on the learning and decision engine of the adaptive dynamic learning system, and based on the message configuration strategy of each stage, message transmission is performed, including: in the message consumption stage, the status information and real-time performance information of the message consumption stage are input into the learning and decision engine of the adaptive dynamic learning system, and the data consumer configuration information output by the learning and decision engine is obtained; based on the data consumer configuration information, the data consumer is controlled to obtain the message to be processed from the target processing node.

[0111] See also Figure 6 , Figure 6 It is a schematic diagram of the processing flow of the adaptive dynamic learning system in the message consumption stage provided in an embodiment of the present application.

[0112] like Figure 6 As shown, after the pending message is assigned to the target processing node, it enters the message consumption stage. At this time, the data consumer can pull messages from the target processing node of the message queue cluster for message consumption processing. In this embodiment, the status information and real-time performance information of the message consumption stage include indicators such as the consumption rate of the data consumer, the message backlog, and the message consumption success rate. ADLS can dynamically adjust the data consumer configuration information according to the above indicators, and control the data consumer to obtain the pending message from the target processing node based on the data consumer configuration information.

[0113] Specifically, the status information and real-time performance information of the message consumption stage include the health status of the data consumer. If ADLS detects that the health status of the data consumer is abnormal or the performance is degraded, it immediately generates the data consumer configuration information and performs self-repair operations based on the data consumer configuration information, such as restarting the consumer instance or migrating it to a better operating environment. According to the message backlog and processing speed, ADLS can automatically adjust the number of consumers and adapt to changes in message load by increasing or decreasing consumer instances.

[0114] For data consumers with slower consumption rates, ADLS can take optimization measures, such as adjusting the number of concurrent processing threads of the consumer or adjusting its message processing logic to increase the consumption speed.

[0115] ADLS can dynamically adjust consumption policies based on the type and priority of messages, such as implementing priority queues to ensure that urgent messages are processed quickly.

[0116] In addition, during the data consumption process, ADLS can dynamically adjust the retry strategy for messages that fail to be consumed, such as adjusting the retry interval and number based on the error type and frequency, thereby improving the success rate of message processing.

[0117] In some embodiments, the adaptive dynamic learning system includes a storage and historical data analysis module, which is used to store historical data. The learning and decision engine includes an LSTM model and a DQN model; the LSTM model is used to predict the message status information and real-time performance information of each stage based on historical data to obtain the configuration parameter prediction value; the DQN model is used to dynamically adjust the configuration parameter prediction value to obtain the message configuration strategy for each stage.

[0118] Compared with the prior art, the message consistency assurance method provided in the embodiment of the present application includes at least the following advantages: (1) High efficiency: The embodiment of the present application adopts an adaptive learning system, which can automatically adjust parameters according to the operating status of the distributed system without human intervention, so that the distributed system can run efficiently and reduce the cost of manual operation and maintenance; (2) Ease of use: The optimized Raft consensus algorithm provided in the embodiment of the present application can improve the fault tolerance of the distributed system, ensure the smooth operation of the distributed system, and reduce the risk of failure.

[0119] The present application also provides a message consistency guarantee device. Figure 7 , Figure 7 7 is a schematic diagram of the structure of the message consistency guarantee device provided in the embodiment of the present application. In the embodiment of the present application, the message consistency guarantee device includes an acquisition module 710 and a consistency guarantee module 720.

[0120] The acquisition module 710 is used to acquire the message status information of each stage and the real-time performance information of the distributed message queue system.

[0121] The consistency assurance module 720 is used to generate message configuration strategies for each stage based on the message status information and real-time performance information of each stage and based on the learning and decision engine of the adaptive dynamic learning system, and to perform message transmission based on the message configuration strategies for each stage.

[0122] Among them, the message configuration strategy aims to minimize the risk of message loss and message duplicate consumption.

[0123] In some embodiments, the message status information and real-time performance information are acquired by a data acquisition module of the adaptive dynamic learning system, the message status information includes status information of the message sending phase, and the message configuration strategy includes data producer configuration information.

[0124] The consistency assurance module 720 is used to input the status information and real-time performance information of the message sending stage into the learning and decision engine of the adaptive dynamic learning system during the message sending stage, and obtain the data producer configuration information output by the learning and decision engine; based on the data producer configuration information, control the data producer to pass the message to be processed to the message queue cluster.

[0125] In some embodiments, the message status information includes status information of a message queue processing stage, and the message configuration strategy includes Raft algorithm parameters.

[0126] The consistency assurance module 720 is used to input the status information and real-time performance information of the message queue processing stage into the learning and decision engine of the adaptive dynamic learning system after the message queue cluster receives the message to be processed during the message queue processing stage, and obtain the Raft algorithm parameters output by the learning and decision engine; determine the target processing node from the message queue cluster based on the Raft consensus algorithm according to the Raft algorithm parameters; allocate the message to be processed to the target processing node, and wait for message delivery.

[0127] In some embodiments, the consistency assurance module 720 is used to determine the fault tolerance threshold and the number of consensus participating nodes of the Raft consensus algorithm according to the Raft algorithm parameters; based on the number of consensus participating nodes, the message queue cluster is grouped and processed to obtain multiple cluster groups; wherein each cluster group includes the same number of consensus participating nodes, and the system fault tolerance probability of the message queue cluster is less than the fault tolerance threshold; determine the leader node of each cluster group; a leader node is elected by all consensus participating nodes in a cluster group through a consistency algorithm; and a target processing node is elected from all leader nodes.

[0128] In some embodiments, the message status information includes status information of the message consumption stage, and the message configuration policy includes data consumer configuration information.

[0129] The consistency assurance module 720 is used to input the status information and real-time performance information of the message consumption stage into the learning and decision engine of the adaptive dynamic learning system during the message consumption stage, and obtain the data consumer configuration information output by the learning and decision engine; based on the data consumer configuration information, control the data consumer to obtain the message to be processed from the target processing node.

[0130] In some embodiments, the adaptive dynamic learning system includes a storage and historical data analysis module, which is used to store historical data. The learning and decision engine includes an LSTM model and a DQN model; the LSTM model is used to predict the message status information and real-time performance information of each stage based on historical data to obtain the configuration parameter prediction value; the DQN model is used to dynamically adjust the configuration parameter prediction value to obtain the message configuration strategy for each stage.

[0131] The present application also provides an electronic device, Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the message consistency guarantee method.

[0132] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0133] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the message consistency assurance method provided by the above methods is implemented.

[0134] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the message consistency assurance method provided by the above methods.

[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A message consistency guarantee method, characterized in that: include: Obtain message status information at each stage and real-time performance information of the distributed message queue system; According to the message status information and the real-time performance information of each stage, based on the learning and decision engine of the adaptive dynamic learning system, a message configuration strategy for each stage is generated, and based on the message configuration strategy for each stage, message delivery is performed; The message configuration strategy aims to minimize the risk of message loss and the risk of message duplicate consumption.

2. The message consistency guarantee method according to claim 1, characterized in that: The message status information and the real-time performance information are acquired by the data acquisition module of the adaptive dynamic learning system, the message status information includes status information of the message sending stage, and the message configuration strategy includes data producer configuration information; The method of generating a message configuration strategy for each stage based on the message status information and the real-time performance information of each stage and based on the learning and decision engine of the adaptive dynamic learning system, and performing message transmission based on the message configuration strategy for each stage, includes: In the message sending stage, the state information of the message sending stage and the real-time performance information are input into the learning and decision engine of the adaptive dynamic learning system to obtain the data producer configuration information output by the learning and decision engine; Based on the data producer configuration information, the data producer is controlled to pass the to-be-processed message to the message queue cluster.

3. The message consistency guarantee method according to claim 2, characterized in that: The message status information includes status information of the message queue processing stage, and the message configuration strategy includes Raft algorithm parameters; The method of generating a message configuration strategy for each stage based on the message status information and the real-time performance information of each stage and based on the learning and decision engine of the adaptive dynamic learning system, and performing message transmission based on the message configuration strategy for each stage, includes: In the message queue processing stage, after the message queue cluster receives the message to be processed, the state information of the message queue processing stage and the real-time performance information are input into the learning and decision engine of the adaptive dynamic learning system to obtain the Raft algorithm parameters output by the learning and decision engine; According to the Raft algorithm parameters, based on the Raft consensus algorithm, determine the target processing node from the message queue cluster; The to-be-processed message is distributed to the target processing node, and the message is waited for to be transmitted.

4. The message consistency guarantee method according to claim 3, characterized in that: The determining the target processing node from the message queue cluster according to the Raft algorithm parameters and based on the Raft consensus algorithm includes: Determine the fault tolerance threshold and the number of consensus participating nodes of the Raft consensus algorithm according to the Raft algorithm parameters; Based on the number of consensus participating nodes, the message queue cluster is grouped to obtain a plurality of cluster groups; wherein each of the cluster groups includes the same number of consensus participating nodes, and the system fault tolerance probability of the message queue cluster is less than the fault tolerance threshold; Determine the leader node of each cluster group; a leader node is elected by all consensus participating nodes in a cluster group through a consensus algorithm; The target processing node is elected from all the leader nodes.

5. The message consistency guarantee method according to claim 3, characterized in that: The message status information includes status information of the message consumption phase, and the message configuration strategy includes data consumer configuration information; The method of generating a message configuration strategy for each stage based on the message status information and the real-time performance information of each stage and based on the learning and decision engine of the adaptive dynamic learning system, and performing message transmission based on the message configuration strategy for each stage, includes: In the message consumption phase, the state information of the message consumption phase and the real-time performance information are input into the learning and decision engine of the adaptive dynamic learning system to obtain the data consumer configuration information output by the learning and decision engine; Based on the data consumer configuration information, the data consumer is controlled to obtain the to-be-processed message from the target processing node.

6. The message consistency guarantee method according to claim 1, characterized in that: The adaptive dynamic learning system includes a storage and historical data analysis module, the storage and historical data analysis module is used to store historical data, and the learning and decision engine includes an LSTM model and a DQN model; The LSTM model is used to predict the message status information and the real-time performance information at each stage according to the historical data to obtain a configuration parameter prediction value; The DQN model is used to dynamically adjust the predicted values ​​of the configuration parameters to obtain the message configuration strategies at each stage.

7. A message consistency assurance device, characterized in that: include: The acquisition module is used to obtain the message status information of each stage and the real-time performance information of the distributed message queue system; A consistency assurance module, configured to generate a message configuration strategy for each stage based on the message status information and the real-time performance information of each stage and based on a learning and decision engine of an adaptive dynamic learning system, and to perform message transmission based on the message configuration strategy for each stage; The message configuration strategy aims to minimize the risk of message loss and the risk of message duplicate consumption.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the message consistency assurance method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the message consistency assurance method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the message consistency assurance method according to any one of claims 1 to 6 is implemented.

Images