Message middleware throughput adjustment method, device and storage medium

Abstract

The present disclosure provides a message middleware throughput adjustment method, device and storage medium, the method comprising: obtaining a message total amount threshold of the message middleware and a time length of a single production or consumption cycle; constructing a mathematical model by taking the production speed and the consumption speed of each cycle as variables and taking the minimization of the overall cost of the system as the target based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle; setting the production speed, the consumption speed and the transmission time of the message of an initial cycle as initial values of the mathematical model; inputting the mathematical model and the initial values into a deep deterministic policy gradient (DDPG) algorithm to solve, and obtaining the optimal production speed and the optimal consumption speed of each cycle to adjust the production speed and the consumption speed of each cycle of the message middleware; compared with a traditional algorithm, the method has stronger computing power, can obtain the optimal efficiency while reducing the overall cost of the system when the data volume is large, and maximizes the possibility of reducing the message backlog.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and particularly relates to a message middleware throughput adjustment method, device and storage medium. BACKGROUND

[0002] The message middleware performs asynchronous delivery of data information by creating producers and consumers, and guarantees that the production and consumption of messages are close to balance within a period of time by means of an algorithm. However, in the case of massive data transmission, the traditional algorithm takes too long to calculate, and new messages may arrive before the calculation is completed, thereby causing the accumulation of messages, which accumulates more and more over time. SUMMARY

[0003] The first aspect of the present disclosure provides a message middleware throughput adjustment method, which comprises: obtaining a message total amount threshold of the message middleware and a time length of a single production or consumption cycle; constructing a mathematical model by taking the production speed and the consumption speed of each cycle as variables and taking the minimization of the overall cost of the system as the goal based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle; setting the production speed, the consumption speed of the initial cycle and the transmission time of the message as the initial values of the mathematical model; inputting the mathematical model and the initial values into a deep deterministic policy gradient (DDPG) algorithm to solve the optimal production speed and the optimal consumption speed of each cycle, so as to adjust the production speed and the consumption speed of each cycle of the message middleware.

[0004] In some embodiments of the present disclosure, the construction of the mathematical model by taking the production speed and the consumption speed of each cycle as variables and taking the minimization of the overall cost of the system as the goal based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle comprises: defining the cost of each cycle according to the production speed, the consumption speed of each cycle and the transmission time delay of the message; summing the costs of all cycles to obtain the overall cost of the system; and constructing the mathematical model by taking the minimization of the overall cost as the goal under the condition of meeting the constraint condition, wherein the constraint condition comprises that the production speed of each cycle is less than or equal to the maximum value of the production speed supported by the message middleware, the consumption speed of each cycle is less than or equal to the minimum value of the consumption speed supported by the message middleware, and the total amount of the messages produced in all cycles is less than or equal to the message total amount threshold.

[0005] In some embodiments of the present disclosure, the mathematical model and initial values are input into the DDPG algorithm to solve the optimal production speed and optimal consumption speed of each period, and adjusting the production speed and consumption speed of the message middleware in each period comprises: inputting the mathematical model and initial values into the DDPG algorithm for multiple rounds of neural network model training until the total cost is minimized to obtain a trained neural network model; calculating the optimal production speed and optimal consumption speed of the message middleware in each period by using the trained neural network model; and adjusting the actual production speed and consumption speed of the message middleware based on the optimal production speed and optimal consumption speed.

[0006] In some embodiments of the present disclosure, the multiple rounds of neural network model training of the mathematical model and initial values input into the DDPG algorithm until the total cost is minimized to obtain a trained neural network model comprises: defining the remaining producible message amount of the system as the system state, determining the system state corresponding to the initial period based on the initial value and the message total amount threshold; in each round of neural network model training, inputting the system state corresponding to the initial period into the policy network and value network of the DDPG algorithm for iterative processing, optimizing the production speed and consumption speed of each period, and updating the parameters of the policy network and value network with the goal of minimizing the total cost of the system until the remaining producible message amount of the system is 0, determining that the system state reaches the final state, and ending the iteration; repeating the multiple rounds of neural network model training to obtain the trained neural network model.

[0007] In some embodiments of the present disclosure, the input of the system state corresponding to the initial period into the policy network and value network of the DDPG algorithm for iterative processing, optimization of the production speed and consumption speed of each period, and updating of the parameters of the policy network and value network with the goal of minimizing the total cost of the system comprises: in each iteration, inputting the system state corresponding to the current period into the policy network, using the policy network to select at least one determined value of the production speed and consumption speed of the period according to the system state corresponding to the current period and the current policy to generate a corresponding at least one system action; evaluating each system action to obtain a corresponding system reward, and obtaining the system state corresponding to the next period based on each system action; storing a tuple composed of the system state corresponding to the current period, the system action, the system reward, and the system state corresponding to the next period in a storage set of the value network; randomly selecting a batch of tuples from the storage set as training samples, and sending the training samples into the policy network and value network for training to update the parameters of the policy network and value network.

[0008] In some embodiments of the present disclosure, the training sample is input into the policy network and the value network for training, and updating the parameters of the policy network and the value network comprises: calculating a loss value of an iterative process according to the training sample and a loss function, updating the parameters of the value network by using a gradient descent method with the goal of minimizing the loss value; and updating the parameters of the policy network by using a gradient ascent method with the goal of optimizing the performance of the policy.

[0009] The second aspect embodiment of the present disclosure provides a message middleware throughput adjustment device, which comprises: an acquisition module configured to acquire a message total amount threshold of the message middleware and a time length of a single production or consumption cycle; a model construction module configured to construct a mathematical model by taking the production speed and the consumption speed of each cycle as variables and minimizing the overall cost of the system as a goal based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle; a setting module configured to set the production speed and the consumption speed of an initial cycle and the transmission time of the message as initial values of the mathematical model; and a solution module configured to input the mathematical model and the initial values into a deep deterministic policy gradient (DDPG) algorithm to solve the mathematical model and obtain the optimal production speed and the optimal consumption speed of each cycle to adjust the production speed and the consumption speed of each cycle of the message middleware.

[0010] The third aspect embodiment of the present disclosure provides a communication device, which comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect embodiment of the present disclosure.

[0011] The fourth aspect embodiment of the present disclosure provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the method described in the first aspect embodiment of the present disclosure.

[0012] The fifth aspect embodiment of the present disclosure provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the method described in the first aspect embodiment of the present disclosure.

[0013] In summary, the message middleware throughput adjustment method provided by the disclosure includes obtaining a message total amount threshold of the message middleware and a time length of a single production or consumption cycle; based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle, taking the production speed and the consumption speed of each cycle as variables and taking the minimization of the overall cost of the system as a target, a mathematical model is constructed; the production speed, the consumption speed of the initial cycle and the transmission time of the message are set as the initial values of the mathematical model; the mathematical model and the initial values are input into a deep deterministic policy gradient (DDPG) algorithm for solving, and the optimal production speed and the optimal consumption speed of each cycle are obtained to adjust the production speed and the consumption speed of each cycle of the message middleware. The scheme is based on the DDPG algorithm, takes the minimization of the overall cost of the system as a target, takes the production speed and the consumption speed as variables, constructs a message middleware adaptive throughput model, and dynamically adjusts the production speed of the producer and the consumption speed of the consumer in each cycle by using the model. Compared with the traditional algorithm, the scheme has stronger computing power, can obtain the optimal efficiency while reducing the overall cost of the system when the data volume is large, and maximizes the possibility of reducing the message backlog.

[0014] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing undue limitations on the disclosure.

[0016] Figure 1 A flowchart of a message middleware throughput adjustment method provided by an embodiment of the disclosure is shown in FIG. 1;

[0017] Figure 2 A flowchart of a message middleware throughput adjustment method provided by an embodiment of the disclosure is shown in FIG. 1;

[0018] Figure 3 A flowchart of a message middleware throughput adjustment method provided by an embodiment of the disclosure is shown in FIG. 1;

[0019] Figure 4 A framework example diagram of a deep deterministic policy gradient (DDPG) algorithm provided by an embodiment of the disclosure is shown in FIG. 4;

[0020] Figure 5 A structure diagram of a message middleware throughput adjustment device provided by an embodiment of the disclosure is shown in FIG. 5;

[0021] Figure 6 A hardware composition structure diagram of a communication device provided by an embodiment of the disclosure is shown in FIG. 6. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present disclosure, and cannot be understood as a limitation of the present disclosure.

[0023] Firstly, the technical problems to be solved by the technical solutions of the present disclosure and related technologies are described in detail.

[0024] The message middleware is an intermediary that can asynchronously deliver data information, such as kafka, rocketmq, etc. The message middleware needs to create at least one producer and consumer. The producer sends messages to the message middleware, the message middleware forwards or stores the messages, and the consumer obtains the messages from the message middleware. In this process, the producer and the consumer have certain production speed and consumption speed, and have certain transmission delay. If the speed of the producer and the consumer is not set reasonably, it may cause message backlog or low transmission efficiency.

[0025] At present, traditional mathematical models are mainly used to solve this problem, such as leak bucket algorithm, token bucket algorithm, etc. These algorithms mainly adjust the production speed and consumption speed of the producer and consumer next time according to the real-time traffic, to ensure that the production and consumption of messages are close to balance within a period of time, so as to minimize the message backlog and improve the transmission efficiency. For example, a leak bucket algorithm adaptive flow control method is disclosed in the related art, which adjusts the message processing speed of different periods to maximize the message throughput while ensuring the stability of system operation.

[0026] With the increasing amount and frequency of Internet message transmission, the existing message middleware algorithm for improving efficiency faces more severe challenges, including: in the case of massive data transmission, the algorithm needs to calculate for a long time, at this time, even if the message is not accumulated, the experience will be greatly reduced; the traditional algorithm needs to calculate the data of each time in real time according to the specific situation, there is no unified method that can be used directly, that is, the input parameters can quickly get the output, and the applicability is not wide; when the data volume is large, the traditional mathematical method takes too long to calculate, and new messages may come before the calculation is completed, which will continuously cause message backlog, and the backlog will increase over time.

[0027] To solve the above problems, the disclosure proposes a new message middleware throughput adjustment scheme, based on the DDPG algorithm, taking the minimum overall system cost as the target, taking the production speed and consumption speed as variables, constructing a message middleware adaptive throughput model, and using the model to dynamically adjust the production speed of each cycle producer and the consumption speed of the consumer. Compared with the traditional algorithm, it has stronger computing power, can get the optimal efficiency while reducing the overall system cost when the data volume is large, maximizes the possibility of reducing message backlog, avoids using additional storage devices to store these backlog messages; At the same time, the neural network model trained by the DDPG algorithm can be used for most existing message middleware without being limited to the type of message, and has a wide range of applications.

[0028] The disclosure will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

[0029] Figure 1 A flowchart of a message middleware throughput adjustment method provided by an embodiment of the disclosure.

[0030] As Figure 1 shown, the message middleware throughput adjustment method provided by the embodiment of the disclosure includes the following steps 101-104:

[0031] Step 101, obtain the message total amount threshold of the message middleware and the duration of a single production or consumption cycle.

[0032] In some embodiments of the disclosure, the message middleware includes Apache Kafka, RabbitMQ, ActiveMQ, Redis, etc., which is not limited by the disclosure.

[0033] In some embodiments of the disclosure, the message total amount threshold of the message middleware can be set based on the system capacity, including the storage capacity of the message queue or topic, the processing capacity of the consumer, and the delay requirement of the system for message processing.

[0034] In some embodiments of the disclosure, the duration of the production or consumption cycle can be set according to specific business requirements and system performance, taking into account factors such as system processing capacity, delay requirements, and message quantity.

[0035] Step 102, based on the message total amount threshold of the message middleware and the duration of a single production or consumption cycle, taking the production speed and consumption speed of each cycle as variables, taking the minimum overall system cost as the target, constructing a mathematical model.

[0036] In some embodiments of the disclosure, the production speed refers to the speed at which the producer produces messages, and the consumption speed refers to the speed at which the consumer consumes messages.

[0037] In some embodiments of the present disclosure, the costs of all periods are summed to obtain the overall cost of the system; and a mathematical model is constructed with the objective of minimizing the overall cost under the constraint conditions, wherein the constraint conditions include that the production speed of each period is less than or equal to the maximum value of the production speed supported by the message middleware, the consumption speed of each period is less than or equal to the minimum value of the consumption speed supported by the message middleware, and the total number of messages produced in all periods is less than or equal to the message total amount threshold.

[0038] In some embodiments of the present disclosure, the cost of each period is defined according to the production speed, the consumption speed and the transmission time delay of the message of each period, specifically, the remaining unconsumed quantity of each period can be calculated according to the consumption speed, the consumption speed and the period length of each period, and the weighted average of the transmission time and the remaining unconsumed quantity of each period is calculated as the cost of each period.

[0039] In some embodiments of the present disclosure, given the message total amount threshold P 总 and the length of a single production or consumption period T, i∈{0,1,2,3...} represents the i th production and consumption period. Let the production speed of the message middleware producer be: Let the consumption speed of the message middleware consumer be: V i c , i∈{0,1,2,3...}.

[0040] In some embodiments of the present disclosure, the weighted average of the transmission time and the remaining unconsumed quantity of each period is calculated as the cost of each period, and the cost can be represented as wherein p is a weighting coefficient, is the number of messages not consumed in the current period, and the transmission time of the message is t i , i∈{0,1,2,3...}.

[0041] In some embodiments of the present disclosure, a mathematical model P1 is defined, and the minimum cost from all message production to the end of consumption in the system is minimized under the constraint conditions, and the specific formula of P1 is as follows:

[0042] P1:

[0043] i∈{0,1,2,3...} Formula 2

[0044]

[0045] Wherein, formula 1 is a target function, formula 2 represents the i-th consumption or production cycle, formula 3 represents that the production speed is less than or equal to the maximum value of the production speed supported by the message middleware, formula 4 represents that the consumption speed is less than or equal to the minimum value of the consumption speed supported by the message middleware, and formula 5 represents that the total amount of messages produced by the producer is less than or equal to the given message threshold.

[0046] In step 103, the initial production speed, consumption speed and message transmission time of the initial cycle are set as the initial values of the mathematical model.

[0047] In some embodiments of the present disclosure, a given message total amount threshold P 总 and a single production or consumption cycle length T, i∈{0,1,2,3...} represents the i-th production and consumption cycle. The production speed of the message middleware producer is set as: The consumption speed of the message middleware consumer is set as: V i c , i∈{0,1,2,3...}, when i=0, and respectively represent the initial production speed and consumption speed of the producer and the consumer, which are set according to the different message middleware in specific applications.

[0048] In step 104, the mathematical model and the initial values are input into the deep deterministic policy gradient (DDPG) algorithm for solving, and the optimal production speed and optimal consumption speed of each cycle are obtained to adjust the production speed and consumption speed of each cycle of the message middleware.

[0049] In some embodiments of the present disclosure, the DDPG (Deep Deterministic Policy Gradient) algorithm belongs to an algorithm under DRL (Deep Reinforcement Learning). The DDPG algorithm is suitable for solving decision-making problems in dynamic environments. First, the DDPG algorithm can optimize the long-term decision-making performance, which is better than the static one-time optimization. Second, the DDPG algorithm learns and optimizes decisions by directly interacting with the environment without any prior knowledge, avoiding many requirements when using traditional solutions. Third, the DDPG algorithm can fully utilize the powerful representation ability of DNN (Deep Neural Network), and can fully approximate the optimal decision with the lowest cost even in complex problems with a wide state and action space.

[0050] In some embodiments of the present disclosure, a message total amount threshold is set, and a single cycle length is given; the initial production speed and consumption speed of the producer and the consumer in the initial cycle are set as: And the consumption speed Message transmission time A mathematical model P1 for adaptive adjustment of the throughput of the message middleware is constructed; the model P1 is solved by a DDPG algorithm to obtain the production speed and consumption speed decision of each period When the total production reaches a threshold, a round of training is completed; during the training, the algorithm automatically adjusts the parameters of the neural network in the direction of minimizing the overall cost of the system, and after multiple rounds of DDPG algorithm training, a neural network model that makes the overall cost smaller is obtained, which is applied to the message middleware to improve the transmission efficiency of the message, as shown in the flow Figure 2 .

[0051] In summary, the embodiments provided by the present disclosure include obtaining a message total amount threshold of a message middleware and a length of a single production or consumption period; based on the message total amount threshold of the message middleware and the length of the single production or consumption period, taking the production speed and consumption speed of each period as variables and minimizing the overall cost of the system as a target, a mathematical model is constructed; the production speed and consumption speed of an initial period and the message transmission time are set as initial values of the mathematical model; the mathematical model and the initial values are input into a deep deterministic policy gradient (DDPG) algorithm for solving to obtain the optimal production speed and optimal consumption speed of each period, so as to adjust the production speed and consumption speed of each period of the message middleware. The present scheme is based on the DDPG algorithm, takes minimizing the overall cost of the system as a target, takes the production speed and consumption speed as variables, constructs a message middleware adaptive throughput model, and dynamically adjusts the production speed of the producer and the consumption speed of the consumer in each period. Compared with the traditional algorithm, the present scheme has stronger computing power and can obtain the optimal efficiency while reducing the overall cost of the system and the possibility of message backlog is maximized when the data volume is large.

[0052] Based on Figure 1 As shown in the embodiments, as shown in Figure 3 , Figure 3 is a flowchart of another message middleware throughput adjustment method provided by the embodiments of the present disclosure.

[0053] The message middleware throughput adjustment method provided by the embodiments of the present disclosure includes the following steps:

[0054] In step 201, a message total amount threshold of a message middleware and a length of a single production or consumption period are obtained.

[0055] In step 202, based on the message total amount threshold of the message middleware and the length of the single production or consumption period, a mathematical model is constructed by taking the production speed and consumption speed of each period as variables and minimizing the overall cost of the system as a target.

[0056] In some embodiments of the present disclosure, a mathematical model is constructed based on a message middleware message total amount threshold, a length of a single production or consumption cycle, taking a production speed and a consumption speed of each cycle as variables, and taking minimizing an overall cost of the system as a goal, including: defining a cost of each cycle according to the production speed, the consumption speed of each cycle, and a transmission time delay of the message; summing the costs of all cycles to obtain the overall cost of the system; and constructing the mathematical model with the goal of minimizing the overall cost under the condition of meeting a constraint condition, wherein the constraint condition includes that the production speed of each cycle is less than or equal to a maximum value of a production speed supported by the message middleware, the consumption speed of each cycle is less than or equal to a minimum value of a consumption speed supported by the message middleware, and a total amount of messages produced in all cycles is less than or equal to the message total amount threshold.

[0057] In step 203, an initial production speed, an initial consumption speed, and an initial transmission time of the message of the initial cycle are set as initial values of the mathematical model.

[0058] The specific implementation of steps 201-203 above can refer to the principles of the embodiments implemented in steps 101-103 shown in FIG. 1, which will not be described here. Figure 1 The principles of the embodiments implemented in steps 101-103 shown in FIG. 1 are the same as those of steps 201-203 above, and will not be described here.

[0059] In step 204, the mathematical model and the initial values are input into a DDPG algorithm for multiple rounds of neural network model training until the overall cost is minimized, and a trained neural network model is obtained.

[0060] In some embodiments of the present disclosure, the DDPG algorithm has key elements such as a system state, a system action, and a system reward.

[0061] In some embodiments of the present disclosure, given a message total amount threshold P 总 and a length of a single production or consumption cycle T, i∈{0,1,2,3...} represents a production and consumption cycle. The production speed of the message middleware producer is: The consumption speed of the message middleware consumer is: i c , i∈{0,1,2,3...}.

[0062] In some embodiments of the present disclosure, S represents a current state of the entire system, represents an environmental state space. P 剩 represents a remaining producible message amount, and the remaining producible message amount is A is used to represent a system action, and the action includes a production speed of the producer a consumption speed of the consumer

[0063] In some embodiments of the present disclosure, a reward R is obtained after each action is performed at each time step. After a possible action is selected at the i-th period, a reward is obtained The reward is a negative cost, and the greater the reward, the lower the cost.

[0064] In some embodiments of the present disclosure, the DDPG algorithm is composed of an Actor policy network and a Critic value network, and the algorithm framework is as shown in Figure 4 The Actor and Critic are different in their division of labor. The agent of the Actor part calculates an action according to the neural network, and the Critic is responsible for generating the reward of the action to evaluate the goodness of the action and update the network parameters. In other words, the Actor is responsible for generating the action, and the Critic is responsible for generating the corresponding reward.

[0065] In some embodiments of the present disclosure, the Critic storage set is responsible for storing each obtained sample (S i ,A i ,R i ,S i+1 ), and then a batch of tuples are taken from it for training at each time of training, and w is updated. The loss function is defined as follows:

[0066]

[0067] where ω - is the parameter of the target-Q network in the Critic, and D represents the tuple storage set, i.e. (S i ,A i ,R i ,S i+1 ) ~ D. At the end of each iteration, the target neural network is used to calculate the loss of this iteration. In order to minimize the loss, the neural network is updated using gradient descent to update the parameters ω:

[0068]

[0069] The action output by the Action part is not a single action, but a continuous probability distribution, so the probability of all actions under a certain state should add up to 1. In the Actor, the algorithm uses gradient ascent to update the policy. The policy π(S, A) is set as:

[0070]

[0071] where θ i is the current period network policy parameter, represents the feature vector. The goal of the policy is to optimize the performance indicator, and the optimization objective function is as follows:

[0072] J(π θ )=E{Q π (S,A)}=∑ s∈S d(S)∑ α∈A π θ (A|S)Q π (S,A) Formula 9

[0073] The gradient of the policy with respect to θ is obtained using partial differential equations:

[0074]

[0075] Optimize the objective using the stochastic gradient ascent method, and update the sampled gradient θ:

[0076]

[0077] In some embodiments of this disclosure, in each round of neural network model training, the system state corresponding to the initial cycle is input into the policy network and value network of the DDPG algorithm for iterative processing, optimizing the production speed and consumption speed of each cycle, and updating the parameters of the policy network and value network with the goal of minimizing the overall cost of the system, until the remaining amount of messages that can be produced by the system is 0, determining that the system state has reached the final state, and ending the iteration; repeating multiple rounds of neural network model training to obtain a trained neural network model.

[0078] In some embodiments of this disclosure, in each iteration, the system state corresponding to the current period is input into the policy network. The policy network selects at least one determined value of the production rate and consumption rate of the current period based on the system state and the current policy to generate at least one corresponding system action. Each system action is evaluated to obtain the corresponding system reward, and the system state corresponding to the next period is obtained based on each system action. The system state, system action, system reward, and system state corresponding to the next period are combined into a tuple and stored in the storage set of the value network. A batch of tuples is randomly selected from the storage set as training samples, and the training samples are sent to the policy network and the value network for training to update the parameters of the policy network and the value network.

[0079] In some embodiments of this disclosure, training samples are fed into the policy network and the value network for training. Updating the parameters of the policy network and the value network includes: calculating the loss value of iterative processing based on the training samples and the loss function, updating the parameters of the value network using gradient descent with the goal of minimizing the loss value; and updating the parameters of the policy network using gradient ascent with the goal of optimizing the performance of the policy.

[0080] In some embodiments of this disclosure, such as Figure 4As shown, the input of the neural network model is the initial state S0 of the system, and the output is the optimized variable . The total number of training rounds is initialized as N, and the capacity of the experience pool is initialized as D; the online policy network parameter θ and the target policy network parameter θ in the Actor network are initialized - = θ, and the online Q network parameter ω and the target Q network parameter ω in the Critic network are initialized - = ω. The loop for i = 0, I is executed: S i is initialized; the Actor part selects the current action A i = π t (S θ ) + μ according to S i , calculates R i , and obtains the next state S i+1 ; (S i , A i , R i , S i+1 ) obtained is stored in D; a batch of (S j , A j , R j , S j+1 ) is randomly obtained from the experience pool as the next training sample; if S i+1 is the final state, y i = R i , otherwise: The parameter ω of the online policy network in the Critic is updated based on the above formula 6; the parameter θ of the online Q network in the Actor is updated based on the above formula 11; every C steps, ω - ←τω+(1-τ)ω - , θ - ←τθ+(1-τ)θ - ; if S i+1 is the final state, the loop is ended, otherwise, it jumps to the above Actor part according to S i , selects the current action A t = π θ (S i ) + μ, calculates R i , and obtains the next state S i+1 , and continues to execute, finally outputs the optimal and

[0081] In step 205, the trained neural network model is used to calculate the optimal production speed and optimal consumption speed of the message middleware in each cycle.​

[0082] Step 506, adjusting the actual production speed and consumption speed of the message middleware based on the optimal production speed and the optimal consumption speed.

[0083] In some embodiments of the present disclosure, after multiple rounds of training of the above-mentioned DDPG algorithm, a neural network model close to the minimum cost can be obtained, which can be applied to the message middleware to dynamically adjust the production speed and consumption speed of each cycle of the producer and the consumer, thereby improving the overall efficiency of the system.

[0084] In summary, the scheme provided by the present disclosure has the following improvements and beneficial effects:

[0085] 1. The neural network model obtained by the DDPG algorithm through a large number of messages is not limited to the type of messages and can be used in most existing message middleware, such as Kafka, RocketMq, etc., without the need for retraining in different scenarios.

[0086] 2. The adjustment method of the DDPG algorithm for each calculation result is close to a normal distribution probability model. For example, if the probability model takes a value range of -1 to 1, the decision value can be any decimal value between -1 and 1, and the granularity of decision adjustment is smaller.

[0087] 3. The DDPG algorithm trains the final neural network model through a large number of messages of different sources and different types. The neural network has strong computing power and can quickly obtain the optimal decision, which is more efficient than traditional algorithms when the data volume is large.

[0088] The present disclosure also provides a message middleware throughput adjustment device, as shown in Figure 5 The structure diagram of the message middleware throughput adjustment device provided by the present disclosure is shown in Figure 5 The message middleware throughput adjustment device 300 comprises:

[0089] The acquisition module 310 is configured to acquire the message total amount threshold of the message middleware and the time length of a single production or consumption cycle. The construction module 320 is configured to construct a mathematical model by taking the production speed and the consumption speed of each cycle as variables and taking the minimization of the overall cost of the system as the target based on the message total amount threshold of the message middleware and the time length of a single production or consumption cycle. The setting module 330 is configured to set the production speed, the consumption speed of the initial cycle and the transmission time of the message as the initial values of the mathematical model. The solving module 340 is configured to input the mathematical model and the initial values into the deep deterministic policy gradient (DDPG) algorithm to solve the optimal production speed and the optimal consumption speed of each cycle, so as to adjust the production speed and the consumption speed of each cycle of the message middleware.

[0090] In some embodiments of the present disclosure, the construction model 320 is specifically configured to: define the cost of each period according to the production speed, the consumption speed and the transmission delay of the message of each period; sum the costs of all periods to obtain the overall cost of the system; and construct a mathematical model with the minimum overall cost as the target under the condition that the constraints are met, wherein the constraints include that the production speed of each period is less than or equal to the maximum value of the production speed supported by the message middleware, the consumption speed of each period is less than or equal to the minimum value of the consumption speed supported by the message middleware, and the total amount of messages produced in all periods is less than or equal to the message total amount threshold.

[0091] In some embodiments of the present disclosure, the solving module 340 is specifically configured to: input the mathematical model and the initial value into the DDPG algorithm for multiple rounds of neural network model training until the overall cost is minimized to obtain a trained neural network model; calculate the optimal production speed and the optimal consumption speed of the message middleware in each period by using the trained neural network model; and adjust the real production speed and the consumption speed of the message middleware based on the optimal production speed and the optimal consumption speed.

[0092] In some embodiments of the present disclosure, the solving module 340 is further configured to: define the remaining producible message amount of the system as the system state, and determine the system state corresponding to the initial period based on the initial value and the message total amount threshold; in each round of neural network model training, input the system state corresponding to the initial period into the policy network and the value network of the DDPG algorithm for iterative processing, optimize the production speed and the consumption speed of each period, and update the parameters of the policy network and the value network with the minimum overall cost of the system as the target until the remaining producible message amount of the system is 0, determine that the system state reaches the final state, and end the iteration; and repeat the multiple rounds of neural network model training to obtain the trained neural network model.

[0093] In some embodiments of the present disclosure, the system state corresponding to the initial period is input into the policy network and the value network of the DDPG algorithm for iterative processing, the production speed and the consumption speed of each period are optimized, and the parameters of the policy network and the value network are updated with the minimum overall cost of the system as the target, comprising: in each iteration, the system state corresponding to the current period is input into the policy network, and at least one determined value of the production speed and the consumption speed of the period is selected to generate a corresponding at least one system action according to the system state corresponding to the current period and the current policy by using the policy network; each system action is evaluated to obtain a corresponding system reward, and the system state corresponding to the next period is obtained based on each system action; the system state corresponding to the current period, the system action, the system reward, and the system state corresponding to the next period are combined into a tuple and stored in a storage set of the value network; a batch of tuples are randomly selected from the storage set as training samples, and the training samples are sent into the policy network and the value network for training to update the parameters of the policy network and the value network.

[0094] In some embodiments of the present disclosure, sending the training samples into the policy network and the value network for training to update the parameters of the policy network and the value network comprises: calculating the loss value of the iteration according to the training samples and the loss function, updating the parameters of the value network by using the gradient descent method with the minimum loss value as the target; and updating the parameters of the policy network by using the gradient ascent method with the performance of the policy as the target.

[0095] In summary, the message middleware throughput adjustment device provided by the present disclosure is based on the DDPG algorithm, takes the minimum overall cost of the system as the target, takes the production speed and the consumption speed as variables, constructs a message middleware adaptive throughput model, and dynamically adjusts the production speed of the producer and the consumption speed of the consumer in each period by using the model. Compared with the traditional algorithm, the message middleware throughput adjustment device has stronger computing ability, can obtain the optimal efficiency while reducing the overall cost of the system when the data volume is large, and maximizes the possibility of reducing the message backlog.

[0096] The message middleware throughput adjustment device provided by the above embodiments and the message middleware throughput adjustment method embodiment provided by the present disclosure belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described here.

[0097] Figure 6 The hardware composition structure diagram of the communication device provided by the embodiments of the present disclosure is as follows: Figure 6As shown, the communication device 400 comprises at least one processor 402; and a memory 401 connected with the at least one processor 402 in communication; wherein the memory 401 stores instructions executable by the at least one processor 402, the instructions are executed by the at least one processor 402 to implement the steps of the message middleware throughput adjustment method according to the embodiments of the present disclosure; or the instructions are executed by the at least one processor 402 to implement the steps of the message middleware throughput adjustment method according to the embodiments of the present disclosure.

[0098] Optionally, the communication device can be specifically a control device of the embodiments of the present application, and the communication device can implement the corresponding processes implemented by the control device in various methods of the embodiments of the present application. For the sake of brevity, details are not described herein.

[0099] It can be understood that the communication device further comprises a communication interface 403. The various components in the communication device are coupled together through a bus system 404. It can be understood that the bus system 404 is used to realize the connection communication between the components. The bus system 404 includes not only a data bus, but also a power supply bus, a control bus and a status signal bus. However, for the sake of clear illustration, all kinds of buses are marked as the bus system 404 in the Figure 6 for the sake of clear illustration, all kinds of buses are marked as the bus system 404 in the

[0100] It can be understood that the memory 401 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 401 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable type of memory.

[0101] The method disclosed in the embodiments of the present application can be applied to the processor 402 or implemented by the processor 402. The processor 402 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 402 or the instruction in the form of software. The processor 402 described above can be a general processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 402 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the hardware decoding processor can be directly embodied to execute the above method, or the combination of hardware and software modules in the decoding processor can be executed. The software module can be located in the storage medium, which is located in the memory 401, and the processor 402 reads the information in the memory 401 to complete the above method in combination with the hardware.

[0102] In the exemplary embodiments, the communication device can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs, PLDs (Programmable Logic Devices), CPLDs (Complex Programmable Logic Devices), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic elements, for executing the above method.

[0103] The embodiments of the present application also provide a non-transitory computer readable storage medium storing computer instructions, which are used to make the computer execute the steps of the message middleware throughput adjustment method disclosed in the embodiments of the present application; or the computer instructions are used to make the computer execute the steps of the message middleware throughput adjustment method disclosed in the embodiments of the present application.

[0104] Optionally, the computer readable storage medium can be applied to the control device in the embodiments of the present application, and the computer instructions make the computer execute the corresponding processes implemented by the control device in the various methods of the embodiments of the present application. For brevity, the details are not described here.

[0105] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0106] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or distributed on a plurality of network units; part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0107] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0108] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage devices, ROM, RAM, magnetic discs or optical discs, and various media that can store program codes.

[0109] Alternatively, the integrated unit of the present application, if implemented in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROM, RAM, magnetic discs or optical discs, and various media that can store program codes.

[0110] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of adjusting throughput of a message middleware, the method comprising: The method comprises: acquiring a message total amount threshold of a message middleware, and a time length of a single production or consumption cycle; constructing a mathematical model based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle, taking the production speed and the consumption speed of each cycle as variables, and taking the minimization of the overall cost of the system as the target, wherein the overall cost of the system is obtained by summing the costs of all cycles, and the cost of each cycle is defined based on the production speed, the consumption speed and the transmission time delay of the message of each cycle; setting the production speed, the consumption speed and the transmission time of the message of an initial cycle as initial values of the mathematical model; inputting the mathematical model and the initial values into a deep deterministic policy gradient (DDPG) algorithm to solve, and obtaining the optimal production speed and the optimal consumption speed of each cycle to adjust the production speed and the consumption speed of each cycle of the message middleware.

2. The method of claim 1, wherein, The constructing of the mathematical model based on the message total amount threshold of the message middleware and the time length of the single production or consumption cycle, taking the production speed and the consumption speed of each cycle as variables, and taking the minimization of the overall cost of the system as the target comprises: building the mathematical model with the minimization of the overall cost as the target under the condition of meeting a constraint condition, wherein the constraint condition comprises that the production speed of each cycle is less than or equal to the maximum value of the production speed supported by the message middleware, the consumption speed of each cycle is less than or equal to the maximum value of the consumption speed supported by the message middleware, and the total amount of the messages produced in all cycles is less than or equal to the message total amount threshold.

3. The method of claim 1, wherein, The inputting of the mathematical model and the initial values into the DDPG algorithm to solve, and the obtaining of the optimal production speed and the optimal consumption speed of each cycle to adjust the production speed and the consumption speed of each cycle of the message middleware comprise: inputting the mathematical model and the initial values into the DDPG algorithm for multiple rounds of neural network model training until the overall cost is minimized, to obtain a trained neural network model; calculating the optimal production speed and the optimal consumption speed of the message middleware in each cycle by using the trained neural network model; adjusting the real production speed and the consumption speed of the message middleware based on the optimal production speed and the optimal consumption speed.

4. The method of claim 3, wherein, The inputting of the mathematical model and the initial values into the DDPG algorithm for multiple rounds of neural network model training until the overall cost is minimized, to obtain a trained neural network model comprises: defining the remaining producible message amount of the system as a system state, and determining the system state corresponding to the initial cycle based on the initial values and the message total amount threshold. In each round of the neural network model training, the system state corresponding to the initial period is input into a policy network and a value network of the DDPG algorithm for iterative processing, the production speed and the consumption speed of each period are optimized, and parameters of the policy network and the value network are updated with the goal of minimizing the overall cost of the system until the remaining producible message amount of the system is 0, the system state reaches a final state, and the iteration ends; The neural network model training is repeatedly performed for multiple rounds to obtain a trained neural network model.

5. The method of claim 4, wherein, The inputting of the system state corresponding to the initial period into the policy network and the value network of the DDPG algorithm for iterative processing, the optimization of the production speed and the consumption speed of each period, and the updating of the parameters of the policy network and the value network with the goal of minimizing the overall cost of the system include: In each iteration, the system state corresponding to the current period is input into the policy network, and at least one system action corresponding to at least one determined value of the production speed and the consumption speed of the period is generated by the policy network according to the system state corresponding to the current period and the current policy; Each system action is evaluated to obtain a corresponding system reward, and the system state corresponding to the next period is obtained based on each system action; A tuple composed of the system state corresponding to the current period, the system action, the system reward, and the system state corresponding to the next period is stored in a storage set of the value network; A batch of tuples are randomly selected from the storage set as training samples, and the training samples are sent into the policy network and the value network for training to update the parameters of the policy network and the value network.

6. The method of claim 5, wherein, The sending of the training samples into the policy network and the value network for training to update the parameters of the policy network and the value network includes: The loss value of the iteration is calculated according to the training samples and a loss function, the parameters of the value network are updated by using a gradient descent method with the goal of minimizing the loss value; The parameters of the policy network are updated by using a gradient ascent method with the goal of optimizing the performance of the policy network.

7. A message middleware throughput adjustment apparatus, characterized by, The device includes: An acquisition module is configured to acquire a message total amount threshold of a message middleware and a time length of a single production or consumption period; A model construction module is configured to construct a mathematical model by taking the production speed and the consumption speed of each period as variables and minimizing the overall cost of a system as a goal based on the message total amount threshold of the message middleware and the time length of the single production or consumption period, wherein the overall cost of the system is obtained by summing the costs of all the periods, and the cost of each period is defined based on the production speed, the consumption speed, and the message transmission delay of each period; A setting module is configured to set the production speed, the consumption speed, and the message transmission time of an initial period as initial values of the mathematical model. A solving module is configured to input the mathematical model and the initial value into a deep deterministic policy gradient (DDPG) algorithm to solve the mathematical model and obtain optimal production speed and optimal consumption speed of each period, so as to adjust the production speed and the consumption speed of the message middleware in each period.

8. A communication device, characterized by Comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1 to 6.

10. A computer program product, characterised in that, A computer program is included, and the computer program, when executed by a processor, implements the method of any one of claims 1 to 6.

Images