Method and device for determining flow of message queue and queue flow determination system

Abstract

The invention provides a message queue traffic determination method and device and a queue traffic determination system. The method comprises the following steps: acquiring first data at a first moment, the first data being data of a log of an MQ message queue; according to the first data, determining first traffic at a second moment by adopting an ARIMA algorithm; at least according to the first data, a Prophet algorithm is adopted to determine second traffic at a second moment; fusing the first traffic and the second traffic to obtain target traffic at the second moment; and under the condition that the target traffic is greater than the traffic threshold value, determining to add a server and / or add a queue. According to the method, the problem of relatively low accuracy of predicting the flow of the message queue in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of traffic prediction, and in particular, to a method, device, computer program product, and queue traffic determination system for determining the traffic of a message queue. Background Technique

[0002] Currently, message queues are the mainstream way to achieve communication and data transmission between systems. With advantages such as high performance, asynchrony, and business irrelevance, MQ has always been the preferred tool component in the construction of large-scale system platforms.

[0003] MQ (Message Queue) is essentially a queue, and each message in the queue is stored. A message queue consists of a Broker (message server, the core part), a Producer (message producer), a Consumer (message consumer), a Topic (topic), a Queue (queue), and a Message (message body). It uses an efficient and reliable message passing mechanism for platform-independent data communication and is a container for storing messages during the message transmission process. However, in large and complex systems, the data volume can reach the million, ten million, or even hundreds of millions level.

[0004] Therefore, it is necessary to predict the traffic of the message queue to facilitate developers to cope with the impact of traffic. However, the accuracy of predicting the traffic of the message queue in the current solutions is relatively low. Summary of the Invention

[0005] The main purpose of the present application is to provide a method, device, computer program product, and queue traffic determination system for determining the traffic of a message queue, so as to at least solve the problem of relatively low accuracy in predicting the traffic of the message queue in the prior art.

[0006] To achieve the above object, according to one aspect of the present application, a method for determining the traffic of a message queue is provided, including: obtaining first data at a first moment, where the first data is the data of the log of the MQ message queue; according to the first data, using the ARIMA algorithm to determine the first traffic at a second moment, where the second moment is later than the first moment, and the first traffic is the number of the MQ message queue determined by the ARIMA algorithm; at least according to the first data, using the Prophet algorithm to determine the second traffic at the second moment, where the second traffic is the number of the MQ message queue determined by the Prophet algorithm; fusing the first traffic and the second traffic to obtain the target traffic at the second moment, where the fusion method at least includes weighted average, and the target traffic is used to guide the target object to adjust resources to cope with the traffic peak; in the case where the target traffic is greater than the traffic threshold, determining to increase the server and / or increase the queue.

[0007] Optionally, using the ARIMA algorithm to determine the first traffic at the second moment according to the first data includes: constructing a first model, where the first model is trained by the ARIMA algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical first data obtained within a historical time period and the historical first traffic corresponding to the historical first data; inputting the first data into the first model to obtain the first traffic corresponding to the first data.

[0008] Optionally, at least according to the first data, using the Prophet algorithm to determine the second traffic at the second moment includes: obtaining a preset time point, where the preset time point is the time point of a holiday and / or a special date, and the special date at least includes one or more of a shopping festival, a year-end settlement date, and a meeting date; adding the preset time point to the first data to obtain second data; using the Prophet algorithm to determine the second traffic at the second moment according to the second data.

[0009] Optionally, using the Prophet algorithm to determine the second traffic at the second moment according to the second data includes: constructing a second model, where the second model is trained by the Prophet algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical second data obtained within a historical time period and the historical second traffic corresponding to the historical second data; inputting the second data into the second model to obtain the second traffic corresponding to the second data.

[0010] Optionally, before fusing the first traffic and the second traffic to obtain the target traffic at the second moment, the method further includes: obtaining a third traffic, where the third traffic is the traffic of the MQ message queue at a third moment determined according to the first data by using the ARIMA algorithm, the third moment is later than the first moment and earlier than the second moment; obtaining a fourth traffic, where the fourth traffic is the traffic of the MQ message queue at the third moment determined according to the first data by using the Prophet algorithm; obtaining the actual traffic at the third moment; calculating a first difference between the actual traffic and the third traffic, and calculating a second difference between the actual traffic and the fourth traffic; determining a first weight coefficient of the first traffic according to the first difference, and determining a second weight coefficient of the second traffic according to the second difference, where there is a negative correlation between the first difference and the first weight coefficient, and there is a negative correlation between the second difference and the second weight coefficient.

[0011] Optionally, after fusing the first traffic and the second traffic to obtain the target traffic at the second moment, the method further includes: generating a first adjustment strategy when the target traffic is greater than a first traffic threshold, where the first adjustment strategy is a strategy for increasing the queue; generating a second adjustment strategy when the target traffic is greater than a second traffic threshold, where the second traffic threshold is a strategy for adding a server, and the first traffic threshold is less than the second traffic threshold.

[0012] Optionally, obtaining the first data at the first moment includes: obtaining the initial first data at the first moment; preprocessing the initial first data to obtain the first data, where the preprocessing method includes at least one or more of removing outliers, filling in missing values, and format normalization.

[0013] According to another aspect of the present application, a device for determining the traffic of a message queue is provided, including: a first acquisition unit configured to acquire first data at a first moment, where the first data is the data of the log of the MQ message queue; a first determination unit configured to determine the first traffic at a second moment according to the first data by using the ARIMA algorithm, where the second moment is later than the first moment, and the first traffic is the number of the MQ message queues determined by using the ARIMA algorithm; a second determination unit configured to determine the second traffic at the second moment at least according to the first data by using the Prophet algorithm, where the second traffic is the number of the MQ message queues determined by using the Prophet algorithm; a fusion unit configured to fuse the first traffic and the second traffic to obtain the target traffic at the second moment, where the fusion method includes at least weighted average, and the target traffic is used to guide the target object to adjust resources to cope with the traffic peak; a third determination unit configured to determine to increase the server and / or increase the queue when the target traffic is greater than the traffic threshold.

[0014] According to still another aspect of the present application, a computer program product is provided, including a computer program, where when the computer program is executed by a processor, the steps of any one of the methods for determining the traffic of the message queue are implemented.

[0015] According to yet another aspect of the present application, a queue traffic determination system is provided, including: one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include those for executing any one of the methods for determining the traffic of the message queue.

[0016] Applying the technical solution of the present application, both the ARIMA algorithm and the Prophet algorithm are data prediction algorithms. The ARIMA algorithm can be used to process the time series characteristics of data and can accurately predict the long-term trend of data, while the Prophet algorithm is particularly good at dealing with prediction problems with seasonality, trendiness, and holiday effects and can automatically detect and adapt to the periodic changes in data. This solution combines the prediction results of these two algorithms, and the obtained target traffic can combine the advantages of the two algorithms, which can ensure a relatively high accuracy of the obtained prediction results. Description of the Drawings

[0017] The specification drawings constituting a part of the present application are used to provide a further understanding of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0018] Figure 1The hardware structure block diagram of a mobile terminal showing a method for determining the traffic of a message queue provided in an embodiment of the present application is shown;

[0019] Figure 2 The flowchart showing a method for determining the traffic of a message queue provided in an embodiment of the present application is shown;

[0020] Figure 3 The flowchart showing another method for determining the traffic of a message queue is shown;

[0021] Figure 4 The flowchart showing the calculation of adaptive weights is shown;

[0022] Figure 5 The structure block diagram of a device for determining the traffic of a message queue provided in an embodiment of the present application is shown.

[0023] Wherein, the above-mentioned drawings include the following reference numerals:

[0024] 102, processor; 104, memory; 106, transmission device; 108, input / output device. Detailed implementation manners

[0025] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the protection scope of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data may be interchanged under appropriate circumstances for the embodiments of the present application described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of description, some nouns or terms related to the embodiments of the present application are described below:

[0029] Prophet: An algorithm for time series prediction that can automatically detect trends and seasonality in data, combine them to obtain prediction values, and is a completely open-source algorithm.

[0030] ARIMA: Full name is Autoregressive Integrated Moving Average Model. The ARIMA model mainly consists of three parts. The basic idea is to use the historical information of the data itself to predict the future, and it is a completely open-source model.

[0031] MQ: Message Queue, (Messges Queue) refers to using an efficient and reliable message passing mechanism for platform-independent data communication and integrating distributed systems based on data communication. It is a container for storing messages during the message transmission process.

[0032] In the current solution, during the process of developers building a system, they need to face how to improve the user access quality, that is, reduce the response time of the system as much as possible. More importantly, it is necessary to ensure that the system still runs stably without crashing and the data is complete without loss under the condition of high concurrency with a large amount of data. Especially in the construction of systems with a data volume of millions or tens of millions, this problem is particularly prominent. This requires introducing a method for traffic prediction and traffic peak shaving to meet the above requirements.

[0033] As introduced in the background technology, the accuracy of predicting the traffic of message queues in the prior art is relatively low. To solve the above problems, the embodiments of the present application provide a method, device, computer program product, and queue traffic determination system for determining the traffic of message queues.

[0034] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention.

[0035] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is the hardware structure block diagram of a mobile terminal for a method of determining the traffic of a message queue according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1Only one processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are shown. Among them, the above mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 The structure shown is only illustrative and does not limit the structure of the above mobile terminal. For example, the mobile terminal may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 that shown in the figure.

[0036] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and their combinations. The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] In this embodiment, a method for determining the traffic of a message queue running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0038] Figure 2 is a flowchart diagram of a method for determining the traffic of a message queue according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0039] Step S201, obtain the first data at the first moment, where the first data is the data of the log of the MQ message queue;

[0040] Specifically, collect the log data of the MQ message queue at the first moment through the system log collection mechanism. These data include message time, system type, file type, file size, and any other metrics that may affect the message queue traffic. Ensure the accuracy of the historical data required for the prediction model training and validation, and provide the basic input for the ARIMA and Prophet algorithms.

[0041] Step S202, according to the first data, use the ARIMA algorithm to determine the first traffic at the second moment, where the second moment is later than the first moment, and the first traffic is the number of the MQ message queue determined by the ARIMA algorithm;

[0042] Specifically, use the ARIMA algorithm to perform time series analysis on the collected first data, and predict the traffic of the MQ message queue at the second moment (future moment), that is, predict the number of messages. The ARIMA algorithm can effectively capture the trend changes in the time series, and improve the adaptability of the prediction model to the traffic trend.

[0043] Step S203, at least according to the first data, use the Prophet algorithm to determine the second traffic at the second moment, where the second traffic is the number of the MQ message queue determined by the Prophet algorithm;

[0044] Specifically, in addition to the ARIMA algorithm, also use the Prophet algorithm, which is especially suitable for processing data with seasonal and periodic characteristics, and predict the number of the MQ message queue at the second moment. The high sensitivity of the Prophet algorithm to seasonal changes makes the prediction results more accurate, can accurately capture the seasonal peaks, and reduce the prediction error.

[0045] Step S204, fuse the first traffic and the second traffic to obtain the target traffic at the second moment, where the fusion method includes at least weighted average, and the target traffic is used to guide the target object to adjust resources to cope with the traffic peak;

[0046] Specifically, through the weighted average method, synthesize the prediction results of ARIMA and Prophet to form the final target traffic prediction value, which will be used as the basis for system resource adjustment. The weighted average fusion method takes into account the prediction accuracy of the two algorithms, improves the reliability of traffic prediction, and ensures the reasonable utilization of system resources during peak periods.

[0047] According to the target traffic forecast value, server resource allocation is automatically or semi-automatically adjusted, including adding servers, opening temporary queues or quasi-real-time queues, to ensure that the system runs stably when the traffic peak is predicted. The resource adjustment strategy in advance avoids system crashes and delays during traffic peaks, and improves the overall system response speed and user experience.

[0048] Of course, in addition to the above-mentioned weighted average, fusion can also be performed in the following ways, such as: average fusion, median fusion, Bayesian fusion, etc.

[0049] Step S205: When the target flow is greater than the flow threshold, determine to add a server and / or a queue.

[0050] Specifically, if the predicted target traffic obtained by the fusion algorithm exceeds the pre-set traffic threshold (e.g. 1,500,000 messages / hour), the system will trigger the automatic expansion mechanism to increase server resources or queue resources to cope with the upcoming traffic peak. By setting the traffic threshold, the system can automatically identify high-risk traffic scenarios and adjust resources in a timely manner to avoid message loss or system failures caused by excessive traffic, thereby enhancing the system's stress resistance and data integrity.

[0051] Through this embodiment, the ARIMA algorithm and the Prophet algorithm are both data prediction algorithms. The ARIMA algorithm can be used to process the time series characteristics of data and can accurately predict the long-term trend of data, while the Prophet algorithm is particularly good at processing prediction problems with seasonality, trend and holiday effects, and can automatically detect and adapt to periodic changes in data. This solution combines the prediction results of these two algorithms, and the obtained target traffic can combine the advantages of the two algorithms, which can ensure that the obtained prediction results have a high accuracy rate.

[0052] Specifically, it is necessary to improve the quality of user access, ensure that producer and consumer messages are accurate, and reduce the response time of the system. How to ensure that the system can still run stably and not crash in the case of large amounts of data and high concurrency, while ensuring that the data is intact and not lost, is a difficult problem for every system developer. Usually, an effective method is to increase software and hardware resources, such as the number of queues and the number of hardware servers, to cope with the impact of peak data traffic. However, it has brought about a significant increase in hardware costs. To address such problems, this solution proposes a system peak traffic prediction method based on the ARIMA algorithm and the Prophet algorithm. Through the prediction results, it is possible to adjust the system resource allocation in a targeted manner before the peak traffic arrives to ensure system response time and data is not lost.

[0053] Existing traffic prediction algorithms can generally be divided into two categories: one is the model based on statistical methods. Such as Kalman filtering theory (KF), adaptive spatio-temporal K-nearest neighbor model (adaptive-STKNN), and autoregressive integrated moving average method (ARIMA), which comprehensively consider factors such as the number of adaptive neighbors, time window size, and spatio-temporal weights. The other is the model based on deep learning. Such as using BP neural network to estimate traffic.

[0054] In the existing time series prediction methods, there are more or less drawbacks in various model algorithms, but a suitable method for the applicable scenario can always be found in different situations. Some existing solutions fully consider the problems of special scenarios and propose methods of supplementary models and the first threshold to optimize the model and improve the effect and accuracy. However, they cannot overcome the disadvantages of slow self-optimization speed, poor real-time performance, and low generalization ability of neural networks. Some existing solutions use the non-linear optimization method of support vector machine (SVM) which is comprehensive and has good generalization. It has good performance for small sample data; different data situations can be adapted by choosing kernel functions. Some existing solutions use the ARIMA model for flight traffic prediction, which well adapts to the characteristics of large data volume, complexity, and diversity. For weather and seasonal changes, a regression equation for weather delays is separately constructed, and then the advantages of the function are analyzed, and finally a correct prediction of future values is made. However, this method has very limited effect in making up for the shortcoming of the seasonality and long-term trend of ARIMA itself.

[0055] Therefore, the solution of this application proposes a method for predicting the system traffic peak by combining the ARIMA algorithm and the Prophet algorithm. Prophet has good performance in prediction problems with complex characteristics such as seasonality, trend, and holidays, so as to achieve higher prediction accuracy, shorten the usage time, and make effective predictions.

[0056] The solution of this application is to predict the peak value of the number of messages that MQ will receive or process at a certain future time point or time period. Such prediction is particularly important because it directly affects how the system prepares and allocates resources to cope with possible high-load situations.

[0057] The traffic of the message queue (MQ) refers to the number of messages or the total size of messages transmitted through the message queue system within a specific time period. In a distributed system, the message queue, as middleware for communication between different services or components, undertakes the tasks of message storage, forwarding, and processing.

[0058] Specifically, the processing flow of this solution for predicting the system peak traffic in the scenarios of large data volume, complexity, diversity, and seasonal changes is as Figure 3As shown, collect the MQ logs in the project system, collect the data set, and collect the log MQ data stored in the system logs in the specified format, including at least the message time, the type of the affiliated system (business, core, middleware, etc.), the file type, the file size, and the delay time information; as follows:

[0059] 1) The obtained metric is: istio_requests_total. The condition for filtering when querying using the API is: kubernetes_namespace = ccs - online. The obtained metric data is classified according to the response code: 200 is <successful request>, and other response codes are <failed request>. The sum of successful + failed requests is <total requests>.

[0060] The metric to be obtained is istio_requests_total, which is a metric related to service calls. Usually in a microservices architecture, such as when using Istio service mesh, it is used to monitor the total number of calls between services. The filtering condition is kubernetes_namespace = ccs - online, which means that the service call logs under the Kubernetes namespace named ccs - online are to be concerned about.

[0061] Through API query, the obtained istio_requests_total data will be classified according to the response code. A 200 response code indicates a successful request, and other response codes indicate a failed request. In this way, the original data is divided into two parts: successful requests and failed requests, and then the total number of requests can be calculated.

[0062] 2) Scheduling tasks: Execute tasks every 30s, 1min, 5min, and 10min respectively. Since istio_requests_total is a linearly growing metric, currently, at specified time intervals, the difference between the metrics obtained in the previous and subsequent acquisitions is used to obtain the incremental data as the total call volume within the specified time.

[0063] The solution designs periodic data collection every 30 seconds, 1 minute, 5 minutes, and 10 minutes. The selection of these time points is to capture traffic changes at different frequencies, facilitating subsequent analysis and the construction of prediction models.

[0064] 3) Acquisition method: Prometheus API, pull method.

[0065] Use the Prometheus API for data collection. This is a data collection method of the pull type, which means that the system actively pulls data from the Prometheus server instead of the Prometheus server pushing data.

[0066] 4) Raw acquisition data storage: The raw call volume data collected within the interval is stored in InfluxDB to obtain dataset 1.

[0067] The collected raw call volume data will be stored in InfluxDB, a time - series database that is very suitable for storing and querying large amounts of timestamp - tagged data.

[0068] Dataset pre - processing: The data is classified into different datasets according to message intervals of 30s, 1min, 5min, and 10min respectively to complete data pre - processing. In this step of dataset pre - processing, all the collected MQ message data will be further classified and sorted according to the message intervals (30s, 1min, 5min, 10min) to form multiple datasets differentiated by time intervals. This helps to more carefully observe the traffic patterns at different time scales during subsequent analysis. Pre - processing also includes cleaning the data, that is, removing any outliers, missing values, or data points that do not conform to the format to ensure the integrity and accuracy of the data.

[0069] In the specific implementation process, according to the above - mentioned first data, the ARIMA algorithm is used to determine the first traffic at the second moment, which can be achieved through the following steps: construct the first model, where the first model is trained using multiple sets of training data through the ARIMA algorithm, and each set of training data in the multiple sets of training data includes historical first data obtained within a historical time period and the historical first traffic corresponding to the historical first data; input the first data into the first model to obtain the first traffic corresponding to the first data.

[0070] In this solution, the advantage of the ARIMA model is that it can handle non - stationary data, model trends and periodicity, and can better handle non - stationary data, further improving the accuracy of prediction.

[0071] Specifically, construct an ARIMA model. The establishment of the ARIMA model includes a moving - average process MA, an autoregressive process AR, an autoregressive moving - average process ARMA, and an ARIMA process. Verify the dataset for the established ARIMA model and select the best parameters to obtain prediction value 1.

[0072] The parameters of the ARIMA model include the autoregressive term p, the differencing order d, and the moving - average term q. These parameters need to be determined according to the characteristics of the dataset. The differencing order d is used to make the data stationary, while the autoregressive term p and the moving - average term q are used to capture the historical dependence of the data.

[0073] Use dataset 1 (MQ data stored in the collected system logs in the specified format information) to train the ARIMA model. During the training process, the model will learn the time series characteristics of the data, including its trends, seasonality, and random fluctuations.

[0074] After the training is completed, the model needs to evaluate its prediction ability through dataset validation to determine the accuracy and reliability of the model. The validation steps include using a part of the data for prediction, then comparing the prediction results with the actual values, calculating the prediction errors, and adjusting the model parameters until the model reaches a satisfactory prediction accuracy.

[0075] Finally, the model can be used to predict the future system traffic, and output the prediction value 1, that is, the traffic peak data predicted based on the ARIMA model.

[0076] This embodiment fully considers the characteristics of historical traffic information and uses the ARIMA algorithm to construct a prediction model that can capture data trends and periodicity. When constructing the first model, by analyzing and validating the historical dataset, the optimal ARIMA parameter combination is selected, thus ensuring the accuracy and reliability of the prediction model. Inputting the traffic data at the current time point (the first moment) into the model can predict the system traffic at the future time point (the second moment). This process provides a crucial first step for traffic prediction, and through the accurate prediction of the ARIMA model, it can provide an accurate prediction basis for the reasonable allocation of system resources, thus achieving the purpose of resource optimization and traffic peak shaving, and improving the stability and data processing efficiency of the system. In terms of traffic prediction, through the training of the ARIMA model in the present invention, it can effectively capture the trend changes in the time series and improve the adaptability of the prediction model to traffic trends.

[0077] Before the e-commerce big promotion, the system operation and maintenance team uses the MQ message queue data during the historical peak period as the training set, and constructs a prediction model through the ARIMA algorithm to predict the traffic peak of the upcoming Double Eleven shopping festival. Each set of training data contains the number of MQ messages per minute (historical first data) and the corresponding traffic (historical first traffic) during Double Eleven in the past few years. Through this process, the operation and maintenance team can have a more accurate prediction of the upcoming traffic peak, so as to deploy and optimize resources in advance.

[0078] Based on the constructed ARIMA model, the operation and maintenance team can monitor the system MQ message queue data (the first data) at the current moment in real time and input it into the model to predict the traffic (the first traffic) at the next moment (for example, the next 10 minutes). For example, when the current moment is 1 minute before the start of the Double Eleven Shopping Festival, by inputting the MQ message queue data at this time into the model, the traffic 10 minutes after the start of the shopping festival can be predicted. Assuming that historical data shows that the traffic peak 10 minutes after the start of Double Eleven in the previous year was 1,200,000 messages per minute, the operation and maintenance team can set a traffic threshold, such as 1,500,000 messages / minute, to trigger the mechanism for resource increase or queue adjustment.

[0079] In some embodiments, at least based on the above first data, the Prophet algorithm is used to determine the second traffic at the above second moment, which can be specifically implemented through the following steps: obtain a preset time point, where the above preset time point is the time point of a holiday and / or a special date, and the above special date includes at least one or more of a shopping festival, a year-end settlement date, and a conference date; add the above preset time point to the above first data to obtain second data; according to the above second data, use the Prophet algorithm to determine the above second traffic at the above second moment.

[0080] In this solution, the Prophet algorithm introduces the recognition of holidays and special dates in the prediction, which enables the prediction to more accurately capture the impact of non-working days or special events on traffic, and further improves the prediction accuracy.

[0081] Specifically, add sample data of special holidays and special dates to the data set, such as data during year-end settlement, opening of the year, Double Eleven, 618 Shopping Festival, and important domestic conferences, to obtain data set 2.

[0082] First, collect the MQ traffic data during special holidays or dates (such as shopping festivals, year-end settlements, and important conferences). These data usually reflect the traffic surges or decreases at these time points due to factors such as user behavior, market demand, and emergencies.

[0083] Merge the collected additional data into the existing data set (data set 1) to form data set 2. This step ensures that the model can consider the traffic patterns at special time points during training.

[0084] Perform necessary preprocessing on the traffic data of special holidays or dates, such as normalization, with the aim of matching these data with the regular data in scale and avoiding biases caused by differences in data magnitudes during model training.

[0085] The prediction process needs to be based on a dataset that contains a 'ds' column. The Prophet.make_future_dataframe method extends the dataset dates by a specified number of days to obtain a compliant data frame. The predict method will get a predicted value (referred to as yhat) for each future date row. If the dates of historical data are passed in, it will provide the model fitting values for the samples. The object created by forecast is a new data frame that contains a column of predicted values yhat, as well as component analysis and confidence intervals, ultimately obtaining the predicted value 2.

[0086] Ensure that dataset 2 (the enhanced dataset that includes data on special holidays or dates) meets the requirements of the Prophet algorithm. The Prophet algorithm requires data to be input in a specific format, namely a data frame that contains two columns: date (ds) and observation (y).

[0087] Use the Prophet's make_future_dataframe method to create a future time frame that will contain all the dates within the prediction range. For example, if we want to predict the traffic for the next 30 days, then make_future_dataframe will generate a sequence of dates for the next 30 days.

[0088] Use the Prophet algorithm to train the model and apply the predict method to predict future traffic. Prophet will automatically detect trends and seasonality in the data and take into account the holiday effect, outputting the predicted value (yhat) and the confidence interval. If dataset 2 contains past data, Prophet will also provide the predicted values of the model fitting for validating the effectiveness of the model.

[0089] This embodiment can achieve accurate prediction of traffic changes during holidays and special dates. For example, during the Double Eleven shopping festival, the system traffic will suddenly surge, and the predicted second traffic may reach a peak of 5,000,000 messages per minute, which far exceeds the daily traffic level. Through the Prophet algorithm, not only can this traffic peak be predicted, but also the seasonal changes and future trends of the traffic can be analyzed, providing key information for system maintenance and resource allocation to avoid system crashes or slow responses during peak traffic periods. In addition, the prediction results of the Prophet algorithm can also provide data support for the enterprise's marketing strategies and operation planning. By setting a reasonable threshold, such as taking 5,000,000 messages per minute as the warning threshold, the enterprise can take measures before the traffic reaches this threshold to ensure the stable operation of the system and the timely processing of data.

[0090] In the specific implementation process, according to the above-mentioned second data, the Prophet algorithm is used to determine the above-mentioned second traffic at the above-mentioned second moment, which can be achieved through the following steps: construct a second model, where the above-mentioned second model is trained by the Prophet algorithm using multiple sets of training data, and each set of training data in the above-mentioned multiple sets of training data includes historical second data obtained within a historical time period and the historical second traffic corresponding to the above-mentioned historical second data; input the above-mentioned second data into the above-mentioned second model to obtain the above-mentioned second traffic corresponding to the above-mentioned second data.

[0091] In this solution, more accurate predictions can be made for traffic data including holidays and special dates (such as data during year-end settlement, opening of the year, Double Eleven, 618 Shopping Festival, and important domestic conferences). The Prophet algorithm performs well in dealing with time series data with seasonality, trend, and holiday effects, and can effectively capture the impact of these special dates on traffic, further improving the prediction accuracy.

[0092] Specifically, targeted data for legal holidays or special dates is added to the dataset, and after normalization processing, the above-mentioned dataset will be used as the training data for the Prophet algorithm. When constructing the second model, these training data including historical second data and historical second traffic will be used to train through the Prophet algorithm to predict the second traffic at the second moment.

[0093] Specifically, in the Prophet prediction algorithm, the above-mentioned second data is input into the already trained second model to obtain the predicted value 2. Here, the Prophet algorithm is used to predict the second data including preset time points, so as to obtain the second traffic at the second moment, and this traffic prediction value will be used for comprehensive analysis of traffic trends to guide the optimal allocation of system resources.

[0094] In specific applications, for example, during the Double Eleven Shopping Festival, the system operation and maintenance team can use the Prophet algorithm to construct a second model to predict the traffic peak. Assuming that historical data shows that the traffic peak 10 minutes after the start of Double Eleven in the previous year was 1,500,000 messages per minute, through the technical solution of this application, the operation and maintenance team can, before the preset time point (i.e., this year's Double Eleven), use the training dataset including historical traffic data (historical second data) and traffic peak data (historical second traffic) to construct a second model to predict the traffic trend during this year's Double Eleven. The prediction result shows that the traffic peak 10 minutes after the start of this year's Double Eleven may reach 1,800,000 messages per minute.

[0095] In some embodiments, before fusing the above-mentioned first traffic and the above-mentioned second traffic to obtain the target traffic at the above-mentioned second moment, the above method further includes the following steps: obtaining a third traffic, where the above-mentioned third traffic is the traffic of the above-mentioned MQ message queue at a third moment determined according to the above-mentioned first data by using the ARIMA algorithm, the above-mentioned third moment is later than the above-mentioned first moment, and the above-mentioned third moment is earlier than the above-mentioned second moment; obtaining a fourth traffic, where the above-mentioned fourth traffic is the traffic of the above-mentioned MQ message queue at the above-mentioned third moment determined according to the above-mentioned first data by using the Prophet algorithm; obtaining the actual traffic at the above-mentioned third moment; calculating a first difference between the above-mentioned actual traffic and the above-mentioned third traffic, and calculating a second difference between the above-mentioned actual traffic and the above-mentioned fourth traffic; determining a first weight coefficient of the above-mentioned first traffic according to the above-mentioned first difference, and determining a second weight coefficient of the above-mentioned second traffic according to the above-mentioned second difference, where the above-mentioned first difference and the above-mentioned first weight coefficient are in a negative correlation relationship, and the above-mentioned second difference and the above-mentioned second weight coefficient are in a negative correlation relationship.

[0096] In this solution, the weights of traffic prediction can be dynamically adjusted according to the accuracy of the ARIMA and Prophet algorithms, so as to achieve more accurate target traffic prediction. The process of difference calculation and weight coefficient determination is essentially to calibrate the two prediction algorithms in real time with actual data to ensure that the final target traffic prediction is closer to the real situation and improve the accuracy of prediction.

[0097] For example, in the above example, if the magnitude relationship between the first difference and the second difference changes, the weight coefficients will also be adjusted accordingly, which enables the prediction model to adjust the prediction strategy according to real-time feedback, enhances the adaptability and robustness of the prediction model, effectively avoids the problem of inaccurate prediction caused by the inherent bias of the algorithm, and provides more scientific data support for the reasonable allocation of system resources.

[0098] Specifically, after merging the prediction results of the ARIMA module and the Prophet prediction method, they are input into the adaptive weight model for weighted calculation. The specific process is as Figure 4 shown. For these n data sources, each data source needs to participate in the training of the model. However, due to the limitation of the machine video memory size during training, not all data sources can be included in each training batch. In this solution, the method of randomly activating some data sources (such as: data sources m, n, k) is used for training, and the corresponding loss (such as: loss functions m, n, k) is also used for backpropagation of the loss function.

[0099] Since ResNet50 is used as the basic model A in this solution, the dimension of the feature layer is 2048. To adaptively adjust the importance of different data sources, a weight allocation network structure is designed in this solution, which mainly includes a multi-layer perceptron MLP and a normalization layer SoftMax layer. Among them, MLP is used to reduce the dimension of the feature layer to the number of data sources n, and the SoftMax layer is used to normalize the probabilities of N data sources for subsequent weight allocation. Therefore, the total loss function at this time is:

[0100]

[0101] In the formula refers to the output of the weight allocation network and is also the weight size assigned to different data sources. is a probability value, and the norm of this vector is constrained to 1, that is

[0102] In the formula, L i refers to the loss function used for the i-th data source. Here, the cross-entropy loss is used, and the representation form of the cross-entropy loss is:

[0103]

[0104] where C represents the number of categories of a single data source, and p k is the probability value assigned to the k-th category to obtain the final prediction result.

[0105] In actual operation, to verify the prediction accuracy of ARIMA and Prophet algorithms, a third moment is selected after the first moment and before the second moment. According to the historical data at this moment, ARIMA and Prophet algorithms are respectively used to predict the traffic of the MQ message queue, obtaining the third traffic and the fourth traffic. For example, before the Double Eleven shopping festival, assuming the first moment is 23:00 on November 10th, the second moment is 00:00 on November 11th, and 23:30 on November 10th is selected as the third moment. The third traffic predicted by the ARIMA algorithm is 1,200,000 messages per minute, and the fourth traffic predicted by the Prophet algorithm is 1,300,000 messages per minute.

[0106] Obtain the actual traffic of the above MQ message queue at the third moment. Assume the actual traffic is 1,250,000 messages per minute. Then, calculate the first difference between the actual traffic and the third traffic, and the second difference between the actual traffic and the fourth traffic, which are 50,000 and -50,000 respectively. Determine the weight coefficients of the first traffic and the second traffic according to the differences. The first difference (50,000) is negatively correlated with the first weight coefficient, and the second difference (-50,000) is also negatively correlated with the second weight coefficient. This means that the smaller the difference, the larger the weight coefficient. For example, the first weight coefficient can be set to 0.4 and the second weight coefficient to 0.6. This indicates that the fourth traffic predicted by the Prophet algorithm is closer to the actual traffic, so a higher weight is given when fusing the first traffic and the second traffic.

[0107] In the specific implementation process, after fusing the above first traffic and the above second traffic to obtain the target traffic at the second moment, the above method further includes the following steps: when the above target traffic is greater than the first traffic threshold, generate a first adjustment strategy, where the above first adjustment strategy is a strategy to increase the queue; when the above target traffic is greater than the second traffic threshold, generate a second adjustment strategy, where the above second traffic threshold is a strategy to increase the server, and the above first traffic threshold is less than the above second traffic threshold.

[0108] In this solution, by setting traffic thresholds at different levels, resources can be automatically adjusted according to the predicted target traffic, improving the system's response speed and processing capacity, and ensuring the timeliness and integrity of data.

[0109] Specifically, when the predicted target traffic reaches or exceeds the pre-set first traffic threshold, the system will trigger the first adjustment strategy, that is, increase the number of MQ message queues. For example, set the first traffic threshold to 4,000,000 messages per minute. When the predicted target traffic exceeds this threshold, the system automatically increases the number of queues, such as from 10 queues to 15 queues, to improve the ability and speed of message processing and ensure the timely delivery and processing of messages.

[0110] Specifically, when the predicted target traffic exceeds the higher second traffic threshold, the system will trigger the second adjustment strategy, that is, increase the server resources. For example, set the second traffic threshold to 5,000,000 messages per minute. This means that the target traffic is very likely to exceed the current processing capacity of the system. At this time, the operation and maintenance team will increase the number of servers according to the predicted value, such as from 50 servers to 65 servers, to ensure the stable operation and data processing capacity of the system in a high-concurrency scenario.

[0111] In some embodiments, obtaining the first data at the first moment can be specifically achieved through the following steps: obtaining the initial first data at the above-mentioned first moment; preprocessing the above-mentioned initial first data to obtain the above-mentioned first data, where the preprocessing methods include at least one or more of removing outliers, filling in missing values, and format normalization.

[0112] In this solution, by removing outliers, the prediction deviation caused by individual extreme data can be avoided, and the stability and reliability of the model are improved. Filling in missing values ensures the continuity and integrity of the data, enabling the model to fully utilize all available data for analysis and avoiding inaccurate predictions caused by data missing. The unified processing of data formats enables traffic data from different sources and types to be analyzed under a common framework, simplifies the data processing process, improves the data processing efficiency, and provides a high-quality data foundation for the establishment and analysis of the prediction model.

[0113] In summary, in the solution of this application, the method of multi-model complementarity can perfectly fill the respective shortcomings and can accurately predict in scenarios of seasonal fluctuations, large data volumes, and long-term trends. It can predict in advance the peak time of the large concurrent volume of the system, achieve early perception, respond in advance to possible impacts and risks, and take preventive measures. It can predict the traffic during special times such as weekdays, holidays, and shopping festivals, and guide the targeted adjustment and operation and maintenance guarantee of the existing system state. It realizes the pre-allocation of system resources, improves the utilization rate of system resources, enables quasi-real-time and temporary queues to handle large traffic congestion, and avoids abnormal situations such as system crashes and delays.

[0114] The advantage of the ARIMA model is that it can handle non-stationary data, model trends and periodicity, and can better handle non-stationary data. Its disadvantage is that it is not applicable to seasonal variations. The advantage of the Prophet model is that it can easily introduce seasonal and periodic effects in dealing with curve fitting problems and can be applied to various data types. After the models are fused, the advantages and disadvantages are complementary, and it can well adapt to the problems faced in the scenarios of this solution.

[0115] The system proposed in this solution has a huge traffic volume, reaching the million level, ten million level, or even hundreds of millions level. How to improve the user's access quality, ensure the accuracy of the messages of producers and consumers, and reduce the system's response time. And how to ensure that the system still runs stably without crashing under the condition of high concurrency of large data volumes, and at the same time ensure the integrity of the data without loss. Through the prediction method proposed in this solution, the arrival time of the traffic peak can be predicted in advance, and the system can trigger a peak-shifting mechanism to open temporary queues and quasi-real-time queues to handle the impact of large traffic on the system.

[0116] The embodiments of the present application also provide an apparatus for determining the traffic of a message queue. It should be noted that the apparatus for determining the traffic of the message queue in the embodiments of the present application can be used to execute the method for determining the traffic of the message queue provided in the embodiments of the present application. The apparatus is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0117] The following introduces the apparatus for determining the traffic of the message queue provided in the embodiments of the present application.

[0118] Figure 5 It is a structural block diagram of an apparatus for determining the traffic of a message queue according to an embodiment of the present application. As Figure 5 shown, the apparatus includes:

[0119] A first acquisition unit 10, configured to acquire first data at a first moment, where the first data is the data of the log of the MQ message queue;

[0120] A first determination unit 20, configured to determine a first traffic at a second moment according to the first data by using the ARIMA algorithm, where the second moment is later than the first moment, and the first traffic is the number of the MQ message queues determined by using the ARIMA algorithm;

[0121] A second determination unit 30, configured to determine a second traffic at the second moment by using the Prophet algorithm at least according to the first data, where the second traffic is the number of the MQ message queues determined by using the Prophet algorithm;

[0122] A fusion unit 40, configured to fuse the first traffic and the second traffic to obtain a target traffic at the second moment, where the fusion method includes at least weighted average, and the target traffic is used to guide a target object to adjust resources to cope with traffic peaks;

[0123] A third determination unit 50, configured to determine to add a server and / or add a queue when the target traffic is greater than a traffic threshold.

[0124] Through this embodiment, both the ARIMA algorithm and the Prophet algorithm are data prediction algorithms. The ARIMA algorithm can be used to process the time series characteristics of data and can accurately predict the long-term trend of data. The Prophet algorithm is particularly good at dealing with prediction problems with seasonality, trend, and holiday effects and can automatically detect and adapt to periodic changes in data. This solution combines the prediction results of these two algorithms, and the target traffic obtained can combine the advantages of the two algorithms, ensuring a relatively high accuracy of the obtained prediction results.

[0125] In the specific implementation process, the first determination unit includes a construction module and a first processing module. The construction module is used to construct a first model. Among them, the above-mentioned first model is trained by using multiple sets of training data through the ARIMA algorithm. Each set of the multiple sets of training data includes historical first data obtained within a historical time period and the historical first traffic corresponding to the historical first data. The first processing module is used to input the above-mentioned first data into the above-mentioned first model to obtain the above-mentioned first traffic corresponding to the above-mentioned first data.

[0126] In this solution, the advantage of the ARIMA model is that it can process non-stationary data, model trends and periodicity, can better process non-stationary data, and further improves the accuracy of prediction.

[0127] In some embodiments, the second determination unit includes a first acquisition module, an addition module, and a determination module. The first acquisition module is used to acquire a preset time point, where the above-mentioned preset time point is the time point of a holiday and / or a special date, and the above-mentioned special date includes at least one or more of a shopping festival, a year-end settlement date, and a conference date; the addition module is used to add the above-mentioned preset time point to the above-mentioned first data to obtain second data; the determination module is used to determine the above-mentioned second traffic at the above-mentioned second moment according to the above-mentioned second data by using the Prophet algorithm.

[0128] In this solution, the Prophet algorithm introduces the recognition of holidays and special dates in the prediction, which enables the prediction to more accurately capture the impact of non-working days or special events on traffic, and further improves the prediction accuracy.

[0129] In the specific implementation process, the determination module includes a construction sub-module and a processing sub-module. The construction sub-module is used to construct a second model. Among them, the above-mentioned second model is trained by using multiple sets of training data through the Prophet algorithm. Each set of the multiple sets of training data includes historical second data obtained within a historical time period and the historical second traffic corresponding to the historical second data. The processing sub-module is used to input the above-mentioned second data into the above-mentioned second model to obtain the above-mentioned second traffic corresponding to the above-mentioned second data.

[0130] In this solution, more accurate prediction can be made for traffic data including holidays and special dates (such as data during year-end settlement, start of the year, Double Eleven, 618 Shopping Festival, and important domestic conferences). The Prophet algorithm performs well in processing time series data with seasonality, trend, and holiday effects, and can effectively capture the impact of these special dates on traffic, further improving the prediction accuracy.

[0131] In some embodiments, the above device further includes a second acquisition unit, a third acquisition unit, a fourth acquisition unit, a calculation unit, and a fourth determination unit. The second acquisition unit is configured to acquire a third traffic volume before fusing the first traffic volume and the second traffic volume to obtain the target traffic volume at the second moment. The third traffic volume is the traffic volume of the MQ message queue at the third moment determined by using the ARIMA algorithm based on the first data. The third moment is later than the first moment and earlier than the second moment. The third acquisition unit is configured to acquire a fourth traffic volume, where the fourth traffic volume is the traffic volume of the MQ message queue at the third moment determined by using the Prophet algorithm based on the first data. The fourth acquisition unit is configured to acquire the actual traffic volume at the third moment. The calculation unit is configured to calculate a first difference between the actual traffic volume and the third traffic volume, and calculate a second difference between the actual traffic volume and the fourth traffic volume. The fourth determination unit is configured to determine a first weight coefficient of the first traffic volume according to the first difference, and determine a second weight coefficient of the second traffic volume according to the second difference. The first difference and the first weight coefficient are in a negative correlation relationship, and the second difference and the second weight coefficient are in a negative correlation relationship.

[0132] In this solution, the weights of traffic prediction can be dynamically adjusted according to the accuracy of the ARIMA and Prophet algorithms, so as to achieve more accurate prediction of the target traffic volume. The process of difference calculation and weight coefficient determination is essentially to calibrate the two prediction algorithms in real time through actual data, ensuring that the final prediction of the target traffic volume is closer to the real situation and improving the prediction accuracy.

[0133] In the specific implementation process, the above device further includes a first generation unit and a second generation unit. The first generation unit is configured to generate a first adjustment strategy after fusing the first traffic volume and the second traffic volume to obtain the target traffic volume at the second moment, when the target traffic volume is greater than the first traffic threshold. The first adjustment strategy is a strategy for increasing the queue. The second generation unit is configured to generate a second adjustment strategy when the target traffic volume is greater than the second traffic threshold. The second traffic threshold is a strategy for adding servers, where the first traffic threshold is less than the second traffic threshold.

[0134] In this solution, by setting traffic thresholds at different levels, resources can be automatically adjusted according to the predicted target traffic, which improves the system's response speed and processing capacity, and ensures the timeliness and integrity of data.

[0135] In some embodiments, the first acquisition unit includes a second acquisition module and a second processing module. The second acquisition module is used to acquire the initial first data at the above-mentioned first moment; the second processing module is used to preprocess the above-mentioned initial first data to obtain the above-mentioned first data, where the preprocessing method includes at least one or more of removing outliers, filling in missing values, and format normalization.

[0136] In this solution, by removing outliers, the prediction deviation caused by individual extreme data can be avoided, which improves the stability and reliability of the model. Filling in missing values ensures the continuity and integrity of the data, enabling the model to fully utilize all available data for analysis and avoiding inaccurate predictions caused by data missing. The unified processing of data formats enables traffic data from different sources and types to be analyzed under a common framework, simplifies the data processing process, improves the data processing efficiency, and provides a high-quality data foundation for the establishment and analysis of the prediction model.

[0137] The device for determining the traffic of the above-mentioned message queue includes a processor and a memory. The above-mentioned first acquisition unit, first determination unit, second determination unit, fusion unit, third determination unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned each module is located in different processors in any combination form.

[0138] The processor contains a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and by adjusting the kernel parameters, the problem of low accuracy in predicting the traffic of the message queue in the prior art can be solved.

[0139] The memory may include non-permanent memory in a computer-readable medium, forms such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), and the memory includes at least one storage chip.

[0140] An embodiment of the present invention provides a computer-readable storage medium. The above-mentioned computer-readable storage medium includes a stored program, where, when the above-mentioned program runs, it controls the device where the above-mentioned computer-readable storage medium is located to execute the method for determining the traffic of the above-mentioned message queue.

[0141] An embodiment of the present invention provides a processor for running a program, wherein when the program runs, it executes a method for determining the traffic of the message queue.

[0142] An embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements at least the steps of the method for determining the traffic of the message queue. The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0143] A computer program product includes a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the method for determining the traffic of the message queue in each embodiment of the present application.

[0144] The present application provides a queue traffic determination system, which includes one or more processors, a memory, and one or more programs. Among them, the one or more programs are stored in the memory and are configured to be executed by the one or more processors. The one or more programs include those for executing any one of the methods for determining the traffic of the message queue.

[0145] Obviously, those skilled in the art should understand that the various modules or steps of the present invention described above can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described herein can be executed in a different order, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0146] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0147] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, as well as the combination of flows and / or blocks in the flowchart and / or block diagram. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0148] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction means that implements the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0150] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0151] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0152] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0153] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0154] The above are only the preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for determining the traffic of a message queue, characterized in that, Including: Obtain the first data at the first moment, where the first data is the data of the log of the MQ message queue; According to the first data, use the ARIMA algorithm to determine the first traffic at the second moment, where the second moment is later than the first moment, and the first traffic is the number of the MQ message queue determined by the ARIMA algorithm; At least according to the first data, use the Prophet algorithm to determine the second traffic at the second moment, where the second traffic is the number of the MQ message queue determined by the Prophet algorithm; Fuse the first traffic and the second traffic to obtain the target traffic at the second moment, where the fusion method at least includes weighted average, and the target traffic is used to guide the target object to adjust resources to cope with the traffic peak; In the case where the target traffic is greater than the traffic threshold, determine to add servers and / or queues.

2. The method according to claim 1, wherein According to the first data, using the ARIMA algorithm to determine the first traffic at the second moment includes: Construct a first model, where the first model is trained by the ARIMA algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical first data obtained within a historical time period and the historical first traffic corresponding to the historical first data; Input the first data into the first model to obtain the first traffic corresponding to the first data.

3. The method according to claim 1, characterized in that, At least according to the first data, using the Prophet algorithm to determine the second traffic at the second moment includes: Obtain a preset time point, where the preset time point is the time point of a holiday and / or a special date, and the special date at least includes one or more of a shopping festival, a year-end settlement date, and a conference date; Add the preset time point to the first data to obtain second data; According to the second data, use the Prophet algorithm to determine the second traffic at the second moment.

4. The method according to claim 3, wherein According to the second data, using the Prophet algorithm to determine the second traffic at the second moment includes: Construct a second model, where the second model is trained by the Prophet algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical second data obtained within a historical time period and the historical second traffic corresponding to the historical second data; Input the second data into the second model to obtain the second traffic corresponding to the second data.

5. The method according to claim 1, characterized in that Before fusing the first traffic and the second traffic to obtain the target traffic at the second moment, the method further includes: Obtain a third traffic, where the third traffic is the traffic of the MQ message queue at a third moment determined by the ARIMA algorithm according to the first data, the third moment is later than the first moment, and the third moment is earlier than the second moment; Obtain a fourth traffic, where the fourth traffic is the traffic of the MQ message queue at the third moment determined by the Prophet algorithm according to the first data; Obtain the actual traffic at the third moment; Calculate the first difference between the actual traffic and the third traffic, and calculate the second difference between the actual traffic and the fourth traffic; Determine the first weight coefficient of the first traffic according to the first difference, and determine the second weight coefficient of the second traffic according to the second difference, wherein the first difference and the first weight coefficient are negatively correlated, and the second difference and the second weight coefficient are negatively correlated.

6. The method according to any one of claims 1 to 5, characterized in that, After fusing the first traffic and the second traffic to obtain the target traffic at the second moment, the method further includes: Generate a first adjustment strategy when the target traffic is greater than the first traffic threshold, wherein the first adjustment strategy is a strategy for increasing the queue; Generate a second adjustment strategy when the target traffic is greater than the second traffic threshold, wherein the second traffic threshold is a strategy for increasing the server, and the first traffic threshold is less than the second traffic threshold.

7. The method according to any one of claims 1 to 5, characterized in that Obtain the first data at the first moment, including: Obtain the initial first data at the first moment; Preprocess the initial first data to obtain the first data, wherein the preprocessing method includes at least one or more of removing outliers, filling missing values, and format normalization.

8. A device for determining the traffic of a message queue, characterized in that Include: A first acquisition unit for obtaining the first data at the first moment, wherein the first data is the data of the log of the MQ message queue; A first determination unit for determining the first traffic at the second moment according to the first data by using the ARIMA algorithm, wherein the second moment is later than the first moment, and the first traffic is the number of the MQ message queue determined by using the ARIMA algorithm; A second determination unit for determining the second traffic at the second moment at least according to the first data by using the Prophet algorithm, wherein the second traffic is the number of the MQ message queue determined by using the Prophet algorithm; A fusion unit for fusing the first traffic and the second traffic to obtain the target traffic at the second moment, wherein the fusion method includes at least weighted average, and the target traffic is used to guide the target object to adjust resources to cope with the traffic peak; A third determination unit for determining to increase the server and / or increase the queue when the target traffic is greater than the traffic threshold.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for determining the traffic of the message queue according to any one of claims 1 to 7.

10. A queue traffic determination system, characterized in that, Include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include the method for determining the traffic of the message queue according to any one of claims 1 to 7.

Images