Flow limiting management method, system and device applied to real-time database and medium

Abstract

The invention provides a current limiting management method, system and device applied to a real-time database and a medium, and the method comprises the steps: obtaining the performance index data of a database system and the application load information of each business process, training based on the historical performance index data and the application load information to obtain a prediction model used for predicting the working state of the database, predicting a database working state in a preset time window corresponding to the current data on the basis of the prediction model so as to perform flow limiting management on a plurality of service processes corresponding to the current application load information on the basis of a prediction result; according to the method, the current service processes are subjected to current limiting management by predicting the future working state, so that the important service processes are guaranteed to be smoothly realized, and a more stable, efficient and easy-to-maintain database environment is provided for a user through intelligent management and resource optimization.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a current limiting management method, system, device and medium applied to a real-time database. Background Art

[0002] With the rapid development of information technology, real-time databases are increasingly used in various fields (such as financial transactions, the Internet of Things, industrial automation, etc.). These applications usually need to handle a large number of concurrent requests and ensure the real-time and accuracy of data. However, under high load, when the number of query and write requests exceeds the processing capacity of the system, it may cause the system to respond slowly or even crash, seriously affecting the user experience and service quality; or because of the periodicity and uncertainty of business activities, traditional static current limiting strategies are difficult to adapt to dynamically changing workloads, which can easily cause resource waste or performance bottlenecks.

[0003] Currently, request traffic is usually controlled through predefined rules, but for complex and changing workloads, rule formulation and maintenance are very difficult and lack flexibility. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application provides a flow limiting management method, system, device and medium applied to a real-time database to solve the above-mentioned technical problems.

[0005] The present application provides a flow limiting management method applied to a real-time database, the method comprising: obtaining performance indicator data of a database system and application load information of a business process, the performance indicator data being used to characterize the occupancy status of the database system, and the application load information being used to characterize the application data requirements of each business process; constructing an association relationship between the performance indicator data and the application load information, and obtaining a prediction model for predicting a database working state based on the association relationship and training historical data, the historical data comprising historical performance indicator data and historical application load information; predicting a database working state within a preset time window corresponding to current data based on the prediction model, the current data comprising current performance indicator data and current application load information; and when it is predicted that the database working state within the preset time window is abnormal, performing flow limiting management on multiple business processes corresponding to the current application load information.

[0006] In one embodiment of the present application, a prediction model for predicting the working status of a database is obtained based on historical data training, including: preprocessing historical performance indicator data and historical application load information to obtain standard data; generating a training sample data set and a test sample data set based on the standard data; training a preset initial model according to the sample data set, testing the trained model according to the test sample data set, and optimizing the trained model parameters based on a preset optimization algorithm until the test results meet the preset training indicators to obtain a prediction model for predicting the working status of the database.

[0007] In one embodiment of the present application, a training sample data set and a test sample data set are generated based on the standard data, including: extracting the time series features of the standard data, the time series features are used to reflect the time series characteristics of the data; constructing association features that reflect the interaction between the standard data, the association features are used to reflect the association relationship between each business process and each performance indicator data; dividing the data containing the time series features and the association features into a training sample data set and a test sample data set.

[0008] In one embodiment of the present application, the working state of the database within a preset time window corresponding to the current data is predicted based on the prediction model, including: preprocessing the current data to obtain prediction data, the prediction data and the standard data have the same data format; inputting the prediction data into the trained prediction model to obtain a prediction value of the working state of the database in the next time window; when the prediction value is greater than the preset standard threshold, determining that the working state of the database system is abnormal; when the prediction value is less than or equal to the preset standard threshold, determining that the working state of the database system is normal.

[0009] In one embodiment of the present application, the predicted value includes at least one of the number of disk read and write requests, disk read and write rate, CPU usage, and memory occupancy.

[0010] In one embodiment of the present application, current limiting management is performed on multiple business processes corresponding to the current application load information, including: dividing the multiple business processes into a first priority and a second priority, the first priority being higher than the second priority; reducing the token bucket capacity and token generation rate of the second priority to reduce the resource occupancy of the second priority business process.

[0011] In one embodiment of the present application, the first-priority business process at least includes a data writing operation, and the second-priority business process at least includes a query operation and a delete operation.

[0012] The present application provides a current limiting management system applied to a real-time database, the system comprising: a system indicator collection module, used to obtain performance indicator data of a database system, the performance indicator data being used to characterize the occupancy status of the database system; an application load collection module, used to obtain application load information of a business process, the application load information being used to characterize the application data requirements of each business process; a device current limiting model training module, used to construct an association relationship between performance indicator data and application load information, and to obtain a prediction model for predicting a database working state based on the association relationship and historical data training, the historical data comprising historical performance indicator data and historical application load information; a device current limiting model prediction module, used to predict a database working state within a preset time window corresponding to current data based on the prediction model, the current data comprising current performance indicator data and current application load information; a current limiting management module, used to perform current limiting management on multiple business processes corresponding to the current application load information when it is predicted that the database working state within the preset time window is abnormal.

[0013] The present application provides an electronic device, comprising a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the current limiting management method applied to a real-time database as described above.

[0014] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to enable a computer to execute the current limiting management method applied to a real-time database as described above.

[0015] Beneficial effects of the present application: The current limiting management method applied to real-time databases proposed in the present application obtains a prediction model for predicting the working status of the database through training with historical performance indicator data and application load information, thereby predicting the working status of the database within a preset time window corresponding to the current data according to the prediction model, so as to perform current limiting management on multiple business processes corresponding to the current application load information based on the prediction results; this method significantly reduces the risk of system overload caused by sudden high loads and improves the stability and reliability of the system by accurately predicting future working states and implementing intelligent current limiting management; and, by predicting and preventing possible bottlenecks in advance, reducing system response delays and improving user experience, especially when processing a large number of concurrent requests, ensuring fast and consistent service responses.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 It is a schematic diagram of an implementation environment of a current limiting management method applied to a real-time database shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a flow chart of a current limiting management method applied to a real-time database shown in an exemplary embodiment of the present application;

[0020] Figure 3 is a block diagram of a current limiting management device applied to a real-time database, shown in an exemplary embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of the implementation steps of a current limiting management method applied to a real-time database shown in an exemplary embodiment of the present application;

[0022] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0025] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0026] First of all, it should be noted that in the following embodiments, performance indicator data usually includes CPU usage, memory usage, and disk I / O status; the application load information of the business process covers the number of query SQL requests, the number of write SQL requests, and query request related information. And the first priority business process is set to specifically refer to the data write operation, and the second priority business process includes at least query operations and delete operations. However, in actual use, in view of the characteristics of the real-time database system, its related data also includes but is not limited to network traffic and delay, transaction processing rate, cache hit rate, lock waiting time, log file generation rate, backup and recovery performance and other data. In addition, in the actual application process, the priority level of different businesses can also be set according to actual needs to achieve classified flow control management of different business processes, so as to ensure that important business processes can be executed smoothly. The following embodiments are only exemplary, and this application does not make specific restrictions on the data categories of its performance indicator data and application load information, as well as the priority level of each business process.

[0027] Figure 1 It is a schematic diagram of an implementation environment of a current limiting management method applied to a real-time database shown in an exemplary embodiment of the present application.

[0028] like Figure 1 As shown, the implementation environment diagram of the current limiting management method applied to the real-time database includes a data acquisition module 101 and a computer module 102.

[0029] The data collection module 101 is used to collect performance indicator data reflecting the occupancy status of the database system, such as CPU usage, memory usage, disk I / O status, etc.; and obtain application load information of each business process, including the number of query SQL requests, the number of write SQL requests, query request related information (such as the total number of time series device files, the average query timestamp range, and the aggregate query ratio) and write request related information (such as the average size of time series device files, the average number of data files under the time partition directory, and the proportion of disordered data). Then, the collected data is transmitted to the computer device 102.

[0030] The computer device 102 first cleans and standardizes the data received from the data acquisition module 101 to ensure data quality. Then, using historical performance indicator data and application load information as input, a prediction model for predicting the working state of the database is obtained based on the Transformer architecture training. Then, the currently collected data is encoded and input into the trained prediction model to predict the working state of the database in the next few minutes or hours, especially to accurately predict the number of read and write IO requests for time series device files. Finally, the parameters in the token bucket algorithm (such as token generation rate and token bucket capacity) are dynamically adjusted according to the prediction results to adapt to the needs of different operations (query, delete, write). In particular, a higher priority is given to the write operation, and token bucket current limiting is not implemented for it, but the disk IO requirements of concurrent writes are predicted by the model to ensure that the overall IOPS does not exceed the disk IO upper limit. In addition, if there is a significant difference between the actual collection results and the working state output by the prediction model for N consecutive times, the model training is restarted, and a conservative strategy is used for estimation before the new model is available, and the real-time performance of the system is continuously monitored to form a feedback loop and continuously optimize the current limiting strategy.

[0031] Figure 2 It is a flow chart of a current limiting management method applied to a real-time database shown in an exemplary embodiment of the present application.

[0032] like Figure 2 As shown, in an exemplary embodiment, the current limiting management method applied to the real-time database includes at least steps S210 to S240, which are described in detail as follows:

[0033] Step S210, obtaining performance indicator data of the database system and application load information of the business process, wherein the performance indicator data is used to characterize the occupancy state of the database system, and the application load information is used to characterize the application data requirements of each business process.

[0034] In one embodiment of the present application, in order to ensure the stability and efficiency of the real-time database, the system needs to collect performance indicators reflecting the current occupancy status and application load information representing the application data requirements of each business process. Generally, the system indicator collection module collects the performance indicators of the operating system, including data information such as CPU usage, memory usage, disk I / O status, and process and thread information; the application load collection module collects the load information of the business application, including the number of query SQL and write SQL requests, query request related information, and write request related information.

[0035] The system indicator collection module first obtains the CPU usage percentage regularly (for example, every 5 seconds) through the API or command line tools provided by the operating system (such as top and mpstat in Linux). A high CPU usage rate may indicate that the system is facing a large workload. Secondly, the operating system interface is used to obtain the occupancy of available memory, used memory, and swap space. Then, tools such as iostat are used to monitor the disk read and write speed and queue length, and record the currently active process list and the number of threads.

[0036] In a specific embodiment of the present application, in a real-time database environment, if a high memory usage is detected, the query operation may be blocked because it needs to wait for memory resources, resulting in a reduction in disk read IO operations per unit time. In this case, the system indicators will be used as part of the input data for training the current limiting model, and in order to ensure the effectiveness of the model, an outlier detection algorithm needs to be introduced to remove abnormal data points caused by non-periodic user behavior.

[0037] The application load collection module focuses on collecting load information related to business applications, including: counting the number of queries and write SQL requests that occur within a specific time period (such as one minute); in addition to simple counting, more detailed query characteristics should be captured, such as the total number of device files involved, the time range distribution of queries, the proportion of aggregate queries, etc.; for write operations, record the average size of time series device files, the average number of data files under each time partition directory, and the proportion of out-of-order data.

[0038] In a specific embodiment of the present application, taking the large-screen monitoring program as an example, it executes a last query once a minute. Such periodic activities will form a stable query pattern; and the report generation program may trigger a series of complex aggregate queries every night. In addition, since most time series databases use the LSM Tree structure for storage optimization, most write operations are performed sequentially and accumulated in memory and then written to the disk in batches. Only when data files are merged will it cause a surge in write IO in the short term. Therefore, combining the above two types of collected information can provide a solid foundation for building a more accurate device current limiting model.

[0039] It can be understood that through the collaborative work of the system indicator collection module and the application load collection module, the operating status of the database system and the actual needs of each business process can be fully understood, thereby providing reliable data support for the subsequent formulation of traffic control strategies.

[0040] Step S220, constructing an association relationship between the performance indicator data and the application load information, and obtaining a prediction model for predicting the working state of the database based on the association relationship and historical data training, where the historical data includes historical performance indicator data and historical application load information.

[0041] In one embodiment of the present application, a prediction model for predicting the working state of a database is obtained based on historical data training, including: preprocessing historical performance indicator data and historical application load information to obtain standard data; generating a training sample data set and a test sample data set based on the standard data; training a preset initial model according to the sample data set, testing the trained model according to the test sample data set, and optimizing the parameters of the trained model based on a preset optimization algorithm until the test result meets the preset training index, so as to obtain a prediction model for predicting the working state of the database. Among them, generating a training sample data set and a test sample data set based on standard data includes: extracting the time series features of the standard data, the time series features are used to reflect the time series characteristics of the data; constructing association features that reflect the interaction between each standard data, the association features are used to reflect the association relationship between each business process and each performance indicator data; dividing the data containing the time series features and the association features into a training sample data set and a test sample data set.

[0042] In a specific embodiment of the present application, in order to build a prediction model that can predict the working status of the database, we first need to collect historical performance indicator data and historical application load information from the system indicator collection module and the application load collection module. Take the CPU usage, memory usage, disk I / O, query SQL and write SQL request numbers, query request related information, write request related information, etc. collected in the above embodiment as an example.

[0043] First, the data are cleaned to remove or correct outliers and missing values ​​to ensure the quality of the data set; and data of different scales are converted to the same scale through methods such as minimum-maximum scaling or Z-score standardization to ensure that the importance of each feature during model training is not incorrectly weighted due to different numerical ranges.

[0044] Secondly, in the process of generating training sample and test sample data sets, first extract time series features that can reflect the characteristics of data changing over time, such as moving average, exponential smoothing, autocorrelation, etc., and calculate the mean, variance and other statistics in several time intervals before and after each timestamp as additional time series features. At the same time, analyze the interaction between each business process and performance indicators, and construct association features that reflect these relationships. For example, calculate the proportion of a business process occupying specific resources (such as CPU, memory) in a certain period of time to measure its impact on the overall system performance. In addition, methods such as Pearson correlation coefficient and mutual information are used to quantify the degree of dependence between different features and include them in the association feature set. Finally, the data containing time series features and association features are randomly divided into training sample data sets and test sample data sets, generally following the ratio of 70% training set / 30% test set, but it can also be adjusted according to actual conditions.

[0045] Then, the Transformer model suitable for processing time series data with long-distance dependencies is selected as the basic architecture for predictive model training. The architecture includes three parts: input representation, encoder, and decoder. Among them, the input representation is to convert the prepared time series features and associated features into a high-dimensional vector representation; the encoder is composed of multiple identical layers with self-attention mechanism, feedforward neural network, residual connection and layer normalization; in addition to self-attention, the decoder also includes an attention mechanism for the encoder output to fuse context information and finally output the prediction result.

[0046] Finally, the preset initial model (i.e., Transformer model) is trained using the training sample data set, that is, the model parameters are adjusted to minimize the loss function (such as mean square error MSE), and model snapshots are saved regularly during this process for subsequent evaluation. In addition, the trained model is verified using the test sample data set to check whether the model is overfitting or underfitting, and the model performance is evaluated based on indicators such as cross entropy loss and accuracy. If the test results do not meet the preset training indicators, the model parameters are further optimized using techniques such as gradient descent and Adam optimization algorithm, and it may be necessary to adjust hyperparameters (such as learning rate, batch size) or introduce regularization methods (such as Dropout) until the model's performance on the test set is stable and satisfactory, so as to obtain a prediction model for predicting the working status of the database.

[0047] It is understandable that the prediction model obtained by training in the above embodiment can be deployed in a production environment, receive new performance indicators and application load information as input in real time, and output a prediction of the working state of the database in the future. In addition, in the actual application process, the model can also be updated regularly to ensure that it continues to adapt to the changing workload and environmental conditions. Therefore, based on the method proposed in this embodiment, not only a powerful prediction model is established to predict the working state of the database, but also the flow control of the database is effectively managed and optimized, ensuring the efficient operation and service quality of the system.

[0048] Step S230: predicting the database working state within a preset time window corresponding to current data based on the prediction model, where the current data includes current performance indicator data and current application load information.

[0049] In one embodiment of the present application, the working state of the database within the preset time window corresponding to the current data is predicted based on the prediction model, including: preprocessing the current data to obtain the prediction data, the data format of the prediction data and the standard data is the same; inputting the prediction data into the trained prediction model to obtain the prediction value of the working state of the database in the next time window; when the prediction value is greater than the preset standard threshold, the working state of the database system is determined to be abnormal; when the prediction value is less than or equal to the preset standard threshold, the working state of the database system is determined to be normal. Among them, the prediction value includes but is not limited to the number of disk read and write requests, disk read and write rate, CPU usage, memory occupancy, etc.

[0050] In a specific embodiment of the present application, in order to predict the working status of the database within a preset time window, the latest performance indicator data (such as CPU usage, memory usage, disk I / O, etc.) and application load information (such as the number of query SQL and write SQL requests, query request related information, write request related information, etc.) are first obtained from the system indicator collection module and the application load collection module. Then, these real-time collected data are preprocessed as necessary, including cleaning outliers, filling missing values, and converting them into an N-dimensional vector in the same format as during training (that is, converting the format of the current data into the same data format as the training data used to train the prediction model) through standardization or normalization methods.

[0051] Then, the data of the past M moments are encoded into an M*N matrix as the input of the prediction model, where each row represents all the features at a time point (i.e., N-dimensional vector), and M represents the length of the time series used for prediction. The preprocessed prediction data is input into the trained database working status prediction model, which outputs a two-dimensional vector representing the number of disk read and write IO operations expected to occur in the next time window, specifically the number of time series device files involved in query and write operations.

[0052] Then, the key performance indicators such as disk read and write rate, CPU usage, memory occupancy, etc. in the next period of time are evaluated based on the predicted values ​​output by the model. If the predicted value (such as the number of disk read and write requests, disk read and write rate, etc.) exceeds the preset standard threshold, the working state of the database system is judged to be abnormal; conversely, if the predicted value is less than or equal to the preset standard threshold, the system working state is considered normal. Based on the predicted disk read and write IO upper limit and query / write operation results, an appropriate deletion operation current limiting rate is set in the timing device current limiting module to ensure that the overall performance of the system can be maintained within a controllable range even under high load conditions, while avoiding unnecessary waste of resources.

[0053] Finally, as new data continues to flow in, the above process is repeated to achieve continuous monitoring and timely response to the database working status. In addition, in the actual application process, the model parameters can be regularly updated according to the actual operation situation to ensure that its prediction accuracy does not decrease over time.

[0054] It can be understood that the method proposed in this embodiment can not only use advanced machine learning technology to accurately predict the working state of the database in the future, but also take corresponding flow control measures according to the prediction results, thereby effectively ensuring the stability of the system and the quality of service. This method is suitable for real-time database environments that need to process a large number of concurrent requests and complex business logic, such as financial transactions, online games, e-commerce platforms and other fields.

[0055] In addition, in another embodiment of the present application, if there is a significant difference between the actual acquisition results of N consecutive times (N≥3) and the predicted results of the device current limiting model, it indicates that the current model may not accurately estimate the number of time series device files to be processed. At this time, the model training should be restarted, and conservative default values ​​should be used for estimation before the new model training is completed to ensure system stability and service quality.

[0056] In a specific embodiment of the present application, in order to ensure the accuracy of the trained database system working status prediction model, a periodic task is set to compare the predicted value of the model with the actual performance indicators recorded by the system performance acquisition module at the end of each time window. Its performance indicators cover multiple aspects such as the number of disk read and write requests, disk read and write rates, CPU usage, memory occupancy, etc. At the same time, a reasonable error range Δ is defined to measure the difference between the predicted value and the actual acquisition result. If the deviation between the predicted value and the actual value exceeds this error range Δ each time in a time window of N consecutive times (for example, N=5), it is considered that the prediction accuracy of the current model is insufficient, triggering the conditions for retraining the model.

[0057] Once N consecutive large deviations are detected, the model retraining process will be started immediately. In addition, during the retraining process, the latest historical performance indicator data and application load information will be collected as new training samples to ensure that the new model can adapt to the latest workload pattern. At the same time, before the new model training is completed, in order to avoid the degradation of service quality due to inaccurate predictions, the system will switch to a conservative default current limiting strategy. Its temporary strategy is based on historical averages or empirical values ​​to ensure that the database system can maintain basic system stability and service response speed even under uncertain conditions.

[0058] Once the new model is trained and fully tested to verify its effectiveness and stability, it will be deployed to the production environment to replace the old model, and the performance of the new model will continue to be monitored to ensure that it can accurately predict the future working status of the database.

[0059] It can be understood that the method proposed in this embodiment can quickly respond when the existing equipment current limiting model fails, which not only ensures the system's immediate response capability, but also provides a solid foundation for subsequent more accurate flow control.

[0060] Step S240: When it is predicted that the database working state within the preset time window is abnormal, current limiting management is performed on multiple business processes corresponding to the current application load information.

[0061] In one embodiment of the present application, a plurality of business processes corresponding to the current application load information are subjected to flow limiting management, including: dividing the plurality of business processes into a first priority and a second priority, the first priority being higher than the second priority; reducing the token bucket capacity and token generation rate of the second priority to reduce the resource occupation of the second priority business process. Specifically, the first priority business process includes at least a data write operation, and the second priority business process includes at least a query operation and a delete operation.

[0062] In a specific embodiment of the present application, in order to effectively manage and optimize the resource allocation of the database, we first divide multiple business processes into two priorities: the first priority and the second priority, where the first priority is higher than the second priority. Specifically, the first priority includes write operations of time series data, and the second priority covers query operations and deletion operations. Independent token buckets are set up to perform flow limiting management on the second priority business processes. It should be understood that the token bucket algorithm generates tokens at a constant speed and puts these tokens into a token bucket of fixed capacity; whenever a request arrives, the system will take a token from the token bucket for processing. If the token bucket is empty, the request will be rejected or queued. This method can smoothly control the request flow and avoid the impact of excessive instantaneous flow on the system.

[0063] Then, in order to adapt to the changing workload pattern, the token bucket parameters for query and delete operations are dynamically adjusted according to the predicted values ​​of the device current limiting model prediction module. For example, based on the prediction of disk read and write IO requirements in the future, the speed of token generation can be adjusted. In the case of an expected increase in the number of query requests, the generation rate of the query token bucket is appropriately increased to meet higher query requirements. At the same time, the maximum capacity of the token bucket is set according to the maximum number of concurrent requests of the system and the average size of the request. When a high concurrency scenario is expected, the token bucket capacity is temporarily expanded to better cope with burst traffic.

[0064] In a specific example, first, the default token generation rate and token bucket size are determined based on historical data analysis, and initial current limiting parameters are set for different types of business processes (such as query, deletion). Then, the predicted value of the device current limiting model is compared with the actual collection results regularly to evaluate the accuracy of the existing model. If there is a significant difference between the actual collection results and the predicted results for N consecutive times (for example, N=5), the model retraining process is triggered. Once it is found that the model prediction accuracy is insufficient or the workload pattern has changed, the token bucket parameters of the query and deletion operations are immediately adjusted according to the latest prediction results to reduce the resource usage of the second priority business process. For example, reduce the generation rate and capacity of the query and deletion token buckets to free up more resources for high-priority write operations.

[0065] It can be understood that the method proposed based on this embodiment not only improves the stability and availability of the system, but also ensures the response speed and service quality of key businesses; it can effectively limit the flow of multiple business processes corresponding to the current application load information to ensure that system resources are reasonably allocated. Especially in real-time database environments that need to process a large number of concurrent requests and complex business logic, such as financial transactions, online games, e-commerce platforms, etc., this method can dynamically respond to different business needs and provide a more flexible and efficient flow control solution.

[0066] Figure 3 is a block diagram of a current limiting management device applied to a real-time database, shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The device may also be applicable to other exemplary implementation environments and specifically configured in other devices, and this embodiment does not limit the implementation environment to which the device is applicable.

[0067] like Figure 3 As shown, the exemplary current limiting management device applied to the real-time database includes: a system indicator collection module 310, an application load collection module 320, a device current limiting model training module 330, a device current limiting model prediction module 340, and a current limiting management module 350.

[0068] Among them, the system indicator collection module 310 is used to obtain performance indicator data of the database system, and the performance indicator data is used to characterize the occupancy status of the database system; the application load collection module 320 is used to obtain application load information of the business process, and the application load information is used to characterize the application data requirements of each business process; the device current limiting model training module 330 is used to construct the association relationship between the performance indicator data and the application load information, and obtain a prediction model for predicting the working status of the database based on the association relationship and historical data training, and the historical data includes historical performance indicator data and historical application load information; the device current limiting model prediction module 340 is used to predict the database working status within the preset time window corresponding to the current data based on the prediction model, and the current data includes the current performance indicator data and the current application load information; the current limiting management module 350 is used to perform current limiting management on multiple business processes corresponding to the current application load information when it is predicted that the database working status within the preset time window is abnormal.

[0069] Figure 4 It is a schematic diagram of the implementation steps of a current limiting management method applied to a real-time database shown in an exemplary embodiment of the present application.

[0070] like Figure 4 As shown, in the process of implementing the complete database current limiting management method, first, the system indicator collection module and the application load collection module respectively collect the performance indicator data of the database system and the application load information of each business process; then, the device current limiting model training module trains the prediction model of the database system working status through historical data; then, the device current limiting model prediction module predicts the database working status within the preset time window corresponding to the current data; finally, the current limiting management module performs current limiting management on multiple business processes corresponding to the current application load information.

[0071] In a specific embodiment of the present application, in order to achieve efficient management of the database system and optimize resource allocation, a method for current limiting management of the database is proposed. First, the system indicator collection module and the application load collection module are responsible for collecting performance indicator data of the database system (such as CPU usage, memory usage, disk I / O, etc.) and application load information of each business process (such as the number of query SQL and write SQL requests, query request related information, write request related information, etc.).

[0072] Next, the device current limiting model training module uses historical data to train a model that can predict the working status of the database. The model is based on the Transformer architecture and processes time series data through three parts: input representation, encoder, and decoder to capture long-term dependencies. During the training process, we encode the data of the past M moments to form an M*N matrix as input, where the N-dimensional vector represents all the features at each time point, and M represents the length of the time series used for prediction. It should be noted that the model outputs a two-dimensional vector, which represents the number of disk read and write IO operations expected to occur in the next time window, specifically the number of time series device files involved in query and write operations. In addition, the model training process also includes the detection and elimination of outliers to ensure the quality of training data.

[0073] Subsequently, the device current limiting model prediction module predicts the database working status within the preset time window corresponding to the current data based on the latest real-time data and the trained prediction model. Specifically, the latest collected data is preprocessed, converted into an N-dimensional vector in the same format as that used in training, and input into the prediction model to obtain the predicted value of the database working status in the next time window. If there is a significant difference between the actual collection results and the predicted results for N consecutive times (for example, N=5), the model is retrained to ensure the accuracy of the model prediction. At the same time, before the new model training is completed, conservative default values ​​are used for estimation to ensure that system stability and service quality are not affected.

[0074] Finally, the flow limiting management module performs flow limiting management on multiple business processes corresponding to the current application load information based on the prediction results. The business processes are divided into two priorities: the first priority includes the write operation of time series data, which is critical to system performance and usually has a higher priority, so it is not affected by the token bucket flow limiting; the second priority covers query operations and deletion operations, and independent token buckets are set for these two operations for flow control. The token bucket algorithm generates tokens at a constant speed and puts these tokens into a token bucket with a fixed capacity. Whenever a request arrives, the system takes a token from the token bucket for processing. If the token bucket is empty, the request will be rejected or queued. In addition, according to the predicted value provided by the device flow limiting model prediction module, the token bucket parameters of the query and deletion operations are dynamically adjusted, such as reducing the generation rate and capacity of the query and deletion token buckets to free up more resources for high-priority write operations, ensuring that the overall performance of the system can be maintained within a controllable range even under high load conditions, while avoiding unnecessary waste of resources.

[0075] It can be understood that the flow limiting management method applied to real-time databases proposed in this application can accurately predict the working status of the database in the future, and can also take corresponding flow control measures based on the prediction results, thereby effectively ensuring the stability of the system and the quality of service.

[0076] It should be noted that the current limiting management device applied to the real-time database provided in the above embodiment and the current limiting management method applied to the real-time database provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the current limiting management device applied to the real-time database provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0077] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the current limiting management method applied to the real-time database provided in the above-mentioned embodiments.

[0078] Figure 5 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0079] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503, such as executing the method described in the above embodiment. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502 and the RAM 503 are connected to each other through the bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0080] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0081] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the system of the present application are executed.

[0082] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0083] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0085] Another aspect of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, enables the computer to execute the current limiting management method applied to a real-time database as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0086] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the current limiting management method applied to a real-time database provided in each of the above embodiments.

[0087] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A current limiting management method applied to a real-time database, characterized in that: The method comprises: Acquire performance indicator data of the database system and application load information of the business process, wherein the performance indicator data is used to characterize the occupancy status of the database system, and the application load information is used to characterize the application data requirements of each business process; Constructing an association relationship between performance indicator data and application load information, and training a prediction model for predicting a database working state based on the association relationship and historical data, wherein the historical data includes historical performance indicator data and historical application load information; Predicting the database working state within a preset time window corresponding to current data based on the prediction model, wherein the current data includes current performance indicator data and current application load information; When it is predicted that the database working status within the preset time window is abnormal, current limiting management is performed on multiple business processes corresponding to the current application load information.

2. The current limiting management method applied to a real-time database according to claim 1 is characterized in that: The prediction model used to predict the working status of the database is trained based on historical data, including: Preprocess historical performance indicator data and historical application load information to obtain standard data; Generate a training sample data set and a test sample data set based on the standard data; The preset initial model is trained according to the sample data set, the trained model is tested according to the test sample data set, and the trained model parameters are optimized based on a preset optimization algorithm until the test results meet the preset training indicators, so as to obtain a prediction model for predicting the working status of the database.

3. The current limiting management method applied to a real-time database according to claim 2 is characterized in that: Generating a training sample data set and a test sample data set based on the standard data includes: Extracting time series features of the standard data, where the time series features are used to reflect the time series characteristics of the data; Constructing a correlation feature that reflects the interaction between each standard data, wherein the correlation feature is used to reflect the correlation relationship between each business process and each performance indicator data; The data containing time series features and correlation features are divided into a training sample data set and a test sample data set.

4. The current limiting management method applied to a real-time database according to claim 2 is characterized in that: Predicting the database working state within a preset time window corresponding to the current data based on the prediction model includes: Preprocessing the current data to obtain predicted data, wherein the predicted data has the same data format as the standard data; Inputting the predicted data into the trained prediction model to obtain a predicted value of the database working state in the next time window; When the predicted value is greater than a preset standard threshold, the working state of the database system is determined to be abnormal; when the predicted value is less than or equal to the preset standard threshold, the working state of the database system is determined to be normal.

5. The current limiting management method applied to a real-time database according to claim 4 is characterized in that: The predicted value includes at least one of the number of disk read / write requests, disk read / write rate, CPU usage, and memory occupancy.

6. The current limiting management method applied to a real-time database according to claim 1 is characterized in that: Perform current limiting management on multiple business processes corresponding to the current application load information, including: Classifying a plurality of business processes into a first priority and a second priority, wherein the first priority is higher than the second priority; Reduce the token bucket capacity and token generation rate of the second priority to reduce the resource usage of the second priority business process.

7. The current limiting management method applied to a real-time database according to claim 6 is characterized in that: The business process of the first priority level includes at least a data writing operation, and the business process of the second priority level includes at least a query operation and a delete operation.

8. A current limiting management system applied to a real-time database, characterized in that: The system comprises: A system indicator collection module is used to obtain performance indicator data of the database system, wherein the performance indicator data is used to characterize the occupancy status of the database system; An application load acquisition module is used to obtain application load information of business processes, where the application load information is used to characterize application data requirements of each business process; A device current limiting model training module is used to build an association relationship between performance indicator data and application load information, and to obtain a prediction model for predicting the working state of the database based on the association relationship and historical data training, wherein the historical data includes historical performance indicator data and historical application load information; A device current limiting model prediction module, used to predict the database working state within a preset time window corresponding to current data based on the prediction model, wherein the current data includes current performance indicator data and current application load information; The current limiting management module is used to perform current limiting management on multiple business processes corresponding to the current application load information when it is predicted that the database working status within a preset time window is abnormal.

9. An electronic device, characterized in that: It includes a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the current limiting management method applied to the real-time database as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to enable a computer to execute the current limiting management method applied to a real-time database as described in any one of claims 1-7.

Images