Memory database adjustment method and device, equipment and storage medium

Abstract

The invention discloses a memory database adjustment method and device, equipment and a storage medium, and relates to the field of cloud computation.The method comprises the steps that memory database creation parameters are obtained, and a plurality of memory database instances are created in a preset database cluster according to the memory database creation parameters; collecting monitoring index data of the memory database instance, and outputting the monitoring index data to a preset index database through a data pipeline; training a preset recurrent neural network model by using the monitoring index data to obtain a target recurrent neural network model; and calling current monitoring index data of the memory database instance, performing real-time prediction on the memory database instance according to the current monitoring index data by utilizing the target recurrent neural network model, and performing corresponding capacity expansion operation or capacity reduction operation on the memory database instance according to a prediction result. The problem of capacity expansion and shrinkage of the current memory database is solved, and the service avalanche phenomenon caused by large load of the memory database due to resource waste and resource shortage is avoided.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a method, device, equipment and storage medium for adjusting a memory database. Background Art

[0002] With the development of the Internet and big data applications, the use of in-memory databases in China is currently very extensive. Many Internet companies use in-memory databases to accelerate data access and processing, such as caching frequently accessed data, accelerating real-time data analysis, and improving the response speed of websites and applications. In the field of e-commerce, in-memory databases are widely used to manage and cache product information, user behavior data, and handle tasks such as real-time transactions and inventory management; in massively multiplayer online games and real-time games, in-memory databases can handle rapidly changing game states, player data, and real-time events, providing high-performance gaming experience; in the financial industry, in-memory databases are used for fast data storage and analysis of high-frequency transactions, as well as real-time risk management and market data analysis. In the field of big data, in-memory databases are used for real-time data stream processing, real-time analysis, and event processing, supporting complex real-time data query and calculation. In the advertising technology industry, in-memory databases are used for real-time advertising, user behavior tracking, and advertising effect analysis, supporting efficient real-time advertising bidding systems. In general, in-memory databases are widely used in various industries in China, mainly because of their fast read and write speeds and high performance, which can meet the needs of large-scale data processing and real-time applications.

[0003] With the rise of cloud computing, cloud computing platforms such as AWS, Azure, and Google Cloud have become the mainstream choice for enterprises. First, because the data is stored in memory, the in-memory database has faster read and write speeds and lower access latency than the traditional disk database. Deploying the in-memory database on the cloud platform can make full use of the high-performance computing resources provided by cloud computing, further improve the response speed and processing capabilities of the database, and meet the needs of real-time data processing and high concurrent access; secondly, the in-memory database usually requires a large amount of memory resources to support high-speed data processing, and the cloud platform provides a pay-as-you-go model, which can be paid according to actual usage, avoiding the high cost of traditional data center construction and maintenance; in addition, the resource sharing and automated management capabilities of the cloud platform can also reduce management costs. Finally, the cloud platform provides integrated management tools and monitoring services, which greatly simplifies the deployment, configuration, and operation and maintenance of the in-memory database. Through the cloud console, administrators can monitor the performance indicators, automated backup, and fault recovery of the database in real time, thereby reducing the complexity of management and the risk of operation.

[0004] The current domestic Internet market has more than 1 billion Internet users. At the same time, users like to use various online services and applications, such as social media, online shopping platforms, video and music streaming, online payment, etc. At this time, the emergence of a hot news or a popular game will cause a surge in the number of visits and concurrency of online services. However, since the in-memory database is mostly used to store the authentication information and hot data of the system, the surge in service visits and concurrency will cause the in-memory database to place extremely high requirements on memory resources, which will lead to insufficient memory resources affecting the performance of the database and even causing the database system to crash. However, most in-memory database cluster architectures are master-slave or sharding architectures. Both of these architectures require a certain amount of expansion time. The cluster of the master-slave synchronization architecture needs to copy all data to the newly expanded node; the cluster of the sharding architecture needs to recalculate the data slots of all nodes, and then move the data of the newly expanded node slots from each node to the newly expanded node. The expansion process has an impact on the normal use of the database. Moreover, currently, the operation and maintenance personnel can only know whether the database is currently under pressure by checking the database monitoring, and then analyze and judge whether the database needs to be expanded. Usually, the best time for expansion has been missed, which leads to database system crash and application service unavailability. Therefore, how to improve the use effect of the in-memory database when the amount of data increases is a problem to be solved in this field. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for adjusting a memory database, which solves the current problem of scaling up and down the memory database, and avoids the service avalanche phenomenon caused by resource waste and resource shortage when the memory database is overloaded. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a method for adjusting a memory database, comprising:

[0007] Acquire the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and use the preset memory database management platform to create a corresponding number of memory database instances in the preset database cluster according to the memory database creation parameters;

[0008] Collect monitoring indicator data of the memory database instance based on preset data collection rules, and output the monitoring indicator data to a preset indicator database through a data pipeline;

[0009] Pulling the monitoring indicator data from the preset indicator database, and using the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model;

[0010] The current monitoring indicator data of the memory database instance is called, and the target recurrent neural network model is used to perform real-time prediction on the memory database instance according to the current monitoring indicator data, and the memory database instance is expanded or reduced accordingly according to the obtained prediction result and the preset database adjustment threshold.

[0011] Optionally, after using the preset memory database management platform to create a corresponding plurality of memory database instances in a preset database cluster according to the memory database creation parameters, the method further includes:

[0012] Generate a unified identifier using the preset in-memory database management platform, and mark each in-memory database instance using the unified identifier;

[0013] Correspondingly, the collecting of monitoring indicator data of the memory database instance based on the preset data collection rule includes:

[0014] Identify the in-memory database instance based on the unified identifier according to a preset database identification interval, and maintain the identified in-memory database instance in a preset database memory;

[0015] The monitoring indicator data of the memory database instance in the preset database memory is collected based on the preset data collection rule.

[0016] Optionally, collecting the monitoring indicator data of the memory database instance based on a preset data collection rule, and outputting the monitoring indicator data to a preset indicator database through a data pipeline, includes:

[0017] Collecting the monitoring indicator data of the memory database instance based on a preset data collection interval, storing the monitoring indicator data in a preset cache area, and determining whether the amount of the monitoring indicator data currently collected in the preset cache area reaches a preset data amount threshold;

[0018] If the data volume reaches the preset data volume threshold, the monitoring indicator data currently collected in the preset buffer area is output to the preset indicator database through the data pipeline;

[0019] If the data volume does not reach the preset data volume threshold, after the monitoring indicator data of all the memory database instances are collected, the monitoring indicator data collected in the preset cache area is output to the preset indicator database through the data pipeline.

[0020] Optionally, before collecting the monitoring indicator data of the memory database instance based on the preset data collection rule, the method further includes:

[0021] Acquire a plurality of custom data collection rules corresponding to each of the in-memory database instances, and configure the preset data collection rules according to the plurality of custom data collection rules;

[0022] Or, obtaining the general data collection rules corresponding to each of the memory database instances, and configuring the preset data collection rules according to the general data collection rules;

[0023] Or, the preset data collection rule is configured based on default data collection parameters.

[0024] Optionally, the using the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model includes:

[0025] Using the monitoring indicator data to train the preset recurrent neural network model to adjust the model parameters of the preset recurrent neural network model;

[0026] Evaluating the preset recurrent neural network model based on the model parameters, optimizing the preset recurrent neural network model based on the evaluation result, and obtaining the target recurrent neural network model;

[0027] Correspondingly, after using the target recurrent neural network model to perform real-time prediction on the memory database instance according to the current monitoring indicator data, it also includes:

[0028] Determine a data deviation between the predicted result and an actual result corresponding to the in-memory database instance, and adjust the target recurrent neural network model according to the data deviation.

[0029] Optionally, the performing a corresponding capacity expansion operation on the memory database instance includes:

[0030] Identify the cluster mode of the preset database cluster corresponding to the in-memory database instance;

[0031] If the cluster mode is the master-slave mode, create a database slave node, and add the database slave node to the master node of the memory database instance;

[0032] If the cluster mode is sentinel mode, create a database slave node and a new sentinel node, and add the node information of the database slave node and the new sentinel node to the configuration file of the sentinel node in the memory database instance;

[0033] If the cluster mode is cluster mode, create a database slave node, and modify the configuration files of all cluster nodes in the memory database instance to add the new database slave node to the memory database instance, and recalculate the slots of all nodes corresponding to the memory database instance, and schedule the slots of the memory database instance to the new database slave node.

[0034] Optionally, performing a corresponding scaling-down operation on the memory database instance includes:

[0035] Identify the cluster mode of the preset database cluster corresponding to the in-memory database instance;

[0036] If the cluster mode is the master-slave mode, modify the master node configuration of the memory database instance to remove the target slave node information, and destroy the slave node to be destroyed after the slave node to be destroyed corresponding to the target slave node information is disconnected; the target slave node information is the node information of the slave node to be destroyed determined according to the prediction result;

[0037] If the cluster mode is the sentinel mode, removing the slave node to be destroyed from the sentinel cluster of the in-memory database instance, and destroying the slave node to be destroyed and the sentinel node corresponding to the slave node to be destroyed;

[0038] If the cluster mode is the cluster mode, the slot of the to-be-destroyed slave node is migrated to other nodes of the in-memory database instance, and the to-be-destroyed slave node is deleted and destroyed from the cluster of the in-memory database instance.

[0039] In a second aspect, the present application provides a memory database adjustment device, comprising:

[0040] A database creation module, used to obtain memory database creation parameters input by a user through a visual interface of a preset memory database management platform, and use the preset memory database management platform to create a corresponding number of memory database instances in a preset database cluster according to the memory database creation parameters;

[0041] A data collection module, used to collect monitoring indicator data of the memory database instance based on preset data collection rules, and output the monitoring indicator data to a preset indicator database through a data pipeline;

[0042] A model training module, used to pull the monitoring indicator data from the preset indicator database, and use the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model;

[0043] The database adjustment module is used to call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to make real-time predictions on the memory database instance according to the current monitoring indicator data, and to perform corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold.

[0044] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned in-memory database adjustment method.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned memory database adjustment method when executed by a processor.

[0046] In this application, first, the memory database creation parameters input by the user through the visual interface of the preset memory database management platform are obtained, and the preset memory database management platform is used to create a corresponding number of memory database instances in the preset database cluster according to the memory database creation parameters, and then the monitoring indicator data of the memory database instance is collected based on the preset data collection rules, and the monitoring indicator data is output to the preset indicator database through the data pipeline, and then the monitoring indicator data in the preset indicator database is pulled, and the preset recurrent neural network model is trained using the monitoring indicator data to obtain a target recurrent neural network model, so as to call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to perform real-time prediction of the memory database instance according to the current monitoring indicator data, and perform corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold. In this way, the present application can uniformly collect and store monitoring data of all memory database instances, and then automatically determine whether to scale up or down through the prediction model, and trigger the corresponding scaling operations to achieve unified management of all memory database instances. By combining the advantages of cloud computing and neural network prediction, real-time prediction of the memory database can be achieved, and scaling can be automatically determined based on the prediction results. In addition, data can be actively obtained from the environment to train the model, and the model prediction accuracy can be continuously optimized to achieve the expansion of the memory database before the peak of access and concurrency, and the scaling of the memory database when the access and concurrency drop to a specific level, ensuring that the memory database is automatically expanded or reduced without human intervention, and saving resources under the premise of stable operation of the memory database, significantly enhancing the stability and robustness of the memory database, and avoiding software system avalanches due to database problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flow chart of a method for adjusting a memory database provided in this application;

[0049] Figure 2 An overall architecture diagram of a memory database adjustment system provided for this application;

[0050] Figure 3 A flow chart of indicator collection provided for this application;

[0051] Figure 4 A specific flow chart of the memory database adjustment method provided in this application;

[0052] Figure 5 A data analysis and prediction system architecture diagram provided for this application;

[0053] Figure 6 A schematic diagram of the structure of a memory database adjustment device provided in this application;

[0054] Figure 7 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Since in-memory databases are mostly used to store system authentication information, hot data, etc., if there is a surge in service access and concurrency, the in-memory database will place extremely high demands on memory resources, which will lead to insufficient memory resources, affecting database performance and even causing the database system to crash. This application uniformly collects and stores monitoring data of all in-memory database instances, and then automatically determines whether to scale up or down through a prediction model, and triggers corresponding scaling operations. It combines the advantages of cloud computing and neural network prediction to achieve real-time prediction of in-memory databases, and expands the in-memory database before the peak access and concurrency peaks arrive, effectively avoiding software system avalanches caused by database problems.

[0057] See also Figure 1 As shown, an embodiment of the present invention discloses a method for adjusting a memory database, comprising:

[0058] Step S11: obtaining the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and using the preset memory database management platform to create a corresponding plurality of memory database instances in the preset database cluster according to the memory database creation parameters.

[0059] In this embodiment, Figure 2 As shown, the memory database adjustment method for realizing automatic scaling of the memory database in a cloud native environment to solve the problem of database overload, unavailability or resource waste mainly includes: an indicator collection system, an indicator storage system, a data analysis and prediction system, and a memory database management system. Among them, the memory database management system is a system for uniformly managing the entire life cycle of the memory database instance, including the ability to create a database, modify database specifications, delete a database, expand the database, reduce the database, view slow logs, view monitoring data, etc., and at the same time provide a restful style Open API (Open Application Programming Interface) operation interface to the external application system. Therefore, in this embodiment, the memory database management system can first obtain the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and use the preset memory database management platform to create a corresponding number of memory database instances in the preset database cluster according to the memory database creation parameters.

[0060] Specifically, the name and description of the database instance set by the user through the visual interface can be obtained, and after adjusting the database size and number of copies accordingly, the memory database management system is used to create a memory database instance according to the content set by the user. In addition, the user can adjust the specifications of the memory database such as GPU (Graphics Processing Unit) and memory in the later use process to meet actual needs, so as to use the memory database management system to adjust the memory database instance according to the specifications set by the user. At the same time, the user can also delete the memory database instance created by the memory database management system. It should be pointed out that after creating the memory database instance, a unified identifier can be generated by using the preset memory database management platform, and each memory database instance can be marked with a unified identifier. That is to say, the memory database instances created by the memory database management system will be marked with a unified mark, and the operation and maintenance users with only indicator query permissions are built-in in the memory database instance, so that the database identification module of the indicator collection system can identify the memory database instances under all namespaces in the kubernetes cluster through the unified identifier at regular intervals, and then maintain the identified memory database instance information in the memory for use by the input module.

[0061] Step S12: Collect monitoring indicator data of the memory database instance based on preset data collection rules, and output the monitoring indicator data to a preset indicator database through a data pipeline.

[0062] In this embodiment, first, a number of custom data collection rules corresponding to each memory database instance can be obtained, and preset data collection rules can be configured according to the custom data collection rules; or, the general data collection rules corresponding to each memory database instance can be obtained, and the preset data collection rules can be configured according to the general data collection rules; or, the preset data collection rules can be configured based on the default data collection parameters. After the data collection rules are configured, the monitoring indicator data of the memory database instance is collected based on the preset data collection rules, and the monitoring indicator data is output to the preset indicator database through the data pipeline.

[0063] It can be understood that the indicator collection system in this embodiment is the source system of the monitoring indicator data. The system can actively identify all memory databases and connect to the memory databases to regularly collect monitoring indicator data. The indicator collection system includes three modules: database identification module, input module, and output module. The database identification module performs database identification; the input module obtains the memory database instance from the memory, and then connects to each memory database instance to collect monitoring indicator data; the input module supports parameter configuration to optimize the collection performance, and the configuration includes: data collection interval, batch sending data volume, data buffer area, indicator renaming, indicator name prefix, indicator name suffix, and reserved tag items. Specifically, the data collection rules can be configured through the above input module, where the data collection interval configuration item is the time interval for the input module to collect monitoring data for the memory database instance, which is the interval from the end of one collection to the start of the next collection; the batch sending data volume configuration item is the maximum amount of data that the input module collects monitoring data and sends to the output module; the data buffer area configuration item is the memory area where the input module caches the collected monitoring indicator data, and the data exists in this buffer area before being sent to the output module; the indicator rename configuration item is that the input module will use this parameter to rename the monitoring indicator; the indicator name prefix configuration item is that the input module adds this parameter to the name prefix of the monitoring indicator; the indicator name suffix configuration item is that the input module adds this parameter to the name suffix of the monitoring indicator. In this way, the indicator collection system has built-in default configurations for these input configurations, and also supports custom configuration of collection parameters for one or all memory database instances to achieve down-frequency collection or up-frequency collection of monitoring data and customized scenario requirements.

[0064] like Figure 3As shown, through the above-mentioned input module, when the monitoring indicator data of the memory database instance is collected based on the preset data collection rules, the memory database instance can be identified based on the unified identifier according to the preset database identification interval through the data cache area configuration item, and the identified memory database instance can be maintained in the preset database memory, and then the monitoring indicator data of the memory database instance in the preset database memory is collected based on the preset data collection rules. At the same time, based on the above-mentioned batch sending data volume configuration item, the maximum data volume of the monitoring data collected by the input module and sent to the output module can be defined. If the data volume of the collected monitoring data reaches this threshold, it will be sent to the output module; otherwise, it will be sent to the output module after all indicators of all instances are collected. In other words, the monitoring indicator data of the memory database instance can be collected based on the preset data collection interval, the monitoring indicator data can be stored in the preset cache area, and it can be determined whether the data volume of the monitoring indicator data currently collected in the preset cache area reaches the preset data volume threshold; if the preset data volume threshold is reached, the monitoring indicator data currently collected in the preset cache area is output to the preset indicator database through the data pipeline; if the preset data volume threshold is not reached, after the monitoring indicator data of all memory database instances are collected, the monitoring indicator data collected in the preset cache area is output to the preset indicator database through the data pipeline.

[0065] And when the above-mentioned output module outputs the monitoring data to the indicator storage system, the parameter configurations supported by the module are: data refresh time interval, data cache configuration, and data cache size. Among them, the data refresh time interval configuration item is the time interval for the output module to deliver the monitoring data to the indicator storage system; the data cache configuration configuration item is used for the output module to configure the cache monitoring data, and this configuration supports two storage modes: memory and local storage; the data cache size configuration item is used for the output module to configure the cache size, and the early monitoring data will be discarded after exceeding this configuration. Correspondingly, through the output module, the indicator collection system has built-in default configurations for these output configurations, and also supports custom configuration parameters to meet custom scenario requirements.

[0066] It should be pointed out that the above-mentioned indicator storage system is used to store the monitoring indicators of the memory database, and provides a data source for the memory database management system to query the monitoring indicator data of each instance, the training data of the data analysis and prediction system, and the trend prediction. In this embodiment, since influxDB (time series database) is designed for cloud native environment and containerized deployment, and supports running in kubernetes or other container orchestration tools, this component is used as the indicator storage system, and influxDB is a database specially designed for processing time series data. It performs well in processing time series data and supports horizontal expansion. It can increase storage capacity and processing power by adding nodes, and also provides a simple and convenient query language. For the above reasons, influxDB is used as a system for storing monitoring indicators in this embodiment. It can also be understood that the use of the influxDB database in this embodiment is not limited, and other alternative databases can also be used to achieve the corresponding technical effects.

[0067] Step S13: Pull the monitoring indicator data from the preset indicator database, and use the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model.

[0068] In this embodiment, the monitoring indicator data in the preset indicator database can be pulled through the data analysis and prediction system, and the preset recurrent neural network model can be trained using the monitoring indicator data to obtain the target recurrent neural network model.

[0069] Step S14: calling the current monitoring indicator data of the memory database instance, and using the target recurrent neural network model to perform real-time prediction on the memory database instance according to the current monitoring indicator data, and performing corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold.

[0070] In this embodiment, the data analysis and prediction system can call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to perform real-time prediction of the memory database instance based on the current monitoring indicator data, and perform corresponding expansion or reduction operations on the memory database instance based on the obtained prediction results and the preset database adjustment threshold.

[0071] This embodiment first obtains the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and uses the preset memory database management platform to create several corresponding memory database instances in the preset database cluster according to the memory database creation parameters, and then collects the monitoring indicator data of the memory database instance based on the preset data collection rules, and outputs the monitoring indicator data to the preset indicator database through the data pipeline, and then pulls the monitoring indicator data in the preset indicator database, and uses the monitoring indicator data to train the preset recurrent neural network model to obtain the target recurrent neural network model, so as to call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to perform real-time prediction of the memory database instance according to the current monitoring indicator data, and perform corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold. In this way, the monitoring data of all memory database instances can be uniformly collected and stored, and then the prediction model can be used to automatically determine whether to expand or shrink, and trigger the corresponding expansion and shrink operations, so as to realize the unified management of all memory database instances, and realize the real-time prediction of the memory database by combining the advantages of cloud computing and neural network prediction, and automatically determine the expansion and shrink according to the prediction results, so as to significantly enhance the stability and robustness of the memory database, and avoid software system avalanche caused by database problems.

[0072] Based on the previous embodiment, it can be seen that the present application can uniformly collect and store the monitoring data of all memory database instances, and then automatically determine whether to scale up or down through the prediction model, and trigger the corresponding scaling operation. Next, this embodiment will elaborate on the above model prediction and database scaling operation in detail. Figure 4 As shown, the embodiment of the present application discloses a specific method for adjusting a memory database, including:

[0073] Step S21: Collect monitoring indicator data of the memory database instance based on preset data collection rules, and output the monitoring indicator data to the preset indicator database through a data pipeline.

[0074] Step S22: Pull the monitoring indicator data from the preset indicator database, and use the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model.

[0075] In this embodiment, Figure 5As shown, the preset recurrent neural network model is trained mainly by pulling the monitoring indicator data in the preset indicator database through the data analysis and prediction system. Among them, the data analysis and prediction system not only provides real-time data analysis and prediction services, but also has the ability to dynamically train and optimize models. Since the recurrent neural network (RNN, Recurrent Neural Network) can capture the time dependency in the data and is particularly suitable for processing time series data, the data analysis and prediction system in this embodiment can specifically use the recurrent neural network model to process and predict monitoring data. Among them, the data analysis and prediction system is mainly divided into two modules: a fully automatic model training and tuning module and a fully automatic online prediction module.

[0076] That is to say, this embodiment can use monitoring index data to train a preset recurrent neural network model to adjust the model parameters of the preset recurrent neural network model, and evaluate the preset recurrent neural network model based on the model parameters, so as to optimize the preset recurrent neural network model based on the evaluation results to obtain a target recurrent neural network model.

[0077] In the specific model training process, the above-mentioned fully automatic model training and tuning module can be used to first acquire and process data. The latest monitoring data can be pulled from the memory database regularly through this module. The above-mentioned data includes but is not limited to the current number of connections of the memory database, the current memory usage data of the memory database, the current CPU (Central Processing Unit) usage data of the memory database, etc., and it can be understood that the data will be pre-processed before training, such as normalization, missing value processing, etc., to ensure data quality. Then, the fully automatic model training and tuning module is used to train and tune the model, that is, the acquired data is trained using a recurrent neural network. During the training process, the system will automatically adjust relevant parameters, such as learning rate, hidden layer size, etc., to optimize the model performance. At the same time, the system will regularly evaluate the performance of the model to ensure the effectiveness and accuracy of the model. After that, the fully automatic model training and tuning module is used to iterate and optimize the model. Specifically, the model can be iterated and optimized according to the model evaluation results to improve the prediction accuracy. Correspondingly, real-time prediction can be performed according to the above-mentioned fully automatic online prediction module. The monitoring data in the current memory database is used by this module to perform real-time prediction through the trained model. Then the prediction result analysis is performed, where the prediction result is used to determine whether the memory database needs to be expanded or reduced, and then the memory database management system interface is called back to perform related operations, and compared with the actual monitoring data to evaluate the prediction accuracy. It can be understood that the above data analysis and prediction system supports custom settings for expansion and reduction thresholds, which can be set globally or individually for a memory database instance. At the same time, the priority set for the memory database instance is higher than the priority set globally. In addition, the data analysis and prediction system in this embodiment also includes a feedback mechanism, which helps to further optimize the model parameters and structure by feeding back the prediction results to the model training and tuning module.

[0078] In addition, the model can be analyzed for deviations through the system optimization and iteration module of the data analysis and prediction system, and the deficiencies in the prediction can be identified by analyzing the deviation between the predicted data and the actual data. That is to say, in this embodiment, after real-time prediction of the memory database instance, the data deviation between the predicted result and the actual result corresponding to the memory database instance can be determined, and the target recurrent neural network model can be adjusted according to the data deviation. When adjusting the model, model reinforcement training can be performed based on the analysis results. According to the deviation analysis results, the data analysis and prediction system can adjust the model training strategy and perform targeted reinforcement training to reduce the prediction error and improve the overall prediction accuracy through continuous model iteration and optimization. In this way, the workflow of the entire data analysis and prediction system in this embodiment is highly automated, reducing the need for manual intervention, and at the same time, through continuous self-learning and optimization, the accuracy and efficiency of the prediction are improved.

[0079] Step S23, calling the current monitoring indicator data of the memory database instance, and using the target recurrent neural network model to perform real-time prediction on the memory database instance according to the current monitoring indicator data, and performing corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold.

[0080] In this embodiment, the memory database management system supports three cluster mode expansions: master-slave, sentinel, and cluster. The system will automatically identify the cluster mode and perform expansion operations. Therefore, when performing corresponding expansion operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold, the cluster mode of the preset database cluster corresponding to the memory database instance can be identified first. If the cluster mode is the master-slave mode, a database slave node is created, and the database slave node is added to the master node of the memory database instance. That is to say, by creating a database slave node and then adding the slave node to the master node, the slave node will perform data synchronization. When the data synchronization is completed, the slave node can provide services to the outside. If the cluster mode is the sentinel mode, a database slave node and a new sentinel node are created, and the node information of the database slave node and the new sentinel node is added to the configuration file of the sentinel node in the memory database instance. That is to say, a database slave node and a new sentinel node can be created, and then all sentinel node configuration files are updated to add the information of the new node to ensure that the sentinel can monitor the new memory database instance, and then the slave node performs data synchronization. When the data synchronization is completed, the slave node can provide services to the outside and can also select the master. If the cluster mode is cluster mode, create a database slave node and modify the configuration files of all cluster nodes in the memory database instance to add the new database slave node to the memory database instance, and recalculate the slots of all nodes corresponding to the memory database instance, and schedule the slots of the memory database instance to the new database slave node. In this way, by creating a database slave node, then modifying all cluster node configuration files, adding the new node to the cluster, and recalculating the slots of all nodes of the memory database instance, the slots of other nodes of the memory database instance are scheduled to the new slave node, so that it can be distributed as evenly as possible to all database nodes.

[0081] Correspondingly, the memory database management system supports three cluster modes of scaling down: master-slave, sentinel, and cluster. The system will automatically identify the cluster mode and perform scaling down operations. Specifically, the cluster mode of the preset database cluster corresponding to the memory database instance can be first identified. If the cluster mode is the master-slave mode, the master node configuration of the memory database instance is modified to remove the target slave node information, and after the slave node to be destroyed corresponding to the target slave node information is disconnected, the slave node to be destroyed is destroyed, wherein the above-mentioned target slave node information is the node information of the slave node to be destroyed determined according to the prediction result; the slave node information to be reduced is removed by modifying the master node configuration, and the replication status is verified to ensure that the removed slave node has been disconnected before the slave node to be reduced is destroyed. If the cluster mode is the sentinel mode, the slave node to be destroyed to be reduced is removed from the sentinel cluster of the memory database instance, and the sentinel node corresponding to the slave node to be destroyed and the sentinel node to be destroyed are destroyed after the sentinel processes the master-slave node configuration. If the cluster mode is the cluster mode, the slots of the slave node to be destroyed to be reduced are migrated to other nodes of the memory database instance, and then the slave node to be destroyed is deleted and destroyed from the cluster of the memory database instance.

[0082] It can also be understood that in this embodiment, the in-memory database management system can also view slow logs and monitoring data. Specifically, the slow log can be enabled by default when creating an in-memory database instance, and the slow log can be collected through a log collection tool. The log of the in-memory database instance can be displayed through the in-memory database management system interface, and the monitoring data stored in the indicator storage system can be displayed through the in-memory database management system interface.

[0083] For a more specific processing procedure of the above step S21, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0084] Through the above technical solution, in the cloud native field, this embodiment combines the advantages of cloud computing and neural network prediction to make real-time predictions on the usage of the memory database, automatically determine the expansion and contraction based on the prediction results, and the system actively obtains data from the environment to train the model, and continuously optimizes the model prediction accuracy, so as to achieve the expansion of the memory database before the peak of the access volume and concurrency, and the reduction of the memory database when the access volume and concurrency drop to a specific level, ensuring that the memory database automatically expands and shrinks without human intervention, and saving resources under the premise of stable operation of the memory database, significantly improving the utilization rate of computing and storage resources, and solving the current problem of expansion and contraction of the memory database, thereby avoiding resource waste and resource shortages that cause service avalanche when the memory database is overloaded, and significantly enhancing the stability of the memory database.

[0085] See also Figure 6As shown, the embodiment of the present application also discloses a memory database adjustment device, including:

[0086] The database creation module 11 is used to obtain the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and use the preset memory database management platform to create a corresponding number of memory database instances in the preset database cluster according to the memory database creation parameters;

[0087] The data collection module 12 is used to collect the monitoring indicator data of the memory database instance based on the preset data collection rules, and output the monitoring indicator data to the preset indicator database through the data pipeline;

[0088] A model training module 13 is used to extract the monitoring indicator data from the preset indicator database, and use the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model;

[0089] The database adjustment module 14 is used to call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to perform real-time prediction on the memory database instance according to the current monitoring indicator data, and perform corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold.

[0090] This embodiment can obtain the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and use the preset memory database management platform to create several corresponding memory database instances in the preset database cluster according to the memory database creation parameters, and then collect the monitoring indicator data of the memory database instance based on the preset data collection rules, and output the monitoring indicator data to the preset indicator database through the data pipeline, and then pull the monitoring indicator data in the preset indicator database, and use the monitoring indicator data to train the preset recurrent neural network model to obtain the target recurrent neural network model, so as to call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to make real-time predictions on the memory database instance according to the current monitoring indicator data, and perform corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold. In this way, the monitoring data of all memory database instances can be uniformly collected and stored, and then the prediction model can be used to automatically determine whether to expand or shrink, and trigger the corresponding expansion and shrink operations, so as to realize the unified management of all memory database instances, and ensure that the memory database is automatically expanded or shrunk without human intervention, so as to save resources under the premise of stable operation of the memory database, significantly enhance the stability and robustness of the memory database, and avoid software system avalanche caused by database problems.

[0091] In some specific embodiments, the in-memory database adjustment device further includes:

[0092] An identification generation module, used to generate a unified identification using the preset in-memory database management platform, and mark each in-memory database instance using the unified identification;

[0093] Accordingly, the data acquisition module 12 specifically includes:

[0094] An instance identification unit, configured to identify the memory database instance based on the unified identifier according to a preset database identification interval, and maintain the identified memory database instance in a preset database memory;

[0095] A data collection unit is used to collect the monitoring indicator data of the memory database instance in the preset database memory based on the preset data collection rule.

[0096] In some specific embodiments, the data acquisition module 12 specifically includes:

[0097] A data storage unit, configured to collect the monitoring indicator data of the memory database instance based on a preset data collection interval, store the monitoring indicator data in a preset cache area, and determine whether the amount of the monitoring indicator data currently collected in the preset cache area reaches a preset data amount threshold;

[0098] A first data output unit, configured to output the monitoring indicator data currently collected in the preset buffer area to the preset indicator database through the data pipeline if the data volume reaches the preset data volume threshold;

[0099] A second data output unit is used to output the monitoring indicator data collected in the preset cache area to the preset indicator database through the data pipeline if the data volume does not reach the preset data volume threshold after the monitoring indicator data of all the memory database instances are collected.

[0100] In some specific embodiments, the in-memory database adjustment device further includes:

[0101] A first rule configuration module is used to obtain a plurality of custom data collection rules corresponding to each of the memory database instances, and configure the preset data collection rules according to the plurality of custom data collection rules;

[0102] A second rule configuration module is used to obtain the general data collection rules corresponding to each of the memory database instances, and configure the preset data collection rules according to the general data collection rules;

[0103] The third rule configuration module is used to configure the preset data collection rules based on default data collection parameters.

[0104] In some specific embodiments, the model training module 13 specifically includes:

[0105] A parameter adjustment unit, used to train the preset recurrent neural network model using the monitoring indicator data to adjust the model parameters of the preset recurrent neural network model;

[0106] A model optimization unit, configured to evaluate the preset recurrent neural network model based on the model parameters, and optimize the preset recurrent neural network model based on the evaluation result to obtain the target recurrent neural network model;

[0107] Correspondingly, the memory database adjustment device further includes:

[0108] The model adjustment module is used to determine the data deviation between the predicted result and the actual result corresponding to the memory database instance, and adjust the target recurrent neural network model according to the data deviation.

[0109] In some specific embodiments, the database adjustment module 14 specifically includes:

[0110] A first pattern recognition unit, configured to recognize a cluster pattern of the preset database cluster corresponding to the in-memory database instance;

[0111] A first database adjustment unit, configured to create a database slave node if the cluster mode is a master-slave mode, and add the database slave node to the master node of the memory database instance;

[0112] A second database adjustment unit is used to create a database slave node and a new sentinel node if the cluster mode is a sentinel mode, and add the node information of the database slave node and the new sentinel node to a configuration file of the sentinel node in the memory database instance;

[0113] The third database adjustment unit is used to create a database slave node if the cluster mode is the cluster mode, and modify the configuration files of all cluster nodes in the memory database instance to add the new database slave node to the memory database instance, and recalculate the slots of all nodes corresponding to the memory database instance, and schedule the slots of the memory database instance to the new database slave node.

[0114] In some specific embodiments, the database adjustment module 14 specifically includes:

[0115] A second pattern recognition unit, configured to recognize a cluster pattern of the preset database cluster corresponding to the in-memory database instance;

[0116] A fourth database adjustment unit is used for modifying the master node configuration of the memory database instance to remove the target slave node information if the cluster mode is the master-slave mode, and destroying the slave node to be destroyed after the slave node to be destroyed corresponding to the target slave node information is disconnected; the target slave node information is the node information of the slave node to be destroyed determined according to the prediction result;

[0117] a fifth database adjustment unit, configured to remove the slave node to be destroyed from the sentinel cluster of the memory database instance, and destroy the slave node to be destroyed and the sentinel node corresponding to the slave node to be destroyed if the cluster mode is the sentinel mode;

[0118] The sixth database adjustment unit is used to migrate the slot of the to-be-destroyed slave node to other nodes of the in-memory database instance if the cluster mode is the cluster mode, and delete and destroy the to-be-destroyed slave node from the cluster of the in-memory database instance.

[0119] Furthermore, the present application also discloses an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0120] Figure 7 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the memory database adjustment method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0121] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0122] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0123] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the memory database adjustment method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0124] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed method for adjusting the memory database. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0125] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0126] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0128] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0129] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for adjusting a memory database, characterized in that: include: Acquire the memory database creation parameters input by the user through the visual interface of the preset memory database management platform, and use the preset memory database management platform to create a corresponding number of memory database instances in the preset database cluster according to the memory database creation parameters; Collect monitoring indicator data of the memory database instance based on preset data collection rules, and output the monitoring indicator data to a preset indicator database through a data pipeline; Pulling the monitoring indicator data from the preset indicator database, and using the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model; The current monitoring indicator data of the memory database instance is called, and the target recurrent neural network model is used to perform real-time prediction on the memory database instance according to the current monitoring indicator data, and the memory database instance is expanded or reduced accordingly according to the obtained prediction result and the preset database adjustment threshold.

2. The method for adjusting the in-memory database according to claim 1, characterized in that: After using the preset memory database management platform to create a corresponding plurality of memory database instances in a preset database cluster according to the memory database creation parameters, the method further includes: Generate a unified identifier using the preset in-memory database management platform, and mark each in-memory database instance using the unified identifier; Correspondingly, the collecting of monitoring indicator data of the memory database instance based on the preset data collection rule includes: Identify the in-memory database instance based on the unified identifier according to a preset database identification interval, and maintain the identified in-memory database instance in a preset database memory; The monitoring indicator data of the memory database instance in the preset database memory is collected based on the preset data collection rule.

3. The method for adjusting the in-memory database according to claim 1, characterized in that: The step of collecting the monitoring indicator data of the memory database instance based on the preset data collection rule and outputting the monitoring indicator data to the preset indicator database through the data pipeline includes: Collecting the monitoring indicator data of the memory database instance based on a preset data collection interval, storing the monitoring indicator data in a preset cache area, and determining whether the amount of the monitoring indicator data currently collected in the preset cache area reaches a preset data amount threshold; If the data volume reaches the preset data volume threshold, the monitoring indicator data currently collected in the preset buffer area is output to the preset indicator database through the data pipeline; If the data volume does not reach the preset data volume threshold, after the monitoring indicator data of all the memory database instances are collected, the monitoring indicator data collected in the preset cache area is output to the preset indicator database through the data pipeline.

4. The method for adjusting the in-memory database according to claim 1, characterized in that: Before collecting the monitoring indicator data of the memory database instance based on the preset data collection rule, the method further includes: Acquire a plurality of custom data collection rules corresponding to each of the in-memory database instances, and configure the preset data collection rules according to the plurality of custom data collection rules; Or, obtaining the general data collection rules corresponding to each of the memory database instances, and configuring the preset data collection rules according to the general data collection rules; Or, the preset data collection rule is configured based on default data collection parameters.

5. The method for adjusting the in-memory database according to claim 1, characterized in that: The method of using the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model includes: Using the monitoring indicator data to train the preset recurrent neural network model to adjust the model parameters of the preset recurrent neural network model; Evaluating the preset recurrent neural network model based on the model parameters, optimizing the preset recurrent neural network model based on the evaluation result, and obtaining the target recurrent neural network model; Correspondingly, after using the target recurrent neural network model to perform real-time prediction on the memory database instance according to the current monitoring indicator data, it also includes: Determine a data deviation between the predicted result and an actual result corresponding to the in-memory database instance, and adjust the target recurrent neural network model according to the data deviation.

6. The method for adjusting the in-memory database according to any one of claims 1 to 5, characterized in that: The corresponding expansion operation on the memory database instance includes: Identify the cluster mode of the preset database cluster corresponding to the in-memory database instance; If the cluster mode is the master-slave mode, create a database slave node, and add the database slave node to the master node of the memory database instance; If the cluster mode is sentinel mode, create a database slave node and a new sentinel node, and add the node information of the database slave node and the new sentinel node to the configuration file of the sentinel node in the memory database instance; If the cluster mode is cluster mode, create a database slave node, and modify the configuration files of all cluster nodes in the memory database instance to add the new database slave node to the memory database instance, and recalculate the slots of all nodes corresponding to the memory database instance, and schedule the slots of the memory database instance to the new database slave node.

7. The method for adjusting the in-memory database according to any one of claims 1 to 5, characterized in that: Performing corresponding scaling operations on the memory database instance includes: Identify the cluster mode of the preset database cluster corresponding to the in-memory database instance; If the cluster mode is the master-slave mode, modify the master node configuration of the memory database instance to remove the target slave node information, and destroy the slave node to be destroyed after the slave node to be destroyed corresponding to the target slave node information is disconnected; the target slave node information is the node information of the slave node to be destroyed determined according to the prediction result; If the cluster mode is the sentinel mode, removing the slave node to be destroyed from the sentinel cluster of the in-memory database instance, and destroying the slave node to be destroyed and the sentinel node corresponding to the slave node to be destroyed; If the cluster mode is the cluster mode, the slot of the slave node to be destroyed is migrated to other nodes of the in-memory database instance, and the slave node to be destroyed is deleted and destroyed from the cluster of the in-memory database instance.

8. A memory database adjustment device, characterized in that: include: A database creation module, used to obtain memory database creation parameters input by a user through a visual interface of a preset memory database management platform, and use the preset memory database management platform to create a corresponding number of memory database instances in a preset database cluster according to the memory database creation parameters; A data collection module, used to collect monitoring indicator data of the memory database instance based on preset data collection rules, and output the monitoring indicator data to a preset indicator database through a data pipeline; A model training module, used to pull the monitoring indicator data from the preset indicator database, and use the monitoring indicator data to train a preset recurrent neural network model to obtain a target recurrent neural network model; The database adjustment module is used to call the current monitoring indicator data of the memory database instance, and use the target recurrent neural network model to make real-time predictions on the memory database instance according to the current monitoring indicator data, and to perform corresponding expansion or reduction operations on the memory database instance according to the obtained prediction results and the preset database adjustment threshold.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the in-memory database adjustment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the in-memory database adjustment method according to any one of claims 1 to 7.

Images