A database data processing method, device, equipment and storage medium

Abstract

The present application discloses a database data processing method, device, equipment and storage medium, the method comprising: obtaining a first load parameter and a second load parameter of a database system; when determining that a first load condition is met based on the first load parameter and the second load parameter, adjusting the storage strategy of a first type of data table in the database system from the first storage strategy to the second storage strategy; when determining that a second load condition is met based on the first load parameter and the second load parameter, adjusting the storage strategy of a second type of data table in the database system from the second storage strategy to the first storage strategy. In this way, when the imbalance of the system is determined by using the first load parameter and the second load parameter, the storage strategy of some of its data tables is dynamically adjusted according to the satisfied load condition, which alleviates the load pressure of the data system and thereby improves the resource utilization of the database system.

Description

Technical Field

[0001] The present application relates to the field of database performance optimization, and in particular to a database data processing method, device, equipment and storage medium. Background Art

[0002] With the rapid development of the information technology industry, massive amounts of data are being generated in various fields. However, it is often difficult for a single database to store and calculate this data. Distributed databases have emerged. Some distributed database systems can have hundreds or thousands of server nodes. There are often multiple database nodes on these server nodes. Each database node stores table data according to the table data storage strategy. It can store the corresponding part of the table data according to the distribution strategy of the set distribution key, or store a full copy of the data according to the set replication table storage strategy.

[0003] In a distributed database system, it is usually necessary to manually set a distribution strategy or a replication strategy in advance when creating a table. Only one of the two can be set to determine the table data storage method. Once the setting is determined, the storage method of the table data is also determined. If the storage method of the table data needs to be changed later, it can only be changed manually, which is not convenient or flexible enough.

[0004] In actual production scenarios, if table data is stored according to the distribution strategy of manually setting the distribution key, each database node only needs to store a part of the data, and the disk space and disk IO usage are relatively small, but data redistribution and broadcast are prone to occur when tasks are running, resulting in a high network load. If table data is stored according to the replication table strategy, each database node stores a copy of the full table data, and the disk space and disk IO usage are large, but data redistribution and broadcast will not occur when executing related tasks, and the network load is low. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a database data processing method, device, equipment and storage medium, the purpose of which is to alleviate the load pressure of the data system and thereby improve the resource utilization of the database system.

[0006] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0007] In a first aspect, a method for processing database data is provided, the method comprising:

[0008] Obtaining a first load parameter and a second load parameter of the database system;

[0009] When it is determined based on the first load parameter and the second load parameter that a first load condition is met, adjusting a storage strategy of a first type of data table in the database system from a first storage strategy to a second storage strategy;

[0010] When it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy.

[0011] In the above scheme, the first load parameter is the system's disk IO resource occupancy rate, and the second load parameter is the system's network bandwidth resource occupancy rate; the first load condition is: the first load parameter is greater than the first threshold and the second load parameter is less than the second threshold; the second load condition is: the first load parameter is less than the third threshold and the second load parameter is greater than the fourth threshold.

[0012] In the above scheme, the method also includes: obtaining a third load parameter, a fourth load parameter and a usage frequency of at least one data table; wherein the third load parameter is the disk IO resource occupancy rate of the data table, and the fourth load parameter is the network bandwidth resource occupancy rate of the data table; if the first load parameter and the second load parameter are determined to meet the first load condition, determining the first type of data table in the database system based on the third load parameter and the usage frequency; if the first load parameter and the second load parameter are determined to meet the second load condition, determining the second type of data table in the database system based on the fourth load parameter and the usage frequency.

[0013] In the above scheme, the determining of the first type of data table in the database system based on the third load parameter and the usage frequency includes: arranging the at least one data table from large to small according to the third load parameter and the usage frequency to obtain a first arrangement order of the data table; selecting the first N data tables stored with a first storage strategy from the at least one data table based on the first arrangement order of the data table; wherein N is a positive integer; and using the first N data tables stored with the first storage strategy as the first type of data tables;

[0014] The method of determining the second category of data tables in the database system based on the fourth load parameter and the usage frequency includes: arranging the at least one data table from large to small according to the fourth load parameter and the usage frequency to obtain a second arrangement order of the data tables; selecting the first M data tables stored with a second storage strategy from the at least one data table based on the second arrangement order of the data tables; wherein M is a positive integer; and using the first M data tables stored with the second storage strategy as the second category of data tables.

[0015] In the above solution, the step of adjusting the storage strategy of the first type of data table in the database system from the first storage strategy to the second storage strategy includes: adjusting the first type of data table stored with the first storage strategy to be stored with the second storage strategy according to the first adjustment strategy;

[0016] The adjusting the storage strategy of the second type of data table in the database system from the second storage strategy to the first storage strategy includes: adjusting the second type of data table stored with the second storage strategy to be stored with the first storage strategy according to the second adjustment strategy.

[0017] In the above scheme, the first storage strategy is stored as storing all data in the data table into the database node; the second storage strategy is stored as dividing the data in the data table into at least two sub-data and storing the at least two sub-data in at least two database nodes respectively.

[0018] In the above solution, the obtaining of the first load parameter and the second load parameter of the database system includes: when the database cluster meets an adjustment cycle, obtaining the first load parameter and the second load parameter of the database system, and adjusting the storage strategy of the data table.

[0019] In a second aspect, a database data processing device is provided, the device comprising:

[0020] An acquiring unit, used for acquiring a first load parameter and a second load parameter of the database system;

[0021] an adjusting unit, configured to adjust a storage strategy of a first type of data table in the database system from a first storage strategy to a second storage strategy when determining that a first load condition is met based on the first load parameter and the second load parameter;

[0022] The adjustment unit is configured to adjust the storage strategy of the second type of data table in the database system from the second storage strategy to the first storage strategy when determining that a second load condition is met based on the first load parameter and the second load parameter.

[0023] In the above scheme, the first load parameter is the system's disk IO resource occupancy rate, and the second load parameter is the system's network bandwidth resource occupancy rate; the first load condition is: the first load parameter is greater than the first threshold and the second load parameter is less than the second threshold; the second load condition is: the first load parameter is less than the third threshold and the second load parameter is greater than the fourth threshold.

[0024] In the above scheme, the device also includes: a determination unit, used to obtain a third load parameter, a fourth load parameter and a usage frequency of at least one data table; wherein the third load parameter is the disk IO resource occupancy rate of the data table, and the fourth load parameter is the network bandwidth resource occupancy rate of the data table; if the first load parameter and the second load parameter are determined to meet the first load condition, the first type of data table in the database system is determined based on the third load parameter and the usage frequency; if the first load parameter and the second load parameter are determined to meet the second load condition, the second type of data table in the database system is determined based on the fourth load parameter and the usage frequency.

[0025] In the above scheme, the device includes: the determining unit is specifically used to arrange the at least one data table from large to small according to the third load parameter and the usage frequency to obtain a first arrangement order of the data table; based on the first arrangement order of the data table, select the first N data tables stored with the first storage strategy from the at least one data table; wherein N is a positive integer; and use the first N data tables stored with the first storage strategy as the first type of data tables;

[0026] It is also specifically used to arrange the at least one data table in descending order according to the fourth load parameter and the usage frequency to obtain a second arrangement order of the data table; based on the second arrangement order of the data table, select the first M data tables stored with the second storage strategy from the at least one data table; wherein M is a positive integer; and use the first M data tables stored with the second storage strategy as the second category data tables.

[0027] In the above solution, the device includes: the adjustment unit, which is used to adjust the first type of data table stored with the first storage strategy to be stored with the second storage strategy according to the first adjustment strategy;

[0028] It is also specifically used to adjust the second type of data table stored with the second storage strategy to storage with the first storage strategy according to the second adjustment strategy.

[0029] In the above scheme, the first storage strategy is stored as storing all data in the data table into the database node; the second storage strategy is stored as dividing the data in the data table into at least two sub-data and storing the at least two sub-data in at least two database nodes respectively.

[0030] In the above solution, the device further includes: an operating unit, which is used to obtain the first load parameter and the second load parameter of the database system and adjust the storage strategy of the data table when the database cluster meets an adjustment cycle.

[0031] According to a third aspect, a database data processing device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0032] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0033] By adopting the above technical solution, the first load parameter and the second load parameter of the database system are obtained; when it is determined based on the first load parameter and the second load parameter that the first load condition is met, the storage strategy of the first type of data table in the database system is adjusted from the first storage strategy to the second storage strategy; when it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy. In this way, when the imbalance of the system is determined by using the first load parameter and the second load parameter, the storage strategy of some of its data tables is dynamically adjusted according to the satisfied load condition, which alleviates the load pressure of the data system and thus improves the resource utilization of the database system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of a first flow chart of a method for processing database data in an embodiment of the present application;

[0035] Figure 2 This is a second flow chart of the database data processing method in the embodiment of the present application;

[0036] Figure 3 This is a third flow chart of the database data processing method in the embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of storage strategies for three data tables in the database in the embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of the composition structure of the database data processing device in the embodiment of the present application;

[0039] Figure 6 It is a schematic diagram of the composition structure of the database data processing device in the embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable a more detailed understanding of the features and technical contents of the present application, the implementation of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the present application.

[0041] In a distributed database system, it is usually necessary to manually set a distribution strategy or a replication table strategy in advance when creating a table. Only one of the two can be set to determine the table data storage method. If the table data is stored according to the distribution strategy with manually set distribution keys, each database node only stores a part of the data, and the disk space and disk IO usage are relatively small, but data redistribution and broadcasting are prone to occur when tasks are running, resulting in a high network load. If the table data is stored according to the replication table strategy, each database node stores a copy of the full table data, and the disk space and disk IO usage are large, but data redistribution and broadcasting will not occur when executing related tasks, and the network load is low.

[0042] In order to solve the above problems, the present invention provides a method for processing database data. Figure 1 Schematic diagram of the first process of the database data processing method in the embodiment of the present application. Figure 1 As shown, the execution subject of steps 101 to 103 may be a processor of a database data processing device. The database data processing method may specifically include:

[0043] Step 101: Acquire a first load parameter and a second load parameter of a database system;

[0044] Specifically, when table data is stored according to a distribution strategy with manually set distribution keys, data redistribution and broadcasting are prone to occur when the task is running. Once this occurs, it will cause a network storm and a sharp increase in network load, which will make the network a performance bottleneck that limits task execution. When stored according to the replication table strategy, each database node stores a copy of the full table data, and the disk space and disk IO occupy a large amount of space. Based on the above-mentioned problems, the database system load is unbalanced, so that the load resources cannot be optimally utilized. To this end, the present application adjusts the storage strategy of each data table through a load dynamic adjustment control program, so that the database system load is in a balanced state.

[0045] Before starting the load dynamic adjustment control program, the adjustment cycle of the database cluster is pre-set. When the database cluster meets an adjustment cycle, the first load parameter and the second load parameter of the database system are obtained, and the storage strategy of the data table is adjusted. After that, the load dynamic adjustment control program is run once every adjustment cycle until each data table no longer stores data, that is, the running program is completely terminated.

[0046] Step 102: When it is determined based on the first load parameter and the second load parameter that a first load condition is met, adjusting the storage strategy of the first type of data table in the database system from the first storage strategy to the second storage strategy;

[0047] The first load parameter here is the system's disk IO resource occupancy rate, and the second load parameter is the system's network bandwidth resource occupancy rate.

[0048] Whether the database system is in a balanced state can be determined based on the first load parameter and the second load parameter. Specifically, when the first load parameter and the second load parameter determine that the first load condition is satisfied, that is, the disk IO resource occupancy rate is greater than the first threshold and the network bandwidth resource occupancy rate is less than the second threshold, then the database system is in an unbalanced state, and the root cause lies in the storage strategy of the data table in the database system.

[0049] Based on the above load conditions, when the storage strategy of certain data tables is changed to adjust the balance of the database system, it is necessary to adjust the storage strategy of data tables with high disk IO resource occupancy, low network bandwidth resource occupancy and high usage frequency. Among them, when storing data according to the first storage strategy, all data of the data table are stored in multiple database nodes on the server node, that is, each database node stores a copy of all data; that is, the data table stored according to the first storage strategy has a large disk IO resource occupancy rate and a small network bandwidth resource occupancy rate. In other words, it is the imbalance caused by too many data tables stored in the database with the first storage strategy. Therefore, it is necessary to adjust some data tables stored with the first storage strategy.

[0050] Specifically, the storage strategy of each data table in the database system and the respective third load parameter, fourth load parameter and usage frequency need to be obtained, wherein the third load parameter is the disk IO resource occupancy rate of the data table, and the fourth load parameter is the network bandwidth resource occupancy rate of the data table. According to the first load condition, combined with the storage strategy and parameter status of each data table, some data tables with large disk IO resource occupancy rate, small network bandwidth resource occupancy rate and high usage frequency are selected from the database system, that is, the first type of data tables stored with the first storage strategy.

[0051] The selected first type of data table stored in the first storage strategy is adjusted to the second storage strategy according to the first adjustment strategy, thereby alleviating the load pressure caused by the first load condition. The first adjustment strategy function is to adjust the first storage strategy to the second storage strategy.

[0052] Step 103: When it is determined based on the first load parameter and the second load parameter that a second load condition is met, adjusting the storage strategy of the second type of data table in the database system from the second storage strategy to the first storage strategy.

[0053] According to the first load parameter and the second load parameter, it can be determined whether the database system is in a balanced state. Specifically, when the first load parameter and the second load parameter determine that the second load condition is satisfied, that is, the disk IO resource occupancy rate is less than the third threshold and the network bandwidth resource occupancy rate is greater than the fourth threshold, then the database system is in an unbalanced state, and the root cause lies in the storage strategy of the data table in the database system.

[0054] Based on the above load conditions, it changes the storage strategy of certain data tables to adjust the balance of the database system, and needs to adjust the storage strategy of data tables with low disk IO resource occupancy, high network bandwidth resource occupancy and high usage frequency. Among them, when storing data according to the second storage strategy, the data in the data table is divided into at least two sub-data, and stored in at least two database nodes respectively; wherein, the data collection of the database node about the data table is a copy of all the data of the data table; that is, the data table stored according to the second storage strategy has a low disk IO resource occupancy rate and a high network bandwidth resource occupancy rate. In other words, it is the imbalance caused by too many data tables stored in the database with the second storage strategy. Therefore, it is necessary to adjust some of the data tables stored with the second storage strategy.

[0055] Specifically, based on the second load condition, combined with the storage strategy and parameter conditions of each data table, some data tables with low disk IO resource occupancy, high network bandwidth resource occupancy and high usage frequency are selected from the database system, that is, the second type of data tables stored with the second storage strategy.

[0056] The selected second type of data table stored in the second storage strategy is adjusted to the first storage strategy storage according to the second adjustment strategy, thereby alleviating the load pressure caused by satisfying the second load condition. The function of the second adjustment strategy is to adjust the second storage strategy to the first storage strategy.

[0057] At the same time, since only one storage strategy can usually be set when creating a table in a database system and it is manually set in advance, when the storage strategy needs to be changed, it can only be changed through manual modification. The technical solution of the present application, which dynamically adjusts the data storage strategy of some tables, reduces the workload of staff and realizes certain automated operations.

[0058] By adopting the above technical solution, the first load parameter and the second load parameter of the database system are obtained; when it is determined based on the first load parameter and the second load parameter that the first load condition is met, the storage strategy of the first type of data table in the database system is adjusted from the first storage strategy to the second storage strategy; when it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy. In this way, when the imbalance of the system is determined by using the first load parameter and the second load parameter, the storage strategy of some of its data tables is dynamically adjusted according to the satisfied load condition, which alleviates the load pressure of the data system and thus improves the resource utilization of the database system.

[0059] Based on the above embodiments, the present application provides another database data processing method. Figure 2 : is a second flow chart of the database data processing method in the embodiment of the present application, such as Figure 2 As shown, the database data processing method may specifically include:

[0060] Step 201: Acquire a first load parameter and a second load parameter of a database system;

[0061] Before running the dynamic adjustment control program, the adjustment cycle of the database cluster needs to be set in advance. When the database cluster meets an adjustment cycle, the first load parameter and the second load parameter of the database system are obtained, and the storage strategy of the data table is adjusted. After that, the load dynamic adjustment control program is run once every adjustment cycle until each data table no longer stores data, that is, the running program is completely terminated.

[0062] After obtaining the first load parameter and the second load parameter of the database system, the present application can find the root cause of the load imbalance in the database system through the first load parameter and the second load parameter; since the load imbalance in the database system is caused by the storage strategy, usage frequency, etc. of each data table, in order to improve the imbalance phenomenon, in addition to obtaining the storage strategy of each data table in the database system, it is also necessary to obtain the third load parameter, the fourth load parameter and the usage frequency of at least one data table in the database; and then determine the data table whose storage strategy needs to be adjusted based on the third load parameter, the fourth load parameter and the usage frequency.

[0063] The load parameters mentioned above, specifically, the first load parameter is the system's disk IO resource occupancy rate, the second load parameter is the system's network bandwidth resource occupancy rate; the third load parameter is the data table's disk IO resource occupancy rate, and the fourth load parameter is the data table's network bandwidth resource occupancy rate.

[0064] Step 202: if the first load parameter and the second load parameter are determined to satisfy a first load condition, determining a first type of data table in the database based on the third load condition and the usage frequency;

[0065] When it is determined based on the first load parameter and the second load parameter that the database system is unbalanced, the first load condition or the second load condition is satisfied. The following is a detailed description of satisfying the first load condition:

[0066] When the first load parameter of the database system is greater than the first threshold and the second load parameter is less than the second threshold, that is, when the disk IO resource occupancy rate is greater than the first threshold and the network bandwidth resource occupancy rate is less than the second threshold, some data tables with large disk IO resource occupancy rate and high usage frequency are searched from at least one data table in the database, and the storage strategy of the selected data tables is adjusted to alleviate the load pressure caused by the excessively large disk IO resource occupancy rate.

[0067] In some embodiments, determining the first category of data tables in the database system based on the third load parameter and the usage frequency includes: arranging the at least one data table from large to small according to the third load parameter and the usage frequency to obtain a first arrangement order of the data tables; based on the first arrangement order of the data tables, selecting the first N data tables stored with a first storage strategy from the at least one data table; wherein N is a positive integer; and using the first N data tables stored with the first storage strategy as the first category of data tables.

[0068] When storing data according to the first storage strategy, all data in the data table is stored in multiple database nodes on the server node, that is, each database node stores a copy of all data, that is, the data table stored according to the first storage strategy has a large disk IO resource occupancy rate and a small network bandwidth resource occupancy rate. In other words, the imbalance is caused by too many data tables stored in the database according to the first storage strategy. Therefore, when the first load parameter and the second load parameter meet the first load condition, some data tables stored according to the first storage strategy need to be adjusted.

[0069] The first category of data tables is a data table for which the storage strategy needs to be adjusted based on the third load parameter and the frequency of use. Specifically, at least one data table in the database system is arranged in descending order based on the third load parameter and the frequency of use to obtain a first arrangement order of at least one data table. Adjusting the balance of the database system is a slow process. When N takes a value of 1, the previous data table stored with the first storage strategy in at least one data table in the first arrangement order is selected as the first category of data tables to dynamically adjust the database system. When N takes a value of 2, the first two data tables stored with the first storage strategy in at least one data table in the first arrangement order are selected as the first category of data tables to dynamically adjust the database system. And so on.

[0070] Step 203: adjusting the storage strategy of the first type of data table in the database system from the first storage strategy to the second storage strategy;

[0071] The selected first type of data table stored in the first storage strategy is adjusted to the second storage strategy according to the first adjustment strategy, thereby alleviating the load pressure caused by the first load condition. The first adjustment strategy function is to adjust the first storage strategy to the second storage strategy.

[0072] Step 204: if the first load parameter and the second load parameter are determined to satisfy a second load condition, determining a second type of data table in the database based on the fourth load condition and the usage frequency;

[0073] A specific description is given for satisfying the second load condition:

[0074] When the first load parameter of the database system is less than the third threshold and the second load parameter is greater than the fourth threshold, that is, when the disk IO resource occupancy rate is less than the third threshold and the network bandwidth resource occupancy rate is greater than the fourth threshold, some data tables with large network bandwidth resource occupancy rate and high usage frequency are searched from at least one data table in the database, and the storage strategy of the selected data tables is adjusted to alleviate the load pressure caused by the excessive network bandwidth resource occupancy rate.

[0075] In some embodiments, determining the second category of data tables in the database system based on the fourth load parameter and the usage frequency includes: arranging the at least one data table from large to small according to the fourth load parameter and the usage frequency to obtain a second arrangement order of the data tables; based on the second arrangement order of the data tables, selecting the first M data tables stored with a second storage strategy from the at least one data table; wherein M is a positive integer; and using the first M data tables stored with the second storage strategy as the second category of data tables.

[0076] When storing data according to the second storage strategy, the data in the data table is divided into at least two sub-data and stored in at least two database nodes respectively; wherein, the data collection of the database node about the data table is a copy of all the data of the data table; that is, the data table stored according to the second storage strategy has a small disk IO resource occupancy rate and a large network bandwidth resource occupancy rate. In other words, it is the imbalance caused by too many data tables stored in the database according to the second storage strategy. Therefore, when the first load parameter and the second load parameter meet the second load condition, some data tables stored according to the second storage strategy need to be adjusted.

[0077] The second category of data tables is a data table for which the storage strategy needs to be adjusted based on the fourth load parameter and the frequency of use. Specifically, at least one data table in the database system is arranged in descending order based on the fourth load parameter and the frequency of use to obtain a second arrangement order of at least one data table. Adjusting the balance of the database system is a slow process. When the value of M is 1, the first data table stored with the second storage strategy in at least one data table in the second arrangement order is selected as the second category of data tables to dynamically adjust the database system. When the value of M is 2, the first two data tables stored with the second storage strategy in at least one data table in the second arrangement order are selected as the second category of data tables to dynamically adjust the database system. And so on.

[0078] Exemplarily, when the database system is in an unbalanced state and needs to be dynamically adjusted for the first time, at this time, if the first load parameter and the second load parameter determine that they meet the first load condition, that is, the first load parameter is greater than the first threshold value and the second load parameter is less than the second threshold value, then the first N data tables stored in the first storage strategy in at least one data table in the first arrangement order are selected as the first type of data tables to perform the first dynamic adjustment on the database system; after waiting for an interval period, the first load parameter and the second load parameter of the database system are re-acquired again, and at this time, the first load parameter and the second load parameter determine that they still meet the first load condition, and it is necessary to adjust the at least one data table based on the third load parameter and the usage frequency again. A data table is arranged from large to small, and the first N data tables stored with the first storage strategy in at least one data table in the arrangement order are selected as first-category data tables to perform a second dynamic adjustment on the database system; if at this time the first load parameter and the second load parameter are determined to meet the second load condition, that is, the first load parameter is less than the third threshold and the second load parameter is greater than the fourth threshold, it is necessary to arrange at least one data table from large to small based on the fourth load parameter and the frequency of use, and select the first M data tables stored with the second storage strategy in the second arrangement order as second-category data tables to perform a second dynamic adjustment on the database system; and so on, until the entire program completely ends its running state.

[0079] Step 205: Adjust the storage strategy of the second type of data table in the database system from the second storage strategy to the first storage strategy.

[0080] The selected second type of data table stored in the second storage strategy is adjusted to the first storage strategy storage according to the second adjustment strategy, thereby alleviating the load pressure caused by satisfying the second load condition. The function of the second adjustment strategy is to adjust the second storage strategy to the first storage strategy.

[0081] Since usually only one storage strategy can be set when creating a table in a database system and it is manually set in advance, when the storage strategy needs to be changed, it can only be changed manually. The technical solution of the present application, which dynamically adjusts the data storage strategy of some tables, reduces the workload of staff and realizes certain automated operations.

[0082] By adopting the above technical solution, the first load parameter and the second load parameter of the database system are obtained; when it is determined based on the first load parameter and the second load parameter that the first load condition is met, the storage strategy of the first type of data table in the database system is adjusted from the first storage strategy to the second storage strategy; when it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy. In this way, when the imbalance of the system is determined by using the first load parameter and the second load parameter, the storage strategy of some of its data tables is dynamically adjusted according to the satisfied load condition, which alleviates the load pressure of the data system and thus improves the resource utilization of the database system.

[0083] In order to better reflect the purpose of the present invention, the above scheme is specifically illustrated by examples. Figure 3 This is a third flow chart of the database data processing method in the embodiment of the present application.

[0084] The following uses the first storage strategy as the replication table strategy and the second storage strategy as the distribution key strategy to illustrate the distributed database data processing method. The specific distributed database processing process is as follows: Figure 3 As shown, the process includes:

[0085] Step 301: Start;

[0086] This application uses a load dynamic adjustment control program to adjust the storage strategy of each data table so that the load of the distributed database system can be in a balanced state. Before running the dynamic adjustment process, it is necessary to pre-set the adjustment cycle of the database cluster and then start the dynamic adjustment control program; when the database cluster meets an adjustment cycle, the load dynamic adjustment control program is run once.

[0087] Step 302: Obtain the system's disk IO resource occupancy and network bandwidth resource occupancy;

[0088] Specifically, after starting the dynamic adjustment control program, the disk IO resource occupancy, network bandwidth resource occupancy and usage frequency of each data table in the distributed database system, as well as the disk IO resource occupancy and network bandwidth resource occupancy of the distributed database system are monitored in real time.

[0089] Since the present application determines whether the distributed database system is in a balanced state based on the system's disk IO resource occupancy rate and network bandwidth resource occupancy rate, it is necessary to obtain the system's disk IO resource occupancy rate and network bandwidth resource occupancy rate before making a judgment; based on the system's disk IO resource occupancy rate and network bandwidth resource occupancy rate, find the root cause of the load imbalance in the distributed database system.

[0090] Step 303: Determine whether the system is in a balanced state; if it is in a balanced state, execute step 312; if it is in an unbalanced state, execute step 304;

[0091] Step 304: determine whether the first load condition is met; if so, execute step 305; if not, that is, the second load condition is met, execute step 306;

[0092] When the disk IO resource occupancy rate and the network bandwidth resource occupancy rate of the distributed database system are determined to meet the first load condition, the system is in an unbalanced state; wherein the first load condition is: the disk IO resource occupancy rate is greater than the first threshold, and the network bandwidth resource occupancy rate is less than the second threshold.

[0093] Exemplarily, when the disk IO resource occupancy rate of the current distributed database system is greater than 80% and the network bandwidth resource occupancy rate is less than 50%, the distributed database system is in an unbalanced state, and step 305 is then executed.

[0094] When the disk IO resource occupancy rate and the network bandwidth resource occupancy rate of the distributed database system are determined to meet the second load condition, the system is in an unbalanced state; wherein the second load condition is: the disk IO resource occupancy rate is less than the third threshold, and the network bandwidth resource occupancy rate is greater than the fourth threshold.

[0095] Exemplarily, when the disk IO resource occupancy rate of the current distributed database system is less than 50% and the network bandwidth resource occupancy rate is greater than 80%, the distributed database system is in an unbalanced state, and step 306 is then executed.

[0096] Step 305: Obtain the disk IO resource occupancy rate and usage frequency of each data table;

[0097] Since the load imbalance in the distributed database system is caused by the load parameters, usage times, etc. of each data table, in order to improve the imbalance, it is necessary to obtain the disk IO resource occupancy rate, network bandwidth resource occupancy rate and usage frequency of at least one data table in the distributed database; and then determine the data table for which the storage strategy needs to be adjusted based on the disk IO resource occupancy rate, network bandwidth resource occupancy rate and usage frequency.

[0098] Here, the purpose of obtaining the disk IO resource occupancy rate and usage frequency of each data table is to prepare for step 307 .

[0099] Step 306: Obtain the network bandwidth resource occupancy rate and usage frequency of each data table;

[0100] The purpose of obtaining the network bandwidth resource occupancy rate and usage frequency of each data table is to prepare for step 308 .

[0101] Step 307: Determine the first type of data table of the distributed database system based on the disk IO resource occupancy rate and usage frequency;

[0102] Since the present application adjusts the storage strategy of each data table in the distributed database according to the load dynamic adjustment control program, it is necessary to obtain the current storage strategy of each data table before running the dynamic adjustment control program each time.

[0103] For example, a distributed database system may have hundreds or thousands of server nodes, and these server nodes often have multiple database nodes, each of which is configured according to a preset storage strategy, such as Figure 4 This is a schematic diagram of the storage strategy for three data tables in the distributed database in the embodiment of the present application. The first data table stores data according to the distribution key strategy, the second data table stores data according to the replication table strategy, and the third data table stores data according to the distribution key strategy. Wherein, when storing data according to the distribution key strategy, the data in the data table is divided into at least two sub-data, and stored in at least two database nodes respectively; wherein the data collection of the database node about the data table is a copy of all the data of the data table; when storing table data according to the replication table strategy, all the data of the data table is stored in multiple database nodes on the server node.

[0104] When the disk IO resource occupancy rate of the distributed database system is greater than 80% and the network bandwidth resource occupancy rate is less than 50%, the distributed database system is in an unbalanced state. It can be seen that the disk IO resource occupancy rate of the current distributed database system is too high, resulting in an unbalanced system load. Since each database node stores a full copy of the data when the replication table strategy is used to store data, the high disk resource occupancy rate of the current system is caused by the fact that too many data tables in the distributed database are stored according to the replication table strategy. At this time, it is necessary to relieve the system load pressure. Specifically, the disk IO resource occupancy rate and usage frequency of each data table are obtained, and they are sorted in order from large to small. At least the previous data table stored according to the replication table strategy, that is, the first type of data table, is selected from the sorted sequence table.

[0105] Step 308: Determine the first type of data table of the distributed database system based on the network bandwidth resource occupancy rate and usage frequency;

[0106] When the disk IO resource occupancy rate of the distributed database system is less than 50% and the network bandwidth resource occupancy rate is greater than 80%, the distributed database system is in an unbalanced state. It can be seen that the network bandwidth resource occupancy rate of the current distributed database system is too high, resulting in an unbalanced system load. Since the distributed key strategy stores data, data redistribution and broadcasting are prone to occur during task execution, which in turn causes a higher network load. Therefore, the high network bandwidth resource occupancy rate of the current system originates from the fact that too many data tables in the distributed database are stored according to the distributed key strategy. At this time, it is necessary to alleviate the system load pressure. Specifically, the network bandwidth resource occupancy rate and usage frequency of each data table are obtained, and they are sorted in order from large to small. At least the previous data table stored according to the distributed key strategy is selected from the sorted sequence table, that is, the second type of data table.

[0107] Step 309: adjusting the storage strategy of the first type of data table in the distributed database system from the replication table storage strategy to the distribution key storage strategy;

[0108] According to the adjustment strategy of converting the distribution key strategy according to the replication table strategy, the first type of data table selected for storage according to the replication table strategy is adjusted to be stored according to the distribution key strategy.

[0109] Step 310: adjusting the storage strategy of the second type of data table in the distributed database system from the distribution key storage strategy to the replication table storage strategy;

[0110] According to the adjustment strategy of converting the replication table strategy into the distribution key strategy, the second type of data table selected for storage according to the distribution key strategy is adjusted to be stored according to the replication table strategy.

[0111] Step 311: Obtain the system load parameter again; then, execute step 303 to determine again whether the system is in a balanced state;

[0112] Specifically, every other adjustment cycle, the disk IO resource occupancy rate and network bandwidth resource occupancy rate of the distributed database system need to be re-obtained to prepare for determining whether the system is in a balanced state again. Continue to repeat the previous workflow until the entire system program is completely completed.

[0113] Step 312: End.

[0114] The end here does not mean that the system will not be judged again when it is in balance. As long as the program is running, it will be judged once every adjustment cycle, and it will only end when the entire program is completely finished.

[0115] By adopting the above technical solution, the first load parameter and the second load parameter of the database system are obtained; when it is determined based on the first load parameter and the second load parameter that the first load condition is met, the storage strategy of the first type of data table in the database system is adjusted from the first storage strategy to the second storage strategy; when it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy. In this way, when the imbalance of the system is determined by using the first load parameter and the second load parameter, the storage strategy of some of its data tables is dynamically adjusted according to the satisfied load condition, which alleviates the load pressure of the distributed data system and thus improves the resource utilization of the distributed database system.

[0116] The present application also provides a database data processing device, such as Figure 5 As shown, the device comprises:

[0117] An acquisition unit 501 is used to acquire a first load parameter and a second load parameter of a database system;

[0118] An adjusting unit 502 is configured to adjust a storage strategy of a first type of data table in the database system from a first storage strategy to a second storage strategy when determining that a first load condition is met based on the first load parameter and the second load parameter;

[0119] The adjusting unit 502 is further configured to adjust the storage strategy of the second type of data table in the database system from the second storage strategy to the first storage strategy when determining that a second load condition is met based on the first load parameter and the second load parameter.

[0120] In some embodiments, the first load parameter is the system's disk IO resource occupancy rate, and the second load parameter is the system's network bandwidth resource occupancy rate; the first load condition is: the first load parameter is greater than a first threshold and the second load parameter is less than a second threshold; the second load condition is: the first load parameter is less than a third threshold and the second load parameter is greater than a fourth threshold.

[0121] In some embodiments, the device also includes: a determination unit, used to obtain a third load parameter, a fourth load parameter and a usage frequency of at least one data table; wherein the third load parameter is the disk IO resource occupancy rate of the data table, and the fourth load parameter is the network bandwidth resource occupancy rate of the data table; if the first load parameter and the second load parameter are determined to meet the first load condition, a first type of data table in the database system is determined based on the third load parameter and the usage frequency; if the first load parameter and the second load parameter are determined to meet the second load condition, a second type of data table in the database system is determined based on the fourth load parameter and the usage frequency.

[0122] In some embodiments, the device includes: the determining unit is specifically used to arrange the at least one data table from large to small according to the third load parameter and the usage frequency to obtain a first arrangement order of the data table; based on the first arrangement order of the data table, select the first N data tables stored with a first storage strategy from the at least one data table; wherein N is a positive integer; and use the first N data tables stored with the first storage strategy as the first type of data tables;

[0123] It is also specifically used to arrange the at least one data table in descending order according to the fourth load parameter and the usage frequency to obtain a second arrangement order of the data table; based on the second arrangement order of the data table, select the first M data tables stored with the second storage strategy from the at least one data table; wherein M is a positive integer; and use the first M data tables stored with the second storage strategy as the second category data tables.

[0124] In some embodiments, the apparatus includes: the adjustment unit 502, specifically configured to adjust the first type of data table stored with the first storage policy to be stored with the second storage policy according to the first adjustment policy;

[0125] It is also specifically used to adjust the second type of data table stored with the second storage policy to storage with the first storage policy according to the second adjustment policy.

[0126] In some embodiments, the first storage strategy is stored as storing all data in the data table into a database node; the second storage strategy is stored as dividing the data in the data table into at least two sub-data, and storing the at least two sub-data in at least two database nodes respectively.

[0127] In some embodiments, the device further includes: an operating unit, configured to obtain a first load parameter and a second load parameter of the database system and adjust a storage strategy of a data table when the database cluster meets an adjustment cycle.

[0128] By adopting the above technical solution, the first load parameter and the second load parameter of the database system are obtained; when it is determined based on the first load parameter and the second load parameter that the first load condition is met, the storage strategy of the first type of data table in the database system is adjusted from the first storage strategy to the second storage strategy; when it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy. In this way, when the imbalance of the system is determined by using the first load parameter and the second load parameter, the storage strategy of some of its data tables is dynamically adjusted according to the satisfied load condition, which alleviates the load pressure of the data system and thus improves the resource utilization of the database system.

[0129] The present application also provides a database data processing device, such as Figure 6 As shown, the device includes: a processor 601 and a memory 602 configured to store a computer program that can be run on the processor; when the processor 601 runs the computer program in the memory 602, the following steps are implemented:

[0130] Obtaining a first load parameter and a second load parameter of the database system;

[0131] When it is determined based on the first load parameter and the second load parameter that a first load condition is met, adjusting a storage strategy of a first type of data table in the database system from a first storage strategy to a second storage strategy;

[0132] When it is determined based on the first load parameter and the second load parameter that the second load condition is met, the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy.

[0133] In some embodiments, the first load parameter is the system's disk IO resource occupancy rate, and the second load parameter is the system's network bandwidth resource occupancy rate; the first load condition is: the first load parameter is greater than a first threshold and the second load parameter is less than a second threshold; the second load condition is: the first load parameter is less than a third threshold and the second load parameter is greater than a fourth threshold.

[0134] In some embodiments, when the processor 601 runs the computer program in the memory 602, the following steps are also implemented: obtaining a third load parameter, a fourth load parameter and a usage frequency of at least one data table; wherein the third load parameter is the disk IO resource occupancy rate of the data table, and the fourth load parameter is the network bandwidth resource occupancy rate of the data table; if the first load parameter and the second load parameter are determined to meet the first load condition, determining the first type of data table in the database system based on the third load parameter and the usage frequency; if the first load parameter and the second load parameter are determined to meet the second load condition, determining the second type of data table in the database system based on the fourth load parameter and the usage frequency.

[0135] In some embodiments, when the processor 601 runs the computer program in the memory 602, the following steps are further implemented: arranging the at least one data table from large to small according to the third load parameter and the usage frequency to obtain a first arrangement order of the data table; selecting the first N data tables stored with the first storage strategy from the at least one data table based on the first arrangement order of the data table; wherein N is a positive integer; using the first N data tables stored with the first storage strategy as the first type of data tables;

[0136] Arrange the at least one data table in descending order according to the fourth load parameter and the usage frequency to obtain a second arrangement order of the data tables; select the first M data tables stored with the second storage strategy from the at least one data table based on the second arrangement order of the data tables; wherein M is a positive integer; and use the first M data tables stored with the second storage strategy as the second category data tables.

[0137] In some embodiments, when the processor 601 runs the computer program in the memory 602, it also implements the following steps: adjusting the first type of data table stored with the first storage policy to storage with the second storage policy according to the first adjustment strategy; adjusting the second type of data table stored with the second storage policy to storage with the first storage policy according to the second adjustment strategy.

[0138] In some embodiments, the first storage strategy is stored as storing all data in the data table into a database node; the second storage strategy is stored as dividing the data in the data table into at least two sub-data, and storing the at least two sub-data in at least two database nodes respectively.

[0139] In some embodiments, when the processor 601 runs the computer program in the memory 602 , the following steps are also implemented: when the database cluster meets an adjustment cycle, the first load parameter and the second load parameter of the database system are obtained, and the storage strategy of the data table is adjusted.

[0140] Of course, in practical applications, Figure 6 As shown, the various components in the database data processing device are coupled together via a bus system 603. It can be understood that the bus system 603 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 603 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 603.

[0141] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0142] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0143] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0144] Optionally, the computer-readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the processor in each method in the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0146] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0147] In addition, all functional units in the embodiments of the present application can be integrated into one processing module, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiment can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the above storage medium includes: mobile storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0148] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0149] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0150] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A database data processing method, characterized in that: The method comprises: Obtaining a first load parameter and a second load parameter of the database system; wherein the first load parameter is the disk IO resource occupancy rate of the system, and the second load parameter is the network bandwidth resource occupancy rate of the system; In the case where the first load parameter is greater than a first threshold and the second load parameter is less than a second threshold, determining a first type of data table in the database system based on a third load parameter and a usage frequency of the data table, and adjusting the storage strategy of the first type of data table in the database system from a first storage strategy to a second storage strategy; wherein the first storage strategy is a replication table storage strategy, and the second storage strategy is a distribution key storage strategy; and the third load parameter is a disk IO resource occupancy rate of the data table; When the first load parameter is less than a third threshold and the second load parameter is greater than a fourth threshold, a second type of data table in the database system is determined based on the fourth load parameter and usage frequency of the data table, and the storage strategy of the second type of data table in the database system is adjusted from the second storage strategy to the first storage strategy; wherein the fourth load parameter is the network bandwidth resource occupancy rate of the data table.

2. The method according to claim 1, characterized in that The determining the first type of data table in the database system based on the third load parameter and the usage frequency of the data table includes: Arrange at least one data table in descending order according to the third load parameter and the usage frequency to obtain a first arrangement order of the data table; Selecting the first N data tables stored in a first storage strategy from the at least one data table based on the first arrangement order of the data tables; wherein N is a positive integer; The first N data tables stored by the first storage strategy are used as the first type of data tables; The determining the second type of data table in the database system based on the fourth load parameter and the usage frequency of the data table includes: Arrange the at least one data table in descending order according to the fourth load parameter and the usage frequency to obtain a second arrangement order of the data table; Selecting the first M data tables stored in a second storage strategy from the at least one data table based on the second arrangement order of the data tables; wherein M is a positive integer; The first M data tables stored using the second storage strategy are used as the second type of data tables.

3. The method according to claim 1, characterized in that The first storage strategy is stored as storing all data of the data table into the database node; The second storage strategy is stored as dividing the data in the data table into at least two sub-data, and storing the at least two sub-data in at least two database nodes respectively.

4. The method according to claim 1, characterized in that: The obtaining of the first load parameter and the second load parameter of the database system includes: When the database cluster meets an adjustment cycle, the first load parameter and the second load parameter of the database system are obtained, and the storage strategy of the data table is adjusted.

5. A database data processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first load parameter and a second load parameter of a database system; wherein the first load parameter is a disk IO resource occupancy rate of the system, and the second load parameter is a network bandwidth resource occupancy rate of the system; an adjusting unit, configured to determine, when the first load parameter is greater than a first threshold and the second load parameter is less than a second threshold, a first type of data table in the database system based on a third load parameter and a usage frequency of the data table, and adjust the storage strategy of the first type of data table in the database system from a first storage strategy to a second storage strategy; wherein the first storage strategy is a replication table storage strategy, and the second storage strategy is a distribution key storage strategy; and the third load parameter is a disk IO resource occupancy rate of the data table; The adjustment unit is used to determine the second type of data table in the database system based on the fourth load parameter and usage frequency of the data table when the first load parameter is less than a third threshold and the second load parameter is greater than a fourth threshold, and adjust the storage strategy of the second type of data table in the database system from the second storage strategy to the first storage strategy; wherein the fourth load parameter is the network bandwidth resource occupancy rate of the data table.

6. A database data processing device, the device comprising: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 4 when running the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images