Resource processing method and device for database migration

Abstract

The embodiment of the invention provides a resource processing method and device for database migration, and the method is applied to computing equipment, and comprises the steps: obtaining static information of an original database, parameter information of a target database, and request parameters and load parameters in a preset time period, determining an initial resource, a test data model and a simulation request tool, simulating static information of an original database and request parameters in a preset time period to obtain initial test data, adjusting the request parameters to obtain adjusted target request parameters and target test data, deploying the target test data to a target database, adjusting initial resources to determine target resources required by the target database, and sending the target resources to the target database. And finally outputting the target resource. Based on the method provided by the invention, the accuracy rate of the resources required by the target database can be determined.

Description

Technical Field

[0001] The present application relates to the field of database migration, and in particular to a resource processing method and device for database migration. Background Art

[0002] Database migration refers to the process of moving the data of the original database to the target database, which usually includes physical migration and logical migration. Heterogeneous database migration means that the original database and the target database are not the same type of database. Before heterogeneous database migration, you need to determine the resources required in the target database.

[0003] Currently, before heterogeneous database migration, the methods for determining the resources required for the target database mainly include static estimation based on experience and estimation based on reference to similar systems.

[0004] However, the current method of determining the resources required by the target database cannot accurately determine the resources required by the target database, resulting in excessive operating load of the target database after moving the data of the original database to the target database, resulting in low operating efficiency of the target data, or low operating load of the target database, resulting in waste of resources. Summary of the invention

[0005] The embodiments of the present application provide a resource processing method and device for database migration, so as to improve the accuracy of determining the resources required for a target database.

[0006] In a first aspect, an embodiment of the present application provides a resource processing method for database migration, which is applied to a computing device, including:

[0007] Obtain static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period;

[0008] Determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database;

[0009] Determine a test data model and a simulation request tool based on the static information of the original database;

[0010] Initializing the test data model and the simulation request tool;

[0011] Simulating the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain the initial test data corresponding to the original database in the preset time period;

[0012] Taking the load parameters of the original database in the preset time period as a reference, adjusting the simulated request parameters until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configuring the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period;

[0013] Deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period;

[0014] Taking the load parameter of the original database in the preset time period as a reference, adjusting the initial resources until the load parameter corresponding to the target database is consistent with the load parameter of the original database, so as to determine the target resources required by the target database in the preset time period;

[0015] Output the target resources required by the target database within the preset time period.

[0016] In a possible implementation, the static information of the original database includes: the total number of CPU cores, the total memory, the total data volume and the network bandwidth resources, and the parameter information of the target database includes: the number of CPU cores of a single node, the memory of a single node and the data capacity of a single node;

[0017] Accordingly, determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database includes:

[0018] Obtaining a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database;

[0019] Determine the maximum value among the first ratio, the second ratio and the third ratio as the initial number of nodes in the target library;

[0020] Determining the network broadband resources of the original database as the initial network broadband resources;

[0021] The initial number of nodes in the target library and the initial network bandwidth resources are determined as the initial resources.

[0022] In a possible implementation, the static information of the original database includes: database type, query mode, and ratio information of read and write operations on data;

[0023] Accordingly, the determining of the test data model and the simulation request tool based on the static information of the original database includes:

[0024] If the database type of the original database is transactional, and the proportion of the read operation exceeds a preset threshold, the test data model and simulation request tool are determined to be test data models and simulation request tools belonging to the first type of environment type;

[0025] If the database type of the original database is transactional, and the proportion of write operations performed on the original database exceeds a preset threshold, the test data model and simulation request tool are determined to be test data models and simulation request tools belonging to the second type of environment;

[0026] If the database type of the original database is analytical, and the query mode is ad hoc query, the test data model and simulation request tool are determined to be test data models and simulation request tools belonging to the third type of environment;

[0027] If the database type of the original database is analytical, and the query method is data processing, the test data model and simulation request tool are determined as test data models and simulation request tools belonging to the fourth environment type.

[0028] In a possible implementation, the static information of the original database includes: hardware configuration environment information;

[0029] Accordingly, the adjusting the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database includes:

[0030] Deploy the initial test data in the same hardware configuration environment as the original database;

[0031] In the same hardware configuration environment as the original database, the simulated request parameters are adjusted based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

[0032] In a possible implementation manner, after adjusting the initial resources based on the load parameter of the original database in the preset time period until the load parameter corresponding to the target database is consistent with the load parameter of the original database to determine the target resources required by the target database in the preset time period, the method further includes:

[0033] Acquire a first resource occupied by deploying the target test data on the target database;

[0034] Acquire a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database;

[0035] The first resource and the second resource are released.

[0036] In a possible implementation manner, the preset time period includes a peak time period or a non-peak time period.

[0037] In a second aspect, an embodiment of the present application provides a resource processing device for database migration, which is applied to a computing device, including:

[0038] An acquisition module is used to acquire static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period;

[0039] A processing module, used for determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database;

[0040] The processing module is further used to determine a test data model and a simulation request tool based on the static information of the original database;

[0041] The processing module is also used to initialize the test data model and the simulation request tool;

[0042] The processing module is further used to simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain the initial test data corresponding to the original database in the preset time period;

[0043] The processing module is further used to adjust the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period;

[0044] The processing module is further used to deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period;

[0045] The processing module is further used to adjust the initial resources based on the load parameters of the original database in the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database in the preset time period;

[0046] The processing module is further used to output the target resources required by the target database in the preset time period.

[0047] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor;

[0048] Memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0052] The resource processing method and device for database migration provided in the embodiment of the present application determines the target test data in the preset time period by acquiring the static information of the original database, the parameter information of the target database, and the request parameters and load parameters in the preset time period, and obtains the target resources required by the target database in the preset time period by deploying the target test data to the target database. Based on the method provided by the present application, the target resources actually used after deployment can be obtained by deploying the target test data corresponding to the original database on the target database, so as to achieve accurate prediction of the resources required on the target database, thereby improving the accuracy of predicting the resources required by the target database. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 Provide a scenario diagram for this application;

[0055] Figure 2Schematic diagram of the resource processing method for database migration provided in this application Figure 1 ;

[0056] Figure 3 Schematic diagram of the resource processing method for database migration provided in this application Figure 2 ;

[0057] Figure 4 Schematic diagram of the resource processing method for database migration provided in this application Figure 3 ;

[0058] Figure 5 Schematic diagram of the resource processing method for database migration provided in this application Figure 4 ;

[0059] Figure 6 Schematic diagram of the resource processing method for database migration provided in this application Figure 5 ;

[0060] Figure 7 A schematic diagram of the structure of a resource processing device for database migration provided by the present application;

[0061] Figure 8 A schematic diagram of the structure of a resource processing system for database migration provided in this application;

[0062] Fig. 9 A schematic diagram of the structure of the computer device provided for this application.

[0063] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] First, the terms involved in this application are explained:

[0066] Slow SQL: refers to Structured Query Language (SQL), including P99, P95, and P90 slow SQL statistics or top time-consuming SQL statistics.

[0067] Figure 1 Provide a scenario diagram for this application, such as Figure 1 As shown, database migration refers to the process of moving the data of the original database to the target database, which usually includes physical migration and logical migration. Heterogeneous database migration means that the original database and the target database are not the same database. Before heterogeneous database migration, you need to determine the resources required in the target database.

[0068] Currently, before heterogeneous database migration, the methods for determining the resources required for the target database mainly include static estimation based on experience and estimation based on reference to similar systems.

[0069] However, the current method of determining the resources required by the target database cannot accurately determine the resources required by the target database, resulting in excessive operating load of the target database after moving the data of the original database to the target database, resulting in low operating efficiency of the target data, or low operating load of the target database, resulting in waste of resources.

[0070] The resource processing method for database migration provided by the present application obtains the static information of the original database, the parameter information of the target database, and the request parameters and load parameters in a preset time period, and presets the initial resources, test data model and simulation request tool required in the target database. Based on the test data model and the simulation request tool, the static information of the original database and the request parameters in the preset time period are simulated to obtain the corresponding initial test data, and the target test data is obtained by adjusting the request parameters simulated by the initial test data. The corresponding load parameters are obtained by deploying the target test data to the target database, and the initial resources are adjusted until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the required target resources. Based on the method provided by the present application, the target test data corresponding to the original database can be deployed on the target database to obtain the target resources actually used after deployment, so as to achieve accurate prediction of the resources required on the target database, and improve the accuracy of predicting the resources required for the target database.

[0071] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0072] Figure 2 Schematic diagram of the resource processing method for database migration provided in this application Figure 1 , applied to computing devices, such as Figure 2 As shown, the method includes:

[0073] S201, obtaining static information of an original database, parameter information of a target database, and request parameters and load parameters in a preset time period.

[0074] The execution subject of this embodiment is a computing device, which can be implemented through a computer program, such as application software, etc.; or, it can also be implemented as a medium storing relevant computer programs, such as a USB flash drive, a cloud disk, etc.; or, it can also be implemented through a physical device integrated or installed with relevant computer programs, such as a chip, etc.

[0075] Combined with the scenario example, the static information includes: CPU information, memory information, disk storage type, disk capacity, data volume and daily increment, log volume and daily increment, network bandwidth, network latency, database deployment architecture, read-write operation ratio, database type, query method, field data type, hardware configuration environment information and data warehouse model on the original database deployment server. Among them, CPU information includes the number of cores, frequency and architecture, memory information includes capacity and speed, and disk storage types include Serial Advanced Technology Attachment (SATA), Solid State Disk (SSD), Non-Volatile Memory Express (NVME), Solid-State Drive (SSD) or traditional Hard Disk Drive (Hard Disk). Drive, referred to as HDD), database deployment architecture includes single-node deployment, master-slave deployment and distributed deployment, database types include transactional and analytical, query methods include Adhoc ad hoc query and ETL (Extract, Transform, Load) data processing, field data types include mainly character type, mainly numeric type and mainly binary data, data warehouse models include snowflake type and star type. The parameter information of the target database includes the number of CPU cores per node, single node memory and single node data capacity.

[0076] Request parameters and load parameters are dynamically changing parameters that change with different time periods. Request parameters include the number of clients, transactions per second (TPS), and queries per second (QPS).

[0077] Load parameters include CPU usage, memory usage, disk read and write operations per second (Input / Output Operations Per Second, IOPS), network traffic, request latency, and other information.

[0078] S202: Determine the initial resources required in the target database according to the static information of the original database and the parameter information of the target database.

[0079] Combined with the scenario example, the initial resources required in the target database include the initial number of nodes in the target database and the initial network bandwidth resources.

[0080] S203: Determine a test data model and a simulation request tool based on the static information of the original database.

[0081] Combined with the scenario examples, the working scenario of the original database can be determined based on the static information of the original database, and the test data model and simulation request tool corresponding to the working scenario of the original database can be determined through the working scenario of the original database. The working scenarios of the original database include: multi-read scenarios of transactional business systems, multi-write scenarios of transactional business systems, ad hoc query scenarios of analytical business systems, and data processing scenarios of analytical business systems. Therefore, the test data model and simulation request tool can be determined based on the four working scenarios of the original database. For example, if the working scenario of the original database is a multi-read scenario of a transactional business system, the test data model and simulation request tool are determined to be a test data model and simulation request tool that conforms to the multi-read scenario of a transactional business system.

[0082] S204: Initialize the test data model and the simulation request tool.

[0083] In combination with the scenario example, before configuring the parameters of the test data model and the simulation request tool, the parameters therein may be initialized first.

[0084] S205 , simulating the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period.

[0085] Combined with the scenario example, based on the static information of the original database, the total data volume of the original database and the data volume, proportion and number of partitions of each field type can be obtained.

[0086] If the determined test data model and simulation request tool are multi-read scenarios of a transactional business system, the test data model simulates the total data volume of the original database and the data volume, proportion, and number of partitions of each field type. The simulation request tool simulates the ratio of read and write operations of the original database, and configures the number of clients, TPS, and QPS to be the same as the original database.

[0087] If the determined test data model and simulation request tool are multi-write scenarios of a transactional business system, the test data model simulates the total data volume of the original database and the data volume, proportion, and number of partitions of each field type. The simulation request tool simulates the ratio of read and write operations of the original database, and configures the number of clients, TPS, and QPS to be the same as the original database.

[0088] If the determined test data model and simulation request tool are for ad hoc query scenarios of analytical business systems, the test data model simulates the total data volume of the original database and the data volume, proportion, and number of partitions of each field type, and simulates the data warehouse model. The specific data warehouse model selects the same snowflake or star type as the original database. The simulation request tool simulates the proportion of the query method of the original database, and configures the number of clients, TPS, and QPS to be the same as the original database.

[0089] If the determined test data model and simulation request tool are data processing scenarios for analytical business systems, the test data model simulates the total data volume of the original database and the data volume, proportion, and number of partitions of each field type, and simulates the data warehouse model. The specific data warehouse model selects the same snowflake or star type as the original database. The simulation request tool simulates the proportion of the query method of the original database, and configures the number of clients, TPS, and QPS to be the same as the original database.

[0090] S206. Based on the load parameters of the original database in the preset time period, adjust the simulated request parameters until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period.

[0091] In combination with the scenario example, taking the load parameters of the above-mentioned original database as a benchmark, adjust the number of clients, TPS and QPS configured by the simulation request tool to adjust the load parameters of the initial test data. When the load parameters of the initial test data are consistent with the load parameters of the original database, obtain the number of clients, TPS and QPS at this time, and determine the number of clients, TPS and QPS at this time as the target request parameters, and determine the test data configured with the target request parameters, the static information of the original database and the load parameters under the preset time period as the target test data.

[0092] S207: deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period.

[0093] Combined with the scenario example, the obtained target test data is deployed on the target database, and the target request parameters obtained above are used to request the target database. That is, on the target database, the obtained target test data is run with the obtained initial resources to obtain the load parameters corresponding to the target database under the condition of the initial resources.

[0094] S208. Based on the load parameters of the original database in the preset time period, adjust the initial resources until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database in the preset time period.

[0095] Combined with the scenario example, after the target test data is deployed on the target database, the actual load parameters corresponding to the target database can be obtained. If the actual load parameters corresponding to the target database exceed the load parameters corresponding to the original database, it can be determined that the initial resources are too small, and the resources required on the target database can be elastically expanded. Similarly, if the actual load parameters corresponding to the target database are smaller than the load parameters corresponding to the original database, it can be determined that the initial resources are too large, and the resources required on the target database can be elastically reduced. Through the above adjustment of the resources required on the target database, the load parameters of the target database are consistent with the load parameters of the original database, and the adjusted resources are determined as the final target resources required by the target database.

[0096] S209: Output the target resources required by the target database in the preset time period.

[0097] Optionally, after obtaining the target, the target resources may be output, for example, the target resources may be output to a display screen, so that relevant staff can obtain the target database based on the target resource configuration.

[0098] This example can obtain the target resources actually used after deployment by deploying the target test data corresponding to the original database on the target database, thereby achieving accurate prediction of the resources required on the target database and improving the accuracy of prediction of the resources required for the target database.

[0099] Optionally, the static information of the original database includes: total number of CPU cores, total memory, total data volume and network bandwidth resources, and the parameter information of the target database includes: number of CPU cores per node, memory per node and data capacity per node.

[0100] Accordingly, Figure 3 Schematic diagram of the resource processing method for database migration provided in this application Figure 2 ,like Figure 3 As shown, S202 includes:

[0101] S301. Obtain a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database.

[0102] In combination with the scenario example, the static information of the original database mentioned above includes the total number of CPU cores, total memory, total data volume and network bandwidth resources of the original database, and the parameter information of the target database includes the number of CPU cores per node, the memory per node and the data capacity per node.

[0103] The ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database is determined as a first ratio, the ratio between the total memory of the original database and the memory of a single node of the target database is determined as a second ratio, and the ratio between the total data volume of the source database and the data capacity of a single node of the target database is determined as a third ratio.

[0104] S302: Determine the maximum value among the first ratio, the second ratio and the third ratio as the initial number of nodes in the target library.

[0105] In combination with the scenario example, the first ratio, the second ratio and the third ratio obtained are first rounded up, and then the maximum value among them is determined as the initial number of nodes of the target database, that is, the initial number of nodes of the target database = MAX(CEIL(total number of CPU cores of the original database / number of CPU cores of a single node of the target database, total memory of the original database / memory of a single node of the target database, total data volume of the original database / data capacity of a single node of the target database)), where CEIL represents the rounding-up function.

[0106] S303: Determine the network broadband resources of the original database as the initial network broadband resources.

[0107] In combination with the scenario example, the static information of the original database includes the network bandwidth resources of the original database, and the network bandwidth resources of the original database can be determined as the initial network bandwidth resources.

[0108] S304: Determine the initial number of nodes in the target library and the initial network bandwidth resources as the initial resources.

[0109] In combination with the scenario example, the initial number of nodes of the target database and the initial network bandwidth resources obtained above are determined as the initial resources required for the target database.

[0110] Based on the method provided in this example, the purpose of determining the initial resources required for the target database can be achieved.

[0111] Optionally, the static information of the original database includes: database type, query mode and ratio information of data read and write operations;

[0112] Accordingly, Figure 4 Schematic diagram of the resource processing method for database migration provided in this application Figure 3 ,like Figure 4 As shown, S203 includes:

[0113] S401. If the database type of the original database is transactional, and the proportion of the read operations exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the first type of environment type.

[0114] Combined with the scenario example, first determine whether the original database is transactional or analytical based on the database type in the static information of the original database. If the original database is transactional, then determine the proportion of read operations and the proportion of write operations based on the proportion of read and write operations in the static information of the original database. If the proportion of read operations on data in the original database exceeds a preset threshold, such as more than 60%, then determine the usage scenario of the original database as a multi-read scenario of a transactional business system in the first environment type, and then determine the test data model and simulation request tool as the test data model and simulation request tool corresponding to the multi-read scenario of a transactional business system.

[0115] S402: If the database type of the original database is transactional and the proportion of write operations exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the second type of environment.

[0116] Combined with the scenario example, if the original database is a transaction type, and the proportion of write operations on data exceeds the preset threshold, for example, more than 60%, the usage scenario of the original database is determined as a multi-write scenario of the second type of environment type, which is a transaction type business system, and then the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the multi-write scenario of the transaction type business system.

[0117] S403. If the database type of the original database is analytical, and the query mode is ad hoc query, the test data model and simulation request tool are determined to be test data model and simulation request tool belonging to the third type of environment.

[0118] Combined with the scenario example, if the original database is of business type, the query method of the original database is determined based on the query method in the static information of the original database; if the query method of the source database is Adhoc ad hoc query type, the usage scenario of the original database is determined as the ad hoc query scenario of the analytical business system of the third type of environment, and then the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the ad hoc query scenario of the analytical business system.

[0119] S404. If the database type of the original database is analytical, and the query mode is data processing, the test data model and simulation request tool are determined as test data model and simulation request tool belonging to the fourth environment type.

[0120] Combined with the scenario example, if the original database is business-type and the query method of the source database is ETL data processing type, the usage scenario of the original database is determined as the data processing scenario of the analytical business system of the fourth type of environment, and then the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the data processing scenario of the analytical business system.

[0121] Based on the method provided in this example, the application scenario of the original database can be determined based on the database type, query method, and ratio information of read and write operations on the data of the original database.

[0122] Optionally, the static information of the original database includes: hardware configuration environment information;

[0123] Accordingly, Figure 5 Schematic diagram of the resource processing method for database migration provided in this application Figure 4 ,like Figure 5 As shown, S206 includes:

[0124] S501: deploy the initial test data in the same hardware configuration environment as the original database.

[0125] Combined with the scenario example, according to the hardware configuration of the original database, when the same resource conditions are met, the initial test data can be first deployed in the same hardware configuration environment as the original database, so as to adjust the corresponding request parameters in the initial test data in the same hardware configuration environment as the original database.

[0126] S502. In the same hardware configuration environment as the original database, based on the load parameters of the original database in the preset time period, adjust the simulated request parameters until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

[0127] In combination with the scenario example, in the same hardware configuration environment as the original database, the TPS, QPS and number of clients are adjusted to achieve the purpose of adjusting the load parameters of the initial test data until the load parameters of the initial test data are consistent with the load parameters of the original database.

[0128] Based on the method provided in this example, the request parameters are adjusted in the same hardware configuration environment as the original database, so that the adjusted request parameters are more accurate.

[0129] Optional, Figure 6 Schematic diagram of the resource processing method for database migration provided in this application Figure 5 ,like Figure 6 As shown, after S208, it also includes:

[0130] S601: Acquire a first resource occupied by deploying the target test data on the target database.

[0131] Combined with the scenario example, the purpose of deploying the target test data on the target database is to determine in advance the resources required by the target database if the original database is migrated to the target database. Therefore, after obtaining the resources required by the target database, the first resource used to deploy the target test data on the target database needs to be released, so the first resource needs to be collected.

[0132] S602: Acquire a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database.

[0133] Combined with the scenario example, the purpose of deploying the initial test data in the same hardware configuration environment as the original database is to adjust the request parameters in the initial test data to make the request parameters more accurate. Therefore, after adjusting the request parameters, the second resource used to deploy the initial test data in the same hardware configuration environment as the original database also needs to be released, so the second resource needs to be collected.

[0134] S603: Release the first resource and the second resource.

[0135] In combination with the scenario example, the first group members and the second resources collected above are released to achieve the effect of reusing the test resources.

[0136] Optionally, the preset time period includes a peak time period or a non-peak time period.

[0137] Combined with the scenario example, the corresponding peak time period can be determined according to the business scenario of the original database. On the same day, the preset time is determined as the peak time period, and the rest of the time is determined as the non-peak time period. The load corresponding to the peak time period is greater than the load corresponding to the non-peak time period, so for the peak time period and the non-peak time period, the resources required in the target database need to be determined separately.

[0138] Specifically, for the peak time period, the load parameters corresponding to the original database during the peak time period can be collected, and the load parameters corresponding to the peak time period can be used as a benchmark. Referring to the above method, the target test data corresponding to the peak time period can be obtained, and the target test data corresponding to the peak time period can be deployed to the target database to obtain the resources required in the target database corresponding to the peak time period.

[0139] Similarly, for non-peak time periods, the load parameters corresponding to the original database in the non-peak time period can be collected, and the load parameters corresponding to the non-peak time period can be used as a benchmark. Referring to the above method, the target test data corresponding to the non-peak time period can be obtained, and the target test data corresponding to the non-peak time period can be deployed to the target database to obtain the resources required in the target database corresponding to the non-peak time period. Based on the method provided in this example, the resources required in the target database can be obtained for the peak time period and the non-peak time period respectively, so that the target database can run stably in each time period.

[0140] This embodiment can obtain the target resources actually used after deployment by deploying the target test data corresponding to the original database on the target database, thereby achieving accurate prediction of the resources required on the target database and improving the accuracy of predicting the resources required for the target database.

[0141] Figure 7 The schematic diagram of the structure of the resource processing device for database migration provided in the present application is applied to a computing device, such as Figure 7 As shown, including:

[0142] The acquisition module 71 is used to acquire static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period;

[0143] A processing module 72, configured to determine the initial resources required in the target database according to the static information of the original database and the parameter information of the target database;

[0144] The processing module 72 is further used to determine a test data model and a simulation request tool based on the static information of the original database;

[0145] The processing module 72 is also used to initialize the test data model and the simulation request tool;

[0146] The processing module 72 is further used to simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain the initial test data corresponding to the original database in the preset time period;

[0147] The processing module 72 is further used to adjust the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period;

[0148] The processing module 72 is further used to deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period;

[0149] The processing module 72 is further configured to adjust the initial resources based on the load parameters of the original database in the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database in the preset time period;

[0150] The processing module 72 is further configured to output the target resources required by the target database within the preset time period.

[0151] Optionally, the static information of the original database includes: the total number of CPU cores, the total memory, the total data volume and the network bandwidth resources, and the parameter information of the target database includes: the number of CPU cores of a single node, the memory of a single node and the data capacity of a single node;

[0152] Accordingly, the processing module 72 is specifically configured to obtain a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total amount of data of the original database and the data capacity of a single node of the target database;

[0153] The processing module 72 is further configured to determine the maximum value among the first ratio, the second ratio and the third ratio as the initial number of nodes in the target library;

[0154] The processing module 72 is further configured to determine the network broadband resource of the original database as the initial network broadband resource;

[0155] The processing module 72 is further specifically configured to determine the initial number of nodes in the target library and the initial network bandwidth resources as the initial resources.

[0156] Optionally, the static information of the original database includes: database type, query mode and ratio information of data read and write operations;

[0157] Correspondingly, the processing module 72 is further configured to determine the test data model and simulation request tool as a test data model and simulation request tool belonging to the first type of environment type if the database type of the original database is transactional and the proportion of the read operation performed on the original database exceeds a preset threshold;

[0158] The processing module 72 is further configured to determine the test data model and simulation request tool as a test data model and simulation request tool belonging to the second type of environment type if the database type of the original database is transactional and the proportion of the write operation to the original database exceeds a preset threshold;

[0159] The processing module 72 is further configured to determine the test data model and the simulation request tool as a test data model and a simulation request tool belonging to the third type of environment type if the database type of the original database is analytical and the query mode is ad hoc query;

[0160] The processing module 72 is further specifically used to determine the test data model and simulation request tool as a test data model and simulation request tool belonging to the fourth environment type if the database type of the original database is analytical and the query mode is data processing.

[0161] Optionally, the static information of the original database includes: hardware configuration environment information;

[0162] Accordingly, the processing module 72 is further configured to deploy the initial test data in the same hardware configuration environment as the original database;

[0163] The processing module 72 is also specifically used to adjust the simulated request parameters in the same hardware configuration environment as the original database, based on the load parameters of the original database in the preset time period, until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

[0164] Optionally, the processing module 72 is further configured to obtain a first resource occupied by deploying the target test data on the target database;

[0165] The processing module 72 is further configured to obtain a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database;

[0166] The processing module 72 is further specifically configured to release the first resource and the second resource.

[0167] The resource processing device for database migration provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0168] Figure 8 A schematic diagram of the structure of the resource processing system for database migration provided in this application, such as Figure 8As shown, it includes console module, monitoring module, original database information collection module, benchmark library deployment module, simulation request module, resource management module, parameter configuration management module, database module and infrastructure module. Among them, console module: through the console, you can load the database driver and installation package after selecting the target database product, observe the performance and load indicators of the original database and target database on the console, and simulate the operation of the request tool, and enter the benchmark data model and tool information applicable to different scenarios. Monitoring module: by accessing the database performance monitoring tool, collect the CPU usage, memory usage, disk IO speed, disk usage, network traffic, network delay of the device where the original database and the target database are located, as well as the TPS, QPS, throughput and SQL delay information of the database. Original database information collection module: enter the original database deployment environment information, network environment, runtime load and other performance characteristics, field type data characteristics, SQL type distribution characteristics, performance characteristics information. Benchmark library deployment module: configure the target library simulation environment connection configuration, configure the network access port permissions, and configure the data volume of the benchmark library to be the same as the original library. The data volume and number of data shards. Simulation request module: configure the database connection information of the simulation database SQL simulation request tool; configure its simulation parameters, including the number of simulated clients, the number of concurrent requests for each client, the number of transactions for each client, the number of concurrent requests, the running time of the simulated load, and the read-write ratio in the request, and the simulation of grouping, sorting, aggregation, and subqueries in the query. Resource management module: manage hardware and network resources consistent with the original database environment, and manage hardware and network resources consistent with the target database. Parameter configuration management module: when the original database environment is simulated and calibrated, record the parameter set information during the simulated load, including the number of clients, TPS, QPS, running time, query configuration parameters, record a set of parameter sets for daily load and peak load respectively, and apply these parameters to the target database. Database module and infrastructure module: databases include: Mysql, Oracle, Postgresql, GaussDB, OceanBase and TDSQL, etc., infrastructure includes containers, virtual machines, servers, networks and load balancing, etc., and databases and basic equipment are the basic resource support environment to support this application.

[0169] Fig. 9 A schematic diagram of the structure of the computer device provided for this application, such as Fig. 9 As shown, the computer device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0170] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0171] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0172] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0173] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0174] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0175] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0176] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0177] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0179] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0183] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0184] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A resource processing method for database migration, characterized in that: Applied to computing devices, including: Obtain static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period; Determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database; Determine a test data model and a simulation request tool based on the static information of the original database; Initializing the test data model and the simulation request tool; Simulating the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain the initial test data corresponding to the original database in the preset time period; Taking the load parameters of the original database in the preset time period as a reference, adjusting the simulated request parameters until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configuring the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period; Deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period; Taking the load parameter of the original database in the preset time period as a reference, adjusting the initial resources until the load parameter corresponding to the target database is consistent with the load parameter of the original database, so as to determine the target resources required by the target database in the preset time period; Output the target resources required by the target database within the preset time period.

2. The method according to claim 1, characterized in that The static information of the original database includes: the total number of CPU cores, total memory, total data volume and network bandwidth resources; the parameter information of the target database includes: the number of CPU cores per node, the memory per node and the data capacity per node; Accordingly, determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database includes: Obtaining a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database; Determine the maximum value among the first ratio, the second ratio and the third ratio as the initial number of nodes in the target library; Determining the network broadband resources of the original database as the initial network broadband resources; The initial number of nodes in the target library and the initial network bandwidth resources are determined as the initial resources.

3. The method according to claim 1, characterized in that The static information of the original database includes: database type, query method and ratio information of data read and write operations; Accordingly, the determining of the test data model and the simulation request tool based on the static information of the original database includes: If the database type of the original database is transactional, and the proportion of the read operation exceeds a preset threshold, the test data model and simulation request tool are determined to be test data models and simulation request tools belonging to the first type of environment type; If the database type of the original database is transactional, and the proportion of write operations performed on the original database exceeds a preset threshold, the test data model and simulation request tool are determined to be test data models and simulation request tools belonging to the second type of environment; If the database type of the original database is analytical, and the query mode is ad hoc query, the test data model and simulation request tool are determined to be test data models and simulation request tools belonging to the third type of environment; If the database type of the original database is analytical, and the query method is data processing, the test data model and simulation request tool are determined as test data models and simulation request tools belonging to the fourth environment type.

4. The method according to claim 1, characterized in that The static information of the original database includes: hardware configuration environment information; Accordingly, the adjusting the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database includes: Deploy the initial test data in the same hardware configuration environment as the original database; In the same hardware configuration environment as the original database, the simulated request parameters are adjusted based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

5. The method according to claim 4, characterized in that After adjusting the initial resources based on the load parameters of the original database in the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database to determine the target resources required by the target database in the preset time period, the method further includes: Acquire a first resource occupied by deploying the target test data on the target database; Acquire a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database; The first resource and the second resource are released.

6. The method according to any one of claims 1 to 5, characterized in that: The preset time period includes a peak time period or a non-peak time period.

7. A resource processing device for database migration, characterized in that: Applied to computing devices, including: An acquisition module is used to acquire static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period; A processing module, used for determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database; The processing module is further used to determine a test data model and a simulation request tool based on the static information of the original database; The processing module is also used to initialize the test data model and the simulation request tool; The processing module is further used to simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain the initial test data corresponding to the original database in the preset time period; The processing module is further used to adjust the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period; The processing module is further used to deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period; The processing module is further used to adjust the initial resources based on the load parameters of the original database in the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database in the preset time period; The processing module is further used to output the target resources required by the target database in the preset time period.

8. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images