Resource processing method and device for database migration

Abstract

The present application provides a resource processing method and apparatus for database migration, which is applied to a computing device. The method includes: obtaining static information of the original database, parameter information of the target database, and request parameters and load parameters within a preset time period; determining initial resources, a test data model, and a simulation request tool; simulating the static information of the original database and the request parameters within a preset time period to obtain initial test data; adjusting the request parameters to obtain adjusted target request parameters and target test data; deploying the target test data to the target database; adjusting the initial resources to determine the target resources required by the target database; and finally outputting the target resources. Based on the method provided by the present application, the accuracy of the resources required by the target database can be determined.

Description

Technical Field

[0001] The present application relates to the field of database migration, and in particular to a resource processing method and device for database migration. Background Art

[0002] Database migration is the process of moving data from an original database to a target database. It typically involves physical migration and logical migration. Heterogeneous database migration occurs when the original and target databases are different types. Before migrating heterogeneous databases, you need to determine the resources required in the target database.

[0003] Currently, before migrating heterogeneous databases, the methods for determining the resources required for the target database mainly include static estimation based on experience and estimation based on reference to similar systems.

[0004] However, the current method of determining the resources required by the target database cannot accurately determine the resources required by the target database, resulting in an excessive operating load on the target database after moving the data of the original database to the target database, resulting in low operating efficiency of the target data, or a low operating load on the target database, resulting in a waste of resources. Summary of the Invention

[0005] Embodiments of the present application provide a resource processing method and apparatus for database migration, so as to improve the accuracy of determining the resources required for a target database.

[0006] In a first aspect, an embodiment of the present application provides a resource processing method for database migration, applied to a computing device, comprising:

[0007] Obtain static information of the original database, parameter information of the target database, and request parameters and load parameters within a preset time period;

[0008] Determining the initial resources required in the target database based on the static information of the original database and the parameter information of the target database;

[0009] Determining a test data model and a simulation request tool based on static information of the original database;

[0010] Initializing the test data model and simulation request tool;

[0011] Simulating the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period;

[0012] Adjusting the simulated request parameters based on the load parameters of the original database during the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, thereby obtaining adjusted target request parameters corresponding to the preset time period, and configuring target test data that includes the target request parameters, static information of the original database, and the load parameters during the preset time period;

[0013] Deploy the target test data to the target database, and request the target database with reference to the target request parameters to obtain the load parameters corresponding to the target database in the preset time period;

[0014] Adjusting the initial resources based on the load parameters of the original database during the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, thereby determining target resources required by the target database during the preset time period;

[0015] Output the target resources required by the target database within the preset time period.

[0016] In one possible implementation, the static information of the original database includes: the total number of CPU cores, total memory, total data volume, and network bandwidth resources; the parameter information of the target database includes: the number of CPU cores per node, memory per node, and data capacity per node;

[0017] Accordingly, determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database includes:

[0018] Obtaining a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database;

[0019] Determine the maximum value among the first ratio, the second ratio and the third ratio as the initial node number of the target library;

[0020] Determining the network broadband resources of the original database as initial network broadband resources;

[0021] The initial number of nodes in the target library and the initial network bandwidth resources are determined as the initial resources.

[0022] In a possible implementation, the static information of the original database includes: database type, query mode, and ratio information of read and write operations on data;

[0023] Accordingly, the determining of the test data model and the simulation request tool based on the static information of the original database includes:

[0024] If the database type of the original database is transactional, and the proportion of read operations on the data of the original database exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the first type of environment type;

[0025] If the database type of the original database is transactional, and the proportion of write operations on the data of the original database exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the second type of environment type;

[0026] If the database type of the original database is analytical and the query mode is ad hoc, the test data model and simulation request tool are determined to belong to the third type of environment type;

[0027] If the database type of the original database is analytical and the query mode is data processing, the test data model and simulation request tool are determined to belong to the fourth type of environment.

[0028] In a possible implementation, the static information of the original database includes: hardware configuration environment information;

[0029] Accordingly, adjusting the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database includes:

[0030] Deploying the initial test data in the same hardware configuration environment as the original database;

[0031] In the same hardware configuration environment as the original database, the simulated request parameters are adjusted based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

[0032] In a possible implementation, after adjusting the initial resources based on the load parameters of the original database in the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database to determine the target resources required by the target database in the preset time period, the method further includes:

[0033] Acquire a first resource occupied by deploying the target test data on the target database;

[0034] Acquire a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database;

[0035] The first resource and the second resource are released.

[0036] In a possible implementation, the preset time period includes a peak time period or a non-peak time period.

[0037] In a second aspect, an embodiment of the present application provides a resource processing apparatus for database migration, applied to a computing device, comprising:

[0038] An acquisition module is used to obtain static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period;

[0039] A processing module, configured to determine the initial resources required in the target database based on the static information of the original database and the parameter information of the target database;

[0040] The processing module is further configured to determine a test data model and a simulation request tool based on the static information of the original database;

[0041] The processing module is further used to initialize the test data model and the simulation request tool;

[0042] The processing module is further configured to simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period;

[0043] The processing module is further configured to adjust the simulated request parameters based on the load parameters of the original database during the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, static information of the original database, and target test data of the load parameters during the preset time period;

[0044] The processing module is further configured to deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period;

[0045] The processing module is further configured to adjust the initial resources based on the load parameters of the original database during the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database during the preset time period;

[0046] The processing module is further configured to output target resources required by the target database within the preset time period.

[0047] In a third aspect, an embodiment of the present application provides a computer device, including: a memory, a processor;

[0048] Memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0052] The resource processing method and apparatus for database migration provided in an embodiment of the present application determines target test data for a preset time period by acquiring static information of the original database, parameter information of the target database, and request parameters and load parameters for the preset time period. By deploying the target test data to the target database, the target resources required by the target database for the preset time period are obtained. Based on the method provided in the present application, the target test data corresponding to the original database can be deployed on the target database to obtain the target resources actually used after deployment. This allows for accurate prediction of the resources required for the target database, thereby improving the accuracy of predictions of the resources required for the target database. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 Provide a scenario diagram for this application;

[0055] Figure 2Schematic diagram of the resource processing method for database migration provided in this application Figure 1 ;

[0056] Figure 3 Schematic diagram of the resource processing method for database migration provided in this application Figure 2 ;

[0057] Figure 4 Schematic diagram of the resource processing method for database migration provided in this application Figure 3 ;

[0058] Figure 5 Schematic diagram of the resource processing method for database migration provided in this application Figure 4 ;

[0059] Figure 6 Schematic diagram of the resource processing method for database migration provided in this application Figure 5 ;

[0060] Figure 7 A schematic diagram of the structure of a resource processing device for database migration provided in this application;

[0061] Figure 8 A schematic diagram of the structure of the resource processing system for database migration provided in this application;

[0062] Figure 9 A schematic diagram of the structure of the computer device provided in this application.

[0063] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0065] First, let’s explain the terms involved in this application:

[0066] Slow SQL: refers to Structured Query Language (SQL), including statistics on slow SQL queries at P99, P95, and P90 or statistics on top time-consuming SQL queries.

[0067] Figure 1 Provide a scenario diagram for this application, such as Figure 1 As shown, database migration refers to the process of moving data from the original database to the target database. It usually includes physical migration and logical migration. Heterogeneous database migration means that the original and target databases are different databases. Before heterogeneous database migration, you need to determine the resources required in the target database.

[0068] Currently, before migrating heterogeneous databases, the methods for determining the resources required for the target database mainly include static estimation based on experience and estimation based on reference to similar systems.

[0069] However, the current method of determining the resources required by the target database cannot accurately determine the resources required by the target database, resulting in an excessive operating load on the target database after moving the data of the original database to the target database, resulting in low operating efficiency of the target data, or a low operating load on the target database, resulting in a waste of resources.

[0070] The resource processing method for database migration provided by the present application obtains the static information of the original database, the parameter information of the target database, and the request parameters and load parameters in a preset time period, and presets the initial resources, test data model and simulation request tool required in the target database. Based on the test data model and the simulation request tool, the static information of the original database and the request parameters in the preset time period are simulated to obtain the corresponding initial test data, and the target test data is obtained by adjusting the request parameters simulated by the initial test data. The corresponding load parameters are obtained by deploying the target test data to the target database, and the initial resources are adjusted until the load parameters corresponding to the target database are consistent with the load parameters of the original database to determine the required target resources. Based on the method provided by the present application, the target resources actually used after deployment can be obtained by deploying the target test data corresponding to the original database on the target database, so as to achieve accurate prediction of the resources required on the target database, thereby improving the accuracy of prediction of the resources required for the target database.

[0071] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0072] Figure 2 Schematic diagram of the resource processing method for database migration provided in this application Figure 1 , applied to computing devices, such as Figure 2 As shown, the method includes:

[0073] S201: Obtain static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period.

[0074] The execution subject of this embodiment is a computing device, which can be implemented through a computer program, such as application software, etc.; or, it can be implemented as a medium storing relevant computer programs, such as a USB flash drive, a cloud disk, etc.; or, it can be implemented through a physical device integrated or installed with relevant computer programs, such as a chip, etc.

[0075] Combined with the scenario example, static information includes: CPU information, memory information, disk storage type, disk capacity, data volume and daily increments, log volume and daily increments on the original database deployment server, network bandwidth in the network environment, network latency, database deployment architecture, ratio of read and write operations, database type, query method, field data type, hardware configuration environment information and data warehouse model, etc. Among them, CPU information includes the number of cores, frequency and architecture, memory information includes capacity and speed, and disk storage types include Serial Advanced Technology Attachment (SATA) solid-state drive (SSD), non-volatile memory Express (NVME) solid-state drive (SSD) or traditional hard disk drive (Hard Disk). Drive (HDD), database deployment architectures include single-node, master-slave, and distributed deployments. Database types include transactional and analytical. Query methods include ad hoc queries and ETL (Extract, Transform, Load) data processing. Field data types include primarily character, numeric, and binary. Data warehouse models include snowflake and star schemas. Target database parameters include the number of CPU cores per node, memory per node, and data capacity per node.

[0076] Request parameters and load parameters are dynamically changing parameters that vary over time. Request parameters include the number of clients, transactions per second (TPS), and queries per second (QPS).

[0077] Load parameters include CPU usage, memory usage, disk read / write operations per second (IOPS), network traffic, request latency, and other information.

[0078] S202: Determine the initial resources required in the target database according to the static information of the original database and the parameter information of the target database.

[0079] Based on the scenario example, the initial resources required in the target database include the initial number of nodes in the target database and the initial network bandwidth resources.

[0080] S203: Determine a test data model and a simulation request tool based on the static information of the original database.

[0081] Based on the scenario examples, the original database's working scenario can be determined based on its static information. The corresponding test data model and simulation request tool can then be determined based on the original database's working scenario. The original database's working scenarios include: multi-read scenarios for transactional business systems, multi-write scenarios for transactional business systems, ad hoc query scenarios for analytical business systems, and data processing scenarios for analytical business systems. Therefore, the test data model and simulation request tool can be determined based on the four working scenarios of the original database. For example, if the original database's working scenario is a multi-read scenario for a transactional business system, the test data model and simulation request tool will be determined to be consistent with the multi-read scenario for a transactional business system.

[0082] S204: Initialize the test data model and simulation request tool.

[0083] In combination with the scenario example, before configuring the parameters of the test data model and the simulation request tool, the parameters can be initialized first.

[0084] S205 : Simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period.

[0085] Combined with the scenario example, based on the static information of the original database, the total data volume of the original database and the data volume, proportion, and number of partitions of each field type can be obtained.

[0086] If the test data model and simulation request tool are for a multi-read scenario of a transactional business system, the test data model simulates the total data volume of the original database, as well as the data volume, proportion, and number of partitions of each field type. The simulation request tool simulates the ratio of read and write operations of the original database and configures the number of clients, TPS, and QPS to be the same as the original database.

[0087] If the determined test data model and simulation request tool are for a multi-write scenario of a transactional business system, the test data model simulates the total data volume of the original database and the data volume, proportion, and number of partitions of each field type. The simulation request tool simulates the ratio of read and write operations of the original database and configures the number of clients, TPS, and QPS to be the same as the original database.

[0088] If the determined test data model and simulation request tool are for ad hoc query scenarios of analytical business systems, the test data model simulates the total data volume of the original database, as well as the data volume, proportion, and number of partitions of each field type, and simulates the data warehouse model. The specific data warehouse model is the same snowflake or star type as the original database. The simulation request tool simulates the proportion of the original database's query method, and configures the number of clients, TPS, and QPS to be the same as the original database.

[0089] If the determined test data model and simulation request tool are for the data processing scenario of an analytical business system, the test data model simulates the total data volume of the original database, as well as the data volume, proportion, and number of partitions of each field type, and simulates the data warehouse model. The specific data warehouse model is selected to be the same snowflake or star type as the original database. The simulation request tool simulates the proportion of the original database's query method, and configures the number of clients, TPS, and QPS to be the same as the original database.

[0090] S206. Based on the load parameters of the original database in the preset time period, adjust the simulated request parameters until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain the adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, the static information of the original database and the target test data of the load parameters in the preset time period.

[0091] Combined with the scenario example, taking the load parameters of the above-mentioned original database as a benchmark, adjust the number of clients, TPS and QPS configured by the simulation request tool to adjust the load parameters of the initial test data. When the load parameters of the initial test data are consistent with the load parameters of the original database, the number of clients, TPS and QPS at this time are obtained, and the number of clients, TPS and QPS at this time are determined as the target request parameters. The test data configured with the target request parameters, the static information of the original database and the load parameters under the preset time period is determined as the target test data.

[0092] S207: Deploy the target test data to the target database, and request the target database with reference to the target request parameters to obtain the corresponding load parameters of the target database in the preset time period.

[0093] Based on the scenario example, the obtained target test data is deployed on the target database, and the target request parameters obtained above are used to request the target database. That is, the obtained target test data is run on the target database with the obtained initial resources to obtain the load parameters corresponding to the target database under the condition of the initial resources.

[0094] S208. Based on the load parameters of the original database in the preset time period, adjust the initial resources until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database in the preset time period.

[0095] In conjunction with the scenario example, after deploying the target test data to the target database, the actual load parameters corresponding to the target database can be obtained. If the actual load parameters corresponding to the target database exceed the load parameters corresponding to the original database, it can be determined that the initial resources are too small, and the resources required for the target database can be elastically expanded. Similarly, if the actual load parameters corresponding to the target database are smaller than the load parameters corresponding to the original database, it can be determined that the initial resources are too large, and the resources required for the target database can be elastically reduced. Through the above adjustment of the resources required for the target database, the load parameters of the target database are made consistent with the load parameters of the original database, and the adjusted resources are determined as the final target resources required for the target database.

[0096] S209: Output the target resources required by the target database in the preset time period.

[0097] Optionally, after obtaining the target, the target resources may be output, for example, outputting the target resources to a display screen, so that relevant staff can obtain the target database based on the target resource configuration.

[0098] This example can obtain the target resources actually used after deployment by deploying the target test data corresponding to the original database on the target database, thereby accurately predicting the resources required on the target database and improving the accuracy of the prediction of the resources required for the target database.

[0099] Optionally, the static information of the original database includes: total number of CPU cores, total memory, total data volume and network bandwidth resources, and the parameter information of the target database includes: number of CPU cores per node, memory per node and data capacity per node.

[0100] Accordingly, Figure 3 Schematic diagram of the resource processing method for database migration provided in this application Figure 2 ,like Figure 3 As shown, S202 includes:

[0101] S301. Obtain a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database.

[0102] In the scenario example, the static information of the original database mentioned above includes information such as the total number of CPU cores, total memory, total data volume, and network bandwidth resources of the original database. The parameter information of the target database includes information such as the number of CPU cores per node, memory per node, and data capacity per node.

[0103] The ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database is determined as a first ratio, the ratio between the total memory of the original database and the memory of a single node of the target database is determined as a second ratio, and the ratio between the total data volume of the source database and the data capacity of a single node of the target database is determined as a third ratio.

[0104] S302: Determine the maximum value among the first ratio, the second ratio, and the third ratio as the initial number of nodes in the target library.

[0105] Based on the scenario example, the first, second, and third ratios are rounded up, and the maximum value is used as the initial number of nodes in the target database. That is, the initial number of nodes in the target database = MAX(CEIL(total number of CPU cores in the original database / number of CPU cores in a single node of the target database, total memory in the original database / memory in a single node of the target database, total data volume in the original database / data capacity in a single node of the target database)), where CEIL represents the rounding-up function.

[0106] S303: Determine the network broadband resources of the original database as initial network broadband resources.

[0107] In combination with the scenario example, the static information of the original database includes the network broadband resources of the original database, and the network broadband resources of the original database can be determined as the initial network broadband resources.

[0108] S304: Determine the initial number of nodes in the target library and the initial network broadband resources as the initial resources.

[0109] Combined with the scenario example, the initial number of nodes in the target database and the initial network bandwidth resources obtained above are determined as the initial resources required for the target database.

[0110] Based on the method provided in this example, you can determine the initial resources required for the target database.

[0111] Optionally, the static information of the original database includes: database type, query method, and ratio information of read and write operations on data;

[0112] Accordingly, Figure 4 Schematic diagram of the resource processing method for database migration provided in this application Figure 3 ,like Figure 4 As shown, S203 includes:

[0113] S401. If the database type of the original database is transactional, and the proportion of read operations on the data of the original database exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the first type of environment type.

[0114] Combined with the scenario example, first determine whether the original database is transactional or analytical based on the database type in the static information of the original database. If the original database is transactional, then determine the proportion of read operations and write operations based on the proportion information of data read and write operations in the static information of the original database. If the proportion of data read operations in the original database exceeds the preset threshold, for example, more than 60%, then the usage scenario of the original database is determined as the first type of environment type, which is a multi-read scenario of a transactional business system. Then, the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the multi-read scenario of the transactional business system.

[0115] S402. If the database type of the original database is transactional and the proportion of write operations on the data of the original database exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the second type of environment type.

[0116] Based on the scenario example, if the original database is transactional and the proportion of data write operations exceeds the preset threshold, for example, more than 60%, the usage scenario of the original database is determined as the second type of environment type, which is a multi-write scenario of a transactional business system. Then, the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the multi-write scenario of the transactional business system.

[0117] S403: If the database type of the original database is analytical and the query mode is ad hoc, the test data model and simulation request tool are determined to belong to the third type of environment.

[0118] Combined with the scenario example, if the original database is of business type, the query method of the original database is determined based on the query method in the static information of the original database. If the query method of the source database is Adhoc ad hoc query type, the usage scenario of the original database is determined as the ad hoc query scenario of the analytical business system of the third type of environment, and then the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the ad hoc query scenario of the analytical business system.

[0119] S404: If the database type of the original database is analytical and the query mode is data processing, the test data model and simulation request tool are determined to belong to the fourth type of environment.

[0120] Combined with the scenario example, if the original database is business-type and the query method of the source database is ETL data processing type, the usage scenario of the original database is determined as the data processing scenario of the analytical business system of the fourth type of environment, and then the test data model and simulation request tool are determined as the test data model and simulation request tool corresponding to the data processing scenario of the analytical business system.

[0121] Based on the method provided in this example, the application scenario of the original database can be determined based on the database type, query method, and ratio of read and write operations on the original database.

[0122] Optionally, the static information of the original database includes: hardware configuration environment information;

[0123] Accordingly, Figure 5 Schematic diagram of the resource processing method for database migration provided in this application Figure 4 ,like Figure 5 As shown, S206 includes:

[0124] S501: Deploy the initial test data in the same hardware configuration environment as the original database.

[0125] Combined with the scenario example, based on the hardware configuration of the original database, when the same resource conditions are met, the initial test data can be first deployed in the same hardware configuration environment as the original database, so as to adjust the corresponding request parameters in the initial test data in the same hardware configuration environment as the original database.

[0126] S502. In the same hardware configuration environment as the original database, based on the load parameters of the original database in the preset time period, adjust the simulated request parameters until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

[0127] In combination with the scenario example, in the same hardware configuration environment as the original database, the TPS, QPS and number of clients are adjusted to achieve the purpose of adjusting the load parameters of the initial test data until the load parameters of the initial test data are consistent with the load parameters of the original database.

[0128] Based on the method provided in this example, the request parameters are adjusted in the same hardware configuration environment as the original database, so that the adjusted request parameters are more accurate.

[0129] Optional, Figure 6 Schematic diagram of the resource processing method for database migration provided in this application Figure 5 ,like Figure 6 As shown, after S208, it also includes:

[0130] S601: Acquire a first resource occupied by deploying the target test data on the target database.

[0131] Combined with the scenario example, the purpose of deploying target test data on the target database is to determine in advance the resources required by the target database if the original database is migrated to the target database. Therefore, after obtaining the resources required by the target database, the first resource used to deploy the target test data on the target database needs to be released, so the first resource needs to be collected.

[0132] S602: Acquire a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database.

[0133] Combined with the scenario example, the purpose of deploying the initial test data in the same hardware configuration environment as the original database is to adjust the request parameters in the initial test data to make the request parameters more accurate. Therefore, after adjusting the request parameters, the second resource used to deploy the initial test data in the same hardware configuration environment as the original database also needs to be released, so the second resource needs to be collected.

[0134] S603: Release the first resource and the second resource.

[0135] In combination with the scenario example, the first team member and the second resource collected above are released to achieve the effect of reusing the test resources.

[0136] Optionally, the preset time period includes a peak time period or a non-peak time period.

[0137] Based on the scenario example, you can determine the corresponding peak time period based on the original database's business scenario. Within a given day, the preset time period is designated as the peak time period, and the rest of the time is designated as the off-peak time period. The load during the peak time period is greater than the load during the off-peak time period. Therefore, the resources required in the target database must be determined separately for the peak and off-peak time periods.

[0138] Specifically, for the peak time period, the load parameters corresponding to the original database during the peak time period can be collected. Based on the load parameters corresponding to the peak time period, the target test data corresponding to the peak time period can be obtained by referring to the above method, and the target test data corresponding to the peak time period can be deployed to the target database to obtain the resources required in the target database corresponding to the peak time period.

[0139] Similarly, for non-peak time periods, the load parameters corresponding to the original database during the non-peak time period can be collected. Using the load parameters corresponding to the non-peak time period as a benchmark, referencing the above method, the target test data corresponding to the non-peak time period can be obtained. The target test data corresponding to the non-peak time period can then be deployed to the target database to obtain the resources required for the target database during the non-peak time period. Based on the method provided in this example, the resources required for the target database can be obtained for both peak and non-peak time periods, allowing the target database to operate stably in each time period.

[0140] This embodiment can obtain the target resources actually used after deployment by deploying the target test data corresponding to the original database on the target database, thereby achieving accurate prediction of the resources required on the target database and improving the accuracy of prediction of the resources required for the target database.

[0141] Figure 7 The schematic diagram of the structure of the resource processing device for database migration provided by this application is applied to computing devices, such as Figure 7 Shown, including:

[0142] An acquisition module 71 is used to acquire static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period;

[0143] A processing module 72 is configured to determine the initial resources required in the target database based on the static information of the original database and the parameter information of the target database;

[0144] The processing module 72 is further configured to determine a test data model and a simulation request tool based on the static information of the original database;

[0145] The processing module 72 is further configured to initialize the test data model and the simulation request tool;

[0146] The processing module 72 is further configured to simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period;

[0147] The processing module 72 is further configured to adjust the simulated request parameters based on the load parameters of the original database during the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, static information of the original database, and target test data of the load parameters during the preset time period;

[0148] The processing module 72 is further configured to deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period;

[0149] The processing module 72 is further configured to adjust the initial resources based on the load parameters of the original database during the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database during the preset time period;

[0150] The processing module 72 is further configured to output target resources required by the target database within the preset time period.

[0151] Optionally, the static information of the original database includes: the total number of CPU cores, the total memory, the total data volume and network bandwidth resources; the parameter information of the target database includes: the number of CPU cores per node, the memory per node and the data capacity per node;

[0152] Accordingly, the processing module 72 is specifically configured to obtain a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database.

[0153] The processing module 72 is further configured to determine the maximum value among the first ratio, the second ratio, and the third ratio as the initial number of nodes in the target library;

[0154] The processing module 72 is further configured to determine the network broadband resources of the original database as the initial network broadband resources;

[0155] The processing module 72 is further configured to determine the initial number of nodes in the target library and the initial network bandwidth resources as the initial resources.

[0156] Optionally, the static information of the original database includes: database type, query method, and ratio information of read and write operations on data;

[0157] Accordingly, the processing module 72 is further configured to determine that the test data model and simulation request tool belong to the first type of environment type if the database type of the original database is transactional and the proportion of read operations on the data of the original database exceeds a preset threshold;

[0158] The processing module 72 is further configured to determine that the test data model and simulation request tool belong to the second type of environment type if the database type of the original database is transactional and the proportion of write operations on the data of the original database exceeds a preset threshold;

[0159] The processing module 72 is further configured to determine, if the database type of the original database is analytical and the query mode is ad hoc, that the test data model and simulation request tool are test data models and simulation request tools belonging to the third type of environment;

[0160] The processing module 72 is further specifically used to determine the test data model and simulation request tool as belonging to the fourth type of environment type if the database type of the original database is analytical and the query mode is data processing.

[0161] Optionally, the static information of the original database includes: hardware configuration environment information;

[0162] Accordingly, the processing module 72 is further configured to deploy the initial test data in the same hardware configuration environment as the original database;

[0163] The processing module 72 is specifically used to adjust the simulated request parameters in the same hardware configuration environment as the original database, based on the load parameters of the original database in the preset time period, until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

[0164] Optionally, the processing module 72 is further configured to obtain a first resource occupied by deploying the target test data on the target database;

[0165] The processing module 72 is further configured to obtain a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database;

[0166] The processing module 72 is further configured to release the first resource and the second resource.

[0167] The resource processing device for database migration provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0168] Figure 8 A schematic diagram of the structure of the resource processing system for database migration provided in this application is shown in FIG. Figure 8As shown, it includes a console module, a monitoring module, a source database information collection module, a benchmark library deployment module, a simulation request module, a resource management module, a parameter configuration management module, a database module, and an infrastructure module. The console module allows users to load database drivers and installation packages after selecting the target database product. This allows users to monitor the performance and load metrics of the source and target databases, simulate the operation of the request tool, and input benchmark data models and tool information for different scenarios. The monitoring module connects to database performance monitoring tools to collect CPU usage, memory usage, disk I / O speed, disk usage, network traffic, network latency, and database TPS, QPS, throughput, and SQL latency information. The source database information collection module collects source database deployment environment information, network environment, runtime load, and other performance characteristics, as well as field type data characteristics, SQL type distribution characteristics, and performance characteristics. The benchmark library deployment module configures the target library simulation environment connection configuration, network access port permissions, and sets the benchmark library's data volume and number of data shards to the same size as the source library. Simulation request module: configure the database connection information of the simulation database SQL simulation request tool; configure its simulation parameters, including the number of simulated clients, the number of concurrent requests for each client, the number of transactions for each client, the number of concurrent requests, the running time of the simulated load, and the read-write ratio in the request, and the simulation of grouping, sorting, aggregation, and subqueries in the query. Resource management module: manage the hardware and network resources consistent with the original database environment, and manage the hardware and network resources consistent with the target database. Parameter configuration management module: when the original database environment is simulated and calibrated, the parameter set information during the simulation load is recorded, including the number of clients, TPS, QPS, running time, query configuration parameters, and a set of parameter sets are recorded for daily load and peak load respectively, and these parameters are applied to the target database. Database module and infrastructure module: databases include: Mysql, Oracle, Postgresql, GaussDB, OceanBase and TDSQL, etc., infrastructure includes containers, virtual machines, servers, networks and load balancing, etc. The database and basic equipment are the basic resource support environment to support this application.

[0169] Figure 9 A schematic diagram of the structure of the computer equipment provided for this application, such as Figure 9 As shown, the computer device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0170] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that the at least one processor 501 performs the above method.

[0171] The specific implementation process of the processor 501 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0172] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0173] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0174] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0175] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0176] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0177] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0179] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0180] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0183] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0184] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A resource processing method for database migration, characterized in that: Applied to computing devices, including: Obtain static information of the original database, parameter information of the target database, and request parameters and load parameters within a preset time period; Determining the initial resources required in the target database based on the static information of the original database and the parameter information of the target database; Determining a test data model and a simulation request tool based on static information of the original database; Initializing the test data model and simulation request tool; Simulating the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period; Adjusting the simulated request parameters based on the load parameters of the original database during the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, thereby obtaining adjusted target request parameters corresponding to the preset time period, and configuring target test data that includes the target request parameters, static information of the original database, and the load parameters during the preset time period; Deploy the target test data to the target database, and request the target database with reference to the target request parameters to obtain the load parameters corresponding to the target database in the preset time period; Adjusting the initial resources based on the load parameters of the original database during the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, thereby determining target resources required by the target database during the preset time period; Output the target resources required by the target database within the preset time period.

2. The method according to claim 1, characterized in that The static information of the original database includes: the total number of CPU cores, total memory, total data volume and network bandwidth resources; the parameter information of the target database includes: the number of CPU cores per node, memory per node and data capacity per node; Accordingly, determining the initial resources required in the target database according to the static information of the original database and the parameter information of the target database includes: Obtaining a first ratio between the total number of CPU cores of the original database and the number of CPU cores of a single node of the target database, a second ratio between the total memory of the original database and the memory of a single node of the target database, and a third ratio between the total data volume of the original database and the data capacity of a single node of the target database; Determine the maximum value among the first ratio, the second ratio and the third ratio as the initial node number of the target library; Determining the network broadband resources of the original database as initial network broadband resources; The initial number of nodes in the target library and the initial network bandwidth resources are determined as the initial resources.

3. The method according to claim 1, characterized in that The static information of the original database includes: database type, query method and ratio information of data read and write operations; Accordingly, the determining of the test data model and the simulation request tool based on the static information of the original database includes: If the database type of the original database is transactional, and the proportion of read operations on the data of the original database exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the first type of environment type; If the database type of the original database is transactional, and the proportion of write operations on the data of the original database exceeds a preset threshold, the test data model and simulation request tool are determined to belong to the second type of environment type; If the database type of the original database is analytical and the query mode is ad hoc, the test data model and simulation request tool are determined to belong to the third type of environment type; If the database type of the original database is analytical and the query mode is data processing, the test data model and simulation request tool are determined to belong to the fourth type of environment.

4. The method according to claim 1, wherein The static information of the original database includes: hardware configuration environment information; Accordingly, adjusting the simulated request parameters based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database includes: Deploying the initial test data in the same hardware configuration environment as the original database; In the same hardware configuration environment as the original database, the simulated request parameters are adjusted based on the load parameters of the original database in the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database.

5. The method according to claim 4, characterized in that After adjusting the initial resources based on the load parameters of the original database in the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database to determine the target resources required by the target database in the preset time period, the method further includes: Acquire a first resource occupied by deploying the target test data on the target database; Acquire a second resource occupied by deploying the initial test data in the same hardware configuration environment as the original database; The first resource and the second resource are released.

6. The method according to any one of claims 1 to 5, characterized in that The preset time period includes a peak time period or a non-peak time period.

7. A resource processing device for database migration, characterized in that: Applied to computing devices, including: An acquisition module is used to obtain static information of the original database, parameter information of the target database, and request parameters and load parameters in a preset time period; a processing module, configured to determine the initial resources required in the target database according to the static information of the original database and the parameter information of the target database; The processing module is further configured to determine a test data model and a simulation request tool based on the static information of the original database; The processing module is further used to initialize the test data model and the simulation request tool; The processing module is further configured to simulate the static information of the original database and the request parameters in the preset time period based on the test data model and the simulation request tool to obtain initial test data corresponding to the original database in the preset time period; The processing module is further configured to adjust the simulated request parameters based on the load parameters of the original database during the preset time period until the load parameters corresponding to the initial test data are consistent with the load parameters of the original database, so as to obtain adjusted target request parameters corresponding to the preset time period, and configure the target request parameters, static information of the original database, and target test data of the load parameters during the preset time period; The processing module is further configured to deploy the target test data to the target database, and request the target database with reference to the target request parameter to obtain the load parameter corresponding to the target database in the preset time period; The processing module is further configured to adjust the initial resources based on the load parameters of the original database during the preset time period until the load parameters corresponding to the target database are consistent with the load parameters of the original database, so as to determine the target resources required by the target database during the preset time period; The processing module is further configured to output target resources required by the target database within the preset time period.

8. A computer device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images