A tablespace storage capacity estimation method, device, equipment and medium

Abstract

The application provides a table space storage capacity estimation method, device, equipment and medium. The method comprises the following steps: creating a temporary table in a target database according to the table structure of a source database table where to-be-loaded data is located, and loading sample data from the source database table to the temporary table; calculating the table space storage capacity occupied by the sample data in the temporary table, and calculating the number of the sample data; determining the attribute value of the sample data as the attribute value of the to-be-loaded data according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data; estimating the number of the to-be-loaded data according to the sample data, and obtaining an estimated value of the number of the to-be-loaded data; and calculating an estimated value of the table space storage capacity occupied by the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data. Therefore, the application can calculate the estimated value of the table space storage capacity occupied by the to-be-loaded data in the source database table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a table space storage capacity estimation method, device, equipment and medium. BACKGROUND

[0002] In recent years, with the rapid development of China, the banking industry is also expanding rapidly. Bank business is becoming more and more extensive, the number of customers is becoming larger and larger, and the transaction system is becoming more and more complex. The data accumulated by commercial banks also grows exponentially, and the collection, cleaning, mining, sorting and reporting of these massive data pose great challenges to traditional banks in storing and utilizing these massive data. For example, in the regulatory reporting application, massive reporting data are involved, which need to be loaded from the upstream source system to the downstream big data platform, and then processed to generate a reporting table that meets the requirements of the regulatory agency and uploaded to the regulatory agency. When loading data from the upstream system to the big data platform, the size of the data to be loaded cannot be directly obtained from the upstream system, and there is a certain risk in importing data because the size of the loaded data is unknown.

[0003] In summary, there is an urgent need for a method for estimating the table space storage capacity occupied by data to be loaded into the downstream big data platform. SUMMARY

[0004] Therefore, the present application provides a table space storage capacity estimation method, device, equipment and medium for estimating the table space storage capacity occupied by data to be loaded into the downstream big data platform, and the technical solution is as follows:

[0005] A table space storage capacity estimation method comprises:

[0006] According to the table structure of the source database table where the to-be-loaded data is located, a temporary table is created in the target database, and sample data is randomly loaded from the source database table to the temporary table, wherein the sample data is part of the to-be-loaded data;

[0007] The table space storage capacity occupied by the sample data in the temporary table is calculated, and the number of sample data is calculated;

[0008] According to the table space storage capacity occupied by the sample data in the temporary table and the number of sample data, the attribute value of the sample data is determined as the attribute value of the to-be-loaded data, wherein the attribute value of the to-be-loaded data represents the table space storage capacity occupied by a piece of to-be-loaded data in the source database table;

[0009] The number of to-be-loaded data is estimated according to the sample data, and an estimated value of the number of to-be-loaded data is obtained;

[0010] According to the attribute value of the to-be-loaded data and the quantity estimation value of the to-be-loaded data, an estimation value of table space storage capacity occupied by the to-be-loaded data in the source database table is calculated.

[0011] Optionally, the sample data comprises first sample data and second sample data, wherein the first sample data refers to sample data randomly loaded to the temporary table for the first time, and the second sample data refers to sample data randomly loaded to the temporary table for the second time.

[0012] The table space storage capacity occupied by the sample data in the temporary table is calculated, and the quantity of the sample data is calculated, comprising:

[0013] The table space storage capacity occupied by the first sample data in the temporary table is calculated as the first table space storage capacity.

[0014] The table space storage capacity occupied by the second sample data in the temporary table is calculated as the second table space storage capacity.

[0015] The quantity of the first sample data is calculated as the first quantity.

[0016] The quantity of the second sample data is calculated as the second quantity.

[0017] Optionally, the quantity estimation value of the to-be-loaded data is obtained by estimating the quantity of the to-be-loaded data according to the sample data, comprising:

[0018] The quantity of the same sample data contained in the first sample data and the second sample data is calculated as the third quantity.

[0019] The quantity estimation value of the to-be-loaded data is calculated according to the first quantity, the second quantity and the third quantity.

[0020] Optionally, the quantity estimation value of the to-be-loaded data is calculated according to the first quantity, the second quantity and the third quantity, comprising:

[0021] The first quantity and the second quantity are multiplied to obtain a product value.

[0022] The quotient of the product value and the third quantity is calculated, and the quotient value is taken as the quantity estimation value of the to-be-loaded data.

[0023] Optionally, the attribute value of the sample data is determined according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, comprising:

[0024] The attribute value of the sample data is calculated according to the first quantity, the second quantity, the first table space storage capacity and the second table space storage capacity.

[0025] Optionally, the attribute value of the sample data is calculated according to the first quantity, the second quantity, the first table space storage capacity and the second table space storage capacity, comprising:

[0026] calculate a quotient of the first table space storage capacity and the first number, and the quotient value as the first attribute value;

[0027] calculate a quotient of the second table space storage capacity and the second number, and the quotient value as the second attribute value;

[0028] calculate an average of the first attribute value and the second attribute value, and obtain the attribute value of the sample data.

[0029] Optionally, according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data, the table space storage capacity estimated value of the to-be-loaded data in the source database table is calculated, comprising:

[0030] multiply the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data, and the product value as the table space storage capacity estimated value of the to-be-loaded data in the source database table.

[0031] A table space storage capacity estimation device, comprising: a sample data loading module, an information calculation module, an attribute value determination module, a number estimated value determination module and a table space storage capacity estimation module;

[0032] The sample data loading module is configured to create a temporary table in the target database according to a table structure of a source database table in which the to-be-loaded data is located, and load sample data from the source database table to the temporary table randomly, wherein the sample data is part of the to-be-loaded data.

[0033] The information calculation module is configured to calculate a table space storage capacity occupied by the sample data in the temporary table, and calculate a number of the sample data.

[0034] The attribute value determination module is configured to determine an attribute value of the sample data according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data, and the attribute value of the sample data is taken as an attribute value of the to-be-loaded data, wherein the attribute value of the to-be-loaded data represents a table space storage capacity occupied by a piece of to-be-loaded data in the source database table.

[0035] The number estimated value determination module is configured to estimate a number of the to-be-loaded data according to the sample data, and obtain an estimated value of the number of the to-be-loaded data.

[0036] The table space storage capacity estimation module is configured to calculate a table space storage capacity estimated value of the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data.

[0037] A table space storage capacity estimation device, comprising a memory and a processor;

[0038] The memory is configured to store a program;

[0039] A processor is configured to execute a program to implement each step of the table space storage capacity estimation method according to any one of the above.

[0040] A readable storage medium stores a computer program, and the computer program is configured to be executed by a processor to implement each step of the table space storage capacity estimation method according to any one of the above.

[0041] According to the above technical solution, the table space storage capacity estimation method provided by the application first creates a temporary table in the target database according to the table structure of the source database table where the to-be-loaded data is located, and then loads sample data from the source database table to the temporary table, calculates the table space storage capacity occupied by the sample data in the temporary table, and calculates the number of the sample data, and then determines the attribute value of the sample data according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data, and takes the attribute value of the sample data as the attribute value of the to-be-loaded data, and then estimates the number of the to-be-loaded data according to the sample data to obtain the estimated value of the number of the to-be-loaded data, and finally calculates the table space storage capacity estimation value of the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data. Therefore, the application can determine the attribute value of the to-be-loaded data according to the sample data loaded into the temporary table, and can estimate the estimated value of the number of the to-be-loaded data according to the sample data, and then according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data, the table space storage capacity estimation value of the to-be-loaded data in the source database table can be accurately and quickly calculated, the estimation efficiency of the table space storage capacity is improved, and a large amount of time is saved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0043] Figure 1 A flowchart of the table space storage capacity estimation method provided by the embodiment of the application is shown.

[0044] Figure 2 A table space storage capacity estimation framework provided by the embodiment of the application is shown.

[0045] Figure 3 A structure diagram of the table space storage capacity estimation device provided by the embodiment of the application is shown.

[0046] Figure 4 A hardware structure block diagram of the table space storage capacity estimation device provided by the embodiment of the application is shown. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0048] This application provides a method for estimating tablespace storage capacity. The following embodiments will provide a detailed description of the tablespace storage capacity estimation method provided in this application.

[0049] Please see Figure 1 The diagram illustrates a flowchart of a tablespace storage capacity estimation method provided in an embodiment of this application. This tablespace storage capacity estimation method may include:

[0050] Step S101: Based on the table structure of the source database table containing the data to be loaded, create a temporary table in the target database and randomly load sample data from the source database table into the temporary table.

[0051] Here, the source database table refers to the table in the source database, the data to be loaded refers to the data in the source database table that needs to be loaded into the target database, and the sample data is a portion of the data to be loaded.

[0052] The source database table may contain a massive amount of data to be loaded. It is difficult to directly calculate the table space storage capacity occupied by the massive amount of data to be loaded in the source database table. However, if the target database contains a table with the same table structure as the source database table, the table space storage capacity occupied by the small amount of data to be loaded can be easily calculated when loading a small amount of data into that table.

[0053] Based on this, before loading the massive amount of data to be loaded into the target database, this step first creates a temporary table in the target database with the same table structure as the source database table. Then, sample data is randomly loaded from the source database table into the temporary table so that the table space storage capacity occupied by the data to be loaded in the source database table can be estimated based on the sample data in the temporary table. In other words, the physical resource capacity occupied by the data to be loaded in the source database table can be estimated based on the sample data in the temporary table.

[0054] Optionally, the source database can be a database in an upstream source system, and the target database can be a database in a downstream big data platform.

[0055] In an alternative embodiment, the step of loading the sample data from the source database table to the temporary table can comprise: loading the sample data from the source database table to the temporary table for multiple times. For example, the step of loading the sample data from the source database table to the temporary table can comprise: loading the sample data from the source database table to the temporary table for two times.

[0056] In step S102, the table space storage capacity occupied by the sample data in the temporary table is calculated, and the number of the sample data is calculated.

[0057] Since the number of the sample data is small, the table space storage capacity occupied by the sample data in the temporary table can be easily calculated, and the number of the sample data can be easily calculated.

[0058] In step S103, the attribute value of the sample data is determined as the attribute value of the data to be loaded according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data.

[0059] The attribute value of the data to be loaded represents the table space storage capacity occupied by a piece of data to be loaded in the source database table.

[0060] The table space storage capacity occupied by the sample data in the temporary table and the number of the sample data have been calculated in the foregoing steps, and thus the table space storage capacity occupied by a piece of sample data in the temporary table can be easily estimated according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data. For the convenience of description, the table space storage capacity occupied by a piece of sample data in the temporary table is referred to as the attribute value of the sample data in this step.

[0061] Since the temporary table has the same table structure as the source database table, the attribute value of the sample data is the same as the attribute value of the data to be loaded, and thus the attribute value of the sample data can be used as the attribute value of the data to be loaded in this step.

[0062] In step S104, the number of the data to be loaded is estimated according to the sample data, and an estimated value of the number of the data to be loaded is obtained.

[0063] Alternatively, it has been described above that the sample data can be loaded from the source database table to the temporary table for multiple times, and thus the number of the data to be loaded can be estimated according to the sample data loaded for multiple times in this step. For example, if the sample data is loaded from the source database table to the temporary table for two times, the number of the data to be loaded can be estimated according to the sample data loaded for two times.

[0064] In step S105, the table space storage capacity estimate value occupied by the data to be loaded in the source database table is calculated according to the attribute value of the data to be loaded and the estimated value of the number of the data to be loaded.

[0065] In summary, the table space storage capacity estimation method provided by the present application first creates a temporary table in the target database according to the table structure of the source database table where the to-be-loaded data is located, and then randomly loads sample data from the source database table to the temporary table, and calculates the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data. Next, according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data, the attribute value of the sample data is determined as the attribute value of the to-be-loaded data. Then, according to the sample data, the number of the to-be-loaded data is estimated to obtain an estimated value of the number of the to-be-loaded data. Finally, according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data, the estimated value of the table space storage capacity occupied by the to-be-loaded data in the source database table is calculated. As can be seen, the present application can determine the attribute value of the to-be-loaded data according to the sample data loaded to the temporary table, and can estimate the estimated value of the number of the to-be-loaded data according to the sample data. Then, according to the attribute value of the to-be-loaded data and the estimated value of the number of the to-be-loaded data, the estimated value of the table space storage capacity occupied by the to-be-loaded data in the source database table can be accurately and quickly calculated, thereby improving the estimation efficiency of the table space storage capacity and the controllability of the data and the maintainability and stability of the data platform where the target database is located.

[0066] Meanwhile, the table space storage capacity estimation method provided by the present application makes the estimation of the table space storage capacity simple, and saves a lot of time.

[0067] Through experiments, it is found that the deviation of the table space storage capacity estimated by the table space storage capacity estimation method provided by the present application from the result obtained by querying the target database after the to-be-loaded data is actually imported into the target database is not more than 5%, which also proves that the table space storage capacity estimation method provided by the present application can more accurately evaluate the table space storage capacity occupied by the to-be-loaded data loaded into the target database.

[0068] In an embodiment of the present application, steps S102-S105 provided in the above embodiment are described in detail taking the step S101 of randomly loading twice sample data from the source database table to the temporary table as an example.

[0069] For the convenience of subsequent description, the two times of randomly loaded sample data are respectively denoted as first sample data and second sample data, that is, the sample data in the above embodiment includes the first sample data and the second sample data, wherein the first sample data refers to the sample data randomly loaded to the temporary table for the first time, and the second sample data refers to the sample data randomly loaded to the temporary table for the second time.

[0070] Based on this, first, the process of "step S102, calculating the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data" is introduced.

[0071] Optionally, the implementation of step S102 can include: calculating the table space storage capacity occupied by the first sample data in the temporary table as the first table space storage capacity; calculating the table space storage capacity occupied by the second sample data in the temporary table as the second table space storage capacity; calculating the number of the first sample data as the first number; and calculating the number of the second sample data as the second number.

[0072] Based on the calculation of the first table space storage capacity, the second table space storage capacity, the first number and the second number in step S102, the process of step S103, i.e., determining the attribute value of the sample data according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data, specifically refers to: calculating the attribute value of the sample data according to the first number, the second number, the first table space storage capacity and the second table space storage capacity.

[0073] Optionally, the process of calculating the attribute value of the sample data according to the first number, the second number, the first table space storage capacity and the second table space storage capacity can include: calculating the quotient of the first table space storage capacity and the first number, and the quotient value is the first attribute value; calculating the quotient of the second table space storage capacity and the second number, and the quotient value is the second attribute value; and calculating the average of the first attribute value and the second attribute value to obtain the attribute value of the sample data.

[0074] Specifically, as introduced above, the first number refers to the number of the first sample data, and the first table space storage capacity refers to the table space storage capacity occupied by the first sample data in the temporary table. Therefore, according to the first table space storage capacity and the first number, the table space storage capacity occupied by each sample data included in the first sample data in the temporary table can be obtained, that is, the quotient of the first table space storage capacity and the first number is calculated, and the quotient value is the table space storage capacity occupied by each sample data included in the first sample data in the temporary table. For the convenience of subsequent description, the quotient value is taken as the first attribute value. In this step, the first attribute value can be represented by B1, and the unit of the first attribute value B1 is GB.

[0075] Similarly, the second quantity refers to the number of the second sample data, and the second tablespace storage capacity refers to the tablespace storage capacity occupied by the second sample data in the temporary table. Therefore, according to the second tablespace storage capacity and the second quantity, the tablespace storage capacity occupied by each piece of sample data contained in the second sample data in the temporary table can be obtained, that is, the quotient of the second tablespace storage capacity and the second quantity is obtained, and the quotient value is the tablespace storage capacity occupied by each piece of sample data contained in the second sample data in the temporary table. For the convenience of subsequent description, the quotient value is taken as the second attribute value. In this step, the first attribute value can be represented by B2, and the unit of the second attribute value B2 is GB.

[0076] After obtaining the first attribute value B1 and the second attribute value B2, the attribute value of the sample data can be calculated according to the first attribute value B1 and the second attribute value B2. For example, optionally, the average value of the first attribute value B1 and the second attribute value B2 can be calculated in this step, and the average value is taken as the attribute value of the sample data, that is, the attribute value of the sample data is represented by P, and P = 0.5 * (B1 / N1 + B2 / N2).

[0077] Correspondingly, the process of the above-mentioned "step S104, estimating the number of the to-be-loaded data according to the sample data to obtain the number estimation value of the to-be-loaded data" can include the following steps S1041-S1042:

[0078] Step S1041, calculating the number of the same sample data contained in the first sample data and the second sample data as a third quantity.

[0079] It can be understood that when the sample data is randomly loaded twice, the same sample data can be contained in the first sample data and the second sample data. At this time, the number of the same sample data can be calculated through this step.

[0080] For the convenience of subsequent description, the number calculated in this step is referred to as the third quantity.

[0081] Step S1042, calculating the number estimation value of the to-be-loaded data according to the first quantity, the second quantity and the third quantity.

[0082] The process of this step can include various implementation manners, and one of the implementation manners is given below. It should be noted that the embodiment is only an example and does not limit the step.

[0083] Optionally, the process of the step "calculating the number estimation value of the to-be-loaded data according to the first quantity, the second quantity and the third quantity" can include: multiplying the first quantity and the second quantity to obtain a product value; calculating the quotient of the product value and the third quantity, and the quotient value is taken as the number estimation value of the to-be-loaded data.

[0084] That is, the first quantity is denoted as N1, the second quantity is denoted as N2, and the third quantity is denoted as N 12 Therefore, the quantity estimation value N of the to-be-loaded data is calculated in this step as:

[0085] The quantity estimation value of the to-be-loaded data has been calculated in this step, and the attribute value of the to-be-loaded data has been calculated in step S102, i.e., the table space storage capacity occupied by a piece of to-be-loaded data in the source database table, so that the table space storage capacity occupied by the massive to-be-loaded data in the source database table can be easily calculated.

[0086] Based on this, in an optional embodiment, after the above introduction, the process of "step S105, calculating the table space storage capacity estimation value occupied by the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the quantity estimation value of the to-be-loaded data" can include: multiplying the attribute value of the to-be-loaded data and the quantity estimation value of the to-be-loaded data, and the product value is taken as the table space storage capacity estimation value occupied by the to-be-loaded data in the source database table. In this step, the attribute value of the to-be-loaded data and the quantity estimation value of the to-be-loaded data can be multiplied, and the product value is taken as the table space storage capacity estimation value occupied by the to-be-loaded data in the source database table, i.e., the table space storage capacity estimation value occupied by the to-be-loaded data in the source database table is denoted as Storage_size, and Storage_size=P*N.

[0087] This embodiment provides a method for calculating the table space storage capacity estimation value occupied by the to-be-loaded data in the source database table in the case of randomly loading sample data twice. Through the attribute value calculation and the quantity estimation of the to-be-loaded data, the table space storage capacity occupied by the to-be-loaded data can be calculated more quickly and accurately, the uncertainty caused by the unknown size of the table space storage capacity occupied by the to-be-loaded data is avoided, and the data controllability and the maintainability and stability of the platform where the target database is located are improved.

[0088] In order for those skilled in the art to better understand the present application, the table space storage capacity estimation method provided by the present application is described in detail below through a specific embodiment.

[0089] Optionally, the table space storage capacity estimation method provided by the present application can be applied to a program script get_stroage.sql, which can automatically load sample data and can query and calculate the table space storage capacity estimation value of the loading source.

[0090] Specifically, the program script get_stroage.sql can obtain the table space storage capacity estimation value of the to-be-loaded data in the source database table by calling two subscripts get_number.sql and get_property.sql. The functions of the program scripts are introduced below based on the table space storage capacity estimation framework diagram shown in FIG. 1. Figure 2

[0091] The function of get_number.sql includes: creating a temporary table (for example, M1_T) in the target database according to the table structure of the source database table where the to-be-loaded data is located, and loading sample data from the source database table to the temporary table; calculating the number of the sample data, and estimating the number of the to-be-loaded data according to the sample data to obtain the number estimation value of the to-be-loaded data.

[0092] Optionally, the code implementation of get_number.sql for calculating the number of the sample data can be: select count(*) from M1_T.

[0093] The function of get_property.sql includes: calculating the table space storage capacity occupied by the sample data in the temporary table; and calculating the property value of the sample data according to the table space storage capacity occupied by the sample data in the temporary table and the number of the sample data, as the property value of the to-be-loaded data.

[0094] Optionally, the code implementation of get_property.sql for calculating the table space storage capacity occupied by the sample data in the temporary table can be:

[0095] Select table_schema||',',table_name||',',(table_data_size / 1000 / 1000 / 1000)||',',(table_storage_size / 1000 / 1000 / 1000)||',',table_type frominformation_schema.CLUSTER_TABLES where TABLE_SCHEMA='mdb'and TABLE_NAME='M1_T'。

[0096] The function of get_stroage.sql includes: calling get_number.sql to calculate the number estimation value of the to-be-loaded data; calling get_property.sql to calculate the property value of the to-be-loaded data; and calculating the table space storage capacity estimation value of the to-be-loaded data in the source database table according to the property value of the to-be-loaded data and the number estimation value of the to-be-loaded data. ​

[0097] Optionally, in the embodiment, the get_number.sql can also record the estimated value of the number of the to-be-loaded data in the table of data description in the form of metadata, and the get_property.sql can also record the attribute value of the to-be-loaded data in the table of data description in the form of metadata, and then the get_stroage.sql can calculate the estimated value of the table space storage capacity occupied by the to-be-loaded data in the source database table according to the attribute value and the estimated value of the number recorded in the table of data description.

[0098] In summary, for the to-be-loaded data needed to be loaded into the target database, the embodiment first creates a temporary table in the target database, and loads a small amount of sample data into the temporary table in the target database, then calculates the table space storage capacity size occupied by an average data in the sample data as the attribute value of the table itself, and estimates the number of the to-be-loaded data, when the capacity of the massive to-be-loaded data contained in the source database table needs to be estimated, the storage capacity of the to-be-loaded data can be estimated through the attribute value and the number of the to-be-loaded data, experiments prove that the table space storage capacity calculated by the embodiment is relatively accurate, and the embodiment makes the estimation of the table space storage capacity simple, saves a lot of time, and improves the data controllability and the maintainability and stability of the big data platform.

[0099] The embodiment of the application further provides a table space storage capacity estimation device, the table space storage capacity estimation device provided by the embodiment of the application is described below, and the table space storage capacity estimation device described below can be correspondingly referred to the table space storage capacity estimation method described above.

[0100] Please refer to Figure 3 , which shows the structure schematic diagram of the table space storage capacity estimation device provided by the embodiment of the application, as Figure 3 shown, the table space storage capacity estimation device can include: a sample data loading module 301, an information calculation module 302, an attribute value determination module 303, a number estimation value determination module 304 and a table space storage capacity estimation module 305.

[0101] The sample data loading module 301 is configured to create a temporary table in the target database according to the table structure of the source database table where the to-be-loaded data is located, and load sample data from the source database table to the temporary table randomly, wherein the sample data is part of the to-be-loaded data.

[0102] The information calculation module 302 is configured to calculate the table space storage capacity occupied by the sample data in the temporary table, and calculate the number of the sample data.

[0103] The attribute value determination module 303 is configured to determine an attribute value of the sample data as the attribute value of the to-be-loaded data according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, where the attribute value of the to-be-loaded data represents the table space storage capacity occupied by a piece of to-be-loaded data in the source database table.

[0104] The quantity estimation value determination module 304 is configured to estimate the quantity of the to-be-loaded data according to the sample data, to obtain an estimated quantity of the to-be-loaded data.

[0105] The table space storage capacity estimation module 305 is configured to calculate an estimated table space storage capacity of the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the estimated quantity of the to-be-loaded data.

[0106] The table space storage capacity estimation device provided by the present application first creates a temporary table in the target database according to the table structure of the source database table in which the to-be-loaded data is located, and then randomly loads sample data from the source database table to the temporary table, and calculates the table space storage capacity occupied by the sample data in the temporary table, and calculates the quantity of the sample data, and then determines the attribute value of the sample data as the attribute value of the to-be-loaded data according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, and then estimates the quantity of the to-be-loaded data according to the sample data, to obtain an estimated quantity of the to-be-loaded data, and finally calculates an estimated table space storage capacity of the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the estimated quantity of the to-be-loaded data. As can be seen, the present application can determine the attribute value of the to-be-loaded data according to the sample data loaded to the temporary table, and can estimate the estimated quantity of the to-be-loaded data according to the sample data, and then according to the attribute value of the to-be-loaded data and the estimated quantity of the to-be-loaded data, the estimated table space storage capacity of the to-be-loaded data in the source database table can be accurately and quickly calculated, thereby improving the estimation efficiency of the table space storage capacity, and improving the data controllability and the maintainability and stability of the data platform in which the target database is located.

[0107] Meanwhile, the table space storage capacity estimation method provided by the present application makes the estimation of the table space storage capacity simple, and saves a lot of time.

[0108] In a possible implementation, the sample data randomly loaded by the sample data loading module 301 includes first sample data and second sample data, where the first sample data refers to sample data randomly loaded to the temporary table for the first time, and the second sample data refers to sample data randomly loaded to the temporary table for the second time.

[0109] Based on this, the information calculation module 302 can include a first calculation module, a second calculation module, a third calculation module and a fourth calculation module.

[0110] The first calculating module is configured to calculate the table space storage capacity occupied by the first sample data in the temporary table as the first table space storage capacity.

[0111] The second calculating module is configured to calculate the table space storage capacity occupied by the second sample data in the temporary table as the second table space storage capacity.

[0112] The third calculating module is configured to calculate the number of the first sample data as the first number.

[0113] The fourth calculating module is configured to calculate the number of the second sample data as the second number.

[0114] In a possible implementation, the number estimation value determining module 304 can include a fifth calculating module and a sixth calculating module.

[0115] The fifth calculating module is configured to calculate the number of the same sample data contained in the first sample data and the second sample data as the third number.

[0116] The sixth calculating module is configured to calculate the number estimation value of the to-be-loaded data according to the first number, the second number and the third number.

[0117] In a possible implementation, the sixth calculating module can include a first product calculating module and a first quotient calculating module.

[0118] The first product calculating module is configured to multiply the first number and the second number to obtain a product value.

[0119] The first quotient calculating module is configured to calculate the quotient of the product value and the third number, and the quotient value is the number estimation value of the to-be-loaded data.

[0120] In a possible implementation, the attribute value determining module 303 can be specifically configured to calculate the attribute value of the sample data according to the first number, the second number, the first table space storage capacity and the second table space storage capacity.

[0121] In a possible implementation, when calculating the attribute value of the sample data according to the first number, the second number, the first table space storage capacity and the second table space storage capacity, the attribute value determining module 303 can include a second quotient calculating module, a third quotient calculating module and an average value calculating module.

[0122] The second quotient calculating module is configured to calculate the quotient of the first table space storage capacity and the first number, and the quotient value is the first attribute value.

[0123] The third quotient calculation module is configured to calculate a quotient of the second table space storage capacity and the second quantity, and the quotient value is taken as the second attribute value.

[0124] The average value calculation module is configured to calculate an average value of the first attribute value and the second attribute value, and obtain the attribute value of the sample data.

[0125] In a possible implementation, the table space storage capacity estimation module 305 can be specifically configured to multiply the attribute value of the to-be-loaded data and the quantity estimation value of the to-be-loaded data, and take a product value as an estimation value of the table space storage capacity occupied by the to-be-loaded data in the source database table.

[0126] The embodiments of the present application further provide a table space storage capacity estimation device. Optionally, Figure 4 A hardware structure block diagram of the table space storage capacity estimation device is shown, and the hardware structure of the table space storage capacity estimation device can include at least one processor 401, at least one communication interface 402, at least one memory 403 and at least one communication bus 404. Figure 4

[0127] In the embodiments of the present application, the number of the processor 401, the communication interface 402, the memory 403 and the communication bus 404 is at least one, and the processor 401, the communication interface 402 and the memory 403 complete mutual communication through the communication bus 404;

[0128] The processor 401 can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.

[0129] The memory 403 can include a high-speed RAM memory, and can also include a non-volatile memory, etc., for example, at least one disk memory.

[0130] The memory 403 stores a program, and the processor 401 can call the program stored in the memory 403, and the program is used for:

[0131] According to the table structure of the source database table where the to-be-loaded data is located, a temporary table is created in the target database, and sample data is randomly loaded from the source database table to the temporary table, wherein the sample data is part of the to-be-loaded data;

[0132] The table space storage capacity occupied by the sample data in the temporary table is calculated, and the quantity of the sample data is calculated.

[0133] ​According to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, attribute values of the sample data are determined as attribute values of the to-be-loaded data, wherein the attribute values of the to-be-loaded data represent the table space storage capacity occupied by a piece of to-be-loaded data in the source database table;

[0134] According to the sample data, the quantity of the to-be-loaded data is estimated to obtain an estimated value of the quantity of the to-be-loaded data;

[0135] According to the attribute values of the to-be-loaded data and the estimated value of the quantity of the to-be-loaded data, an estimated value of the table space storage capacity occupied by the to-be-loaded data in the source database table is calculated.

[0136] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0137] The embodiment of the present application further provides a readable storage medium, which has a computer program stored thereon, and the computer program is executed by a processor to realize the table space storage capacity estimation method.

[0138] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0139] Finally, it needs to be noted that, in this document, relationship terms such as and the second and the like are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or sequence between the entities or operations. Moreover, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitation, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0140] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0141] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of estimating a storage capacity of a tablespace, characterized by, The method comprises the following steps: creating a temporary table in a target database according to a table structure of a source database table where data to be loaded is located, and loading sample data from the source database table to the temporary table randomly, wherein the sample data is part of the data to be loaded; the sample data comprises first sample data and second sample data; the first sample data refers to sample data loaded to the temporary table for the first time, and the second sample data refers to sample data loaded to the temporary table for the second time; calculating a table space storage capacity occupied by the first sample data in the temporary table as a first table space storage capacity; calculating a table space storage capacity occupied by the second sample data in the temporary table as a second table space storage capacity; calculating a quantity of the first sample data as a first quantity; calculating a quantity of the second sample data as a second quantity; determining an attribute value of the sample data as an attribute value of the data to be loaded according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, wherein the attribute value of the data to be loaded represents a table space storage capacity occupied by a piece of data to be loaded in the source database table; calculating a quantity of the same sample data contained in the first sample data and the second sample data as a third quantity; calculating an estimated value of the quantity of the data to be loaded according to the first quantity, the second quantity and the third quantity; calculating an estimated value of the table space storage capacity occupied by the data to be loaded in the source database table according to the attribute value of the data to be loaded and the estimated value of the quantity of the data to be loaded.

2. The tablespace storage capacity estimation method of claim 1, wherein, The method of calculating the estimated value of the quantity of the data to be loaded according to the first quantity, the second quantity and the third quantity comprises the following steps: multiplying the first quantity and the second quantity to obtain a product value; calculating a quotient of the product value and the third quantity, and taking the quotient value as the estimated value of the quantity of the data to be loaded.

3. The tablespace storage capacity estimation method of claim 1, wherein, The method of determining the attribute value of the sample data according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data comprises the following steps: calculating the attribute value of the sample data according to the first quantity, the second quantity, the first table space storage capacity and the second table space storage capacity.

4. The tablespace storage capacity estimation method of claim 3, wherein, The method of calculating the attribute value of the sample data according to the first quantity, the second quantity, the first table space storage capacity and the second table space storage capacity comprises the following steps: calculating a quotient of the first table space storage capacity and the first quantity, and taking the quotient value as a first attribute value; calculating a quotient of the second table space storage capacity and the second quantity, and taking the quotient value as a second attribute value; calculating an average value of the first attribute value and the second attribute value to obtain the attribute value of the sample data.

5. The tablespace storage capacity estimation method of claim 1, wherein, The method of calculating the estimated value of the table space storage capacity occupied by the data to be loaded in the source database table according to the attribute value of the data to be loaded and the estimated value of the quantity of the data to be loaded comprises the following steps: The attribute value of the to-be-loaded data is determined according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, as the attribute value of the to-be-loaded data, wherein the attribute value of the to-be-loaded data represents the table space storage capacity occupied by a piece of to-be-loaded data in the source database table.

6. A tablespace storage capacity estimation apparatus characterized by comprising: The method comprises the following steps: The sample data loading module is configured to create a temporary table in the target database according to the table structure of the source database table in which the to-be-loaded data is located, and load sample data from the source database table to the temporary table randomly, wherein the sample data is part of the to-be-loaded data; the sample data comprises first sample data and second sample data; the first sample data refers to sample data loaded to the temporary table for the first time, and the second sample data refers to sample data loaded to the temporary table for the second time; The first computing module is configured to calculate the table space storage capacity occupied by the first sample data in the temporary table, as a first table space storage capacity; The second computing module is configured to calculate the table space storage capacity occupied by the second sample data in the temporary table, as a second table space storage capacity; The third computing module is configured to calculate the quantity of the first sample data, as a first quantity; The fourth computing module is configured to calculate the quantity of the second sample data, as a second quantity; The attribute value determining module is configured to determine the attribute value of the sample data according to the table space storage capacity occupied by the sample data in the temporary table and the quantity of the sample data, as the attribute value of the to-be-loaded data, wherein the attribute value of the to-be-loaded data represents the table space storage capacity occupied by a piece of to-be-loaded data in the source database table; The fifth computing module is configured to calculate the quantity of the same sample data contained in the first sample data and the second sample data, as a third quantity; The sixth computing module is configured to calculate the quantity estimation value of the to-be-loaded data according to the first quantity, the second quantity and the third quantity; The table space storage capacity estimation module is configured to calculate the table space storage capacity estimation value of the to-be-loaded data in the source database table according to the attribute value of the to-be-loaded data and the quantity estimation value of the to-be-loaded data. The computer program is executed by the processor to implement the steps of the table space storage capacity estimation method according to any one of claims 1-5. The computer program is executed by the processor to implement the steps of the table space storage capacity estimation method according to any one of claims 1-5.

7. A tablespace storage capacity estimation device characterized by comprising: ​ ​ ​ 8. A readable storage medium, having stored thereon a computer program, characterized in that, ​

Images