Data barreling method of database and related products

Abstract

The invention provides a data barreling method of a database and a related product. The method comprises the steps of obtaining a bucket dividing instruction for a database, wherein the bucket dividing instruction is used for performing data bucket dividing on the database; determining a permission list of an initiator of the bucket dividing instruction, wherein the permission list is used for indicating permissions owned by the initiator in a database; and according to the permission list, selecting and executing a corresponding data bucket dividing scheme to obtain a database query statement of each bucket of data. Through the method, a proper barrel dividing scheme can be selected according to configuration and owned permissions, so that the overall barrel dividing data extraction efficiency and the data reading performance are improved, the performance consumption of a database is reduced, and the influence on production is reduced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a database data bucketing method and related products. Background Art

[0002] When performing heterogeneous migration of data in certain databases, it is necessary to read all table data in the original database. Since the amount of data may be large, it will affect the data migration time and have a certain impact on production. A common solution is to migrate data in large tables by buckets to improve migration efficiency, reduce migration time, shorten the time window, and reduce the impact on production.

[0003] In the current data bucket migration solution, primary keys or indexes are used when obtaining SQL statements and bucket query data, but this consumes too much performance. The specific performance issues are as follows:

[0004] 1. Reading primary keys or indexes consumes disk, CPU, memory and other resources;

[0005] 2. Using primary key or index sorting will take up more memory and temporary table space, resulting in insufficient sorting performance;

[0006] 3. When reading data sorted by primary key or index, random reading may occur, increasing the query response time.

[0007] Therefore, the above performance issues need to be optimized. Summary of the invention

[0008] In view of the above problems, the present invention proposes a database data bucketing method and related products that overcome the above problems or at least partially solve the above problems.

[0009] One objective of the present invention is to bucket data based on ROWID.

[0010] A further object of the present invention is to reduce the performance consumption of the database when data is bucketed.

[0011] A further object of the present invention is to select different schemes for data bucketing according to different permissions.

[0012] In particular, the present invention provides a method for bucketing data in a database, comprising:

[0013] Get the bucketing instruction for the database. The bucketing instruction is used to bucket the data in the database.

[0014] Determine the permission list of the initiator of the bucketing instruction. The permission list is used to indicate the permissions that the initiator has in the database;

[0015] Select the corresponding data bucketing scheme according to the permission list and execute it to obtain the database query statement for each bucket of data.

[0016] Optionally, the steps of selecting and executing a corresponding data bucketing scheme according to the permission list include:

[0017] When the permission list meets the first preset condition, the first bucketing solution is executed. The first preset condition includes that the initiator has the read permission of the dynamic performance view, the create task permission, and the execute permission of the data operation function, and the system change number is not used for this data bucketing;

[0018] When the permission list meets the second preset condition, the second bucketing scheme is executed, and the second preset condition includes that the initiator has the permission to read the data storage allocation information in the database and the permission to read the object information in the database;

[0019] When the first bucketing scheme and the second bucketing scheme cannot be executed or are not adopted, the third bucketing scheme is executed.

[0020] Optionally, the step of executing the first bucketing scheme includes:

[0021] Generate task names based on bucketing instructions;

[0022] Delete the existing invalid tasks through the delete function in the data operation function;

[0023] Create the corresponding target task according to the task name through the creation function in the data operation function;

[0024] The target task is controlled by the execution function in the data operation function to execute the bucketing task based on ROWID;

[0025] Obtain the execution result of the bucketing task in the system table corresponding to the dynamic performance view according to the task name. The execution result includes the start and end pairs of the data in each bucket. The start and end pairs are the start and end ROWIDs of the buckets where the start and end pairs are located.

[0026] Delete the target task through the delete function in the data operation function;

[0027] Generate a database query statement for each bucket of data based on the execution results.

[0028] Optionally, the preset parameters in the bucketing task include: task name, mode name of the data table to be bucketed, name of the data table to be bucketed, whether to bucket by data row, and number of rows per bucket;

[0029] The steps to perform the ROWID-based bucketing task include:

[0030] Fill the preset parameters according to the bucketing instructions;

[0031] Get the data to be bucketed according to the schema name of the data table to be bucketed and the name of the data table to be bucketed;

[0032] Sort the data to be bucketed according to its own ROWID;

[0033] Use aggregate functions to determine the total number of rows of data to be bucketed based on the sorting results;

[0034] When the preset parameters show that the data is divided into buckets by rows, the sorting result is divided according to the total number of rows and the number of rows in each bucket in the preset parameters to obtain the actual number of buckets and the start and end pairs of data in each bucket.

[0035] Optionally, the step of generating a database query statement for querying data according to the execution result includes:

[0036] Determine the preset number of buckets according to the bucketing instruction;

[0037] Determine the actual number of buckets based on the execution results;

[0038] Determine whether the actual number of buckets exceeds the preset number of buckets;

[0039] If yes, merge the execution results so that the actual number of buckets meets the requirements of the bucketing instruction;

[0040] Generate a database query statement based on the start and end pairs after the merge is completed.

[0041] Optionally, the step of executing the second bucketing scheme includes:

[0042] Obtain the object identifier corresponding to the query table to be bucketed according to the bucketing instruction in the system table where the object information is located in the database;

[0043] In the case where the object identifier exists, the table name and the schema name of the table to be bucketed are filtered in the system table where the data storage allocation information is located according to the object identifier, and the total number of data blocks of the data block where the data to be bucketed is located is determined according to the table name and the schema name;

[0044] Determine the preset number of buckets according to the bucketing instruction, and calculate the number of data blocks corresponding to each bucket according to the total number of data blocks and the preset number of buckets;

[0045] Obtain the file number and block number of the data to be queried by bucketing according to the table name and mode name in the system table where the object information is located;

[0046] Arrange the data to be queried in buckets according to the file number and block number, and generate ROWIDs one by one in sequence;

[0047] The start and end pairs corresponding to each bucket are calculated based on the number of data blocks corresponding to each bucket. The start and end pairs are the start ROWID and end ROWID of the bucket where the start and end pairs are located.

[0048] Generate a database query statement for each bucket of data based on the start and end pairs.

[0049] Optionally, the step of executing the second bucketing scheme further includes:

[0050] Obtain the object identifier corresponding to the query table to be bucketed according to the bucketing instruction in the system table where the object information is located in the database;

[0051] In the case where an object identifier exists, the table name and the schema name of the table to be bucketed are filtered in the system table where the data storage allocation information is located according to the object identifier, and the total number of data blocks of the data block where the data to be bucketed is located and the data volume of each data block are determined according to the table name and the schema name;

[0052] Determine the preset number of data rows corresponding to each bucket according to the bucketing instruction;

[0053] The data blocks are divided according to the preset number of data rows corresponding to each bucket and the data volume of each data block, so as to determine the corresponding relationship between each bucket and the data block;

[0054] Obtain the file number and block number of the data to be queried by bucketing according to the table name and mode name in the system table where the object information is located;

[0055] Arrange the data to be queried in buckets according to the file number and block number, and generate ROWIDs one by one in sequence;

[0056] The start and end pairs corresponding to each bucket are calculated based on the correspondence between each bucket and the data block. The start and end pairs are the start ROWID and end ROWID of the bucket where the start and end pairs are located.

[0057] Generate a database query statement for each bucket of data based on the start and end pairs.

[0058] Optionally, after the step of determining the total number of data blocks of the data block where the to-be-bucketed query data is located and the data amount of each data block according to the table name and the schema name, the step further includes: determining the to-be-bucketed query table according to the table name of the to-be-bucketed query table, and querying the minimum ROWID and the maximum ROWID in the to-be-bucketed query table, wherein the ROWID records the file number and the block number;

[0059] After the step of obtaining the file number and block number of the data to be bucketed and queried according to the table name and the schema name in the system table where the object information is located, the method also includes: judging whether the obtained file number and block number of the data to be bucketed and queried include the minimum file number and the minimum block number corresponding to the minimum ROWID and the maximum file number and the maximum block number corresponding to the maximum ROWID; if not, performing data compensation based on the minimum ROWID and the maximum ROWID, so as to obtain the file number and block number of all the data to be bucketed and queried between the minimum ROWID and the maximum ROWID.

[0060] Optionally, the steps of executing the third bucketing scheme include:

[0061] Determine the data to be bucketed, the total number of buckets, and the amount of data in each bucket according to the bucketing instruction;

[0062] Perform paging query on the bucketed data based on the ROWID of the bucketed data;

[0063] The demarcation point data of each bucket is determined by combining the total number of buckets and the amount of data in each bucket through the preset database query statement. The demarcation point ROWNUM corresponding to the paging query result is determined;

[0064] According to the demarcation point ROWNUM, filter the paging query results to obtain the demarcation point ROWID corresponding to the demarcation point data of each bucket of data;

[0065] The start and end pairs of each bucket of data are calculated based on the demarcation point ROWID. The start and end pairs are the start and end ROWID of the bucket where the start and end pairs are located.

[0066] Generate a database query statement for each bucket of data based on the start and end pairs.

[0067] Optionally, the dynamic performance views include: SYS.DBA_PARALLEL_EXECUTE_CHUNKS view;

[0068] The database functions corresponding to creating tasks include: CREATE JOB;

[0069] Data manipulation functions include: DBMS_PARALLEL_EXECUTE;

[0070] The views corresponding to the data storage allocation information in the database include: SYS.DBA_EXTENTS view;

[0071] The views corresponding to the object information in the database include: SYS.ALL_OBJECTS view;

[0072] The default database query statement is:

[0073] SELECT(LEVEL-1)*{$rowsPerChunk}+1FROM SYS.DUAL CONNECT BY LEVEL<={splitNum};

[0074] {$rowsPerChunk} is the amount of data in each bucket, and {splitNum} is the total number of buckets.

[0075] Optionally, after the step of selecting and executing a corresponding data bucketing scheme according to the permission list, the following steps are further included:

[0076] Create a set number of database connections and use database query statements to extract data.

[0077] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned database data bucketing methods are implemented.

[0078] According to another aspect of the present invention, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned database data bucketing methods are implemented.

[0079] According to another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a machine executable program stored in the memory and running on the processor, and the processor implements the steps of any of the above-mentioned database data bucketing methods when executing the machine executable program.

[0080] The data bucketing method of the database of the present invention first obtains a bucketing instruction for the database, and the bucketing instruction is used to bucket the database; determines the permission list of the initiator of the bucketing instruction, and the permission list is used to indicate the permissions the initiator has in the database; selects and executes the corresponding data bucketing scheme according to the permission list to obtain a database query statement for each bucket of data. Through this method, an appropriate bucketing scheme can be selected according to the configuration and the permissions already possessed, and finally a bucketing method based on ROWID filtering conditions is used, which improves the efficiency of overall bucketing data extraction, reduces the performance consumption of the database, improves the performance of data reading, and reduces the impact on production.

[0081] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0083] Figure 1 is a flowchart of a method for bucketing data in a database according to an embodiment of the present invention;

[0084] Figure 2 is a flowchart of a method for bucketing data in a database according to another embodiment of the present invention;

[0085] Figure 3 1 is a schematic diagram of a first bucketing solution flow chart of a method for bucketing data in a database according to an embodiment of the present invention;

[0086] Figure 4 2 is a flowchart of a second bucketing solution of a method for bucketing data in a database according to an embodiment of the present invention;

[0087] Figure 5 is a schematic diagram of a second bucketing solution flow chart of a method for bucketing data in a database according to another embodiment of the present invention;

[0088] Figure 6 1 is a schematic diagram of a third bucketing solution flow chart of a method for bucketing data in a database according to an embodiment of the present invention;

[0089] Figure 7 is a schematic diagram of a computer program product according to an embodiment of the present invention;

[0090] Figure 8 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; and

[0091] Fig. 9 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] It should be understood by those skilled in the art that the embodiments described below are only some embodiments of the present invention, rather than all embodiments of the present invention, and these embodiments are intended to explain the technical principles of the present invention, rather than to limit the protection scope of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present invention.

[0093] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment.

[0094] In order to overcome the defects and shortcomings of the prior art, the present invention provides a method to optimize the SQL statement for obtaining bucket query data and the performance of bucket query data, thereby improving the efficiency of data extraction. The present invention mainly uses ROWID as the condition field for bucket judgment, so that the method of the present invention has the following advantages:

[0095] 1. Faster sorting performance. ROWID sorting is performed according to the physical storage order of data, which is usually faster than sorting in logical order (such as primary key sorting) because ROWID sorting only needs to compare fixed-length binary values;

[0096] 2. Higher query efficiency. When using ROWID for sequential query, random reads during the query process will be reduced, improving the read performance.

[0097] 3. Reduced resource consumption. Data filtering uses ROWID direct positioning and efficient sorting features, which occupies less CPU and memory resources than using primary keys or indexes.

[0098] Therefore, the present invention provides a method for bucketing data in a database. Figure 1 FIG. 1 is a flow chart of a method for bucketing database data according to an embodiment of the present invention. Figure 1 As shown, the data bucketing method of the database includes at least the following steps S101 to S103.

[0099] Step S101, obtain the bucketing instruction for the database. When performing heterogeneous data migration in the database, it is usually necessary to read all the table data in the original database. Since the amount of data may be large, it is generally chosen to migrate the data in buckets for large tables. At this time, it is necessary to send bucketing instructions to the database. It should be noted that the bucketing instruction can be a manually input control instruction or a bucketing instruction generated by the database in response to a specific operation during operation. And the bucketing instruction generally needs to be configured with corresponding configuration parameters before execution, such as the preset number of buckets or the number of data rows per bucket, etc. Those skilled in the art can set the configuration parameters required for the bucketing instruction according to the actual situation.

[0100] Step S102, determine the permission list of the initiator of the bucketing instruction. In some optional embodiments, the database sets different permissions for different users, so when receiving the bucketing instruction, the database will check the permissions of the initiator (that is, the user currently logged in to the database) to determine whether certain operations can be performed.

[0101] Step S103, select the corresponding data bucketing scheme according to the permission list and execute it to obtain the database query statement for each bucket of data. Since different initiators have different permissions, and different data bucketing schemes require different permissions, it is necessary to select different data bucketing schemes according to different permission lists, so as to flexibly adjust the scheme according to the permissions of the initiator.

[0102] In some optional embodiments, after selecting the corresponding data bucketing scheme according to the permission list and executing the steps, it can generally also include: creating a set number of database connections and using database query statements to extract data. When the data bucketing scheme is executed, a database query statement for each bucket of data will generally be obtained. At this time, each database query statement can directly query the bucket data in the database to achieve data extraction. Therefore, you can choose to create a set number of database connections and use database query statements to extract data, that is, concurrent operations, so as to quickly achieve data extraction. The extracted data can generally be handed over to the data conversion and data loading functions to import the data into the target database, thereby realizing the data bucketing migration of the database.

[0103] Through this method, you can select the appropriate bucketing scheme based on the configuration and existing permissions, which improves the efficiency of overall bucketing data extraction, reduces database performance consumption, improves data reading performance, and reduces the impact on production.

[0104] In some optional embodiments, the steps of selecting and executing a corresponding data bucketing scheme according to a permission list may generally include: when the permission list meets a first preset condition, executing a first bucketing scheme, the first preset condition including that the initiator has read permission for dynamic performance views, permission to create tasks, and permission to execute data operation functions, and that the current data bucketing does not use a system change number; when the permission list meets a second preset condition, executing a second bucketing scheme, the second preset condition including that the initiator has read permission for data storage allocation information in the database and read permission for object information in the database; when both the first bucketing scheme and the second bucketing scheme cannot be executed or are not adopted, executing a third bucketing scheme.

[0105] In order to clearly explain the above preset conditions, the current database is used as an example. The first preset condition includes: the first bucketing scheme cannot use the system change number, and the initiator has the read permission of the dynamic performance view, the creation permission of the task, and the execution permission of the data operation function. Figure 1 Generally, it may include: SYS.DBA_PARALLEL_EXECUTE_CHUNKS view; the database function corresponding to the creation task may generally include: CREATE JOB; the data operation function may generally include: DBMS_PARALLEL_EXECUTE function. Those skilled in the art may determine the specific names or function names corresponding to different permissions based on the design of the actual database. The system change number is generally a core concept used in the database for transaction management, recovery mechanism and consistent reading. It is an increasing counter used to track transactions and changes in the database. An optional example of the system change number in the database is SCN (System Change Number). When the database enables SCN, the database will implement dynamic data storage, that is, record different versions of data through the system change number. Since the reading function, task creation function and data operation function of the dynamic performance view are usually used to process the data of the static database, the first bucketing scheme of the present invention cannot use the system change number. If the above-mentioned first preset condition is met, the first bucketing scheme can be selected for execution.

[0106] Optionally, the steps for executing the first bucketing scheme may generally include: generating a task name according to the bucketing instruction; deleting an existing invalid task through a delete function in a data operation function; creating a corresponding target task according to the task name through a create function in a data operation function; controlling the target task to execute the ROWID-based bucketing task through an execute function in a data operation function; obtaining the execution result of the bucketing task in a system table corresponding to the dynamic performance view according to the task name, the execution result including the start and end pairs of the data in each bucket, the start and end pairs being the starting ROWID and ending ROWID of the bucket where the start and end pairs are located; deleting the target task through a delete function in a data operation function; and generating a database query statement for each bucket of data according to the execution result.

[0107] Optionally, the preset parameters in the bucketing task include: task name, schema name of the data table to be bucketed, name of the data table to be bucketed, whether to bucket by data row, and number of rows per bucket. The steps of executing the bucketing task based on ROWID include: filling the preset parameters according to the bucketing instruction; obtaining the data to be bucketed according to the schema name of the data table to be bucketed and the name of the data table to be bucketed; sorting the data to be bucketed according to its own ROWID; using an aggregate function to determine the total number of rows of the data to be bucketed according to the sorting result; when the preset parameters show that bucketing is performed by data row, the sorting result is divided according to the total number of rows and the number of rows per bucket in the preset parameters to obtain the actual number of buckets and the start and end pairs of data in each bucket. An optional example of a schema name in a database is Schema. Generally, Schema is a namespace for organizing database objects (such as tables, views, stored procedures, etc.). The basic purpose is to manage and organize objects in the database. Schema provides a namespace so that objects with the same name can coexist in different Schemas. When objects with the same name in the database exist in different schemas, it is necessary to determine the schema name where the data to be bucketed is located, so as to find the corresponding data in the corresponding namespace according to the name of the data table to be bucketed as the data to be bucketed.

[0108] Since the first bucketing scheme mentioned above buckets the data according to the preset number of rows per bucket, the number of start and end pairs obtained thereby will vary according to the different data volumes of the data to be bucketed, that is, the actual number of buckets obtained in the end is uncertain, and the bucketing instruction generally sets a preset number of buckets according to the actual situation. In order to make the execution result meet the requirements of the bucketing instruction, optionally, the step of generating a database query statement for querying data according to the execution result may also generally include: determining the preset number of buckets according to the bucketing instruction; determining the actual number of buckets according to the execution result; judging whether the actual number of buckets exceeds the preset number of buckets; if so, merging the execution result so that the actual number of buckets meets the requirements of the bucketing instruction; generating a database query statement according to the start and end pairs after the merging is completed. The merging operation may generally be to splice the excess bucket data into the preset bucket, or to evenly divide the data into each bucket without data block limitation. Those skilled in the art may select the specific merging operation according to the actual situation.

[0109] In some other optional embodiments, when the permission list meets the second preset condition, the second bucketing scheme is executed, and the second preset condition generally includes that the initiator has the read permission of the data storage allocation information in the database and the read permission of the object information in the database. Figure 1 Generally, it can include: SYS.DBA_EXTENTS view; the view corresponding to the object information in the database Figure 1 Generally, it can include: SYS.ALL_OBJECTS view. SYS.DBA_EXTENTS view provides allocation information of all tablespaces in the database, including the space occupied by tables, indexes, clusters and other objects. SYS.ALL_OBJECTS view provides information of all objects in the database, including tables, views, stored procedures, etc. In addition, the second bucketing scheme is not limited by the system change number, that is, the second bucketing scheme can be executed when the database uses the system change number.

[0110] Optionally, the steps of executing the second bucketing scheme may generally include: obtaining the object identifier corresponding to the table to be bucketed and queried according to the bucketing instruction in the system table where the object information is located in the database; if the object identifier exists, filtering the table name and mode name of the table to be bucketed and queried according to the object identifier in the system table where the data storage allocation information is located, and determining the total number of data blocks in the data block where the data to be bucketed and queried is located according to the table name and mode name; determining the preset number of buckets according to the bucketing instruction, and calculating the number of data blocks corresponding to each bucket according to the total number of data blocks and the preset number of buckets; obtaining the file number and block number of the data to be bucketed and queried according to the table name and mode name in the system table where the object information is located; arranging the data to be bucketed and queried according to the file number and block number, and generating ROWID one by one in sequence; calculating and generating the start and end pairs corresponding to each bucket according to the number of data blocks corresponding to each bucket, the start and end pairs being the starting ROWID and the ending ROWID of the bucket where the start and end pairs are located; generating a database query statement for each bucket of data according to the start and end pairs.

[0111] Optionally, due to different data volumes of data blocks, the data volumes of buckets obtained by the second bucketing scheme may differ greatly. Therefore, in order to avoid such a situation, another second bucketing scheme may be selected. The steps of executing the second bucketing scheme may generally further include: obtaining the object identifier corresponding to the to-be-bucketed query table according to the bucketing instruction in the system table where the object information is located in the database; if the object identifier exists, filtering the table name and mode name of the to-be-bucketed query table according to the object identifier in the system table where the data storage allocation information is located, and determining the total number of data blocks of the data block where the to-be-bucketed query data is located and the data volume of each data block according to the table name and mode name; determining the preset number of data rows corresponding to each bucket according to the bucketing instruction; dividing the data blocks according to the preset number of data rows corresponding to each bucket and the data volume of each data block, so as to determine the corresponding relationship between each bucket and the data block; obtaining the file number and block number of the to-be-bucketed query data according to the table name and mode name in the system table where the object information is located; arranging the to-be-bucketed query data according to the file number and block number, and generating ROWID one by one in sequence; calculating and generating the start and end pairs corresponding to each bucket according to the corresponding relationship between each bucket and the data block, the start and end pairs being the starting ROWID and the ending ROWID of the bucket where the start and end pairs are located; generating a database query statement for each bucket of data according to the start and end pairs. The data block division based on the preset number of data rows corresponding to each bucket and the data volume of each data block generally refers to establishing a corresponding relationship between the data block and the corresponding bucket when the cumulative number of rows of continuous data blocks reaches the number of rows in each bucket, rather than splitting by specifying the number of blocks. For example, when the number of rows per bucket is required to be 100, the first data block has 90 rows of data and the second data block has 20 rows of data. Since the data blocks themselves are generally not split, the split is selected between the second data block and the third data block, so that the first data block and the second data block are established together with the bucket. In the end, there are 110 rows of data in the bucket, and the third data block will participate in the processing process of the next bucket.

[0112] In addition, since the second bucketing scheme can use the system change number (SCN) compared to the first bucketing scheme, this means that the second bucketing scheme can be dynamically processed based on the database, that is, data bucketing is performed during database iteration. Therefore, data changes may occur during the execution process. For example, due to certain deletion operations, the amount of data when obtaining the file number and block number of the data to be bucketed and queried may be less than the data when determining the total amount of data blocks.

[0113] Therefore, in order to avoid such situations, after the step of determining the total number of data blocks of the data block where the data to be bucketed is located and the data amount of each data block according to the table name and the mode name, it can generally also include: determining the query table to be bucketed according to the table name of the query table to be bucketed, and querying the minimum ROWID and the maximum ROWID in the query table to be bucketed, and the file number and the block number are recorded in the ROWID. An optional example of the composition of ROWID is as follows: data object number (occupying 1-6 bits in ROWID), file number (occupying 6-9 bits in ROWID), block number (occupying 9-13 bits in ROWID), and row number (occupying 13-16 bits in ROWID). Those skilled in the art can determine the specific composition of ROWID according to actual conditions. After the step of obtaining the file number and block number of the data to be bucketed and queried according to the table name and the schema name in the system table where the object information is located, it can generally also include: judging whether the file number and block number of the obtained data to be bucketed and queried include the minimum file number and the minimum block number corresponding to the minimum ROWID and the maximum file number and the maximum block number corresponding to the maximum ROWID; if not, data compensation is performed based on the minimum ROWID and the maximum ROWID, so as to obtain the file number and block number of all the data to be bucketed and queried between the minimum ROWID and the maximum ROWID. Through the compensation mechanism of this method, when the database uses the system change number (SCN), it can also ensure that the database buckets the data according to the data at the specified time. Those skilled in the art can decide whether to add this operation to the second bucketing scheme according to the actual situation of the database.

[0114] In some other optional embodiments, when both the first bucketing scheme and the second bucketing scheme cannot be executed or are not adopted, the third bucketing scheme is executed. The steps of executing the third bucketing scheme may generally include: determining the data to be bucketed, the total number of buckets and the amount of data per bucket according to the bucketing instruction; performing paging query on the data to be bucketed according to the ROWID of the data to be bucketed; determining the demarcation point ROWNUM corresponding to the demarcation point data of each bucket data in the paging query result by combining the total number of buckets and the amount of data per bucket through a preset database query statement; filtering the demarcation point ROWID corresponding to the demarcation point data of each bucket data in the paging query result according to the demarcation point ROWNUM; calculating and generating the start and end pairs of each bucket data according to the demarcation point ROWID, the start and end pairs are the starting ROWID and the ending ROWID of the bucket where the start and end pairs are located; generating a database query statement for each bucket data according to the start and end pairs. Among them, when performing paging query in the database, the sorting function is usually combined to ensure the order consistency and predictability of the data, so the sorting operation can be synchronously implemented through paging query. ROWNUM is a pseudo column used to limit the number of rows in the query result. It is assigned when the query result set is generated, and is an integer that increases from 1. The preset database query statement can be:

[0115] SELECT(LEVEL-1)*{$rowsPerChunk}+1FROM SYS.DUAL CONNECT BY LEVEL<={splitNum}; Formula 1

[0116] {$rowsPerChunk} is the amount of data in each bucket, and {splitNum} is the total number of buckets.

[0117] The SQL query statement shown in Formula 1 is generally used to generate a series of starting row numbers, which are used to divide the data into multiple buckets. Each bucket contains a fixed number of rows ($rowsPerChunk), and the total number of buckets is {splitNum}. The specific meaning is as follows:

[0118] 1.LEVEL pseudo-column: LEVEL is a pseudo-column in the database, used to represent the hierarchical level of a recursive query. In the CONNECT BY clause, LEVEL will increase from 1.

[0119] 2. CONNECT BY LEVEL <= {splitNum}: This clause specifies the number of recursive levels, that is, the total number of generated buckets is {splitNum}.

[0120] 3. (LEVEL-1)*{$rowsPerChunk}+1: Calculate the starting row number of each bucket. LEVEL-1 starts at 0, multiply it by the number of rows per bucket {$rowsPerChunk}, and add 1 to get the starting row number of each bucket.

[0121] Through this preset database query statement, you can get the ROWNUM corresponding to the demarcation point data of each bucket of data in the paging query results, and then filter the paging query results according to ROWNUM to get the ROWID of the demarcation point data of each bucket of data. Then, the start and end pairs of each bucket of data can be calculated by combining the ROWID of the demarcation point data with the amount of data in each bucket.

[0122] Figure 2 FIG. 1 is a flow chart of a method for bucketing database data according to another embodiment of the present invention. Figure 2 As shown, the data bucketing method of the database includes at least the following steps S201 to S207.

[0123] Step S201, read the bucket configuration and determine the permission list.

[0124] Step S202: determine whether the permission list meets a first preset condition.

[0125] Step S203: When the answer in step S202 is yes, execute the first bucketing scheme.

[0126] Step S204: if the judgment in step S202 is no, whether the permission list meets the second preset condition.

[0127] Step S205: When the result of step S204 is yes, execute the second bucketing scheme.

[0128] Step S206: If the result of the determination in step S204 is no, execute the third bucketing scheme.

[0129] Step S207: Create a set number of database connections and use database query statements to extract data.

[0130] Through this method, you can choose the appropriate bucketing scheme according to the configuration and existing permissions, and finally use the bucketing method based on the ROWID filter condition, which improves the efficiency of overall bucketing data extraction, reduces performance consumption, improves data reading performance, and reduces the impact on production.

[0131] Figure 3 1 is a schematic diagram of a first bucketing solution flow chart of a method for bucketing data in a database according to an embodiment of the present invention; Figure 3 As shown, the data bucketing method of the database includes at least the following steps S301 to S307.

[0132] It should be noted that the first bucketing solution has the following restrictions: the system change number cannot be used, and the read permission of the dynamic performance view, the permission to create tasks, and the execution permission of data operation functions are required. Figure 1 Generally, it may include: SYS.DBA_PARALLEL_EXECUTE_CHUNKS view; the database function corresponding to the creation task may generally include: CREATE JOB; the data operation function may generally include: DBMS_PARALLEL_EXECUTE function.

[0133] Step S301, generating a task name according to the bucketing instruction. In some optional embodiments, a unique task name may be generated according to the table and rule (table name and fixed prefix or other parameters) of the bucketing query data as needed.

[0134] Step S302, deleting the existing invalid task through the delete function in the data operation function. In some optional embodiments, the operation of deleting the existing invalid task can be performed through DBMS_PARALLEL_EXECUTE.DROP_TASK in the DBMS_PARALLEL_EXECUTE function to delete the invalid task.

[0135] Step S303, create a corresponding target task according to the task name through the create function in the data operation function. In some optional embodiments, the operation of step S303 can create a new target task through DBMS_PARALLEL_EXECUTE.CREATE_TASK in the DBMS_PARALLEL_EXECUTE function combined with the task name generated in step S301.

[0136] Step S304, control the target task to execute the bucketing task based on ROWID through the execution function in the data operation function. In some optional embodiments, the operation of step S304 can be controlled by the DBMS_PARALLEL_EXECUTE.CREATE_CHUNKS_BY_ROWID function in the DBMS_PARALLEL_EXECUTE function to execute the bucketing task based on ROWID.

[0137] The preset parameters in the bucketing task generally include: task name (TASK_NAME), schema name of the data table to be bucketed (TABLE_OWNER), name of the data table to be bucketed (TABLE_NAME), whether to bucket by data row (BY_ROW), and number of rows per bucket (CHUNK_SIZE).

[0138] The execution process of the bucketing task is generally as follows: fill in the preset parameters according to the bucketing instruction; obtain the data to be bucketed according to the schema name of the data table to be bucketed and the name of the data table to be bucketed; sort the data to be bucketed according to its own ROWID; use the aggregate function to determine the total number of rows of the data to be bucketed based on the sorting result; when the preset parameters show that bucketing is performed by data rows, divide the sorting result according to the total number of rows and the number of rows in each bucket in the preset parameters to obtain the actual number of buckets and the start and end pairs of data in each bucket.

[0139] Step S305, obtain the execution result of the bucket task in the system table corresponding to the dynamic performance view according to the task name. In some optional embodiments, the operation of step S305 is to obtain the execution result of the task by querying the dynamic performance view SYS.DBA_PARALLEL_EXECUTE_CHUNKS) system table, and obtain START_ROWID and END_ROWID, that is, the start and end pair of each bucket.

[0140] Step S306, deleting the target task through the delete function in the data operation function. In some optional embodiments, the operation of step S306 can be performed through the DBMS_PARALLEL_EXECUTE.DROP_TASK function in the DBMS_PARALLEL_EXECUTE function to delete the target task, thereby releasing the storage space of the database.

[0141] Step S307, generating a database query statement for each bucket of data according to the execution result. In some optional embodiments, since the first bucketing scheme buckets the data according to the preset number of rows per bucket, the number of start and end pairs obtained thereby will vary according to the different data volumes of the data to be bucketed, that is, the actual number of buckets finally obtained is uncertain, and the bucketing instruction generally sets a preset number of buckets according to the actual situation. In order to make the execution result meet the requirements of the bucketing instruction, step S307 generally may also include: determining the preset number of buckets according to the bucketing instruction; determining the actual number of buckets according to the execution result; judging whether the actual number of buckets exceeds the preset number of buckets; if so, merging the execution result so that the actual number of buckets meets the requirements of the bucketing instruction; generating a database query statement according to the start and end pairs after the merging is completed. The merging operation may generally be to splice the excess bucket data into the preset bucket, or to evenly divide the data into each bucket without data block limitation. Those skilled in the art may select a specific merging operation according to the actual situation.

[0142] Through this method, the problem of inaccurate number of buckets of the built-in function was solved in the first bucketing solution based on CREATE_CHUNKS_BY_ROWID. Finally, the bucketing method based on ROWID filtering conditions was used to improve the efficiency of overall bucketing data extraction, reduce performance consumption, improve data reading performance, and reduce the impact on production.

[0143] Figure 4 1 is a flowchart of a second bucketing solution of a method for bucketing data in a database according to an embodiment of the present invention; Figure 4 As shown, the data bucketing method of the database includes at least the following steps S401 to S408.

[0144] It should be noted that the second bucketing solution has the following restrictions: you must have the permission to read the data storage allocation information in the database and the permission to read the object information in the database. In addition, the second bucketing solution can also use the system change number (SCN). Figure 1 Generally, it can include: SYS.DBA_EXTENTS view; the view corresponding to the object information in the database Figure 1Generally, it can include: SYS.ALL_OBJECTS view. SYS.DBA_EXTENTS view provides allocation information of all tablespaces in the database, including the space occupied by objects such as tables, indexes, and clusters. An optional example of a system change number in a database is SCN (System Change Number). When the database enables SCN, the database will implement dynamic data storage, that is, record different versions of data through system change numbers.

[0145] Step S401, obtain the object identifier corresponding to the query table to be bucketed according to the bucketing instruction in the system table where the object information is located in the database. In some optional embodiments, since the SYS.ALL_OBJECTS view provides information about all objects in the database, including tables, views, stored procedures, etc., the operation of step S401 can obtain the object identifier (objectId) corresponding to the query table to be bucketed through the system table SYS.ALL_OBJECTS.

[0146] Step S402: if there is an object identifier, filter the table name and the schema name of the to-be-bucketed query table in the system table where the data storage allocation information is located according to the object identifier.

[0147] Step S403, determine the total number of data blocks of the data block where the data to be queried by bucketing is located according to the table name and the schema name. In some optional embodiments, since the SYS.DBA_EXTENTS view provides the allocation information of all table spaces in the database, including the space occupied by objects such as tables, indexes, and clusters, the operation of step S402 can filter the table name and its corresponding schema name that need to be queried by bucketing through the system table SYS.DBA_EXTENTS. Then, step S403 is performed to query the total number of data blocks of the data block where the data to be queried by bucketing is located according to the table name and its corresponding schema name.

[0148] Step S404, determining a preset number of buckets according to the bucketing instruction, and calculating the number of data blocks corresponding to each bucket according to the total number of data blocks and the preset number of buckets.

[0149] Step S405, obtain the file number and block number of the data to be queried in buckets according to the table name and mode name in the system table where the object information is located. In some optional embodiments, all file numbers and block numbers can be obtained by querying the system table SYS.DBA_EXTENTS to filter the table name and mode name that need to be queried in buckets. An optional example of the composition of ROWID is as follows: data object number (occupying 1-6 bits in ROWID), file number (occupying 6-9 bits in ROWID), block number (occupying 9-13 bits in ROWID), row number (occupying 13-16 bits in ROWID). Those skilled in the art can determine the specific composition of ROWID according to actual conditions. The file number and block number of the data to be queried in buckets can be obtained through the above ROWID.

[0150] Step S406, arranging the data to be queried in buckets according to the file number and the block number, and generating ROWIDs one by one in sequence. An optional example of the arrangement method is arranging in ascending order according to the file number and the block number, and generating ROWIDs one by one in sequence.

[0151] Step S407, calculating and generating a start and end pair corresponding to each bucket according to the number of data blocks corresponding to each bucket.

[0152] Step S408, generating a database query statement for each bucket of data according to the start and end pairs.

[0153] Through this method, you can choose the appropriate bucketing scheme according to the configuration and existing permissions, and finally use the bucketing method based on the ROWID filter condition, which is the second bucketing scheme. This improves the efficiency of overall bucketing data extraction, reduces performance consumption, improves data reading performance, and reduces the impact on production.

[0154] Optionally, due to different data volumes of data blocks, the data volumes of buckets obtained by the second bucketing scheme may differ greatly. Therefore, in order to avoid such a situation, another second bucketing scheme may be selected. Figure 5 It is a flowchart of a second bucketing scheme of a method for bucketing data in a database according to another embodiment of the present invention. The method for bucketing data in a database at least includes the following steps S501 to S509.

[0155] Figure 5 Bucketing scheme and Figure 4 The restrictions of the bucketing scheme shown are the same and will not be repeated here.

[0156] Step S501, obtaining the object identifier corresponding to the to-be-bucketed query table in the system table where the object information is located in the database according to the bucketing instruction.

[0157] Step S502: if there is an object identifier, filter the table name and the schema name of the table to be bucketed and queried in the system table where the data storage allocation information is located according to the object identifier.

[0158] Step S503, determine the total number of data blocks and the data volume of each data block of the data block where the data to be queried is located according to the table name and the schema name. In step S503, to ensure that the data can be evenly divided into each bucket, it is necessary to query the data volume of each data block when determining the total number of data blocks.

[0159] Step S504, determining the preset number of data rows corresponding to each bucket according to the bucketing instruction.

[0160] Step S505, divide the data blocks according to the preset number of data rows corresponding to each bucket and the data volume of each data block, so as to determine the corresponding relationship between each bucket and the data block. Among them, step S505 generally refers to establishing a corresponding relationship between the data block and the corresponding bucket when the cumulative number of rows of continuous data blocks reaches the number of rows in each bucket, rather than dividing by a specified number of blocks. For example, when the number of rows per bucket is required to be 100, the first data block has 90 rows of data and the second data block has 20 rows of data. Since the data blocks themselves are generally not divided, the second data block and the third data block are selected to be divided, so that the first data block and the second data block are established together with the bucket. Correspondence relationship, finally, there are 110 rows of data in the bucket, and the third data block will participate in the processing process of the next bucket.

[0161] Step S506, obtaining the file number and block number of the data to be bucketed and queried according to the table name and the mode name in the system table where the object information is located.

[0162] Step S507, arranging the data to be queried in buckets according to the file numbers and block numbers, and generating ROWIDs one by one in sequence.

[0163] Step S508, calculating and generating a start and end pair corresponding to each bucket according to the corresponding relationship between each bucket and the data block.

[0164] Step S509: Generate a database query statement for each bucket of data according to the start and end pairs.

[0165] This method solved Figure 4 The problem of unbalanced bucketing in the data block-based bucketing solution shown in the figure makes the amount of data in each bucket more balanced, and uses a bucketing method based on ROWID filtering conditions to improve the efficiency of overall bucket data extraction, reduce performance consumption, improve data reading performance, and reduce the impact on production.

[0166] Figure 6 is a schematic flow chart of a third bucketing solution of a method for bucketing data in a database according to an embodiment of the present invention; Figure 6 As shown, the data bucketing method of the database includes at least the following steps S601 to S605.

[0167] Step S601, determine the data to be bucketed, the total number of buckets, and the amount of data per bucket according to the bucketing instruction. The method of determining the data to be bucketed according to the bucketing instruction is generally to determine the name of the data table to be bucketed according to the bucketing instruction, and then obtain the data in the table as the data to be bucketed. The total number of buckets, the amount of data per bucket, and the name of the data table to be bucketed are generally recorded as configuration parameters of the bucketing instruction, etc., so as to facilitate database reading.

[0168] Step S602 , performing paging query on the data to be bucketed according to the ROWID of the data to be bucketed.

[0169] Step S603, determine the demarcation point ROWNUM corresponding to the demarcation point data of each bucket of data in the paging query result by combining the total number of buckets and the amount of data in each bucket through a preset database query statement. In some optional embodiments, the preset database query statement can be shown as Formula 1. The ROWNUM corresponding to the demarcation point data of each bucket of data in the paging query result can be obtained through this preset database query statement, and then the ROWID of the demarcation point data of each bucket of data is filtered in the paging query result according to ROWNUM. Then, the start and end pairs of each bucket of data can be calculated by combining the ROWID of the demarcation point data with the amount of data in each bucket.

[0170] Step S604, filter the paging query results according to the demarcation point ROWNUM to obtain the demarcation point ROWID corresponding to the demarcation point data of each bucket of data. This step does not require obtaining the ROWID of all data, but only requires obtaining the demarcation point ROWID corresponding to the demarcation point data of each bucket of data, thereby greatly reducing database consumption.

[0171] Step S604, calculating and generating a start and end pair of each bucket of data according to the demarcation point ROWID.

[0172] Step S605, generating a database query statement for each bucket of data according to the start and end pairs.

[0173] Through this method, when the database authority is insufficient to execute the first bucketing scheme or the second bucketing scheme, or when the first bucketing scheme and the second bucketing scheme are not adopted, the problem of poor performance of the paging method can be solved by the third bucketing scheme based on ROWID sorting paging. In addition, the method of obtaining only the ROWID of the bucketing threshold reduces the consumption of the database.

[0174] The flow chart provided by the present embodiment is not intended to indicate that the operation of the method will be performed in any particular order, or that all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical thinking provided by the present embodiment method, additional changes may be made to the above method.

[0175] It should be understood that in some embodiments, each part can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0176] This embodiment also provides a computer program product 10 , a computer readable storage medium 20 , and a computer device 30 . Figure 7 is a schematic diagram of a computer program product 10 according to an embodiment of the present invention, Figure 8 is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention, Fig. 9 is a schematic diagram of a computer device 30 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which implements the steps of any of the above-mentioned database data bucketing methods when executed by a processor 32. The computer-readable storage medium 20 stores the above-mentioned computer program 11, which implements the steps of any of the above-mentioned database data bucketing methods when executed by a processor 32. The computer device 30 may include a memory 31, a processor 32, and a computer program 11 stored in the memory 31 and running on the processor 32.

[0177] The computer program 11 for performing the operation of the present invention may be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, a configuration data of an integrated circuit, or a source code or an object code written in any combination of one or more programming languages ​​and process programming languages. The computer program 11 may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider via the Internet). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0178] For the purposes of the description of the present embodiment, the computer program product 10 is a related product that includes the computer program 11. For the purposes of the description of the present embodiment, the computer readable storage medium 20 is a tangible device capable of retaining and storing the computer program 11, which may be any device that can contain, store, communicate, propagate or transmit the program 11 for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of the computer readable storage medium 20 include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the foregoing.

[0179] The computer device 30 may be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smart phone. In some examples, the computer device 30 may be a cloud computing node. The computer device 30 may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, target programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 30 may be implemented in a distributed cloud computing environment where remote processing devices linked via a communication network perform tasks. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including a storage device.

[0180] The computer device 30 may include a processor 32 adapted to execute stored instructions, and a memory 31 providing temporary storage space for the operation of the instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 31 may include a random access memory (RAM), a read-only memory, a flash memory, or any other suitable storage system.

[0181] The computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows data to be input and output with external devices that may be connected to the computer device. The network adapter / interface may provide communication between the computer device and a network, which is typically shown as a communication network.

[0182] At this point, those skilled in the art should recognize that, although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived based on the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized as covering all these other variations or modifications.

Claims

1. A method for bucketing data in a database, comprising: Obtaining a bucketing instruction for the database, wherein the bucketing instruction is used to bucket data in the database; Determine a permission list of the initiator of the bucketing instruction, where the permission list is used to indicate the permissions that the initiator has in the database; A corresponding data bucketing scheme is selected and executed according to the permission list to obtain a database query statement for each bucket of data.

2. The method for bucketing a database according to claim 1, wherein: The step of selecting and executing a corresponding data bucketing scheme according to the permission list includes: When the permission list meets the first preset condition, the first bucketing scheme is executed, wherein the first preset condition includes that the initiator has the read permission of the dynamic performance view, the create task permission, and the execute permission of the data operation function, and the system change number is not used in this data bucketing; When the permission list meets the second preset condition, executing the second bucketing scheme, wherein the second preset condition includes that the initiator has the permission to read the data storage allocation information in the database and the permission to read the object information in the database; When both the first bucketing scheme and the second bucketing scheme cannot be executed or are not adopted, the third bucketing scheme is executed.

3. The method for bucketing a database according to claim 2, wherein: The step of executing the first bucketing scheme includes: Generate a task name according to the bucketing instruction; Deleting the existing invalid tasks through the deletion function in the data operation function; Create a corresponding target task according to the task name through the creation function in the data operation function; Controlling the target task to execute the bucketing task based on ROWID through the execution function in the data operation function; Acquire the execution result of the bucketing task in the system table corresponding to the dynamic performance view according to the task name, wherein the execution result includes a start and end pair of data in each bucket, and the start and end pairs are the start ROWID and the end ROWID of the bucket where the start and end pairs are located; Deleting the target task through a deletion function in the data operation function; A database query statement for each bucket of data is generated according to the execution result.

4. The method for bucketing a database according to claim 3, wherein: The preset parameters in the bucketing task include: task name, mode name of the data table to be bucketed, name of the data table to be bucketed, whether to bucket by data row, and number of rows per bucket; The steps of performing the bucketing task based on ROWID include: Fill the preset parameters according to the bucketing instruction; Acquire the data to be bucketed according to the schema name of the data table to be bucketed and the name of the data table to be bucketed; Sort the data to be bucketed according to its own ROWID; Use an aggregate function to determine the total number of rows of the data to be bucketed according to the sorting result; When the preset parameters indicate bucketing by data row, the sorting result is divided according to the total number of rows and the number of rows in each bucket in the preset parameters to obtain the actual number of buckets and the start and end pairs of data in each bucket.

5. The method for bucketing a database according to claim 4, wherein: The step of generating a database query statement for query data according to the execution result comprises: Determine the preset number of buckets according to the bucketing instruction; Determine the actual number of buckets according to the execution result; Determine whether the actual number of buckets exceeds the preset number of buckets; If yes, then merge the execution results so that the actual number of buckets meets the requirements of the bucket instruction; The database query statement is generated according to the start and end pairs after the merging is completed.

6. The method for bucketing a database according to claim 2, wherein: The step of executing the second bucketing scheme includes: Obtaining the object identifier corresponding to the to-be-bucketed query table according to the bucketing instruction in the system table where the object information is located in the database; In the case where the object identifier exists, filtering the table name and the schema name of the table to be bucketed and queried according to the object identifier in the system table where the data storage allocation information is located, and determining the total number of data blocks of the data block where the data to be bucketed and queried is located according to the table name and the schema name; Determine a preset number of buckets according to the bucketing instruction, and calculate the number of data blocks corresponding to each bucket according to the total number of data blocks and the preset number of buckets; Obtaining the file number and block number of the data to be queried by bucketing according to the table name and the mode name in the system table where the object information is located; Arrange the to-be-bucketed query data according to the file number and the block number, and generate ROWIDs one by one in sequence; A start and end pair corresponding to each bucket is calculated and generated according to the number of data blocks corresponding to each bucket, where the start and end pairs are the start ROWID and the end ROWID of the bucket where the start and end pairs are located; A database query statement for each bucket of data is generated according to the start and end pairs.

7. The method for bucketing a database according to claim 6, wherein: The step of executing the second bucketing scheme also includes: Obtaining the object identifier corresponding to the to-be-bucketed query table according to the bucketing instruction in the system table where the object information is located in the database; In the case where the object identifier exists, filtering the table name and the schema name of the table to be bucketed and queried according to the object identifier in the system table where the data storage allocation information is located, and determining the total number of data blocks of the data block where the data to be bucketed and queried is located and the data amount of each data block according to the table name and the schema name; Determine the preset number of data rows corresponding to each bucket according to the bucketing instruction; Dividing the data blocks according to the preset number of data rows corresponding to each bucket and the data volume of each data block, thereby determining the corresponding relationship between each bucket and the data block; Obtaining the file number and block number of the data to be queried by bucketing according to the table name and the mode name in the system table where the object information is located; Arrange the data to be bucketed and queried according to the file number and the block number, and generate ROWIDs one by one in sequence; A start-end pair corresponding to each bucket is calculated and generated according to the corresponding relationship between each bucket and the data block, where the start-end pair is a start ROWID and an end ROWID of the bucket where the start-end pair is located; A database query statement for each bucket of data is generated according to the start and end pairs.

8. The method for bucketing a database according to claim 7, wherein: After the step of determining the total number of data blocks of the data block where the to-be-bucketed query data is located and the data amount of each of the data blocks according to the table name and the mode name, the method further includes: determining the to-be-bucketed query table according to the table name of the to-be-bucketed query table, and querying the minimum ROWID and the maximum ROWID in the to-be-bucketed query table, wherein the ROWID records the file number and the block number; After the step of obtaining the file number and block number of the data to be bucketed and queried according to the table name and the schema name in the system table where the object information is located, the step also includes: judging whether the obtained file number and block number of the data to be bucketed and queried include the minimum file number and minimum block number corresponding to the minimum ROWID and the maximum file number and maximum block number corresponding to the maximum ROWID; if not, performing data compensation based on the minimum ROWID and the maximum ROWID, so as to obtain the file numbers and block numbers of all the data to be bucketed and queried between the minimum ROWID and the maximum ROWID.

9. The method for bucketing a database according to claim 2, wherein: The steps of executing the third bucketing scheme include: Determine the data to be bucketed, the total number of buckets, and the amount of data in each bucket according to the bucketing instruction; Perform paging query on the data to be bucketed according to the ROWID of the data to be bucketed; Determine the demarcation point ROWNUM corresponding to the demarcation point data of each bucket in the paging query result by combining the total number of buckets and the amount of data in each bucket through a preset database query statement; According to the demarcation point ROWNUM, the demarcation point ROWID corresponding to the demarcation point data of each bucket of data is obtained by filtering in the paging query result; The start and end pairs of each bucket of data are calculated based on the demarcation point ROWID, where the start and end pairs are the start ROWID and the end ROWID of the bucket where the start and end pairs are located; A database query statement for each bucket of data is generated according to the start and end pairs.

10. The method for bucketing a database according to claim 9, wherein: The dynamic performance views include: SYS.DBA_PARALLEL_EXECUTE_CHUNKS view; The database functions corresponding to the creation task include: CREATE JOB; The data operation functions include: DBMS_PARALLEL_EXECUTE; The views corresponding to the data storage allocation information in the database include: SYS.DBA_EXTENTS view; The views corresponding to the object information in the database include: SYS.ALL_OBJECTS view; The preset database query statement is: SELECT(LEVEL-1)*{$rowsPerChunk}+1FROM SYS.DUAL CONNECT BY LEVEL<={splitNum}; Where {$rowsPerChunk} is the amount of data per bucket, and {splitNum} is the total number of buckets.

11. The method for bucketing a database according to claim 1, wherein: After the step of selecting and executing the corresponding data bucketing scheme according to the permission list, the following step further includes: Create a set number of database connections and use the database query statements to extract data.

12. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the database data bucketing method described in any one of claims 1 to 11 are implemented.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the database data bucketing method described in any one of claims 1 to 11.

Images