Data table partition reorganization method and device, electronic equipment and readable storage medium

Abstract

The application provides a data table partition reorganization method and device, electronic equipment and readable storage medium, which can be applied to the technical field of databases, and comprises the following steps: in response to a data table partition reorganization command, determining a data unit of a target column for indicating a partition key in a data table based on the partition key for the target partition indicated in the data table partition reorganization command; for each data unit, performing intersection analysis on the unit data range and the partition key data range of the data unit to obtain an analysis result; in the case that the analysis result indicates that the unit data range and the partition key data range have a non-empty intersection, and the unit data range has an element that does not exist in the partition key data range, determining reorganization data belonging to the partition key data range from the plurality of data of the data unit; and performing partition reorganization on the data table based on the reorganization data of each data unit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database, in particular to a data table partition reorganization method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] A partition table is a plurality of sub-tables obtained by decomposing a data table of a database according to certain rules. By using the partition table, a large amount of cumbersome business data can be classified and stored in each partition according to business rules according to the business needs of customers. Therefore, the partition table can enhance the availability of the database, reduce the maintenance cost of the database, and effectively improve the query performance of the database.

[0003] In actual business scenarios, as the business needs change continuously, the data table partition needs to be reorganized to adapt to the changing business needs. In related technologies, the partition reorganization is usually realized by directly traversing the data in the data table and selecting each data that meets the partition condition to construct a new partition. This way of traversing each data needs to occupy a large amount of resources, resulting in low efficiency of partition reorganization. SUMMARY

[0004] In view of the above problems, the present application provides a data table partition reorganization method, device, electronic equipment and readable storage medium.

[0005] According to a first aspect of the present application, a data table partition reorganization method is provided, comprising: in response to a data table partition reorganization command, determining a data unit of a target column in a data table for representing a partition key based on the partition key for a target partition indicated in the data table partition reorganization command, wherein the data table comprises a plurality of columns, and each column comprises at least one data unit; for each data unit, performing intersection analysis on a unit data range and a partition key data range of the data unit to obtain an analysis result, wherein the unit data range is determined according to a plurality of data stored in the data unit, the partition key data range is determined from the data table partition reorganization command, and the partition key data range represents the value range of the data to be divided into the target partition on the target column; in the case that the analysis result indicates that the unit data range and the partition key data range have a non-empty intersection, and the unit data range has an element that does not exist in the partition key data range, determining reorganization data belonging to the partition key data range from the plurality of data of the data unit; and based on the reorganization data of each data unit, reorganizing the data table.

[0006] According to an embodiment of the present application, the determining the reorganization data belonging to the partition key data range from the plurality of data of the data unit comprises: determining a cut range for cutting the data unit based on an intersection between the unit data range and the partition key data range; and selecting data belonging to the cut range from the plurality of data of the data unit according to the cut range to obtain the reorganization data.

[0007] According to an embodiment of the present application, the data unit is configured with a data index, the data index is used to sort the plurality of data in the data unit according to the numerical value to obtain the upper limit value and the lower limit value of the unit data range; and the determining the cut range for cutting the data unit based on the intersection between the unit data range and the partition key data range comprises: calculating the intersection between the partition key data range and the unit data range according to the upper limit value and the lower limit value of the partition key data range and the upper limit value and the lower limit value of the unit data range to obtain the cut range.

[0008] According to an embodiment of the present application, the data table partition reorganization method further comprises: in a case where the analysis result indicates that the unit data range is a subset of the partition key data range, determining that the plurality of data in the data unit are all reorganization data; and in a case where the analysis result indicates that there is no non-empty intersection between the unit data range and the partition key data range, determining that the plurality of data in the data unit are not reorganization data.

[0009] According to an embodiment of the present application, the partition reorganization of the data table based on the reorganization data of at least one data unit respectively comprises: for each reorganization data, performing the following operations: recording the row identifier of the reorganization data in the data table into a data log file; inserting the reorganization data into a temporary table for a target partition; deleting the data corresponding to the row identifier in the data table based on the row identifier recorded in the data log file; and persisting the temporary table as the target partition.

[0010] According to an embodiment of the present application, the data table partition reorganization method further comprises: in a case where the partition reorganization is interrupted, saving the database definition language executed before the interruption into a definition log file; in a case where the partition reorganization is resumed, judging the partition reorganization progress based on the definition log file; in a case where the partition reorganization progress indicates that there is reorganization data not inserted into the temporary table, emptying the temporary table and the data log file; re-executing the partition reorganization based on the reorganization data; in a case where the partition reorganization progress indicates that there is no reorganization data not inserted into the temporary table, completing the partition reorganization based on the data log file and the temporary table.

[0011] According to an embodiment of the present application, the data unit of the target column for representing the partition key in the data table is determined based on the partition key indicated in the data table partition reorganization command, including: matching in the plurality of columns in the data table based on the partition key to obtain the target column matching the partition key; and initiating a data acquisition request to a storage medium for storing the data table based on the target column to obtain at least one data unit of the target column.

[0012] The second aspect of the present application provides a data table partition reorganization apparatus, including: a data unit determination module configured to determine, in response to a data table partition reorganization command, a data unit of a target column for representing a partition key in a target partition based on the partition key indicated in the data table partition reorganization command, wherein the data table includes a plurality of columns, and each column includes at least one data unit; an analysis module configured to, for each data unit, perform intersection analysis on a unit data range of the data unit and a partition key data range to obtain an analysis result, wherein the unit data range is determined according to a plurality of data stored in the data unit, and the partition key data range is determined from the data table partition reorganization command, and the partition key data range represents a value range of the data to be divided into the target partition on the target column; a reorganized data determination module configured to, in a case where the analysis result indicates that the unit data range and the partition key data range have a non-empty intersection, and the unit data range has an element not in the partition key data range, determine, from the plurality of data of the data unit, reorganized data belonging to the partition key data range; and a partition reorganization module configured to perform partition reorganization on the data table based on the reorganized data of each of the at least one data unit.

[0013] The third aspect of the present application provides an electronic device, including: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the steps of the above method.

[0015] The fifth aspect of the present application further provides a computer program product including a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the above method.

[0016] According to the embodiment of the present application, by responding to the data table partition reorganization command, the data unit for representing the partition key target column is determined based on the partition key indicated by the data table partition reorganization command for the target partition, the target column corresponding data unit is determined, and the partition reorganization is realized in batches in units of data units. The intersection analysis is performed by using the unit data range and the partition key data range of the data unit, and it is determined that the unit data range and the partition key data range exist non-empty intersection, and the unit data range exists element which does not exist in the partition key data range, and the reorganization data belonging to the partition key data range is accurately determined from the plurality of data of the data unit, compared with the prior art directly loop traversing the data in the data table, the time required to determine the reorganization data can be effectively saved. Based on the reorganization data of each data unit, the data table is partitioned and reorganized, and the reorganization data in each data unit is obtained, and the data table partition reorganization based on the data unit batch is realized, which greatly reduces the time consumption of the partition reorganization process, effectively improves the efficiency of the online data table partition reorganization, and further improves the performance of the database. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above content and other purposes, features and advantages of the present application will be more clearly understood through the following description of the embodiments of the present application with reference to the accompanying drawings.

[0018] Figure 1 An application scenario diagram of the data table partition reorganization method, device, electronic equipment and readable storage medium according to the embodiment of the present application is shown.

[0019] Figure 2 A flowchart of the data table partition reorganization method according to the embodiment of the present application is shown.

[0020] Figure 3 A flowchart of the data table partition reorganization method according to the embodiment of the present application is shown.

[0021] Figure 4 A data table partition reorganization method according to the embodiment of the present application is shown.

[0022] Figure 5 A flowchart of the data table partition reorganization method according to another embodiment of the present application is shown.

[0023] Figure 6 A structural block diagram of the data table partition reorganization device according to the embodiment of the present application is shown.

[0024] Figure 7 A block diagram of the electronic equipment suitable for implementing the data table partition reorganization method according to the embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present application. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have been omitted to avoid unnecessarily complicating the present application with details that will be readily apparent to those skilled in the art.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, mean the term "comprises," as long as the above-mentioned terms do not exclude other features, steps, operations, elements, and / or components.

[0027] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0028] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should be generally interpreted that the meaning of the expression is at least one of the items listed before the conjunction, unless otherwise clearly defined. For example, the expression "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C together, etc.

[0029] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0030] In the scenario of making automated decisions by using personal information, the method, device and system provided by the embodiments of the present application provide corresponding operation entrances for users to select to agree or reject the automated decision result; if the user selects to reject, the expert decision process is entered. The expression "automated decision" refers to the activity of making decisions by automatically analyzing and evaluating the behavior habits, interests and hobbies, or economic, health and credit conditions of individuals by computer programs. The expression "expert decision" refers to the activity of making decisions by personnel who are engaged in a certain field of work, have specialized experience, knowledge and skills, and have reached a certain professional level.

[0031] In the related art, partition reorganization is usually directly looped through the data in the data table, and the data meeting the partition condition is selected piece by piece to construct new partition data. Therefore, in the case that the data table needs to be partitioned and reorganized and the amount of data in the data table is large, the time cost is high and the processing efficiency is low to directly loop through each piece of data in the data table and determine whether each piece of data meets the partition condition piece by piece.

[0032] For example, the data table adopts column storage to store data, that is, each column of data in the data table is stored as a data file, and the data file can be in the format of CXXXXX.seg, where "XXXXX" is a column value. The data file corresponding to the first column of data is C00000.seg, the data file corresponding to the second column of data is C00001.seg, and so on. Such a storage mode will cause the data file to be larger when the amount of data in the data table is larger, and the time of traversing the data file in the partition reorganization process will be longer. In addition, in order to avoid the data in the data table from being changed due to user write operation and affecting the partition reorganization result, a write lock can be added to the data table during the partition reorganization process. Therefore, the write lock of the data table needs to be occupied for a long time, so that the user of the database cannot write or modify the data in the data table in time.

[0033] Figure 1 An application scenario diagram of the data table partition reorganization method, device, electronic device and readable storage medium according to the embodiments of the present application is shown.

[0034] As shown in Figure 1 According to the application scenario 100 of the embodiment, the first terminal device 101, the second terminal device 102, the third terminal device 103, the network 104 and the server 105 can be included. The network 104 is used as a medium to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc.

[0037] The server 105 can be a server providing various services, such as a background management server supporting websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as examples). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data, etc. obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the data table partition reorganization method provided by the embodiments of the present application can generally be executed by the server 105. Correspondingly, the data table partition reorganization apparatus provided by the embodiments of the present application can generally be arranged in the server 105. The data table partition reorganization method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the data table partition reorganization apparatus provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0039] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network, and the server in the above scenario is only illustrative. According to the implementation needs, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network, and the server.

[0040] The data table partition reorganization method of the embodiments of the present application will be described in detail below based on the scenario described above. Figure 1 Figures 2 to 5 The data table partition reorganization method of the embodiments of the present application will be described in detail below based on the scenario described above.

[0041] Figure 2 ​A flowchart of a data table partition reorganization method according to an embodiment of the present application is shown.

[0042] As shown in Figure 2 The data table partition reorganization method of this embodiment includes operations S210-S240.

[0043] In operation S210, in response to a data table partition reorganization command, data units of a target column in a data table for representing a partition key of a target partition are determined based on the partition key indicated in the data table partition reorganization command.

[0044] The data table includes a plurality of columns, and each column includes at least one data unit.

[0045] In this embodiment, the data table partition reorganization command can be used to instruct partition reorganization of the data table. Partition reorganization is re-division of the range or value list of an existing partition, including merging and splitting partitions. The number of partitions to be reorganized is greater than the number of newly defined partitions, which is merging, and vice versa, which is splitting. The partition key represents one or more columns specified, and the database system determines which partition a column of data should be stored in according to the value of this column. The target partition refers to the partition obtained by partition reorganization.

[0046] The data table partition reorganization can include online partition reorganization. Online partition reorganization can reorganize partitions during database system operation without interrupting service, so that users can continue to access data during the reorganization process, and partition reorganization can be performed without downtime maintenance.

[0047] For example, a data unit is the smallest logical unit for storing and managing data in a database, and each data unit stores 65536 rows of data. The data in a data column is saved in at least one data unit. The data unit can be divided into full data units, mixed data units, and empty data units according to the data stored therein. When the data unit includes 65536 rows of data, the data unit is a full data unit. When a portion of the 65536 rows of data in the data unit is deleted, the data unit is a mixed data unit. When there is no data in the data unit, the data unit is an empty data unit.

[0048] Generally, the database state file stores information about each column in the data table, including the size of the data, the number of data units, and the maximum and minimum values of each data unit and row identifier.

[0049] For example, the maximum and minimum values in the data unit and the state of the data unit are recorded in the database state file. When the state flag is 0, it is a full data unit, when the state flag is 1, it is a mixed data unit, and when the state flag is 2, it is an empty data unit.

[0050] In operation S220, for each data unit, an intersection analysis is performed on the unit data range and the partition key data range of the data unit, to obtain an analysis result.

[0051] The unit data range is determined according to the plurality of data stored in the data unit, and the partition key data range is determined from the data table partition reorganization command. The partition key data range represents the value range of the data to be divided into the target partition on the target column. The analysis result can be used to indicate whether there is an intersection between the unit data range and the partition key data range.

[0052] The unit data range can be determined according to the maximum value and the minimum value of the plurality of data in the data unit.

[0053] In operation S230, in the case where the analysis result indicates that there is a non-empty intersection between the unit data range and the partition key data range, and the unit data range has elements that do not exist in the partition key data range, the reorganization data belonging to the partition key data range is determined from the plurality of data of the data unit.

[0054] In the case where there is a non-empty intersection between the unit data range and the partition key data range, and the unit data range has elements that do not exist in the partition key data range, it is equivalent to the case where the unit data range and the partition key data range are partially overlapped. At this time, the plurality of data of the data unit contains data that may not belong to the partition key data range, and therefore the reorganization data belonging to the partition key data range needs to be determined from the data of the data unit.

[0055] In an embodiment, the analysis result is used to determine the data in the data unit that needs to be divided into the target partition.

[0056] In operation S240, the data table is partitioned and reorganized based on the respective reorganization data of the at least one data unit.

[0057] In this embodiment, by responding to the data table partition reorganization command, the data unit for representing the partition key target column is determined based on the partition key of the target partition indicated by the data table partition reorganization command, the target column corresponding data unit is determined, and the partition reorganization is realized in batches in units of data units. The intersection analysis is performed by using the unit data range and the partition key data range, and it is determined that the unit data range and the partition key data range have a non-empty intersection, and the unit data range has an element that does not exist in the partition key data range. The reorganization data belonging to the partition key data range is accurately determined from the multiple data of the data unit, which can effectively save the time required to determine the reorganization data compared with directly looping through the data in the data table. Based on the reorganization data of each data unit, the data table is partitioned and reorganized to obtain the reorganization data in each data unit, and the data table partition reorganization based on the data unit batch is realized, which greatly reduces the time consumption of the partition reorganization process, effectively improves the efficiency of the online data table partition reorganization, and further improves the performance of the database.

[0058] Figure 3 A flowchart of determining the reorganization data belonging to the partition key data range by the data table partition reorganization method according to the embodiment of the application is shown.

[0059] As shown in Figure 3 The data table partition reorganization method of this embodiment includes operations S331-S332.

[0060] In operation S331, the interception range for intercepting the data unit is determined based on the intersection between the unit data range and the partition key data range.

[0061] In operation S332, the reorganization data is obtained by selecting the data belonging to the interception range from the multiple data of the data unit according to the interception range.

[0062] In this embodiment, the interception range is determined, which represents the data belonging to the partition key data range in a data unit.

[0063] In this embodiment, the intersection between the unit data range and the partition key data range can be determined based on the comparison of the overlapping range between the unit data range and the partition key data range.

[0064] In this embodiment, the interception range of the data in the data unit can be determined by comparing the overlapping range between the unit data range and the partition key data range from the several data units respectively, which realizes the batch division of data and the partition reorganization, and greatly improves the partition reorganization efficiency.

[0065] Figure 4A schematic diagram for determining a data interception range according to the data table partition reorganization method of the embodiment of the present application is shown.

[0066] In the embodiment, the data unit is configured with a data index, which is used to sort the plurality of data in the data unit according to the data value size, to obtain the upper limit value and the lower limit value of the unit data range.

[0067] In the embodiment, the interception range for intercepting the data unit is determined based on the intersection between the unit data range and the partition key data range, including: calculating the intersection between the partition key data range and the unit data range according to the upper limit value and the lower limit value of the partition key data range and the upper limit value and the lower limit value of the unit data range, to obtain the interception range.

[0068] In the embodiment, the data in the data unit is sorted according to the data value size, so that the data in the data unit is arranged in sequence, and further, the upper limit value and the lower limit value of the data in the data unit are determined based on the sorted data in the data unit, so that the interval of the data value in the data unit can be directly compared with the partition key data range to determine the interception range of the data. Compared with the existing method of determining whether each piece of data needs to be partitioned, the determination method of intercepting data can partition data in batches, and greatly improves the partition reorganization efficiency.

[0069] The data table partition reorganization method of the embodiment of the present application further includes: in the case that the analysis result indicates that the unit data range is a subset of the partition key data range, determining that the plurality of data in the data unit are all reorganization data; and in the case that the analysis result indicates that there is no non-empty intersection between the unit data range and the partition key data range, determining that the plurality of data in the data unit are not all reorganization data.

[0070] For example, the data in the data unit is sorted according to the data value size, and in the case that the maximum value and the minimum value of the data unit are equal, and the equal maximum value and the minimum value both belong to the partition key data range, the data in the entire data unit is divided into the target partition, and further, the state of the data unit is recorded as a processed data unit, so that the processed data unit is directly skipped when the data unit is iterated next time, and the re-partition efficiency is improved.

[0071] For example, the partition key data range is (2, 5), and the plurality of data in the data unit are all 3, so that the maximum value and the minimum value of the data in the data unit are equal, both being 3, and the maximum value and the minimum value of the data in the data unit both belong to the partition key data range, so that it can be determined that the data in the data unit all belong to the partition key data range, and at this time, the data in the entire data unit is divided into the target partition.

[0072] In the case that the maximum value and the minimum value of the data in the data unit are not equal, and both the maximum value and the minimum value are contained in the data range of the partition key, the data in the entire data unit is divided into the target partition, and further, the state of the data unit is recorded as a processed data unit, so that when the data unit is traversed next time, the processed data unit is directly skipped, thereby improving the efficiency of re-partitioning.

[0073] For example, the data range of the partition key is (2, 5), the minimum value of the data in the data unit is 3, and the maximum value is 4. In this case, the minimum value and the maximum value of the data in the data unit are both contained in the data range of the partition key, so it can be determined that the data in the data unit all belong to the data range of the partition key. At this time, the data in the entire data unit is divided into the target partition.

[0074] In the case that the maximum value and the minimum value of the data in the data unit are not equal, and the minimum value is contained in the data range of the partition key, and the maximum value exceeds the data range of the partition key, the maximum value of the data range of the partition key is taken as a newly determined maximum value, and further, the data in the data unit between the newly determined maximum value and the minimum value is divided into the target partition. Further, the state of the data unit is not recorded as a processed data unit, and when the data unit is traversed next time, it is continuously determined whether there is data in the data unit that needs to be divided into the target partition.

[0075] For example, the data range of the partition key is (2, 5), the minimum value of the data in the data unit is 3, and the maximum value is 6. At this time, the minimum value 3 of the data in the data unit is contained in the data range of the partition key, and the maximum value 6 exceeds the data range of the partition key. In this case, the maximum value 5 of the data range of the partition key is taken as a newly determined maximum value, and further, the data in the data unit between the newly determined maximum value 5 and the minimum value 3 is divided into the target partition.

[0076] Similarly, in the case that the maximum value and the minimum value of the data in the data unit are not equal, and the maximum value is contained in the data range of the partition key, and the minimum value exceeds the data range of the partition key, the minimum value of the data range of the partition key is taken as a newly determined minimum value, and further, the data in the data unit between the newly determined minimum value and the maximum value is divided into the target partition. Further, the state of the data unit is not recorded as a processed data unit, and when the data unit is traversed next time, it is continuously determined whether there is data in the data unit that needs to be divided into the target partition.

[0077] For example, the data range of the partition key is (2, 5), the minimum value of the data in the data unit is 1, and the maximum value is 4. At this time, the minimum value 1 of the data in the data unit exceeds the data range of the partition key, and the maximum value 4 is included in the data range of the partition key. In this case, the minimum value 2 of the data range of the partition key is taken as the newly determined minimum value, and further, the data in the data unit between the newly determined minimum value 2 and the maximum value 4 is divided into the target partition.

[0078] In the case where neither the maximum value nor the minimum value of the data in the data unit is included in the data range of the partition key, it is equivalent to that there is no data in this data unit that needs to be divided into the target partition, and the data unit is directly skipped.

[0079] In the present embodiment, the data in the data unit is divided into the target partition, the row identifier corresponding to the data that needs to be divided in the data unit is determined, and the whole data corresponding to the row identifier is divided into the target partition.

[0080] In the present embodiment, according to the analysis result indicating the comparison between the data range of the unit and the data range of the partition key, the range of the data in the data unit that is subjected to batch division can be determined, and batch division of data and partition reorganization in the data unit are realized, thereby greatly reducing the time consumption of partition reorganization.

[0081] According to an embodiment of the present application, the partition reorganization of the data table based on the respective reorganization data of at least one data unit comprises the following operations for each reorganization data: recording the row identifier of the reorganization data in the data table into a data log file; inserting the reorganization data into a temporary table for the target partition; deleting the data corresponding to the row identifier in the data table based on the row identifier recorded in the data log file; and persisting the temporary table as the target partition.

[0082] In the present embodiment, the method of first establishing a temporary table and then persisting the temporary table as the target partition is used, and the data table of the target partition is established at one time, thereby effectively improving the reliability of establishing the target partition.

[0083] According to an embodiment of the present application, the data table partition reorganization method further comprises a coping method in the case of interruption of reorganization during partition reorganization. Specifically, it can comprise the following operations: in the case of interruption of partition reorganization, saving the database definition language executed before the interruption into a definition log file; in the case of resumption of partition reorganization, judging the partition reorganization progress based on the definition log file; in the case that the partition reorganization progress indicates that there is reorganization data that has not been inserted into the temporary table, emptying the temporary table and the data log file; based on the reorganization data, re-executing the partition reorganization; and in the case that the partition reorganization progress indicates that there is no reorganization data that has not been inserted into the temporary table, completing the partition reorganization based on the data log file and the temporary table.

[0084] According to the embodiment of the present application, the definition log file can record the database definition language executed before the interruption, in the case of partition reorganization interruption, the partition reorganization progress can be determined based on the definition log file, in the case that the partition reorganization progress indicates that there is reorganization data not inserted into the temporary table, the temporary table and the data log file are emptied, and the partition reorganization is re-executed, so that the partition reorganization can be timely and reliably completed in the case of partition reorganization interruption, and the reliability of the data table partition reorganization method is improved.

[0085] According to the embodiment of the present application, based on the partition key for the target partition indicated in the data table partition reorganization command, the data unit of the target column in the data table for indicating the partition key is determined, including: based on the partition key, matching in multiple columns in the data table to obtain the target column matched with the partition key; based on the target column, initiating a data acquisition request to the storage medium for storing the data table to obtain at least one data unit of the target column.

[0086] In the embodiment, obtaining at least one data unit of the target column can be obtaining multiple data units, which can divide the data in batches in units of data units and realize re-partitioning, so that the operation granularity of the partition reorganization is improved, and the efficiency of the partition reorganization is improved.

[0087] Figure 5 A flowchart of a data table partition reorganization method according to another embodiment of the present application is shown.

[0088] As shown in Figure 5 The data table partition reorganization method of the embodiment includes operations S501-S507.

[0089] In operation S501, a definition log file is defined. Wherein, the client executes the reorganization command, and the background starts the definition log file to record all actions and execution status from the beginning to the end of the reorganization process.

[0090] In operation S502, the data table is locked. Wherein, after the reorganization command is issued, the cluster layer adds a write lock to the data table, and adds an exclusive lock to the single machine layer.

[0091] In operation S503, the to-be-created partition is marked. Wherein, the parsing layer parses the information of the reorganization partition and the information of the new partition, the information of the reorganization partition indicates the table definition before reorganization, and the information of the new partition indicates the table structure after reorganization. After parsing, the to-be-reorganized partition state is marked as to-be-reorganized, and the to-be-created partition state is marked as to-be-added.

[0092] In operation S504, a temporary table definition file is generated. Initially, table structure information is saved to the table definition file, which is used to store table structure information. After partition reorganization, the partition distribution in the table structure changes. At this time, the partition table definition content is updated based on the original table definition file, and the temporary table definition file is generated, which will be effective.

[0093] In operation S505, multiple temporary tables are generated. A temporary table is created for the partition to be reorganized. Data that meets the conditions of the partition to be reorganized is marked for deletion. The deletion mark does not affect the original data file, but records the row mark of the data to the deletion table, and inserts the reorganized data into the temporary table. When there are multiple partitions to be reorganized, multiple temporary tables are generated.

[0094] In operation S506, historical data is deleted. The corresponding data is deleted according to the deletion table.

[0095] In operation S507, the definition log file is closed. The temporary table definition file is installed, and the user can see the partition table structure after the partition reorganization. The temporary table name is changed to the formal table name, the partition reorganization is completed, and finally the definition log file is closed.

[0096] Based on the above data table partition reorganization method, the application also provides a data table partition reorganization device, which will be described in detail below. Figure 6 The data table partition reorganization device is described in detail.

[0097] Figure 6 The structure block diagram of the data table partition reorganization device according to the embodiment of the application is shown.

[0098] As shown in Figure 6 The data table partition reorganization device 600 of the embodiment includes a data unit determination module 610, an analysis module 620, a reorganized data determination module 630, and a partition reorganization module 640.

[0099] The data unit determination module 610 is configured to determine, in response to a data table partition reorganization command, a data unit of a target column in a data table for representing a partition key based on the partition key for the target partition indicated in the data table partition reorganization command, wherein the data table includes a plurality of columns, and each column includes at least one data unit. In an embodiment, the data unit determination module 610 can be configured to perform operation S210 described above, and details are not repeated here.

[0100] The analysis module 620 is configured to, for each data unit, perform intersection analysis on a unit data range of the data unit and a partition key data range to obtain an analysis result, where the unit data range is determined according to the plurality of data stored in the data unit, and the partition key data range is determined from the data table partition restructuring command and represents a value range of the data to be divided into the target partition on the target column. In an embodiment, the analysis module 620 can be configured to perform the operation S220 described above, and details are not repeated here.

[0101] The restructuring data determination module 630 is configured to, in a case where the analysis result indicates that there is a non-empty intersection between the unit data range and the partition key data range, and the unit data range has an element that is not in the partition key data range, determine, from the plurality of data of the data unit, restructuring data belonging to the partition key data range. In an embodiment, the restructuring data determination module 630 can be configured to perform the operation S230 described above, and details are not repeated here.

[0102] The partition restructuring module 640 is configured to perform partition restructuring on the data table based on the respective restructuring data of the at least one data unit. In an embodiment, the partition restructuring module 640 can be configured to perform the operation S240 described above, and details are not repeated here.

[0103] According to an embodiment of the present application, the restructuring data determination module 630 includes a truncated range determination sub-module and a restructuring data acquisition sub-module.

[0104] The truncated range determination sub-module is configured to determine a truncated range for truncating the data unit based on the intersection between the unit data range and the partition key data range. In an embodiment, the truncated range determination sub-module is configured to perform the operation S331 described above.

[0105] The restructuring data acquisition sub-module is configured to select, from the plurality of data of the data unit, data belonging to the truncated range according to the truncated range to obtain the restructuring data. In an embodiment, the restructuring data acquisition sub-module is configured to perform the operation S332 described above.

[0106] According to an embodiment of the present application, the truncated range determination sub-module includes a truncated range determination unit.

[0107] The truncated range determination unit is configured to calculate the intersection between the partition key data range and the unit data range according to the upper limit value and the lower limit value of the partition key data range and the upper limit value and the lower limit value of the unit data range to obtain the truncated range.

[0108] The data table partition restructuring apparatus 600 further includes a subset determination module and a restructuring data exclusion module.

[0109] The subset determining module is configured to determine that the plurality of data in the data unit are all reorganization data in a case where the analysis result indicates that the unit data range is a subset of the partition key data range.

[0110] The reorganization data excluding module is configured to determine that the plurality of data in the data unit are all not reorganization data in a case where the analysis result indicates that there is no non-empty intersection between the unit data range and the partition key data range.

[0111] The partition reorganization module 640 comprises an execution submodule, a record submodule, a data insertion submodule, a data deletion submodule and a table persistence submodule.

[0112] The execution submodule is configured to perform the following operations for each reorganization data.

[0113] The record submodule is configured to record the row identification of the reorganization data in the data table into a data log file.

[0114] The data insertion submodule is configured to insert the reorganization data into a temporary table for the target partition.

[0115] The data deletion submodule is configured to delete the data corresponding to the row identification in the data table based on the row identification recorded in the data log file.

[0116] The table persistence submodule is configured to persist the temporary table as the target partition.

[0117] The data table partition reorganization apparatus 600 further comprises a definition log file saving module, a judgment module, a clearing module, a re-execution module and a continued partition reorganization module.

[0118] The definition log file saving module is configured to save the database definition language executed before interruption of the partition reorganization into a definition log file in a case where the partition reorganization is interrupted.

[0119] The judgment module is configured to judge the partition reorganization progress based on the definition log file in a case where the partition reorganization is resumed.

[0120] The clearing module is configured to clear the temporary table and the data log file in a case where the partition reorganization progress indicates that there is reorganization data not inserted into the temporary table.

[0121] The re-execution module is configured to re-execute the partition reorganization based on the reorganization data.

[0122] The continued partition reorganization module is configured to complete the partition reorganization based on the data log file and the temporary table in a case where the partition reorganization progress indicates that there is no reorganization data not inserted into the temporary table.

[0123] According to an embodiment of the present application, any of the modules of the data unit determining module 610, the analyzing module 620, the reorganized data determining module 630, and the partition reorganization module 640 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules, and implemented in one module. According to an embodiment of the present application, at least one of the data unit determining module 610, the analyzing module 620, the reorganized data determining module 630, and the partition reorganization module 640 can be implemented at least in part as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system in package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc., or implemented in hardware or firmware, or implemented in any one of software, hardware, and firmware or in a proper combination of any of them. Alternatively, at least one of the data unit determining module 610, the analyzing module 620, the reorganized data determining module 630, and the partition reorganization module 640 can be implemented at least in part as a computer program module that can perform the corresponding functions when the computer program module is run.

[0124] Figure 7 A block diagram of an electronic device suitable for implementing the data table partition reorganization method according to an embodiment of the present application is shown.

[0125] As shown in Figure 7 The electronic device 700 according to an embodiment of the present application includes a processor 701 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 702 or loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), etc. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0126] In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via the bus 704. The processor 701 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 702 and / or the RAM 703. It should be noted that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0127] According to the embodiments of the present application, the electronic device 700 can further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 can further include one or more of the following components connected to the input / output (I / O) interface 705: an input part 706 including a keyboard, a mouse, and the like; an output part 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 708 including a hard disk, and the like; and a communication part 709 including a network interface card such as a LAN card, a modem, and the like. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is installed in the storage part 708 as necessary.

[0128] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0129] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include the ROM 702 and / or the RAM 703 described above and / or one or more memory other than the ROM 702 and the RAM 703.

[0130] Embodiments of the present application also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the methods provided by the embodiments of the present application.

[0131] The above functions defined in the system / device / apparatus of the embodiments of the present application are performed when the computer program is executed by the processor 701. According to an embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0132] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of signals on network media. The computer program is downloaded and installed through the communication part 709 and / or installed from the detachable medium 711. The program codes contained in the computer program can be transmitted by any appropriate network media, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

[0133] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709 and / or installed from the detachable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0134] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming language includes, but is not limited to, such languages as Java, C++, python, "C" language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0135] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0136] Those skilled in the art will understand that features recited in the various embodiments of the present application can be combined and / or integrated in a variety of ways, even if such combinations or integrations are not expressly noted in the present application. In particular, features recited in the various embodiments of the present application can be combined and / or integrated in ways that do not depart from the spirit and scope of the present application. All such combinations and / or integrations are within the scope of the present application.

[0137] The embodiments of the present application are described above. However, these embodiments are merely for illustration purposes, and are not intended to limit the scope of the present application. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present application, and these substitutions and modifications shall fall within the scope of the present application.

Claims

1. A method for reorganizing a data table partition, characterized by, The method comprises: in response to a data table partition reorganization command, determining, based on a partition key for a target partition indicated in the data table partition reorganization command, a data unit of a target column in a data table for representing the partition key, wherein the data table comprises a plurality of columns, and each column comprises at least one data unit; for each data unit, performing intersection analysis on a cell data range of the data unit and a partition key data range to obtain an analysis result, wherein the cell data range is determined according to a plurality of data stored in the data unit, the partition key data range is determined from the data table partition reorganization command, and the partition key data range represents a value range of data to be divided into the target partition on the target column; in a case where the analysis result indicates that the cell data range and the partition key data range have a non-empty intersection, and the cell data range has an element that does not exist in the partition key data range, determining, from the plurality of data of the data unit, reorganization data belonging to the partition key data range; and based on the reorganization data of at least one data unit, reorganizing the data table.

2. The method of claim 1, wherein, The method further comprises: in a case where the analysis result indicates that the cell data range is a subset of the partition key data range, determining that the plurality of data in the data unit are all the reorganization data; and in a case where the analysis result indicates that the cell data range and the partition key data range have no non-empty intersection, determining that the plurality of data in the data unit are not all the reorganization data.

3. The method of claim 2, wherein, The method further comprises: based on the reorganization data of at least one data unit, reorganizing the data table. The method further comprises:

4. The method of claim 1, wherein, in a case where the analysis result indicates that the cell data range is a subset of the partition key data range, determining that the plurality of data in the data unit are all the reorganization data; and in a case where the analysis result indicates that the cell data range and the partition key data range have no non-empty intersection, determining that the plurality of data in the data unit are not all the reorganization data. The method further comprises:

5. The method according to any one of claims 1 to 4, characterized in that, for each reorganization data, performing the following operations: recording a row identifier of the reorganization data in the data table into a data log file; inserting the reorganization data into a temporary table for the target partition; deleting, based on the row identifier recorded in the data log file, data corresponding to the row identifier in the data table; persisting the temporary table as the target partition. ​ 6. The method of claim 5, wherein, The method further comprises: in case of interruption of the partition reorganization, saving database definition language executed before the interruption into a definition log file; in case of resumption of the partition reorganization, judging a partition reorganization progress based on the definition log file; in case that the partition reorganization progress indicates that there is reorganization data not inserted into the temporary table, emptying the temporary table and the data log file; re-executing the partition reorganization based on the reorganization data; in case that the partition reorganization progress indicates that there is no reorganization data not inserted into the temporary table, completing the partition reorganization based on the data log file and the temporary table.

7. The method of claim 1, wherein, The determining of the data unit of the target column for representing the partition key in the data table based on the partition key indicated in the data table partition reorganization command comprises: matching in multiple columns in the data table based on the partition key to obtain a target column matching the partition key; initiating a data acquisition request to a storage medium for storing the data table based on the target column to obtain at least one data unit of the target column.

8. A data table partition reorganization apparatus, characterized by comprising: The apparatus comprises: a data unit determining module configured to, in response to a data table partition reorganization command, determine a data unit of a target column for representing a partition key in a target partition based on the partition key indicated in the data table partition reorganization command, wherein the data table comprises multiple columns, and each column comprises at least one data unit; an analysis module configured to, for each data unit, perform intersection analysis on a unit data range of the data unit and a partition key data range to obtain an analysis result, wherein the unit data range is determined according to multiple data stored in the data unit, and the partition key data range is determined from the data table partition reorganization command, and the partition key data range represents a value range of data to be divided into the target partition on the target column; a reorganization data determining module configured to, in case that the analysis result indicates that the unit data range and the partition key data range have a non-empty intersection, and the unit data range has an element not in the partition key data range, determine reorganization data belonging to the partition key data range from the multiple data of the data unit; a partition reorganization module configured to perform partition reorganization on the data table based on the reorganization data of each data unit. 9.An electronic device comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-7.

Images