Data processing

Abstract

Provided are a data processing method, an electronic device, a computer-readable storage medium and a computer program product, which relate to the field of databases. The data processing method comprises: determining, from a database, M data groups to be integrated, wherein M is a positive integer greater than 1, each data group to be integrated comprises a plurality of data records having sorting key values and has a corresponding sorting key value interval, a minimum endpoint value and a maximum endpoint value of the sorting key value interval are respectively a minimum sorting key value and a maximum sorting key value in each data record in the corresponding data group, and for any data group to be integrated, there is at least one other data group to be integrated, the sorting key value interval of which overlaps with the sorting key value interval thereof; and reintegrating the M data groups to be integrated into at least one target data group, and adding the at least one reintegrated target data group to the database.

Description

[0001] Data processing technology

[0002]

[0001] This disclosure relates to the field of database technology, and more particularly to data processing. Background Art

[0003]

[0002] With the continuous development of computer technology and cloud storage technology, databases have expanded to include more and more storage architectures to store data information. As the amount of recorded data increases, the time required to query data in the database also increases. Some databases can improve query efficiency by dividing the stored data files into ordered parts according to the range of sort keys. However, for databases using unordered storage architectures, it is difficult to improve data query efficiency by dividing the data into ordered parts. Summary of the Invention

[0004]

[0003] Embodiments of this disclosure provide a data processing method, an electronic device, a computer-readable storage medium, and a computer program product to alleviate or solve one or more technical problems existing in the related art.

[0005]

[0004] In a first aspect, embodiments of this disclosure provide a data processing method applied to a database employing an append-only storage architecture. The method includes: determining M data groups to be integrated from the database; wherein M is a positive integer greater than 1, the data groups to be integrated include multiple data records with sorting key values ​​and corresponding sorting key value intervals, the minimum endpoint value and the maximum endpoint value of the sorting key value interval being the minimum sorting key value and the maximum sorting key value of each data record in the corresponding data group, respectively; any data group to be integrated has at least one other data group to be integrated whose sorting key value interval overlaps with it; re-integrating the M data groups to be integrated into at least one target data group, and adding the re-integrated at least one target data group to the database, wherein the sorting key value intervals of the target data groups in the at least one target data group do not overlap.

[0006]

[0005] In a second aspect, embodiments of the present disclosure provide a database system, including: a database storing multiple data groups; and a server for implementing the method of any one of the embodiments of the present disclosure, and for querying the data records from the multiple data groups of the database in response to a query request for data records.

[0007]

[0006] In a third aspect, embodiments of the present disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of the embodiments of the present disclosure when executing the computer program.

[0008]

[0007] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of the present disclosure.

[0009]

[0008] In a fifth aspect, embodiments of the present disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of the present disclosure.

[0010]

[0009] The data processing method based on the embodiments of this disclosure selects a portion of data groups in the database that have a certain overlap between their sort key value intervals as data groups to be integrated, and integrates the data groups to be integrated into target data groups whose sort key value intervals do not overlap. This can achieve local ordered reorganization of the database, thereby reducing the overlap of sort key value intervals between data groups in the database. This method can be applied to scenarios where data is queried using sort key values ​​as indexes, thereby improving the query efficiency when data is queried using sort key values ​​as indexes.

[0011]

[0010] The above description is merely an overview of the technical solution of this disclosure. In order to better understand the technical means of this disclosure, it can be implemented according to the contents of the specification. Furthermore, in order to make the above and other objects, features, and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. (See attached drawings.)

[0012]

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this disclosure and should not be considered as limiting the scope of this disclosure.

[0013]

[0012] Figure 1 is a schematic diagram of an application scenario of an embodiment of this disclosure;

[0014]

[0013] S2 is a schematic diagram of the overlap of sorting key value ranges of each data group in the database under the Append-Only storage architecture;

[0015]

[0014] Figure 3 is a schematic diagram of the database system according to an embodiment of the present disclosure;

[0016]

[0015] FIG4 shows a flowchart of a data processing method according to an embodiment of the present disclosure;

[0017]

[0016] FIG5 shows a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

[0018]

[0017] FIG6 is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Description

[0019]

[0018] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this disclosure. Therefore, the drawings and description are considered exemplary in nature and not restrictive.

[0019] To facilitate understanding of the technical solutions of the embodiments of this disclosure, related technologies of the embodiments of this disclosure are described below. The following related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this disclosure, all of which fall within the protection scope of the embodiments of this disclosure.

[0020]

[0020] Data in the database can be divided into multiple row groups (data groups). Each data group contains multiple rows of data, and each row of data is a data record. These multiple rows of data records are divided into multiple columns, thus recording data in a columnar storage format. For example, as shown in Figure 1, each column of data in the same data group can be represented as a data block. For example, PackA1 (data block A1) represents the data in column A of row group 1, and PackB1 (data block B1) represents the data in column B of data group row group 1. Each data group usually also stores statistical information of each data block, including the maximum key value, minimum key value, and number of null values ​​(the number of valid key values) of each data block (i.e., a column of data in the data group). For example, in Figure 1, the minimum key value and maximum key value of PackA1 are 6 and 20, respectively. The statistical information stored in the data block corresponding to PackA1 includes Min=6 and Max=20.

[0021]

[0021] Min-Max filtering is a common optimization technique used in database data querying, data analysis and other data processing tasks. It aims to reduce the amount of data to be processed by using the minimum and maximum key values ​​of each data block. When executing a query, the value range of the specified column data can be determined according to the query conditions. If the determined value range does not overlap with the key value range of the specified column (corresponding to a data block) of the data group (the endpoints of which are the maximum and minimum key values ​​of the statistics), then the query for this data group can be skipped, thereby improving query efficiency. For example, if the query condition is that the key value in column A is greater than 15, the key value range of PackA1 is [6, 20], the key value range of PackA2 is [17, 30], and the key value range of PackA3 is [1, 10]. Therefore, it can be determined that there are data records in row group 1 and row group 2 with key values ​​in column A greater than 15, but there are no data records in row group 3 with key values ​​in column A greater than 15. When querying data, the query for row group 3 can be skipped directly based on the fact that the key value range of PackA3 in row group 3 does not meet the query condition. This reduces the number of data groups that need to be queried and improves query efficiency.

[0022]

[0022] Database administrators can set one or more columns of data fields in the database as Order Keys according to query requirements. Usually, data fields whose key value range can be used as query conditions (such as fields representing time or quantity) are used as sort keys. The column containing the sort key can be called the sort column. The data in the database is sorted based on the sort key to minimize the overlap of the key value ranges of the sort keys of each data group. This can achieve better query filtering effect and improve query efficiency when users set query conditions through the key value range of the sort key.

[0023]

[0023] An append-only storage architecture is one in which data, once written to the database, is not modified or deleted. Instead, modifications or deletions to the original data are only represented by appending records to the data. This avoids the overhead of disk addressing and data rewriting, enabling efficient data writing. However, this storage architecture also leads to the disorder of data records. The key value ranges of the sorted sequences of each data group (referred to as the sort key value range, i.e., the key value range of the sorted sequence of a data group is the sort key value range of that data group) may have significant overlap. Therefore, when querying data in a database under this storage architecture using the Min-Max method, the filtering effect on non-target data is poor, resulting in low query efficiency.

[0024]

[24] Figure 2 shows the overlap of sorting key value ranges for each data group in the database under the Append-Only storage architecture.

[0025]

[25] As shown in Figure 2, sorting key value interval 1, sorting key value interval 2, sorting key value interval 3, and sorting key value interval 4 represent the key value range of sorted data in data group 1, data group 2, data group 3, and data group 4, respectively. That is, the sorting key value interval in data group 1 is [0, 4], the sorting key value interval in data group 2 is [1, 5], the sorting key value interval in data group 3 is [1, 6], and the sorting key value interval in data group 4 is [5, 7]. Assuming that the range of sorted data determined according to the query condition is [r, r], that is, the query condition is that the key value of the sorting key is r, each arrow in the figure represents a query when r is a specific value. It can be seen that when the value range of r is [1, 6], two or more data groups will be queried. That is, the sorting key value intervals of the four data groups from data group 1 to data group 4 overlap in [1, 6], which will affect the query efficiency.

[0026]

[26] In addition, new data will be continuously entered and stored in the database system. The data contained in the reorganized data group under the Append-Only storage architecture, as well as the additional new data, need to be recorded at the end. If it is necessary to reorganize the data groups in the database based on the key value of the sort key, it is impossible to enter additional new data during the sorting and reorganization. It is also difficult for the system to perform global sorting and integration of all data groups in the system during the gap of new data. Moreover, the computational load of global sorting is too large, which will also affect other operations of the database system.

[0027]

[27] In view of this, the present disclosure proposes a data processing method that can select only M data groups for integration each time. After each integration, the degree of overlap of the sorting key value range of each data group in the database can be alleviated to a certain extent. The database system can continuously select M data groups for integration in the background to refresh each data group in the database, thereby avoiding the expansion of data volume, which would lead to serious data disorder and affect query efficiency.

[0028]

[28] FIG3 is a schematic diagram of a database system 300 according to an embodiment of the present disclosure. The database system 300 may include a database 301 and a server 302.

[0029]

[29] In the database 301, multiple data groups are stored. Each data group contains multiple columns of data. Each column of data in the data group is stored in a corresponding data block. The data block also stores statistical information for the column of data. The statistical information includes at least the maximum key value and the minimum key value of the column of data, so as to determine the key value range of the column.

[0030]

[30] The server 302 can obtain the query request sent by the user based on the user terminal. The query request may include the query conditions set by the user based on the key value range of the sort key. In response to the query request for data records, the server 302 can query data records that meet the query conditions in multiple data groups of the database 301 based on the key value range of the sort sequence of each data group.

[0031]

[0031] In addition, the server 302 can periodically select M data groups from multiple data groups in the database 301 for reorganization, so that the key value ranges of the sorting sequence in the N data groups (N is less than or equal to M) do not overlap. The N data groups are added to the database and the corresponding data blocks of the original M data groups are deleted. This achieves local ordered reorganization of the database, which reduces the degree of overlap of the key value ranges of the sorting sequence between data groups to a certain extent. This can improve the query efficiency of data records when the query conditions include the key value range of the sorting key.

[0032]

[0032] It should be noted that the above-described application scenarios or examples of the data processing method provided in the embodiments of this disclosure are for ease of understanding, and the embodiments of this disclosure do not specifically limit the application of the data processing method. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to select or edit authorization or refuse.

[0033]

[0033] The technical solutions of this disclosure and how the technical solutions of this disclosure solve the aforementioned technical problems will be described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0034]

[0034] FIG4 shows a flowchart of a data processing method according to an embodiment of the present disclosure. This data processing method can be applied to a database employing an append-only storage architecture. As shown in FIG4, the data processing method may include the following steps.

[0035]

[0035] Step S401: Determine M data groups to be integrated from the database; where M is a positive integer greater than 1, each data group to be integrated includes multiple data records with sorting key values ​​and corresponding sorting key value intervals, the minimum endpoint value and the maximum endpoint value of the sorting key value interval are the minimum sorting key value and the maximum sorting key value in each data record of the corresponding data group, respectively; any data group to be integrated has at least one other data group to be integrated whose sorting key value interval overlaps with it.

[0036]

[0036] Here, any data group to be integrated has at least one other data group to be integrated whose sorting key value range overlaps with it. That is, at least two of the selected M data groups to be integrated have overlapping sorting key value ranges. At most, any data group to be integrated may overlap with other data groups to be integrated. By determining such M data groups as data groups to be integrated, a better de-overlapping effect can be obtained after integration.

[0037] Step S402: Reintegrate the M data groups to be integrated into at least one target data group, and add the reintegrated target data group to the database, wherein the sort key value ranges of the target data groups in the at least one target data group do not overlap.

[0037]

[0038] Understandably, due to the characteristics of append-only storage architectures, it is not possible to directly migrate and replace data between existing data groups like storage architectures such as LSM (log-structured merge-tree). Therefore, the integrated target data group needs to be stored in the database as newly added data.

[0038]

[0039] Here, although the database system is configured so that each data group contains the same number of data records, some of the selected data groups to be integrated may not have actual data stored in certain columns, or the data blocks storing these data groups may have inefficiently used space due to encoding or data alignment reasons. This results in a certain data void rate in the data groups. Therefore, when re-integrating M data groups to be integrated, in addition to ensuring that there is no overlap in sorting key values ​​between the integrated target data groups, adjustments to reduce the data void rate can also be taken into account. The maximum number of the re-integrated target data groups is M. If the data void rate in the data groups to be integrated is high, there may be cases where the number of the integrated target data groups is less than M.

[0039]

[0040] Using the method of this embodiment, a portion of the data groups in the database that have some overlap between their sort key value ranges can be selected as the data groups to be integrated. By integrating the data groups to be integrated into target data groups whose sort key value ranges do not overlap, the database can be partially ordered and reorganized. This reduces the overlap of sort key value ranges between the data groups in the database, improves the query efficiency when querying data using the sort key as an index, and enhances the Min-Max filtering effect.

[0040]

[0041] For example, after adding at least one reorganized target data group to the database in step S402, the method further includes: releasing the storage space where the M data groups to be integrated are located in the database.

[0041]

[0042] Here, the storage space containing the M data groups to be integrated, which are also the multiple data blocks corresponding to the data groups to be integrated, will become null values ​​in the database after the data records in the M data groups to be integrated are extracted and re-integrated into the target data group. Therefore, after the data group is re-integrated, the multiple data blocks corresponding to the data groups to be integrated can be cleared and released, thereby improving the effective utilization of the database storage space.

[0042]

[0043] In one embodiment, the M data groups to be integrated determined in step S401 are the M data groups with the largest overlap length of sorting key value intervals in the database.

[0043]

[0044] Ideally, to maximize the effect of a single local reordering, the selected M data groups to be integrated should be the M data groups with the largest overlap length of their sort key value intervals. Here, if M=4, taking the four data groups in Figure 2 as an example, there are three overlapping data groups in the interval [1, 4], so [1, 4] overlaps twice. There are two overlapping data groups in each of the intervals [4, 5] and [5, 6], so [4, 5] and [5, 6] overlap once each. The overlap length of the sort key value intervals among these four data groups is [1, 4]*2 + [4, 5] + [5, 6] = 8. Therefore, assuming there are 100 data groups in the database, it is necessary to traverse the 100 data groups to select 4 combinations, calculate the overlap length of the sort key value intervals of the 4 data groups under different combinations, and determine the 4 data groups with the largest overlap length as the data groups to be integrated. This allows for the best results from a single restructuring.

[0044]

[45] It is understood that since the exhaustive method is used to determine the M data groups to be integrated in the database, it is necessary to traverse and calculate the combination form of all data groups. When the amount of data in the database is too large, the exhaustive calculation is computationally intensive and time-consuming. Assuming there are T data groups in the database, the computational complexity of determining the M data groups to be integrated from the T data groups is as high as O(TAM). In order to improve the efficiency of selecting the data groups to be integrated in the database and ensure the single reorganization effect, the present disclosure provides a simplified algorithm for determining the M data groups with the largest overlap length of sorting key value intervals.

[0045]

[46] First, the data groups can be classified according to the overlap of the sorting key value intervals between the data groups in the database. The data group that is not covered by the sorting key value intervals of other data groups is the parent data group, and the data group whose sorting key value intervals are covered by at least one other data group is the child data group. The child data group has a corresponding relationship with the parent data group whose sorting key value interval is covered by the parent data group. In this way, a set of data groups is constructed. Each set of data groups contains a parent data group and a child data group corresponding to that data group. If a child data group corresponds to multiple parent data groups, the child data group can be assigned to the set of data groups where any of the corresponding parent data groups are located.

[0046]

[47] After constructing each data group in the database into different data group sets, the data group sets can be numbered and sorted according to the minimum sort key value of the parent data group in the data group set in ascending order of the minimum sort key value. For example, the range of sort key values ​​in the database is [0, 100], and there are a total of 10 data group sets (that is, 10 parent data groups). The data group set containing the parent data group with the minimum sort key value of 0 is set 1, and the data group set containing the parent data group with the largest minimum sort key value among the 10 parent data groups is set 10. The data groups within each data group set are sorted according to the length of the sort key value range, and the parent data group is the first data group within the data group set.

[0047]

[48] ​​Thus, according to the sorting of the data sets, we can traverse the cases where the data sets to be integrated come from different data sets. For example, in the case of only 3 data sets, there are 7 cases: 1) The data sets to be integrated are all data sets in data set 1; 2) The data sets to be integrated are all data sets in data set 2; 3) The data sets to be integrated are all data sets in data set 3; 4) The data sets to be integrated include data sets in data set 1 and data set 2; 5) The data sets to be integrated include data sets in data set 1 and data set 3; 6) The data sets to be integrated include data sets in data set 2 and data set 3; 7) The data sets to be integrated include data sets in data set 1, data set 2 and data set 3.

[0048]

[0049] In addition, to reduce the difficulty of traversal, selection rules for M data groups to be integrated can be configured. The selection rules include: 1) The prerequisite for a sub-data group to be selected is that its parent data group in its data group set has been selected; 2) Data groups in the same data group set are selected according to the length of the sorted key value range, with the longer data group being selected over the shorter data group.

[0049]

[0050] Formula 1 shows the formula for calculating the maximum overlap length of the sorted key value intervals among the selected i data groups when traversing the data group set h in the sorted order. f[h][i] = max{f |j|[ilm] + sum(top_child_len(m)) + (pack[j].R - pack[h].L)} Formula 1

[0050]

[0051] Where h represents the set of data groups to which the i-th selected data group belongs, j represents the set of data groups that have been traversed before reaching the set of data groups h, pack[j].R represents the maximum sort key value in the j-th data group set, pack[h].L represents the minimum sort key value in the set of data groups h, m represents the number of sub-data groups that have been selected in the set of data groups h, and top_child_len(m) represents the sum of the sort key value interval lengths of the m sub-data groups with the largest sort key value interval lengths in the set of data groups h.

[0051]

[0052] Therefore, based on Formula 1, we can gradually traverse the maximum overlap length of the sorting key value intervals when the data groups to be integrated come from different data group sets. If there are 3 data group sets, after traversal, we only need to compare the obtained f[l][M], f[2][M], f[3][M] and select the data group with the maximum value to determine the M data groups with the largest overlap length of sorting key value intervals as the data groups to be integrated.

[0052]

[0053] Using Formula 1 to determine the M data groups to be integrated, the computational complexity can be reduced from O(TAM) to O(T*M*M). To further simplify the calculation, all data groups in the database can be globally sorted according to their data group set number and their order within the data group. Then, each data group is traversed in the global sort order. Distinguishing between the i-th selected data group as a parent or child data group, Formula 1 is expanded to obtain Formula 2 when the i-th data group is a child data group and Formula 3 when the i-th data group is a parent data group. f[k][i] = f[kl][il] + pack[i] .R - pack[i] .L (Formula 2)

[0053]

[0054] Where k represents the global sorting number of the i-th data group, and pack[i].R-pack[i].L is the length of the sorting key value interval of the sub-data group i. Since the parent data group corresponding to the sub-data group i has been selected in this case, and the sub-data groups are sorted and numbered in descending order of the length of the sorting key value interval, we only need to add the length of the sorting key value interval of the sub-data group i itself to the overlap length of the sorting key value interval of the previous (i-1) selected data groups to obtain the maximum overlap length of the sorting key value intervals of the i data groups. f[k][i] = max{f |j|[i-1] + Father (pack[j]) .R} -pack[i] .L Formula 3

[0055] Where j represents the global sorting number of the (i-1)th selected data group. When the i-th selected data group k is the parent data group, it is necessary to calculate the overlap between the parent data group (data group j may be a child data group or it may be its own parent data group) and the i-th selected data group k as the overlap superimposed on the selected (i-1) data groups, so as to calculate the overlap length of the sorting key value interval of the i data groups. This formula can obtain the maximum overlap length of the sorting key value interval of the i data groups. The selection of the i data groups corresponding to the maximum overlap length is taken as the data group to be integrated.

[0054]

[0056] Based on this, we can distinguish the types of data groups in the candidate data groups, and use different calculation methods to calculate the overlap length of the sorting key value intervals of the selected i data groups under different data group combination forms. By selecting the data group selection method corresponding to the maximum value, we can determine the M data groups with the largest overlap length of sorting key value intervals as the data groups to be integrated. This further reduces the computational complexity in Formula 1 to O(T*M), simplifying the calculation process of determining the data groups to be integrated when seeking the best single reorganization effect. Each time reorganization is performed, the M data groups with the largest overlap length of sorting key value intervals are determined in the database, which can maximize the de-overlapping effect of each reorganization.

[0055]

[0057] It is understandable that using an exhaustive search method to determine the M data groups to be integrated in the database requires traversing and calculating the combination forms of all data groups. Even with some computational optimization, the computational load remains large and time-consuming when the amount of data in the database is too large. To further improve the efficiency of selecting data groups to be integrated in the database, this disclosure proposes a simplified calculation method to determine the data groups to be integrated. Specifically, in step S401, determining the M data groups to be integrated from the database includes: selecting any parent data group as the first data group to be integrated; wherein the sorting key value range of the parent data group is not covered by other data groups in the database, and the sorting key value range of the parent data group covers the sorting key value range of its corresponding child data group; in response to the number of already determined data groups to be integrated being less than M, determining the i-th data group to be integrated; wherein i is a positive integer and 1 < i <M o

[0056]

[0058] Here, based on the division of parent data groups and child data groups in the previous embodiment, any parent data group can be selected as the first data group to be integrated. Since this data group is the parent data group, there must be at least a child data group that overlaps with its sorting key value range. The subsequent data groups to be integrated are selected based on this parent data group.

[0057]

[0059] Preferably, in this embodiment, determining the i-th data group to be integrated includes: determining the i-th data group to be integrated from multiple candidate data groups; wherein, the candidate data groups include parent data groups that overlap with the sorting key value range of the already determined data groups to be integrated and / or the corresponding parent data groups are child data groups of the already determined data groups to be integrated, and the candidate data groups do not include the already determined data groups to be integrated.

[0058]

[0060] It is understandable that any subsequent selected data group to be integrated must have overlapping sort key value ranges with at least one already determined data group to be integrated. Only by selecting these as data groups to be integrated can the overlap-removing effect be achieved. Since the data groups were pre-classified based on the overlap of their sort key value ranges, when considering the selection of subsequent data groups to be integrated, the selection scope can be directly limited to the sub-data groups corresponding to the already determined data groups to be integrated, as well as the parent data groups whose sort key value ranges overlap with those of the data groups to be integrated. These data groups that meet the conditions are selected as candidate data groups. It is only necessary to calculate the overlap length of the sort key value ranges between each data group in the candidate data groups and the already determined data groups to be integrated when adding them as new data groups to be integrated, without considering data groups in the database that do not meet the conditions.

[0059]

[0061] By classifying the data groups in the database using the method of this embodiment, the selection range can be narrowed when selecting the data groups to be integrated, thereby improving the speed of determining the data groups to be integrated and thus improving the efficiency of database reorganization.

[0060]

[0062] For example, determining the i-th data group to be integrated from multiple candidate data groups includes: calculating the overlap length of the sorting key value interval between each candidate data group and the already determined data group to be integrated; and determining the candidate data group corresponding to the maximum overlap length as the i-th data group to be integrated.

[0061]

[0063] Here, although the sorting key value range of the sub-data groups in the candidate data group is covered by the parent data group that has been determined as the data group to be integrated, the overlap length of the sorting key value range between the sub-data groups and the parent data groups is only the length of the sorting key value range of the sub-data groups. However, other parent data groups in the candidate data group may have a longer overlap length with the sorting key value range of the already determined data group to be integrated. Therefore, it is necessary to calculate the overlap length of the sorting key value range between the data groups to be integrated when each data group in the candidate data group is the i-th data group to be integrated, so as to determine the candidate data group corresponding to the maximum overlap length as the i-th data group to be integrated.

[0062]

[0064] The method described in this embodiment further simplifies the selection of data groups to be integrated, speeds up the determination of data groups to be integrated, and thus improves the efficiency of database reorganization.

[0063]

[0065] In one embodiment, step S402, which involves re-integrating the M data groups to be integrated into at least one target data group, includes: sorting the data records of the M data groups to be integrated according to the size of the sorting key value; and integrating the sorted data records into a target data group at preset intervals.

[0064]

[0066] Here, since the data records within and between data groups in the append-only storage architecture are not strictly sorted according to the order of key values, when sorting the data records of the M data groups to be integrated, it is not necessary to strictly arrange the data records in descending or ascending order of sorting key values. It is only necessary to ensure that the sorting key value ranges of the preset data records in each group do not overlap, so that the sorting key value ranges between the integrated target data groups do not overlap.

[0065]

[0067] For example, if you need to integrate 10 data records with corresponding sort key values ​​(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) into two target data groups, with the sort key value range of the first target data group being [1, 5] and the sort key value range of the second target data group being (5, 10], then when sorting the data records, it is not necessary to strictly arrange them in ascending or descending order of values. It is only necessary to ensure that the sort key values ​​of the first 5 data records are 1, 2, 3, 4, 5 (in any order) or 6, 7, 8, 9, 10 (in any order) to achieve the effect that the sort key value ranges of the two target data groups do not overlap after reorganization.

[0066]

[0068] Using the method of this embodiment, by extracting and sorting all data records in the data group to be integrated, and then splitting the sorted data records according to a preset number of records, the data group to be integrated can be quickly integrated into a target data group with non-overlapping sorted key value ranges.

[0067]

[0069] Based on the technical solutions of the embodiments of this disclosure, a data processing method suitable for append-only storage architecture is proposed. This method can achieve partial ordered reorganization of the database, reduce the overlap of sort key value intervals between data groups in the database, improve query efficiency when querying data using sort key as index, and improve the Min-Max filtering effect. Furthermore, the embodiments of this disclosure also propose an optimized algorithm for selecting data groups to be integrated when performing partial ordered reorganization, which can reduce the computational amount of determining data groups to be integrated and improve the efficiency of ordered reorganization.

[0068]

[0070] Corresponding to the application scenarios and data processing methods provided in the embodiments of this disclosure, the embodiments of this disclosure also provide a data processing device 500. As shown in FIG5, the device may include: a determining module 501, configured to determine M data groups to be integrated from a database; wherein M is a positive integer greater than 1, the data groups to be integrated include multiple data records with sorting key values ​​and corresponding sorting key value intervals, the minimum endpoint value and the maximum endpoint value of the sorting key value intervals being the minimum sorting key value and the maximum sorting key value in each data record of the corresponding data group, respectively; any data group to be integrated has at least one other data group to be integrated whose sorting key value interval overlaps with its own; an integrating module 502, configured to re-integrate the M data groups to be integrated into at least one target data group, and add the re-integrated at least one target data group to the database, wherein the sorting key value intervals of the target data groups in the at least one target data group do not overlap.

[0069]

[0071] For example, the M data groups to be integrated are the M data groups in the database with the largest overlap length of the sorted key value intervals.

[0070]

[0072] For example, the determining module 501 is configured to: select any parent data group as the first data group to be integrated; wherein the sorting key value range of the parent data group is not covered by other data groups in the database, and the sorting key value range of the parent data group covers the sorting key value range of its corresponding child data group; and, in response to the fact that the number of already determined data groups to be integrated is less than M, determine the i-th data group to be integrated; wherein i is a positive integer and 1 < i <M o

[0071]

[0073] For example, the determining module 501 is further configured to: determine the i-th data group to be integrated from a plurality of candidate data groups; wherein, the candidate data group includes parent data groups that overlap with the sorting key value range of the already determined data group to be integrated and / or the corresponding parent data group is a child data group of the already determined data group to be integrated, and the candidate data group does not include the already determined data group to be integrated.

[0072]

[0074] For example, the determining module 501 is further configured to: calculate the overlap length of the sorting key value interval between each candidate data group and the already determined data group to be integrated; and determine the candidate data group corresponding to the maximum overlap length as the i-th data group to be integrated.

[0073]

[0075] For example, the integration module 502 is used to sort the data records of the M data groups to be integrated according to the size of the sorting key value; and to integrate the sorted data records into a target data group at preset intervals.

[0074]

[0076] For example, after the integration module 502 adds at least one re-integrated target data group to the database, it further includes: in the database, releasing the storage space where the M data groups to be integrated are located.

[0075]

[0077] The functions of each module in the apparatus of this embodiment can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0076]

[0078] Figure 6 is a block diagram of an electronic device used to implement embodiments of the present disclosure. As shown in Figure 6, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the methods in the above embodiments. The number of memories 601 and processors 602 can be one or more.

[0077]

[0079] The electronic device also includes: a communication interface 603 for communicating with external devices and transmitting data.

[0078]

[0080] If the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 6, but this does not indicate that there is only one bus or one type of bus.

[0079]

[0081] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0080]

[0082] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0081]

[0083] This disclosure also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the methods provided in this disclosure.

[0082]

[0084] This disclosure also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0083]

[0085] It should be understood that the aforementioned processor can be a CPU, or it can be other general-purpose processors or digital signal processors.

[0084] (Digital Signal Processor, DSP), Application Specific Integrated Circuit (ASIC), FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0085]

[0086] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. For example, static random access memory.

[0086] (Static RAM SRAM) Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DR RAM).

[0087]

[0087] In the above embodiments, the implementation can be carried out entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0088]

[0088] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0089]

[0090] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0090]

[0091] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0091]

[0092] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0092]

[0093] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0093]

[0094] The above description is merely an exemplary embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this disclosure, and these should all be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

Claims 1. A data processing method, applied to a database using an append storage architecture, comprising: determining M data groups to be integrated from a database; wherein M is a positive integer greater than 1, the data groups to be integrated include multiple data records with sorting key values and correspond to sorting key value intervals, minimum and maximum endpoint values of the sorting key value intervals are minimum and maximum sorting key values of the data records in the corresponding data groups; any data group to be integrated has at least one other data group to be integrated with which the sorting key value intervals overlap; re-integrating the M data groups to be integrated into at least one target data group, and adding the at least one target data group after re-integration to the database, wherein the sorting key value intervals of each target data group in the at least one target data group do not overlap.

2. The method according to claim 1, wherein, The M data groups to be integrated are M data groups with the maximum overlapping length of the sorting key value intervals in the database.

3. The method according to claim 1, wherein, determining M data groups to be integrated from a database, including: selecting any parent data group as a first data group to be integrated; wherein the sorting key value interval of the parent data group is not covered by other data groups in the database, and the sorting key value interval of the parent data group covers the sorting key value intervals of its corresponding child data groups; in response to the number of the determined data groups to be integrated being less than M, determining an i-th data group to be integrated; wherein i is a positive integer and 1 4. The method according to claim 3, wherein, determining an i-th data group to be integrated, including: determining the i-th data group to be integrated from multiple candidate data groups; wherein the candidate data groups include parent data groups with overlapping sorting key value intervals with the determined data groups to be integrated and / or corresponding parent data groups being child data groups of the determined data groups to be integrated, and the candidate data groups do not include the determined data groups to be integrated.

5. The method according to claim 4, wherein, determining the i-th data group to be integrated from multiple candidate data groups, including: calculating the overlapping lengths of the sorting key value intervals of each of the candidate data groups and the determined data groups to be integrated, respectively; determining the candidate data group corresponding to the maximum overlapping length as the i-th data group to be integrated.

6. The method according to claim 1, wherein, re-integrating the M data groups to be integrated into at least one target data group, including: sorting the data records of the M data groups to be integrated according to the sizes of the sorting key values; integrating the sorted data records into one target data group every interval of a preset number of data records.

7. The method according to any one of claims 1-6, wherein, adding the at least one target data group after re-integration to the database, further including: releasing the storage space where the M data groups to be integrated are located in the database. a database storing multiple data groups; 8. A database system comprising: a server for implementing the method in any of claims 1-7, and in response to a query request for a data record, querying the data record from the multiple data groups of the database. ​ 9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images