Data management method and device, electronic device and storage medium

Abstract

The present specification provides a data management method and device, an electronic device and a storage medium, the method comprising: for each balancing group, constructing distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, wherein the distribution information comprises at least one balancing unit, the balancing unit comprises at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets, and the data units comprise partitions of non-partitioned tables and / or partitioned tables; for each balancing group, based on the distribution information of the balancing group on each log stream of the data system, balancing processing is performed on multiple data units in the balancing group; based on the distribution information of each balancing group on each log stream of the data system, balancing processing is performed on data units on all log streams of the data system.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a data management method and device, an electronic device, and a storage medium. Background Art

[0002] With the development and cooperation of artificial intelligence and mobile Internet technologies, more and more data is generated in users' work and life, so data management is becoming more and more important and critical. At present, there are more and more types of databases used to manage data, such as cloud encrypted databases and distributed databases. A distributed database is a data storage system in which data is distributed in multiple physical locations, but logically regarded as a single database; these physical locations can be different computers in the same local area network or different geographical locations in a wide area network; the purpose of the design of a distributed database is to improve the speed, reliability and availability of data access, while supporting large-scale data processing and high concurrent access.

[0003] The data in the distributed database needs to be processed evenly to achieve the above-mentioned effects of the distributed database, such as the support for high concurrent access, etc. However, the effect of the balanced processing of data in the related technology is average. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a data management method and device, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data management method is proposed, the method comprising:

[0007] For each balancing group in at least one balancing group formed by multiple data units of the data to be managed, construct distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, wherein the distribution information includes at least one balancing unit, the balancing unit includes at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets, and the data unit includes a partition of a non-partitioned table and / or a partitioned table;

[0008] For each balancing group, based on the distribution information of the balancing group on each log stream of the data system, balancing processing is performed on multiple data units in the balancing group, so that the data units of the balancing group on different log streams meet the preset balancing requirements;

[0009] Based on the distribution information of each balancing group on each log stream of the data system, the data units on all log streams of the data system are balanced so that the data units on different log streams of the data system meet the preset balancing requirements.

[0010] According to a second aspect of one or more embodiments of this specification, a data management device is provided, the device comprising:

[0011] A construction module is used to construct, for each balancing group in at least one balancing group formed by multiple data units of the data to be managed, distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, wherein the distribution information includes at least one balancing unit, the balancing unit includes at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets, and the data unit includes a partition of a non-partitioned table and / or a partitioned table;

[0012] An intra-group balancing module is used to perform balancing processing on multiple data units in each balancing group based on the distribution information of the balancing group on each log stream of the data system, so that the data units of the balancing group on different log streams meet the preset balancing requirements;

[0013] The inter-group balancing module is used to balance the data units on all log streams of the data system based on the distribution information of each balancing group on each log stream of the data system, so that the data units on different log streams of the data system meet the preset balancing requirements.

[0014] According to a third aspect of one or more embodiments of the present specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0015] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] The processor implements the method as described in the first aspect by running the executable instructions.

[0019] According to a fifth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0020] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0021] The data management method provided in the embodiment of this specification first constructs the distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides; then, based on the distribution information of the balancing group on each log stream of the data system, performs balancing processing on multiple data units in the balancing group, so that the data units of the balancing group on different log streams meet the preset balancing requirements; finally, based on the distribution information of each balancing group on each log stream of the data system, performs balancing processing on the data units on all log streams of the data system, so that the data units on different log streams of the data system meet the preset balancing requirements. Since the distribution information can aggregate or disperse adjacent data units in the balancing group through at least one dimension in the balancing unit and the log stream bucket, the distribution information of the balancing group can be constructed based on information such as the category of the data unit in the balancing group, and then, based on the balancing processing principle, the balancing processing of the data units in the balancing group and the balancing processing of the data units between the balancing groups are performed in sequence, so that the adjacent data units in the balancing group are dispersed in different log streams and the difference in the number of data units on different log streams meets the preset requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the architecture of a distributed database provided by an exemplary embodiment.

[0023] Figure 2 is a flow chart of a data management method provided by an exemplary embodiment.

[0024] Figure 3 is a schematic diagram of distribution information provided by an exemplary embodiment.

[0025] Figure 4 It is a schematic diagram of distribution information of a balancing group composed of multiple non-partition tables provided by an exemplary embodiment.

[0026] Figure 5 It is a schematic diagram of distribution information of a balancing group composed of multiple partitions of a partition table provided by an exemplary embodiment.

[0027] Figure 6 The diagram is a diagram showing the effect of balancing processing of a balancing group composed of multiple non-partition tables provided by an exemplary embodiment.

[0028] Figure 7 The diagram is a diagram showing the effect of balancing processing of a balancing group composed of multiple partitions of a partition table provided by an exemplary embodiment.

[0029] Figure 8It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0030] Fig. 9 is a block diagram of a data management device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0032] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0033] Next, some concepts involved in this specification are explained.

[0034] Partitioning: Divide the database table into multiple groups of data according to specified rules. Each group of data is called a partition.

[0035] Log stream: An organizational structure of logs in a distributed database, which contains several partitions.

[0036] Transfer: Migrate from one log stream to another log stream in partitions.

[0037] Balance group: describes the relationship between partitions in a distributed database. All non-partitioned tables under a tenant form a balance group, all first-level partitions of a first-level partition table form a balance group, and all second-level partitions under each first-level partition of a second-level partition table form a balance group. During the load balancing process, the partitions in each balance group are scattered and distributed on different log streams.

[0038] In the related art, when a distributed database performs data balancing, the number of partitions or non-partitioned tables on different log streams is relatively balanced, but it cannot guarantee that adjacent partitions or non-partitioned tables are scattered, that is, it cannot avoid adjacent non-partitioned tables (such as non-partitioned tables under the same database of tenants) from being aggregated in one log stream, nor can it avoid adjacent partitions (such as adjacent partitions in the partitioned table of the tenant) from being aggregated in one log stream, which causes access hot spots in some log streams, resulting in a decline in system performance.

[0039] Based on the above technical problems, at least one embodiment of the present specification provides a data management method, which aims to enable a distributed database to break up adjacent partitions or non-partition tables on the basis of a balanced number of log streams when performing data balancing (such as partition balancing), thereby avoiding access hot spots caused by adjacent partitions or non-partition tables being aggregated in one log stream.

[0040] Exemplarily, the method can be applied to Figure 1 The distributed database shown.

[0041] Please refer to the attached Figure 2 , which exemplarily shows a flow chart of a data management method, including steps S201 to S203.

[0042] In step S201, for each balancing group in at least one balancing group formed by multiple data units of the data to be managed, distribution information of the balancing group on each log stream of the data system is constructed based on the log stream where each data unit in the balancing group currently resides.

[0043] The distribution information includes at least one balancing unit, the balancing unit includes at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, and adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets. For example, if the data system has n log streams, this step can construct n distribution information corresponding to the n log streams for each balancing group. Each distribution information is attached. Figure 3 The tree structure shown by way of example has a log stream as the root node, a balancing unit as the intermediate node, and a log stream bucket as the leaf node; therefore, the distribution information can also be figuratively called a distribution tree. If a balancing group has several data units on a log stream, each data unit can be added to the log stream bucket in the distribution information corresponding to the log stream of the balancing group.

[0044] The data system can be shared by multiple tenants, and the data of different tenants are isolated from each other at the physical and / or logical levels. The data of tenants can be divided into multiple databases, and the data in different databases under the tenants are isolated from each other at the physical and / or logical levels. For example, the data to be managed mentioned in this step can be the data of a tenant in the data system.

[0045] The data unit includes non-partitioned tables and / or partitions of partitioned tables, that is, the data to be managed may contain a balancing group formed by multiple non-partitioned tables and / or a balancing group formed by multiple partitions of a partitioned table, for example, a balancing group formed by all first-level partitions of a first-level partitioned table, or a balancing group formed by all second-level partitions under each first-level partition table of a second-level partitioned table.

[0046] For a balancing group formed by multiple non-partitioned tables, adjacent data in the balancing group refers to non-partitioned tables belonging to the same database. For example, a tenant has two databases, Database 1 and Database 2. Database 1 contains non-partitioned tables T1, T2, and T3, and Database T2 contains non-partitioned tables T4, T5, and T6. Then, non-partitioned tables T1, T2, and T3 are adjacent data units, and non-partitioned tables T4, T5, and T6 are adjacent data units.

[0047] For example, for a balancing group formed by multiple non-partitioned tables, this step can construct the distribution information of the balancing group on each log stream in the following manner:

[0048] First, for each log stream of the data system, multiple non-partition tables of the balancing group on the log stream are grouped according to the database to which they belong, and each obtained group of non-partition tables is used as a balancing unit.

[0049] Next, for each obtained balancing unit, a log stream bucket is constructed in the balancing unit, and the non-partitioned table in the balancing unit is stored in the log stream bucket.

[0050] In other words, please refer to the attached Figure 4 For a balancing group formed by multiple non-partitioned tables, the distribution information on each log stream is to split the non-partitioned tables of different databases at the balancing unit level, so that the non-partitioned tables under each database are gathered in a unique log stream bucket under a balancing unit.

[0051] All partitions of a partition table are in the same database. For a balancing group formed by multiple partitions of a partition table, adjacent data units in the balancing group refer to adjacent partitions. For example, if a tenant's database 1 contains four first-level partitions P0, P1, P2, and P3 of a first-level partition table T1, then P0 and P1 are adjacent partitions, P1 and P2 are adjacent partitions, and P2 and P3 are adjacent partitions.

[0052] For example, for a balancing group formed by multiple partitions of a partition table, this step can construct the distribution information of the balancing group on each log stream in the following manner:

[0053] First, for each log stream of the data system, a balancing unit is constructed and all partitions of the balancing group on the log stream are stored in the balancing unit.

[0054] Next, multiple log stream buckets are constructed in the constructed balancing unit, and all partitions in the constructed balancing unit are stored in the constructed multiple log stream buckets so that adjacent partitions are stored in different log stream buckets, wherein the number of the constructed multiple log stream buckets is the same as the number of all log streams of the data system.

[0055] In other words, please refer to the attached Figure 5 For a balancing group formed by multiple non-partitioned tables, its distribution information on each log stream is to split adjacent partitions at the log stream bucket level, so that adjacent partitions are distributed in different log stream buckets. For example, a data system has three log streams, and a balancing group has nine first-level partitions P1, P2, P3, P4, P5, P6, P7, P8, and P9 of a first-level partition table on a certain log stream. Then the distribution information of the balancing group on the log stream may include three log stream buckets, namely log stream bucket 1, log stream bucket 2, and log stream bucket 3 under a balancing unit. Log stream bucket 1 contains three non-adjacent partitions P1, P4, and P7, log stream bucket 2 contains three non-adjacent partitions P2, P5, and P8, and log stream bucket 3 contains three non-adjacent partitions P3, P6, and P9.

[0056] It should be understood that when balancing data units, this method aims to break up non-partitioned tables in the same database into different log streams and break up adjacent partitions into different log streams, and the distribution information constructed in this step can aggregate non-partitioned tables in the same database at the balancing unit level and split partitions at the log stream bucket level. In subsequent steps, when balancing data, different balancing strategies can be adopted at the balancing unit level and the log stream bucket level, respectively, so as to achieve the breaking up of adjacent data units in the balancing groups formed by multiple non-grouped tables and the balancing groups formed by multiple partitions, respectively. The specific balancing strategies will be introduced in detail in the subsequent steps and will not be described here.

[0057] In step S202, for each balancing group, based on the distribution information of the balancing group on each log stream of the data system, balancing processing is performed on multiple data units in the balancing group so that the data units of the balancing group on different log streams meet the preset balancing requirements.

[0058] In one possible embodiment, for all data units in the balancing group, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than a preset threshold.

[0059] Among them, the preferred preset threshold is 1.

[0060] That is to say, when performing intra-group balancing, iteration is performed with the number of data units on any two log streams being no greater than a preset threshold as a convergence condition. For example, when the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, data unit transfer can be performed in the following manner to achieve convergence:

[0061] First, the minimum value of the difference between the number of data units on the log stream with the most data units and the log stream mean, and the difference between the log stream mean and the number of data units on the log stream with the least data units, is determined as the first transfer quantity, wherein the log stream mean is the ratio of the number of data units in the balancing group to the number of log streams of the data system.

[0062] That is, the first transfer quantity X is determined according to the following formula:

[0063] X= min(part_num_max - part_num / ls_num, part_num / ls_num - part_num_min)

[0064] In the above formula, part_num_max is the number of data units on the log stream with the most data units, part_num_min is the number of data units on the log stream with the least data units, part_num is the number of data units in the balancing group, and ls_num is the number of log streams of the data system.

[0065] Next, the first transfer quantity of data units on the log stream with the most data units is transferred to the log stream with the least data units.

[0066] For example, each time a data unit on the log stream with the most data units is transferred to the log stream with the least data units in the following manner until the number of transferred data units reaches the first transfer number:

[0067] The balancing unit with the smallest first balancing ratio on the log stream with the most data units is determined as the source balancing unit, the balancing unit corresponding to the source balancing unit on the log stream with the least data units is determined as the destination balancing unit, and a data unit in the source balancing unit is transferred to the destination balancing unit, wherein the first balancing ratio is the ratio of the number of data units in the balancing unit on the log stream with the most data units to the number of data units in the corresponding balancing unit on the log stream with the least data units.

[0068] That is, each time the source equalization unit is determined according to the following formula :

[0069]

[0070] In the above formula, is the number of data units in the balancing unit on the log stream with the most data units, is the number of data units in the balancing unit on the log stream with the least data units.

[0071] For another example, one data unit in the source balancing unit is transferred to the destination balancing unit each time in the following manner:

[0072] Transfer a data unit in the log stream bucket with the largest second balancing ratio in the source balancing unit to the corresponding log stream bucket in the destination balancing unit, wherein the second balancing ratio is the ratio of the number of data units in the log stream bucket in the source balancing unit to the number of data units in the corresponding log stream bucket in the destination balancing unit.

[0073] That is, determine which log stream bucket to transfer according to the following formula: Data unit within:

[0074]

[0075] In the above formula, is the number of data units in the log stream bucket in the source balancing unit, Balances the number of data units in the log stream bucket within the unit for the stated purpose.

[0076] Taking a balancing group formed by multiple non-partitioned tables as an example, the first transfer quantity can be determined in accordance with the method of this embodiment during the intra-group balancing process, and the source balancing unit can be determined in accordance with the method in the above example each time a non-partitioned table is transferred, and a non-partitioned table in the log stream bucket under the source balancing unit is transferred from the log stream with the most non-partitioned tables to the log stream with the least non-partitioned tables. Figure 6 The intra-group balance shown in the figure includes 9 non-partitioned tables, DB1:T1, DB1:T2, DB1:T3, DB2:T1, DB2:T2, DB2:T3, DB3:T1, DB3:T2, DB3:T3. DB1:T1, DB1:T2, DB1:T3 belong to database DB1, DB2:T1, DB2:T2, DB2:T3 belong to database DB2, DB3:T1, DB3:T2, DB3:T3 belong to database DB3, and the data system There are three log streams, LS1, LS2, and LS3. Before balancing, the nine non-partitioned tables are all on log stream LS1. After balancing, DB1:T1, DB2:T1, and DB3:T1 are on log stream LS1, DB1:T2, DB2:T2, and DB3:T2 are on log stream LS2, and DB1:T3, DB2:T3, and DB3:T3 are on log stream LS3. That is, the nine non-partitioned tables are evenly distributed on three log streams, and the non-partitioned tables under the same database are scattered on different log streams.

[0077] Taking the balancing group formed by multiple partitions of the partition table as an example, the first transfer quantity can be determined in the manner of this embodiment during the intra-group balancing process, and the source balancing unit can be determined in the manner of the above example each time a partition is transferred, and a partition in the log stream bucket with the largest second transfer ratio under the source balancing unit is transferred from the log stream with the most partitions to the log stream with the least partitions. Figure 7 As shown in the intra-group balancing, the balancing group includes 9 partitions, namely T1:P1, T1:P2, T1:P3, T1:P4, T1:P5, T1:P6, T1:P7, T1:P8, and T1:P9. The data system has 3 log streams, namely LS1, LS2, and LS3. Before balancing, the 9 partitions are all on the log stream LS1. After balancing, T1:P1, T1:P4, and T1:P7 are on the log stream LS1, T1:P2, T1:P5, and T1:P8 are on the log stream LS2, and T1:P3, T1:P6, and T1:P9 are on the log stream LS3. That is, the 9 partitions are evenly distributed on the 3 log streams, and adjacent partitions are scattered on different log streams.

[0078] In step S203, based on the distribution information of each balancing group on each log stream of the data system, balancing processing is performed on the data units on all log streams of the data system, so that the data units on different log streams of the data system meet the preset balancing requirements.

[0079] In one possible embodiment, for all data units in a data system, if the difference between the number of data units on a log stream with the most data units and the number of data units on a log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than a preset threshold.

[0080] Among them, the preferred preset threshold is 1.

[0081] That is to say, when the inter-group balancing process is performed, the number of data units on any two log streams is not greater than the preset threshold as the convergence condition for iteration. For example, when the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than the preset threshold, the data unit transfer can be performed in the following manner to achieve convergence:

[0082] First, the minimum value of the difference between the number of data units on the log stream with the most data units and the log stream mean, and the difference between the log stream mean and the number of data units on the log stream with the least data units, is determined as the first transfer quantity, wherein the log stream mean is the ratio of the number of data units in the balancing group to the number of log streams of the data system.

[0083] That is, the first transfer quantity X is determined according to the following formula:

[0084] X= min(part_num_max - part_num / ls_num, part_num / ls_num - part_num_min)

[0085] In the above formula, part_num_max is the number of data units on the log stream with the most data units, part_num_min is the number of data units on the log stream with the least data units, part_num is the number of data units in the data to be managed, that is, the sum of the number of data units in all balancing groups, and ls_num is the number of log streams of the data system.

[0086] Next, the first transfer quantity of data units on the log stream with the most data units is transferred to the log stream with the least data units.

[0087] For example, each time a data unit on the log stream with the most data units is transferred to the log stream with the least data units in the following manner until the number of transferred data units reaches the first transfer number:

[0088] Traverse each balancing group in which data units exist on both the log stream with the most data units and the log stream with the least data units. If the number of data units on the log stream with the most data units of the traversed balancing group is greater than the number of data units on the log stream with the least data units, transfer one data unit of the traversed balancing group on the log stream with the most data units to the log stream with the least data units, until the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is no greater than a preset threshold.

[0089] That is, if the traversed balancing group meets the following conditions, a data unit in the balancing group is transferred from the log stream with the most data units to the log stream with the least data units:

[0090] bg_part_num_ls_max>bg_part_num_ls_min

[0091] In the above formula, bg_part_num_ls_max is the number of data units of the traversed balancing group on ls_max, ls_max is the number of data units on the log stream with the most data units, bg_part_num_ls_min is the number of data units of the traversed balancing group on ls_min, ls_min is the number of data units on the log stream with the least data units.

[0092] For another example, each time a data unit of the traversed balancing group that meets the above conditions is transferred from the log stream with the most data units to the log stream with the least data units in the following manner:

[0093] The balancing unit with the smallest first balancing ratio on the log stream with the most data units is determined as the source balancing unit, the balancing unit corresponding to the source balancing unit on the log stream with the least data units is determined as the destination balancing unit, and a data unit in the source balancing unit is transferred to the destination balancing unit, wherein the first balancing ratio is the ratio of the number of data units in the balancing unit on the log stream with the most data units to the number of data units in the corresponding balancing unit on the log stream with the least data units.

[0094] That is, each time the source equalization unit is determined according to the following formula :

[0095]

[0096] In the above formula, is the number of data units in the balancing unit on the log stream with the most data units, is the number of data units in the balancing unit on the log stream with the least data units.

[0097] For another example, one data unit in the source balancing unit is transferred to the destination balancing unit each time in the following manner:

[0098] Transfer a data unit in the log stream bucket with the largest second balancing ratio in the source balancing unit to the corresponding log stream bucket in the destination balancing unit, wherein the second balancing ratio is the ratio of the number of data units in the log stream bucket in the source balancing unit to the number of data units in the corresponding log stream bucket in the destination balancing unit.

[0099] That is, determine which log stream bucket to transfer according to the following formula: Data unit within:

[0100]

[0101] In the above formula, is the number of data units in the log stream bucket in the source balancing unit, Balances the number of data units in the log stream bucket within the unit for the stated purpose.

[0102] It should be understood that this step may further include the following execution conditions: if the difference in the number of data units on any two log streams of each balancing group is not greater than a preset threshold, then based on the distribution information of each balancing group on each log stream of the data system, the data units on all log streams of the data system are balanced so that the difference in the number of data units on any two log streams of the data system meets the preset requirements. That is, after each balancing group completes the intra-group balancing according to step S202, it then completes the inter-group balancing according to this step, or it is called the balancing within the entire data to be managed.

[0103] Figure 8 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 8At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0104] Please refer to Fig. 9 , the data management device can be used for Figure 8 The data management device may include:

[0105] A construction module 901 is used to construct, for each balancing group in at least one balancing group formed by multiple data units of the data to be managed, distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, wherein the distribution information includes at least one balancing unit, the balancing unit includes at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets, and the data unit includes a partition of a non-partitioned table and / or a partitioned table;

[0106] The intra-group balancing module 902 is used to perform balancing processing on multiple data units in each balancing group based on the distribution information of the balancing group on each log stream of the data system, so that the data units of the balancing group on different log streams meet the preset balancing requirements;

[0107] The inter-group balancing module 903 is used to balance the data units on all log streams of the data system based on the distribution information of each balancing group on each log stream of the data system, so that the data units on different log streams of the data system meet the preset balancing requirements.

[0108] In a possible embodiment of the present specification, for a balancing group formed by multiple non-partition tables, the building module is used to:

[0109] For each log stream of the data system, the balancing group groups multiple non-partition tables on the log stream according to the database to which it belongs, and each group of non-partition tables obtained is used as a balancing unit;

[0110] For each obtained balancing unit, a log stream bucket is constructed in the balancing unit, and the non-partitioned table in the balancing unit is stored in the log stream bucket.

[0111] In a possible embodiment of the present specification, for a balancing group formed by multiple partitions of a partition table, the construction module is used to:

[0112] For each log stream of the data system, a balancing unit is constructed and all partitions of the balancing group on the log stream are stored in the balancing unit;

[0113] Multiple log stream buckets are constructed in the constructed balancing unit, and all partitions in the constructed balancing unit are stored in the constructed multiple log stream buckets so that adjacent partitions are stored in different log stream buckets, wherein the number of the constructed multiple log stream buckets is the same as the number of all log streams of the data system.

[0114] In a possible embodiment of the present specification, the intra-group balancing module is used to:

[0115] For all data units in the balancing group, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than the preset threshold.

[0116] In a possible embodiment of the present specification, the inter-group balancing module is used to:

[0117] For all data units in the data system, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than the preset threshold.

[0118] In a possible embodiment of the present specification, the inter-group balancing module is used to transfer at least one data unit on the log stream with the most data units to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and when the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than a preset threshold, to:

[0119] Traverse each balancing group in which data units exist on both the log stream with the most data units and the log stream with the least data units. If the number of data units on the log stream with the most data units of the traversed balancing group is greater than the number of data units on the log stream with the least data units, transfer one data unit of the traversed balancing group on the log stream with the most data units to the log stream with the least data units, until the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is no greater than a preset threshold.

[0120] In a possible embodiment of the present specification, the intra-group balancing module or the inter-group balancing module is used to transfer at least one data unit on the log stream with the most data units to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and when the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than a preset threshold, to:

[0121] Determine the minimum value of the difference between the number of data units on the log stream with the most data units and the log stream mean, and the difference between the log stream mean and the number of data units on the log stream with the least data units as the first transfer quantity, wherein the log stream mean is the ratio of the number of data units in the balancing group to the number of log streams of the data system;

[0122] The first transfer quantity of data units on the log stream with the most data units is transferred to the log stream with the least data units.

[0123] In a possible embodiment of the present specification, when the intra-group balancing module or the inter-group balancing module is used to transfer the first transfer quantity of data units on the log stream with the most data units to the log stream with the least data units, it is used to:

[0124] Each time, a data unit on the log stream with the most data units is transferred to the log stream with the least data units in the following manner until the number of transferred data units reaches the first transfer number:

[0125] The balancing unit with the smallest first balancing ratio on the log stream with the most data units is determined as the source balancing unit, the balancing unit corresponding to the source balancing unit on the log stream with the least data units is determined as the destination balancing unit, and a data unit in the source balancing unit is transferred to the destination balancing unit, wherein the first balancing ratio is the ratio of the number of data units in the balancing unit on the log stream with the most data units to the number of data units in the corresponding balancing unit on the log stream with the least data units.

[0126] In a possible embodiment of the present specification, when the intra-group balancing module or the inter-group balancing module is used to transfer a data unit in the source balancing unit to the destination balancing unit, it is used to:

[0127] Transfer a data unit in the log stream bucket with the largest second balancing ratio in the source balancing unit to the corresponding log stream bucket in the destination balancing unit, wherein the second balancing ratio is the ratio of the number of data units in the log stream bucket in the source balancing unit to the number of data units in the corresponding log stream bucket in the destination balancing unit.

[0128] In a possible embodiment of the present specification, the inter-group balancing module is used to:

[0129] If the difference in the number of data units on any two log streams of each balancing group is not greater than the preset threshold, then based on the distribution information of each balancing group on each log stream of the data system, the data units on all log streams of the data system are balanced so that the difference in the number of data units on any two log streams of the data system meets the preset requirements.

[0130] One or more embodiments of the present specification also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in any of the above embodiments.

[0131] One or more embodiments of the present specification further provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method provided in any of the above embodiments.

[0132] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0133] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0135] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0137] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0138] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0139] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0140] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0141] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data management method, the method comprising: For each balancing group in at least one balancing group formed by multiple data units of the data to be managed, construct distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, wherein the distribution information includes at least one balancing unit, the balancing unit includes at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets, and the data unit includes a partition of a non-partitioned table and / or a partitioned table; For each balancing group, based on the distribution information of the balancing group on each log stream of the data system, balancing processing is performed on multiple data units in the balancing group, so that the data units of the balancing group on different log streams meet the preset balancing requirements; Based on the distribution information of each balancing group on each log stream of the data system, the data units on all log streams of the data system are balanced so that the data units on different log streams of the data system meet the preset balancing requirements; For a balancing group formed by multiple partitions of a partition table, constructing distribution information of the balancing group on each log stream of a data system based on the log stream where each data unit in the balancing group currently resides includes: For each log stream of the data system, a balancing unit is constructed and all partitions of the balancing group on the log stream are stored in the balancing unit; Multiple log stream buckets are constructed in the constructed balancing unit, and all partitions in the constructed balancing unit are stored in the constructed multiple log stream buckets so that adjacent partitions are stored in different log stream buckets, wherein the number of the constructed multiple log stream buckets is the same as the number of all log streams of the data system.

2. The data management method according to claim 1, for a balancing group formed by multiple non-partitioned tables, constructing distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, comprises: For each log stream of the data system, the balancing group groups multiple non-partition tables on the log stream according to the database to which it belongs, and each group of non-partition tables obtained is used as a balancing unit; For each obtained balancing unit, a log stream bucket is constructed in the balancing unit, and the non-partitioned table in the balancing unit is stored in the log stream bucket.

3. The data management method according to claim 1, wherein the step of performing balancing processing on multiple data units in the balancing group based on the distribution information of the balancing group on each log stream of the data system so that the data units of the balancing group on different log streams meet the preset balancing requirements comprises: For all data units in the balancing group, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than the preset threshold.

4. The data management method according to claim 1, wherein the step of performing balancing processing on data units on all log streams of the data system based on the distribution information of each balancing group on each log stream of the data system so that data units on different log streams of the data system meet preset requirements comprises: For all data units in the data system, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than the preset threshold.

5. The data management method according to claim 4, wherein the transferring at least one data unit on the log stream with the most data units to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and making the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units not greater than a preset threshold, comprises: Traverse each balancing group in which data units exist on both the log stream with the most data units and the log stream with the least data units. If the number of data units on the log stream with the most data units of the traversed balancing group is greater than the number of data units on the log stream with the least data units, transfer one data unit of the traversed balancing group on the log stream with the most data units to the log stream with the least data units, until the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is no greater than a preset threshold.

6. The data management method according to claim 3 or 4, wherein the transferring of at least one data unit on the log stream with the most data units to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and making the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units not greater than a preset threshold, comprises: Determine the minimum value of the difference between the number of data units on the log stream with the most data units and the log stream mean, and the difference between the log stream mean and the number of data units on the log stream with the least data units as the first transfer quantity, wherein the log stream mean is the ratio of the number of data units in the balancing group to the number of log streams of the data system; The first transfer quantity of data units on the log stream with the most data units is transferred to the log stream with the least data units.

7. The data management method according to claim 6, wherein the step of transferring the first transfer quantity of data units from the log stream with the most data units to the log stream with the least data units comprises: Each time, a data unit on the log stream with the most data units is transferred to the log stream with the least data units in the following manner until the number of transferred data units reaches the first transfer number: The balancing unit with the smallest first balancing ratio on the log stream with the most data units is determined as the source balancing unit, the balancing unit corresponding to the source balancing unit on the log stream with the least data units is determined as the destination balancing unit, and a data unit in the source balancing unit is transferred to the destination balancing unit, wherein the first balancing ratio is the ratio of the number of data units in the balancing unit on the log stream with the most data units to the number of data units in the corresponding balancing unit on the log stream with the least data units.

8. The data management method according to claim 7, wherein the step of transferring a data unit in the source balancing unit to the destination balancing unit comprises: Transfer a data unit in the log stream bucket with the largest second balancing ratio in the source balancing unit to the corresponding log stream bucket in the destination balancing unit, wherein the second balancing ratio is the ratio of the number of data units in the log stream bucket in the source balancing unit to the number of data units in the corresponding log stream bucket in the destination balancing unit.

9. The data management method according to claim 1, wherein the step of performing balancing processing on data units on all log streams of the data system based on the distribution information of each balancing group on each log stream of the data system so that data units on different log streams of the data system meet preset requirements comprises: If the difference in the number of data units on any two log streams of each balancing group is not greater than the preset threshold, then based on the distribution information of each balancing group on each log stream of the data system, the data units on all log streams of the data system are balanced so that the difference in the number of data units on any two log streams of the data system meets the preset requirements.

10. A data management device, comprising: A construction module is used to construct, for each balancing group in at least one balancing group formed by multiple data units of the data to be managed, distribution information of the balancing group on each log stream of the data system based on the log stream where each data unit in the balancing group currently resides, wherein the distribution information includes at least one balancing unit, the balancing unit includes at least one log stream bucket, the log stream bucket is empty or contains at least one data unit of the balancing group, adjacent data units in the balancing group are distributed in the same balancing unit or different log stream buckets, and the data unit includes a partition of a non-partitioned table and / or a partitioned table; An intra-group balancing module is used to perform balancing processing on multiple data units in each balancing group based on the distribution information of the balancing group on each log stream of the data system, so that the data units of the balancing group on different log streams meet the preset balancing requirements; An inter-group balancing module is used to balance the data units on all log streams of the data system based on the distribution information of each balancing group on each log stream of the data system, so that the data units on different log streams of the data system meet the preset balancing requirements; For a balancing group formed by multiple partitions of a partition table, the building module is used to: For each log stream of the data system, a balancing unit is constructed and all partitions of the balancing group on the log stream are stored in the balancing unit; Multiple log stream buckets are constructed in the constructed balancing unit, and all partitions in the constructed balancing unit are stored in the constructed multiple log stream buckets so that adjacent partitions are stored in different log stream buckets, wherein the number of the constructed multiple log stream buckets is the same as the number of all log streams of the data system.

11. The data management device according to claim 10, wherein for a balancing group formed by a plurality of non-partition tables, the construction module is used to: For each log stream of the data system, the balancing group groups multiple non-partition tables on the log stream according to the database to which it belongs, and each group of non-partition tables obtained is used as a balancing unit; For each obtained balancing unit, a log stream bucket is constructed in the balancing unit, and the non-partitioned table in the balancing unit is stored in the log stream bucket.

12. The data management device according to claim 10, wherein the intra-group balancing module is used for: For all data units in the balancing group, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than the preset threshold.

13. The data management device according to claim 10, wherein the inter-group balancing module is used for: For all data units in the data system, if the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is greater than a preset threshold, at least one data unit on the log stream with the most data units is transferred to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and ensure that the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than the preset threshold.

14. The data management device according to claim 13, wherein the inter-group balancing module is used to transfer at least one data unit on the log stream with the most data units to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and when the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than a preset threshold, to: Traverse each balancing group in which data units exist on both the log stream with the most data units and the log stream with the least data units. If the number of data units on the log stream with the most data units of the traversed balancing group is greater than the number of data units on the log stream with the least data units, transfer one data unit of the traversed balancing group on the log stream with the most data units to the log stream with the least data units, until the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is no greater than a preset threshold.

15. The data management device according to claim 12 or 13, wherein the intra-group balancing module or the inter-group balancing module is used to transfer at least one data unit on the log stream with the most data units to the log stream with the least data units to split adjacent data units on the log stream with the most data units, and when the difference between the number of data units on the log stream with the most data units and the number of data units on the log stream with the least data units is not greater than a preset threshold, to: The minimum value of the difference between the number of data units on the log stream with the most data units and the log stream mean, and the difference between the log stream mean and the number of data units on the log stream with the least data units is determined as the first transfer quantity, wherein, The log stream mean is the ratio of the number of data units in the balancing group to the number of log streams of the data system; The first transfer quantity of data units on the log stream with the most data units is transferred to the log stream with the least data units.

16. The data management device according to claim 15, wherein when the intra-group balancing module or the inter-group balancing module is used to transfer the first transfer quantity of data units on the log stream with the most data units to the log stream with the least data units, it is used to: Each time, a data unit on the log stream with the most data units is transferred to the log stream with the least data units in the following manner until the number of transferred data units reaches the first transfer number: The balancing unit with the smallest first balancing ratio on the log stream with the most data units is determined as the source balancing unit, the balancing unit corresponding to the source balancing unit on the log stream with the least data units is determined as the destination balancing unit, and a data unit in the source balancing unit is transferred to the destination balancing unit, wherein: The first balancing ratio is a ratio of the number of data units in the balancing unit on the log stream with the most data units to the number of data units in the corresponding balancing unit on the log stream with the least data units.

17. The data management device according to claim 16, wherein when the intra-group balancing module or the inter-group balancing module is used to transfer a data unit in the source balancing unit to the destination balancing unit, it is used to: A data unit in the log stream bucket with the largest second balancing ratio in the source balancing unit is transferred to the corresponding log stream bucket in the destination balancing unit, wherein: The second balancing ratio is a ratio of the number of data units in the log stream bucket in the source balancing unit to the number of data units in the corresponding log stream bucket in the destination balancing unit.

18. A computer program product, comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

19. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

20. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images