Data unit management method and apparatus, electronic device, and storage medium

Abstract

This disclosure provides a data unit management method applicable to big data technology, finance, or other fields. The method includes: in response to a first data unit's data volume satisfying a preset splitting threshold condition, obtaining a first current splitting point array corresponding to the first data unit; generating an initial splitting point array based on a data volume threshold represented by the preset splitting threshold condition and a preset number of splits, wherein the initial splitting point array includes values ​​of multiple initial splitting points; adjusting the value of each initial splitting point in the initial splitting point array using the values ​​of the first current splitting points in the first current splitting point array to obtain a first target splitting point array; and splitting the first data unit into multiple second data units based on the first target splitting point array. Furthermore, this disclosure also provides a data unit management device, electronic device, and storage medium.

Description

Technical Field

[0001] This disclosure relates to the fields of big data technology and finance, and more specifically, to a data unit management method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] Splitting and merging data units is one of the core functions of HBase (Hadoop database, a non-relational distributed database) and is the foundation for achieving distributed scalability. By splitting and merging data units, a balance can be maintained between data reading efficiency and the difficulty of data unit management.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problem in the related technology: during the merging and splitting process, the splitting point of the data unit does not match the expected splitting point. Summary of the Invention

[0004] In view of the above, this disclosure provides a data unit management method, a data unit management device, an electronic device, a readable storage medium, and a computer program product.

[0005] One aspect of this disclosure provides a data unit management method, comprising: in response to a first data unit's data volume satisfying a preset splitting threshold condition, obtaining a first current splitting point array corresponding to the first data unit; generating an initial splitting point array based on a data volume threshold represented by the preset splitting threshold condition and a preset splitting quantity, wherein the initial splitting point array includes values ​​of multiple initial splitting points; adjusting the value of each initial splitting point in the initial splitting point array using the values ​​of the first current splitting points in the first current splitting point array to obtain a first target splitting point array; and splitting the first data unit into multiple second data units based on the first target splitting point array.

[0006] According to embodiments of this disclosure, the method of adjusting the value of each initial split point in the initial split point array using the value of the first current split point in the first current split point array to obtain a first target split point array includes: for each initial split point, generating a first value interval corresponding to the initial split point based on the value of the initial split point and a first preset interval value; determining a first target split point corresponding to the initial split point based on the value of at least one first target current split point in the first current split point if the value of at least one first target current split point is within the first value interval; determining a first target split point corresponding to the initial split point based on the value of the initial split point if the values ​​of all first current split points are outside the first value interval; and obtaining the first target split point array based on the values ​​of multiple first target split points.

[0007] According to an embodiment of this disclosure, the method further includes: for each initial split point, if the value of the first current split point is outside the first value range, adding the value of the initial split point to the first current split point array to obtain a second current split point array.

[0008] According to an embodiment of this disclosure, the method further includes: in response to a merge instruction, adjusting the value of each initial split point in the initial split point array using the value of the second current split point in the second current split point array to obtain a second target split point array; and merging multiple second data units into a third data unit based on the second target split point array.

[0009] According to embodiments of this disclosure, the method of adjusting the value of each initial split point in the initial split point array using the values ​​of the second current split points in the second current split point array to obtain a second target split point array includes: for each initial split point, generating a second value interval corresponding to the initial split point based on the value of the initial split point and a second preset interval value; determining a second target split point corresponding to the initial split point based on the value of at least one second target current split point in the second current split point if the value of at least one second target current split point is within the second value interval; adjusting the second preset interval value and generating an adjusted second value interval based on the adjusted second preset interval value, until at least one second target current split point in the second current split point is within the adjusted second value interval; and obtaining the second target split point array based on the values ​​of multiple second target split points.

[0010] According to an embodiment of this disclosure, a value interval corresponding to the initial split point is generated based on the value of the initial split point and a preset interval value, including: determining the lower boundary of the value interval based on the difference between the value of the initial split point and the preset interval value; and determining the upper boundary of the value interval based on the sum of the value of the initial split point and the preset interval value; wherein the preset interval value includes the first preset interval value or the second preset interval value, and the value interval includes the first value interval or the second value interval.

[0011] According to embodiments of this disclosure, determining a target split point corresponding to the initial split point based on the value of at least one of the aforementioned target current split points includes: calculating the difference between the value of at least one of the aforementioned target current split points and the value of the aforementioned initial split point; determining a target difference with the smallest absolute value from the at least one of the aforementioned differences; and determining a target split point corresponding to the aforementioned initial split point based on the value of the aforementioned target current split point corresponding to the aforementioned target difference; wherein the aforementioned target current split point includes the aforementioned first target current split point or the aforementioned second target current split point, and the aforementioned target split point includes the aforementioned first target split point or the aforementioned second target split point.

[0012] According to an embodiment of this disclosure, generating an initial split point array based on the data volume threshold and the preset number of splits, as indicated by the preset split threshold condition, includes: determining an interval width based on the data volume threshold and the preset number of splits; determining a start key and an end key based on the data volume threshold and the interval width, wherein the start key represents the value of the first initial split point in the initial split point array, and the end key represents the value of the last initial split point in the initial split point array; and determining the values ​​of other initial split points in the initial split point array based on the start key, the interval width, and the end key.

[0013] According to embodiments of this disclosure, the first preset interval value is represented as the product of the interval width and the first proportional coefficient; the second preset interval value is represented as the product of the interval width and the second proportional coefficient; adjusting the second preset interval value includes adjusting the second proportional coefficient.

[0014] Another aspect of this disclosure provides a data unit management device, comprising: an acquisition module, configured to acquire a first current split point array corresponding to the first data unit in response to a preset split threshold condition being met; a generation module, configured to generate an initial split point array based on a data volume threshold represented by the preset split threshold condition and a preset number of splits, wherein the initial split point array includes values ​​of a plurality of initial split points; a first adjustment module, configured to adjust the value of each initial split point in the initial split point array using the values ​​of the first current split points in the first current split point array to obtain a first target split point array; and a splitting module, configured to split the first data unit into a plurality of second data units based on the first target split point array.

[0015] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0016] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0017] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0018] According to embodiments of this disclosure, when splitting a data unit, an initial splitting point array is generated based on a preset splitting threshold condition and a preset splitting quantity. The values ​​of existing first current splitting points are then used to adjust each initial splitting point in the initial splitting point array to obtain a first target splitting point array. The first data unit is then split based on this first target splitting point array. Through these technical means, it can be ensured that the splitting point of the first data unit is consistent with the expected splitting point during the splitting process. This allows the splitting of the data unit to maintain pre-partitioning within a certain range, thus at least partially overcoming the technical problem in related technologies where the splitting point of the data unit does not match the expected splitting point during merging and splitting, thereby effectively reducing the management difficulty of the cluster. Attached Figure Description

[0019] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1The illustration schematically depicts an exemplary system architecture to which data unit management methods and apparatus can be applied according to embodiments of the present disclosure.

[0021] Figure 2 A flowchart illustrating the data unit splitting process in a data unit management method according to an embodiment of the present disclosure is shown.

[0022] Figure 3 A flowchart illustrating the data unit splitting process in a data unit management method according to another embodiment of the present disclosure is shown.

[0023] Figure 4 A flowchart illustrating the data unit merging process in a data unit management method according to an embodiment of the present disclosure is shown.

[0024] Figure 5 A flowchart illustrating the data unit merging process in a data unit management method according to another embodiment of the present disclosure is shown.

[0025] Figure 6 A block diagram of a data unit management apparatus according to an embodiment of the present disclosure is shown schematically.

[0026] Figure 7 A block diagram of an electronic device suitable for implementing a data unit management method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0031] In related technologies, splitting and merging data units in HBase typically involves using HBase's built-in split-merge strategy to merge data units in pairs or split them in two. Alternatively, a pre-split method can be used, where split points are defined during table creation and data units are split and merged using these predefined split points. However, in the actual merging and splitting process, the actual split points of data units often differ from the predefined split points, making the effectiveness of the pre-split method unpredictable.

[0032] In view of the above, embodiments of this disclosure provide a data unit management method, a data unit management device, an electronic device, a readable storage medium, and a computer program product. The method includes a data unit splitting process and a data unit merging process. The data unit splitting process includes: in response to a first data unit's data volume satisfying a preset splitting threshold condition, obtaining a first current splitting point array corresponding to the first data unit; generating an initial splitting point array based on a data volume threshold represented by the preset splitting threshold condition and a preset splitting quantity, wherein the initial splitting point array includes values ​​of multiple initial splitting points; adjusting the value of each initial splitting point in the initial splitting point array using the values ​​of the first current splitting points in the first current splitting point array to obtain a first target splitting point array; and splitting the first data unit into multiple second data units based on the first target splitting point array.

[0033] It should be noted that the data unit management method and apparatus determined in the embodiments of this disclosure can be used in the field of big data technology or the financial field, and can also be used in any field other than the field of big data technology and the financial field. The application field of the data unit management method and apparatus determined in the embodiments of this disclosure is not limited.

[0034] Figure 1 The illustration schematically depicts an exemplary system architecture to which the data unit management method and apparatus can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105.

[0036] Terminal devices 101, 102, and 103 can be various electronic devices that support data input and output, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Various client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software.

[0038] Network 104 is a medium used to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired and / or wireless communication links.

[0039] Server 105 can be a server that provides various services, such as a database server that provides storage services for data generated during the use of client applications on terminal devices 101, 102, and 103.

[0040] It should be noted that the data unit management method provided in this embodiment can generally be executed by the server 105. Accordingly, the data unit management device provided in this embodiment can generally be located in the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart illustrating the data unit splitting process in a data unit management method according to an embodiment of the present disclosure is shown.

[0043] like Figure 2 As shown, the data unit splitting process includes operations S201 to S204.

[0044] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0045] In operation S201, in response to the data volume of the first data unit meeting the preset splitting threshold condition, the array of the first current splitting points corresponding to the first data unit is obtained.

[0046] In operation S202, an initial split point array is generated based on the data volume threshold represented by the preset split threshold condition and the preset number of splits. The initial split point array includes the values ​​of multiple initial split points.

[0047] In operation S203, the values ​​of each initial split point in the initial split point array are adjusted using the values ​​of the first current split point in the first current split point array to obtain the first target split point array.

[0048] In operation S204, the first data unit is split into multiple second data units based on the first target split point array.

[0049] According to embodiments of this disclosure, the first data unit may be a data unit (Region) in HBase.

[0050] According to embodiments of this disclosure, the split points in the first current split point array may include split points generated based on HBase's built-in split-merge strategy, split points generated during previous split-merge processes, etc. A split point generated based on HBase's built-in split-merge strategy may be the first rowkey (primary key) of the most central block in the largest file of the largest store (column family) in the data unit.

[0051] According to embodiments of this disclosure, the preset splitting threshold condition can represent the maximum data volume of a set data unit, i.e., the data volume threshold. The data volume satisfying the preset splitting threshold condition can be defined as the data volume being greater than or equal to the data volume threshold.

[0052] According to embodiments of this disclosure, the preset number of splits can be a hyperparameter set by the operator based on the specific application scenario.

[0053] According to embodiments of this disclosure, the process of generating an initial split point array based on a data volume threshold represented by a preset split threshold condition and a preset split number can be a process of pre-partitioning the first data unit. By planning the initial split point array, it can be ensured that the sizes of the sub-data units into which the data unit is split are similar.

[0054] According to embodiments of this disclosure, the process of splitting a first data unit into multiple second data units based on a first target splitting point array can be divided into three stages: a prepare stage, an execute stage, and a rollback stage. In the prepare stage, several second data units are initialized in memory. In the execute stage, firstly, the state of the first data unit is changed to SPLITING; then, a temporary folder is created in the storage directory of the first data unit, and multiple subfolders are created within this temporary folder. Each subfolder can generate a reference file, which can point to the corresponding file in the first data unit; then, the files corresponding to the multiple subfolders are copied to the table name directory of the table corresponding to each second data unit, and the state of the second data unit is modified to complete the data unit splitting. If an exception occurs in the execute stage, the rollback stage can be executed to roll back the splitting operation.

[0055] According to embodiments of this disclosure, when splitting a data unit, an initial splitting point array is generated based on a preset splitting threshold condition and a preset splitting quantity. The values ​​of existing first current splitting points are then used to adjust each initial splitting point in the initial splitting point array to obtain a first target splitting point array. The first data unit is then split based on this first target splitting point array. Through these technical means, it can be ensured that the splitting point of the first data unit is consistent with the expected splitting point during the splitting process. This allows the splitting of the data unit to maintain pre-partitioning within a certain range, thus at least partially overcoming the technical problem in related technologies where the splitting point of the data unit does not match the expected splitting point during merging and splitting, thereby effectively reducing the management difficulty of the cluster.

[0056] The following is for reference. Figure 3 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0057] According to embodiments of this disclosure, operation S202 may include the following operations:

[0058] The interval width is determined based on the data volume threshold and the preset number of splits. The start key and end key are then determined based on the data volume threshold and interval width. The start key represents the value of the first initial split point in the initial split point array, and the end key represents the value of the last initial split point in the initial split point array. Finally, the values ​​of the other initial split points in the initial split point array are determined based on the start key, interval width, and end key.

[0059] According to embodiments of this disclosure, the interval width can be determined, for example, by formula (1), based on a data volume threshold and a preset number of splits:

[0060]

[0061] In formula (1), Interval represents the interval width; RegionSize represents the data volume threshold; and RegionNum represents the preset number of splits.

[0062] According to embodiments of this disclosure, the number of initial split points can be RegionNum-1.

[0063] According to embodiments of this disclosure, the start key can be set to the value of Interval, and the end key can be set to the value of Interval × (RegionNum-1). After determining the values ​​of the first and last initial split points, the values ​​of other initial split points can be obtained by cyclically accumulating the interval width based on the value of the start key. Alternatively, the value of each initial split point can be directly set to Interval × i, where i represents the position of the initial split point in the initial split point array.

[0064] According to embodiments of this disclosure, operation S203 may include the following operations:

[0065] For each initial split point, a first value interval corresponding to the initial split point is generated based on the value of the initial split point and a first preset interval value. If at least one first target current split point's value falls within the first value interval, the first target split point corresponding to the initial split point is determined based on the value of at least one first target current split point. If all first current split point values ​​are outside the first value interval, the first target split point corresponding to the initial split point is determined based on the value of the initial split point. Furthermore, an array of first target split points is obtained based on the values ​​of multiple first target split points.

[0066] According to embodiments of this disclosure, the first preset interval value can be expressed as the product of the interval width and a first proportional coefficient. The first proportional coefficient can be set according to specific application scenarios, for example, it can be set to 1%, 5%, etc.

[0067] According to embodiments of this disclosure, generating a first value range corresponding to the initial split point based on the value of the initial split point and a first preset interval value may include the following operations:

[0068] The lower boundary of the first value interval is determined based on the difference between the initial split point value and the first preset interval value. The upper boundary of the value interval is determined based on the sum of the initial split point value and the first preset interval value.

[0069] According to embodiments of this disclosure, determining the first target split point corresponding to the initial split point based on the value of at least one current first target split point may include the following operations:

[0070] Calculate the difference between the value of at least one current split point of the first target and the value of the initial split point. Determine the target difference with the smallest absolute value from the at least one difference. And, determine the first target split point corresponding to the initial split point based on the value of the current split point of the first target corresponding to the target difference.

[0071] For example, for an initial split point with a value of 1000, the first value interval corresponding to this initial split point is [950, 1050]; the values ​​of multiple first target current split points determined based on the first current split point array are 953, 971, and 1010, respectively; from the above three values ​​of the first target current split points, it can be determined that the value of the first target current split point of the movie with the target difference with the smallest absolute value is 1010, and therefore the value of the first target split point corresponding to this initial split point can be determined to be 1010.

[0072] According to embodiments of this disclosure, determining the target difference with the smallest absolute value from at least one difference can be accomplished by methods such as bubble sort.

[0073] For example, you can declare a list `list` with two parameters `minDis` and `minIndex`. `list` can be used to store the values ​​of all current split points of the first target. Assign the index of the first element to `minIndex`, then take the absolute value of the difference between the first element and the initial split point value, and assign this absolute value to `minDis`. Afterward, you can iterate through each subsequent element in `list`. If the absolute value of the difference between a subsequent element and the initial split point value is less than `minDis`, assign that difference to `minDis` and assign the index of that element to `minIndex`. The final result, `minDis`, is the target difference value.

[0074] According to an embodiment of this disclosure, when the values ​​of the first current split points are all outside the first value range, the value of the first target split point corresponding to the initial split point can be determined to be the value of the initial split point.

[0075] According to an embodiment of this disclosure, when the values ​​of the first current split points are all outside the first value range, the values ​​of the initial split points are added to the first current split point array. After traversing each initial split point, a second current split point array can be obtained.

[0076] According to the embodiments of this disclosure, by determining the first target split point based on the first value range and adjusting the current split point array during the splitting process, it can be ensured that the data unit can maintain pre-partitioning within a certain range during the splitting and merging process, and will not automatically split into multiple sub-data units in the cluster, thus avoiding the problem of cluster management difficulties.

[0077] Figure 3 A flowchart illustrating the data unit splitting process in a data unit management method according to another embodiment of the present disclosure is shown.

[0078] like Figure 3 As shown, the data unit splitting process includes operations S301 to S309.

[0079] In operation S301, the initial split point array is determined.

[0080] In operation S302, determine whether the initial split point array has been traversed completely. If it is determined that the initial split point array has not been completely traversed, execute operation S303; if it is determined that the initial split point array has been completely traversed, execute operation S308.

[0081] In operation S303, the initial split point is obtained and the first value range of the initial split point is calculated.

[0082] In operation S304, traverse the array of the first current split point.

[0083] In operation S305, determine whether there is a current split point of the first target located within the first value interval. If it is determined that there is at least one current split point of the first target, execute operation S306; if it is determined that there is no current split point of the first target, execute operation S307.

[0084] In operation S306, the first target split point is determined from the current split point of the first target.

[0085] In operation S307, the first target split point is created based on the initial split point.

[0086] After completing operation S306 or S307, return to execute operation S302.

[0087] By operating S308, the first target split point array is obtained.

[0088] In operation S309, the first data unit is split into multiple second data units based on the first target split point array.

[0089] According to the embodiments of this disclosure, the specific description of the methods of operating S301 to S309 can be found in the description of the methods of operating S201 to S204, and will not be repeated here.

[0090] Figure 4 A flowchart illustrating the data unit merging process in a data unit management method according to an embodiment of the present disclosure is shown.

[0091] like Figure 4 As shown, the data unit merging process includes operations S401 to S402.

[0092] In operation S401, in response to the merge instruction, the values ​​of each initial split point in the initial split point array are adjusted using the values ​​of the second current split points in the second current split point array to obtain the second target split point array.

[0093] In operation S402, multiple second data units are merged into a third data unit based on the second target split point array.

[0094] According to embodiments of this disclosure, a merge instruction can be an instruction to merge multiple second data units. The merge instruction can be generated in scenarios including, but not limited to, the following: for example, when multiple second data units are not invoked within a preset time period, the preset time period can be set according to a specific application scenario; another example, when the total data of the data units in the cluster exceeds a threshold, the threshold can be set according to a specific application scenario; yet another example, the merge instruction can be triggered by a command input by an operator.

[0095] According to embodiments of this disclosure, the split points in the second current split point array may include the split points originally in the first current split point array, as well as split points created based on the initial split point during the splitting process.

[0096] According to embodiments of this disclosure, adjusting the value of each initial split point in the initial split point array using the value of the second current split point in the second current split point array can be achieved by finding a second current split point in the second current split point array that is compatible with each initial split point.

[0097] According to embodiments of this disclosure, by adjusting the initial split point array of the pre-partition and merging multiple second data units using the adjusted second target split point array, the split point closest to the desired split point can be found, thereby effectively improving the accuracy of merging. Furthermore, the merged third data unit can maintain the pre-partition within a certain range during subsequent splitting and merging processes, avoiding the problem of cluster management difficulties.

[0098] The following is for reference. Figure 5 In conjunction with specific embodiments, Figure 4 The method shown will be further explained.

[0099] According to embodiments of this disclosure, operation S401 may include the following operations:

[0100] For each initial split point, a second value interval corresponding to the initial split point is generated based on the value of the initial split point and a second preset interval value. If at least one value of the second target current split point is within the second value interval, the second target split point corresponding to the initial split point is determined based on the value of at least one second target current split point. If the values ​​of all second current split points are outside the second value interval, the second preset interval value is adjusted, and an adjusted second value interval is generated based on the adjusted second preset interval value, until at least one value of the second target current split point is within the adjusted second value interval. Finally, an array of second target split points is obtained based on the values ​​of multiple second target split points.

[0101] According to embodiments of this disclosure, the second preset interval value can be expressed as the product of the interval width and the second proportional coefficient. The second proportional coefficient can be set according to specific application scenarios, for example, it can be set to 1%, 5%, etc.

[0102] According to embodiments of this disclosure, generating a second value range corresponding to the initial split point based on the value of the initial split point and a second preset interval value may include the following operations:

[0103] The lower boundary of the second value interval is determined based on the difference between the initial split point value and the second preset interval value. The upper boundary of the value interval is determined based on the sum of the initial split point value and the second preset interval value.

[0104] According to embodiments of this disclosure, determining a second target split point corresponding to an initial split point based on the value of at least one current split point of a second target may include the following operations:

[0105] Calculate the difference between the value of at least one current split point of the second target and the value of the initial split point. Determine the target difference with the smallest absolute value from the at least one difference. And, determine the second target split point corresponding to the initial split point based on the value of the current split point of the second target corresponding to the target difference.

[0106] According to embodiments of this disclosure, adjusting the second preset interval value may include adjusting the second proportional coefficient. The method for adjusting the second proportional coefficient is not limited here. For example, it may be done by accumulating a preset proportional coefficient each time; if the original second proportional coefficient is 1%, the adjusted second proportional coefficient would be 1% + 0.1% = 1.1%. Alternatively, it may be done by increasing the second proportional coefficient proportionally; if the original second proportional coefficient is 1%, the adjusted second proportional coefficient would be 1% * 1.1 = 1.1%.

[0107] Figure 5 A flowchart illustrating the data unit merging process in a data unit management method according to another embodiment of the present disclosure is shown.

[0108] like Figure 5 As shown, the data unit merging process includes operations S501 to S509.

[0109] In operation S501, the initial split point array is determined.

[0110] In operation S502, determine whether the initial split point array has been traversed completely. If it is determined that the initial split point array has not been completely traversed, execute operation S503; if it is determined that the initial split point array has been completely traversed, execute operation S508.

[0111] In operation S503, the initial split point is obtained and the second value range of the initial split point is calculated.

[0112] In operation S504, traverse the second current split point array.

[0113] In operation S505, determine whether there is a second target current split point located within the second value interval. If it is determined that there is no second target current split point, execute operation S506; if it is determined that there is at least one second target current split point, execute operation S507.

[0114] In operation S506, the second proportional coefficient is adjusted to redetermine the second value range. After completing operation S506, the process returns to execute operation S504.

[0115] In operation S507, the second target split point is determined from the current split point of the second target. After completing operation S507, the process returns to execute operation S502.

[0116] By operating S508, the second target split point array is obtained.

[0117] In operation S509, multiple second data units are merged into a third data unit based on the second target split point array.

[0118] According to the embodiments of this disclosure, the specific description of the methods of operation S501 to S509 can be found in the description of the methods of operation S401 to S402, and will not be repeated here.

[0119] Figure 6 A block diagram of a data unit management apparatus according to an embodiment of the present disclosure is shown schematically.

[0120] like Figure 6As shown, the data unit management device 600 includes an acquisition module 610, a generation module 620, a first adjustment module 630, and a splitting module 640.

[0121] The acquisition module 610 is used to acquire the first current split point array corresponding to the first data unit in response to the data volume of the first data unit meeting the preset split threshold condition.

[0122] The generation module 620 is used to generate an initial split point array based on the data volume threshold represented by the preset split threshold condition and the preset number of splits. The initial split point array includes the values ​​of multiple initial split points.

[0123] The first adjustment module 630 is used to adjust the value of each initial split point in the initial split point array using the value of the first current split point in the first current split point array, so as to obtain the first target split point array.

[0124] The splitting module 640 is used to split a first data unit into multiple second data units based on a first target splitting point array.

[0125] According to embodiments of this disclosure, when splitting a data unit, an initial splitting point array is generated based on a preset splitting threshold condition and a preset splitting quantity. The values ​​of existing first current splitting points are then used to adjust each initial splitting point in the initial splitting point array to obtain a first target splitting point array. The first data unit is then split based on this first target splitting point array. Through these technical means, it can be ensured that the splitting point of the first data unit is consistent with the expected splitting point during the splitting process. This allows the splitting of the data unit to maintain pre-partitioning within a certain range, thus at least partially overcoming the technical problem in related technologies where the splitting point of the data unit does not match the expected splitting point during merging and splitting, thereby effectively reducing the management difficulty of the cluster.

[0126] According to embodiments of this disclosure, the first adjustment module 630 includes a first adjustment unit, a second adjustment unit, a third adjustment unit, and a fourth adjustment unit.

[0127] The first adjustment unit is used to generate a first value interval corresponding to each initial split point, based on the value of the initial split point and a first preset interval value.

[0128] The second adjustment unit is used to determine the first target split point corresponding to the initial split point based on the value of at least one first target current split point when the value of at least one first target current split point is within the first value range.

[0129] The third adjustment unit is used to determine the first target split point corresponding to the initial split point based on the value of the initial split point when the values ​​of the first current split points are all outside the first value range.

[0130] The fourth adjustment unit is used to obtain the first target split point array based on the values ​​of multiple first target split points.

[0131] According to embodiments of this disclosure, the data unit management device 600 further includes a second adjustment module.

[0132] The second adjustment module is used to add the value of the initial split point to the first current split point array to obtain the second current split point array when the value of the first current split point is outside the first value range for each initial split point.

[0133] According to embodiments of this disclosure, the data unit management device 600 further includes a third adjustment module and a merging module.

[0134] The third adjustment module is used to adjust the value of each initial split point in the initial split point array respectively using the value of the second current split point in the second current split point array in response to the merge instruction, so as to obtain the second target split point array.

[0135] The merging module is used to merge multiple second data units into a third data unit based on the second target split point array.

[0136] According to embodiments of this disclosure, the third adjustment module includes a fifth adjustment unit, a sixth adjustment unit, a seventh adjustment unit, and an eighth adjustment unit.

[0137] The fifth adjustment unit is used to generate a second value interval corresponding to each initial split point, based on the value of the initial split point and the second preset interval value.

[0138] The sixth adjustment unit is used to determine the second target split point corresponding to the initial split point based on the value of at least one second target current split point when the value of at least one second target current split point is within the second value range.

[0139] The seventh adjustment unit is used to adjust the second preset interval value when the values ​​of the second current split points are all outside the second value interval, and to generate an adjusted second value interval based on the adjusted second preset interval value, until there is at least one second target current split point whose value is within the adjusted second value interval.

[0140] The eighth adjustment unit is used to obtain an array of second target split points based on the values ​​of multiple second target split points.

[0141] According to embodiments of this disclosure, the first adjustment unit or the fifth adjustment unit includes a first adjustment subunit and a second adjustment subunit.

[0142] The first adjustment subunit is used to determine the lower boundary of the value interval based on the difference between the value of the initial split point and the preset interval value.

[0143] The second adjustment subunit is used to determine the upper boundary of the value interval based on the sum of the initial split point value and the preset interval value.

[0144] According to embodiments of this disclosure, the second adjustment unit or the sixth adjustment unit includes a third adjustment subunit, a fourth adjustment subunit, and a fifth adjustment subunit.

[0145] The third adjustment subunit is used to calculate the difference between the value of the current split point of at least one target and the value of the initial split point.

[0146] The fourth adjustment subunit is used to determine the target difference with the smallest absolute value from at least one difference.

[0147] The fifth adjustment subunit is used to determine the target split point corresponding to the initial split point based on the value of the target current split point corresponding to the target difference.

[0148] According to embodiments of this disclosure, the generation module 620 includes a first generation unit, a second generation unit, and a third generation unit.

[0149] The first generation unit is used to determine the interval width based on the data volume threshold and the preset number of splits.

[0150] The second generation unit determines the start key and end key based on the data volume threshold and the interval width. The start key is represented by the value of the first initial split point in the initial split point array, and the end key is represented by the value of the last initial split point in the initial split point array.

[0151] The third generation unit is used to determine the values ​​of other initial split points in the initial split point array based on the start key, interval width, and end key.

[0152] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0153] For example, any plurality of the acquisition module 610, generation module 620, first adjustment module 630, and splitting module 640 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the acquisition module 610, generation module 620, first adjustment module 630, and splitting module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 610, generation module 620, first adjustment module 630, and splitting module 640 may be implemented at least partially as a computer program module that can perform corresponding functions when the computer program module is run.

[0154] It should be noted that the data unit management device part in the embodiments of this disclosure corresponds to the data unit management method part in the embodiments of this disclosure. The specific description of the data unit management device part is referred to in the data unit management method part, and will not be repeated here.

[0155] Figure 7A block diagram of an electronic device suitable for implementing a data unit management method according to an embodiment of the present disclosure is shown schematically. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0156] like Figure 7 As shown, a computer electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0157] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0158] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0159] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0160] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0161] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0163] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the data unit management method provided in the embodiments of this disclosure.

[0164] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0165] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0166] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0168] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data unit management method, comprising: In response to the fact that the amount of data in the first data unit meets the preset splitting threshold condition, the array of the first current splitting points corresponding to the first data unit is obtained; An initial split point array is generated based on the data volume threshold represented by the preset split threshold condition and the preset number of splits, wherein the initial split point array includes the values ​​of multiple initial split points; The values ​​of each initial split point in the initial split point array are adjusted using the values ​​of the first current split point in the first current split point array to obtain the first target split point array; and Based on the first target split point array, the first data unit is split into multiple second data units; wherein, the step of adjusting the value of each initial split point in the initial split point array using the value of the first current split point in the first current split point array to obtain the first target split point array includes: For each initial split point, a first value interval corresponding to the initial split point is generated based on the value of the initial split point and the first preset interval value. If at least one of the first target current split points has a value that is within the first value range, the first target split point corresponding to the initial split point is determined based on the value of at least one of the first target current split points. When the values ​​of the first current split points are all outside the first value range, a first target split point corresponding to the initial split point is determined based on the value of the initial split point; and Based on the values ​​of multiple first target split points, an array of first target split points is obtained.

2. The method according to claim 1, further comprising: For each initial split point, if the value of the first current split point is outside the first value range, the value of the initial split point is added to the first current split point array to obtain the second current split point array.

3. The method according to claim 2, further comprising: In response to the merge instruction, the values ​​of each initial split point in the initial split point array are adjusted using the values ​​of the second current split points in the second current split point array to obtain the second target split point array; as well as Based on the second target split point array, multiple second data units are merged into a third data unit.

4. The method according to claim 3, wherein, The step of adjusting the value of each initial split point in the initial split point array using the value of the second current split point in the second current split point array to obtain the second target split point array includes: For each initial split point, a second value interval corresponding to the initial split point is generated based on the value of the initial split point and the second preset interval value. If at least one of the second target current split points has a value that is within the second value range, the second target split point corresponding to the initial split point is determined based on the value of at least one of the second target current split points. When the values ​​of all second current split points are outside the second value interval, adjust the second preset interval value, and generate an adjusted second value interval based on the adjusted second preset interval value, until at least one value of the second target current split point is within the adjusted second value interval; and Based on the values ​​of multiple second target split points, an array of second target split points is obtained.

5. The method according to claim 1 or 4, wherein, Based on the value of the initial split point and the preset interval value, a value interval corresponding to the initial split point is generated, including: Based on the difference between the initial split point value and the preset interval value, the lower boundary of the value interval is determined; and Based on the value of the initial split point and the sum of the preset interval values, the upper boundary of the value interval is determined; The preset interval value includes either the first preset interval value or the second preset interval value, and the value range includes either the first value range or the second value range.

6. The method according to claim 1 or 4, wherein, Determining the target split point corresponding to the initial split point based on the value of at least one of the target current split points includes: Calculate the difference between the value of at least one of the target's current split points and the value of the initial split point; Determine the target difference with the smallest absolute value from at least one of the said differences; and The target split point corresponding to the initial split point is determined based on the value of the current target split point corresponding to the target difference; Wherein, the target current split point includes the first target current split point or the second target current split point, and the target split point includes the first target split point or the second target split point.

7. The method according to claim 1, wherein, The step of generating an initial split point array based on the data volume threshold represented by the preset split threshold condition and the preset split number includes: The interval width is determined based on the data volume threshold and the preset number of splits; Based on the data volume threshold and the interval width, a start key and an end key are determined, wherein the start key represents the value of the first initial split point in the initial split point array, and the end key represents the value of the last initial split point in the initial split point array; and Based on the start key, the interval width, and the end key, determine the values ​​of other initial split points in the initial split point array.

8. The method according to claim 1, wherein, The first preset interval value is represented as the product of the interval width and the first proportional coefficient; The second preset interval value is expressed as the product of the interval width and the second proportional coefficient; Adjusting the second preset range value includes adjusting the second proportional coefficient.

9. A data unit management device, comprising: The acquisition module is used to acquire the first current split point array corresponding to the first data unit in response to the data volume of the first data unit meeting the preset split threshold condition; The generation module is used to generate an initial split point array based on the data volume threshold represented by the preset split threshold condition and the preset number of splits, wherein the initial split point array includes the values ​​of multiple initial split points; The first adjustment module is used to adjust the value of each initial split point in the initial split point array using the value of the first current split point in the first current split point array, so as to obtain a first target split point array; and A splitting module is used to split the first data unit into multiple second data units based on the first target splitting point array; The first adjustment module includes: The first adjustment unit is used to generate a first value range corresponding to each initial split point based on the value of the initial split point and a first preset range value. The second adjustment unit is used to determine the first target split point corresponding to the initial split point based on the value of at least one first target current split point when the value of at least one first target current split point is within the first value range. The third adjustment unit is configured to determine a first target split point corresponding to the initial split point based on the value of the initial split point, provided that the values ​​of the first current split points are all outside the first value range; and The fourth adjustment unit is used to obtain an array of the first target split points based on the values ​​of multiple first target split points.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1 to 8.

Images