Online data skew adjustment method and device for stream computing job

Abstract

The application provides an online data skew adjustment method and device for stream computing jobs, comprising: obtaining data volume statistical information of a Task in a time period to determine data skew degrees of each data shard in the Task; if the data skew degree of a data shard is greater than a skew threshold, analyzing a data volume deviation state of a Key in the data shard to adjust a data shard to which the Key belongs, and constructing a data shard-Key mapping table; all upstream Tasks of the Task perform data distribution to the target Task in combination with the data shard-Key mapping table. The online data skew adjustment method and device for stream computing jobs provided by the application dynamically optimize data skew by changing the mapping relationship between the Key and the data shard based on the data shard mode and the data volume statistical information, realize online migration of the Key state, and ensure that relevant data is distributed to the same Task, thereby effectively improving the rate of stream computing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an online data skew adjustment method and apparatus for stream computing jobs. Background Technology

[0002] With the widespread application of big data, batch processing has become very mature, while the real-time nature of data is becoming increasingly important in the future, and the low latency brought by stream computing is gaining more and more attention. In stream computing, data skew generally occurs in scenarios that require data distribution based on keys, such as: group by statistical operations, key-based data splitting and join operations, and key-based complex event processing (CEP) partition by operations. Due to the large difference in data volume between different keys, the execution time of one or several tasks may be longer, resulting in overall data latency.

[0003] Currently, there are many solutions to address data skew, such as increasing hardware resources (JVM memory), key filtering, local-global two-stage aggregation, data sharding, random prefixing, and pre-sampling.

[0004] JVM memory, while increasing JVM memory and CPU resources can improve computation speed, is only a temporary solution and does not fundamentally address the data skew problem. Key filtering filters out some keys that cause data skew based on prior experience; while this method prevents execution failure, it can lead to data loss. Local-global two-stage aggregation is suitable for statistical operations of aggregation classes and can reduce the network cost of data shufies caused by data skew, but it cannot reduce memory and CPU costs. Data sharding can partially solve the data skew problem, but it does not consider the difference in data volume for each key and cannot be adjusted based on the statistical results of the running data volume. Random prefixing scatters data for the same key, so the related data within the data shards is incomplete, resulting in incorrect calculation results; it is only suitable for statistical operations. Pre-sampling cannot be used in stream processing. The dataset must already exist. Developers analyze the degree of data skew by sampling and, based on the sampled data, use encoding or configuration files to prevent data skew in advance.

[0005] In summary, current methods for data stream operations with correlations cannot achieve online dynamic optimization of data skew when data skew is unpredictable. Furthermore, it is impossible to distribute correlated data to the same task to ensure the integrity of the related data. Summary of the Invention

[0006] To address the problems existing in the prior art, embodiments of the present invention provide an online data skew adjustment method and apparatus for stream computing jobs.

[0007] This invention provides an online data skew adjustment method for stream computing jobs, comprising: acquiring data volume statistics for each computing thread Task within a time period; determining the data skewness of each data shard in the target Task based on the data volume statistics when the overall data volume statistics of any target Task are unbalanced; analyzing the data volume deviation state of the Key in any data shard when the data skewness of any data shard is greater than a skew threshold; adjusting the data shard to which the Key belongs in any data shard based on the data volume deviation state of the Key, and adding the mapping relationship between the adjusted Key and the corresponding data shard to a data shard-Key mapping table; and controlling all upstream Tasks of the target Task to perform data distribution to the target Task in conjunction with the data shard-Key mapping table.

[0008] According to the present invention, an online data skew adjustment method for stream computing jobs is provided, wherein the data volume statistics include: the total data volume of each data fragment and the total data volume of each key in each time period.

[0009] According to an online data skew adjustment method for a stream computing job provided by the present invention, the step of adjusting the data shard to which the Key belongs in any data shard according to the data volume deviation state of the Key includes: when the data volume deviation state of the Key is balanced, adjusting the Key with the smallest data volume in any data shard to other data shards in the target Task; when the data volume deviation state of the Key is unbalanced, determining the data shard with the largest data volume and the data shard with the smallest data volume in the target Task, and adjusting at least one Key in the data shard with the largest data volume to the data shard with the smallest data volume.

[0010] According to the present invention, an online data skew adjustment method for a stream computing job is provided, wherein adding the mapping relationship between the key to be adjusted and the corresponding data shard to the data shard-key mapping table includes: for any key to be adjusted, if the mapping relationship between the adjusted key and the corresponding data shard is inconsistent with the calculation result of the mapping relationship function, then adding the mapping relationship between the key and the corresponding data shard to the data shard-key mapping table.

[0011] According to the present invention, an online data skew adjustment method for a stream computing job controls all upstream tasks of a target task to perform data distribution to the target task in conjunction with the shard-key mapping table. The method includes: sending a first notification to all upstream tasks based on the data shard-key mapping relationship to control all upstream tasks to suspend data distribution; sending a second notification to all downstream tasks containing the target task to determine the state corresponding to the key that needs to be migrated and the location of that state; after all downstream tasks respond to the second notification and migrate the state corresponding to the key, updating the mapping relationship between the task, data shard, and key, and sending the mapping relationship to all upstream tasks; and controlling all upstream tasks to distribute data to the target task according to the mapping relationship.

[0012] According to the present invention, an online data skew adjustment method for stream computing jobs, wherein controlling all upstream tasks to distribute data to the target task according to the mapping relationship includes: if the upstream task determines that the key to be distributed is in the data shard-key mapping table, then determining the data shard corresponding to the key to be distributed according to the data shard-key mapping table, and allocating the key to be distributed to the data shard; if the upstream task determines that the key to be distributed is not in the data shard-key mapping table, calculating the data shard number corresponding to the key to be distributed using a data shard-key mapping function; and finding the downstream task number corresponding to the data shard number from the Task-data shard mapping relationship according to the data shard number, to complete the allocation of the key to be distributed.

[0013] According to the present invention, an online data skew adjustment method for stream computing jobs includes, before obtaining the data volume statistics of each computing thread Task within a time period, the method further includes: allocating data shards to each Task based on a hash algorithm and constructing a Task-data shard mapping table; each upstream Task distributing the corresponding stream computing job data to the corresponding downstream Task according to the mapping relationship between its downstream Task and the data shards; and determining the data shard number to which each Key belongs through a hash function, so as to evenly distribute all Keys corresponding to the stream computing job data to each data shard.

[0014] This invention also provides an online data skew adjustment device for stream computing jobs, comprising: a data acquisition unit for acquiring data volume statistics of each computing thread Task within a time period; a data skew calculation unit for determining the data skew of each data shard in the target Task based on the data volume statistics when the overall data volume statistics of any target Task are unbalanced; a deviation state analysis unit for analyzing the data volume deviation state of the Key in any data shard when the data skew of any data shard is greater than a skew threshold; a data skew adjustment method for adjusting the data shard to which the Key belongs in any data shard according to the data volume deviation state of the Key, and adding the mapping relationship between the adjusted Key and the corresponding data shard to a data shard-Key mapping table; and a data distribution unit for controlling all upstream Tasks of the target Task and, in conjunction with the data shard-Key mapping table, performing data distribution to the target Task.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the online data skew adjustment method for any of the above-described stream computing jobs.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the online data skew adjustment method for the stream computing job as described above.

[0017] The present invention provides an online data skew adjustment method and apparatus for stream computing jobs. Based on data sharding, it dynamically optimizes data skew by changing the mapping relationship between the key and the data shard according to data volume statistics, realizes online migration of key state, and ensures that related data are distributed to the same task, effectively improving the speed of stream computing. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the online data skew adjustment method for stream computing jobs provided by the present invention;

[0020] Figure 2 This is a schematic diagram of the stream computing cluster configuration provided by the present invention;

[0021] Figure 3 This is a schematic diagram of the three-level mapping relationship between Task, data sharding, and Key provided by the present invention;

[0022] Figure 4 This is a schematic diagram of the data fragment to which the Key belongs in the adjusted data fragmentation provided by the present invention;

[0023] Figure 5 This is a general schematic diagram of a data skew adjustment method provided by the present invention, as shown below. Figure 5 As shown;

[0024] Figure 6 This is a detailed schematic diagram of an online data tilt adjustment method provided by the present invention;

[0025] Figure 7 This is a schematic diagram of data distribution using Task-data sharding provided by the present invention;

[0026] Figure 8 This is a schematic diagram of the data sharding-key mapping table provided by the present invention;

[0027] Figure 9 This is a schematic diagram of the distribution decision process provided by the present invention;

[0028] Figure 10 This is a schematic diagram of the structure of the online data skew adjustment device for stream computing jobs provided by the present invention;

[0029] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention; Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0032] Data skew is very common in the MapReduce programming model. Data skew simply means that a large number of identical keys are assigned to the same partition, causing one node to bear a huge load, while other nodes have to wait for this busy node after they have finished computing, which also slows down the overall computing time. It can be said that the efficiency is very low.

[0033] For CEP's Partition By semantics, the data of each key is related, so its data skew handling method is different from that of general data skew handling. First, it is necessary to ensure that the data of the same key is distributed to the same computing node, while also taking into account the adjustment of data skew.

[0034] Because existing technologies cannot dynamically and online adjust data skew for the scenarios described above in the Partition By semantics, data distribution strategies must be set manually, and modifying the strategy requires restarting the job, which interrupts the data processing flow.

[0035] The present invention provides an online data skew adjustment method and apparatus for stream computing jobs. When data needs to be distributed in a stream computing cluster, the online data stream of the stream computing job is divided into data shards according to the key, and each data shard is then assigned to a Task computing task. When data skew is determined to exist, the data skew is reduced by adjusting the data shard to which the relevant key belongs. Throughout the process, the state corresponding to the key remains on the same Task.

[0036] Therefore, the online data skew adjustment method for stream computing jobs provided by this invention can provide a solution for dynamically optimizing data skew online and ensuring that related data is distributed to the same task when data skew is unpredictable, especially when data skew is based on KeyShuffle and has correlations.

[0037] The following is combined Figures 1-11 This invention describes the online data skew adjustment method and apparatus for stream computing jobs provided in embodiments of the present invention.

[0038] Figure 1 This is a flowchart illustrating the online data skew adjustment method for stream computing jobs provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps:

[0039] Step 101: Obtain the data volume statistics of each computing thread Task within a time period.

[0040] Figure 2 This is a schematic diagram of the stream computing cluster configuration provided by the present invention, as shown below. Figure 2 As shown, the entire cluster architecture mainly includes a control node (JobManager), worker nodes (TaskManager), and Tasks.

[0041] This invention assigns two new functions to the control node: data scheduling and data skew analysis. This can be viewed as adding two new virtual modules to the control node: a data skew scheduling center and a data skew analysis center. The data skew scheduling center is responsible for controlling the online optimization process of data skew. The data skew analysis center periodically receives data traffic statistics from each computing node, analyzes the data skew degree, determines whether the data skew degree exceeds a threshold, and if it does, calculates the mapping relationship between data shards and keys, adjusting the data distribution relationship through the data skew scheduling center.

[0042] Worker nodes are primarily responsible for receiving scheduling from control nodes and for receiving, starting, and destroying Tasks. A job contains multiple Tasks, which are executed on worker nodes. Each Task contains one operator or a chain of operators (OperatorChain), and each operator contains the specific business logic for data computation. Operators within a Task are treated as a single unit.

[0043] This invention expands upon the monitoring information reported by conventional Tasks by providing statistical information on the amount of data executed by the Tasks. Specifically, Tasks report statistical information on the amount of data executed within each time period (i.e., each statistical period) to the control node. For example, the total data throughput of each Key and the total data throughput of each data shard within a time period.

[0044] Step 102: When the overall data volume statistics of any target task are unbalanced, determine the data skewness of each data shard in the target task based on the data volume statistics.

[0045] Figure 3 This is a schematic diagram of the three-level mapping relationship between Task, data sharding, and Key provided by the present invention, as shown below. Figure 3 As shown, one Task holds multiple data shards, and one data shard contains multiple Keys. Here, m is the parallelism of the job, and the number of shards is the product of m and the data shard ratio.

[0046] This invention collects the total data volume of data shards for each Task within each time period. Since the total number of data shards is determined before the job starts, the number of data shards remains constant regardless of the execution method. It also analyzes the total data volume processed by each Task. Furthermore, it uses statistical methods to calculate the mean squared error of the total data volume processed by each Task, using this value to determine whether the overall data volume statistics for each Task are balanced.

[0047] If it is balanced, it means that there is no data skew during the stream computing job within this time period, so no adjustment is needed.

[0048] If it is unbalanced, it means that data skew has occurred during the stream computing job. The Task with data skew can be used as the target Task.

[0049] Furthermore, if the overall data volume statistics of the target task are unbalanced, it is necessary to further calculate and determine the data skewness of each data segment in the target task in order to identify which data segment in the target task has data skewness.

[0050] Specifically, the data skewness can be determined by statistically analyzing the total data volume of each data shard and the total data volume of each key within each data shard, and then calculating the mean squared error of the total data volume of each data shard.

[0051] Step 103: If the data skewness of any data shard is greater than the skewness threshold, analyze the data volume deviation status of the Key in the data shard.

[0052] The skew threshold can be set according to the actual situation. For example, the average skewness of all data shards can be used as the skew threshold, or 1.5 times the average skewness of all data can be used as the skew threshold. This invention does not make specific limitations on this.

[0053] If the total amount of data processed by each Task is balanced overall, no adjustment is needed; otherwise, it is necessary to analyze the deviation in the total amount of data processed by each data shard. Specifically, this includes the following two cases:

[0054] If the data volume deviation is less than or equal to the skew threshold, that is, the data volume deviation is within the allowable range, it indicates that the data volume processed by each Task is balanced, and no dynamic adjustment is made.

[0055] If the data volume deviation is greater than the skew threshold, the data volume deviation status of the Key in each data shard is analyzed, including two cases: balanced data volume deviation and unbalanced data volume deviation.

[0056] Step 104: Based on the data volume deviation status of the Key, adjust the data shard to which the Key belongs in any data shard, and add the mapping relationship between the adjusted Key and the corresponding data shard to the data shard-Key mapping table.

[0057] The data shard-key mapping table does not need to be pre-built; by default, it determines which task directly downstream a key should distribute data to based on the data shard-key mapping relationship. The control node has a global data shard-key mapping table, and each task stores its own portion of the mapping relationship. Only after a skew adjustment is performed will the mapping relationship between a specific key (the key that experienced the adjustment event) and the data shard be recorded, preventing excessive computational load and memory consumption due to a large number of keys.

[0058] Specifically, the data skew analysis center periodically analyzes data skew. If the data skew exceeds a threshold, a new data shard-key mapping relationship is planned in the data skew analysis center.

[0059] Step 105: Control all upstream Tasks of the target Task, and in conjunction with the data sharding-key mapping table, perform data distribution to the target Task.

[0060] Finally, the data skew scheduling center notifies the upstream task of the data skewed task of the new data shard and key mapping relationship, and moves the data cache.

[0061] The online data skew adjustment method for stream computing jobs provided by this invention is based on data sharding. According to data volume statistics, it dynamically optimizes data skew by changing the mapping relationship between the key and the data shard, realizes online migration of the key state, and ensures that related data are distributed to the same task, effectively improving the speed of stream computing.

[0062] Based on the above embodiments, as an optional embodiment, the data volume statistics information includes: the total data volume of each data shard and the total data volume of each key in each time period.

[0063] It should be noted that in the online data skew adjustment method for streaming computing jobs provided by this invention, the total number of data fragments is determined before the job starts and is equal to the parallelism of the job. Therefore, no matter how the online data skew adjustment is performed, the number of data fragments remains unchanged.

[0064] This invention utilizes the total data volume of data shards in each Task within each cycle, as well as the total data volume of each Key, to analyze the total data volume processed by each Task and the data volume deviation of each data shard in the Task. It also provides a basis for determining the data volume deviation status of the Key in the later stage (i.e., judging whether the data in each Key has been skewed).

[0065] The online data skew adjustment method for stream computing jobs provided by this invention provides data support for online data skew analysis and adjustment by collecting online data volume statistics.

[0066] Based on the above embodiments, as an optional embodiment, adjusting the data shard to which the Key belongs in any data shard according to the data volume deviation status of the Key includes:

[0067] If the data volume deviation of the Key is balanced, the Key with the smallest data volume in any data shard will be adjusted to other data shards in the target Task.

[0068] When the data volume deviation of the Key is unbalanced, the data shard with the largest data volume and the data shard with the smallest data volume in the target Task are determined, and at least one Key in the data shard with the largest data volume is adjusted to the data shard with the smallest data volume.

[0069] Specifically, the data volume deviation status can be determined based on the deviation of the total data volume of each key. That is, if the deviation of the total data volume of any key is greater than the threshold, its data volume deviation status is set to unbalanced; if the deviation of the total data volume of any key is not greater than the threshold, its data volume deviation status is set to balanced.

[0070] It should be noted that the basis for judging the data volume deviation of a Key is that the overall data volume statistics of the target Task to which it belongs are in an unbalanced state.

[0071] At this point, if the data volume deviation state of the Key is balanced, it means that the data volume of the Key is relatively balanced overall, but the data volume of each data shard is unbalanced (i.e., there is a skew). This may be because the Key was unbalanced in the previous execution but is now relatively balanced. Therefore, you can choose to allocate the Key with the smaller (or smallest) data volume in the data shard to other data shards to ensure the data volume of the data shards is balanced as much as possible.

[0072] If the data volume deviation of a key is unbalanced, then select one or more keys from the data shard with the larger (or largest) data volume and move them to the data shard with the smallest data volume.

[0073] Figure 4 This is a schematic diagram of the data fragment to which the Key belongs in the adjusted data fragment provided by the present invention, as shown below. Figure 4 As shown, assuming that the initial data volume in a certain data shard m+1 is the largest (including Key 19 and Key 34), and the data volume in data shard 2m+1 is smaller (including Key 7 and Key 23), then Key 34 is migrated from data shard m+1 to data shard 2m+1 to reduce the degree of data skew and obtain the data shard to which the adjusted Key belongs.

[0074] It should be noted that when selecting the key and data sharding, the principle of minimizing cost should be followed, that is, while moving the key, the movement of state data and the pressure of balancing computation should be minimized as much as possible.

[0075] The online data skew adjustment method for stream computing jobs provided by this invention targets data streams with correlation based on key distribution. It uses a data sharding-based strategy, which does not require prior knowledge of the key distribution. It only needs to adjust the data flow of each computing node by changing the mapping relationship between the key and the data shards based on the data volume statistics during the operation, thereby achieving dynamic optimization of data skew.

[0076] Based on the above embodiments, as an optional embodiment, adding the mapping relationship between the adjusted Key and the corresponding data shard to the data shard-Key mapping table includes:

[0077] For any adjusted Key, if the mapping relationship between the adjusted Key and the corresponding data shard is inconsistent with the calculation result of the mapping relationship function, then the mapping relationship between the Key and the corresponding data shard is added to the data shard-Key mapping table.

[0078] Regardless of whether the data volume deviation of a key is balanced or unbalanced, the data shard-key mapping table may swell during key movement. Moving a large number of keys will generate a large number of data shard-key mapping records. To minimize the size of the data shard-key mapping table, for each key to be moved, if the data shard-key mapping relationship is inconsistent with the calculation result of the mapping relationship function, it is recorded in the data shard-key mapping relationship; if they are consistent, it is not recorded in the data shard-key mapping relationship, thus avoiding the continuous expansion of the data shard-key mapping table.

[0079] Based on the above embodiments, as an optional embodiment, controlling all upstream Tasks of the target Task, in conjunction with the shard-key mapping table, to perform data distribution to the target Task includes: sending a first notification to all upstream Tasks according to the data shard-key mapping relationship to control all upstream Tasks to suspend data distribution; sending a second notification to all downstream Tasks where the target Task is located to determine the state corresponding to the key that needs to be migrated and the location of the state; after all downstream Tasks respond to the second notification and migrate the state corresponding to the key, updating the mapping relationship between Task, data shard, and key, and sending the mapping relationship to all upstream Tasks; and controlling all upstream Tasks to implement data distribution to the target Task according to the mapping relationship.

[0080] Figure 5 This is a general schematic diagram of a data skew adjustment method provided by the present invention, as shown below. Figure 5 As shown, the entire process mainly includes three major steps: generating a skew adjustment plan, executing data skew adjustment, and distributing the data after skew adjustment.

[0081] (1) Generate tilt adjustment plan

[0082] The data skew in a task is not a problem with the current task, but rather the skew already occurred when the upstream task sent data to the downstream task.

[0083] Therefore, in this step, the data skew analysis center periodically determines whether data skew has occurred based on the data skewness of the data shards.

[0084] If data skew is confirmed, a new data shard-key mapping relationship is planned in the data skew analysis center. The data skew scheduling center then notifies the upstream task of the skewed task to move the data cache according to the new data shard-key mapping relationship.

[0085] (2) Perform data skew adjustment

[0086] Figure 6 This is a detailed schematic diagram of an online data tilt adjustment method provided by the present invention, as shown below. Figure 6 As shown, including but not limited to the following steps:

[0087] 1) Distribution of notification data

[0088] According to the data skew adjustment plan, the data skew scheduling center where the target task is located sends the first notification to each upstream task. After receiving the first notification, the upstream tasks suspend data distribution.

[0089] 2) Notify the migration key status

[0090] The upstream Task that suspended data distribution sends a second notification to the downstream Task where the target Task is located, informing the corresponding downstream Task that it needs to migrate the state corresponding to the Key and the location of the state (Task 2.2) information.

[0091] 3) Task migration and removal status

[0092] Migrate the state data corresponding to the key between tasks, such as Figure 6 As shown, downstream Task 2.1 establishes a communication connection with downstream Task 2.2. Downstream Task 2.1 requests the status data corresponding to the Key from downstream Task 2.2 to read the status corresponding to the Key online from downstream Task 2.2, and removes the status corresponding to the Key in downstream Task 2.2. The control center notifies the upstream Tasks (Task 1.n) to update the Key and data sharding mapping relationship.

[0093] 4) Update the mapping relationship and resume distribution.

[0094] Once the Task migration and removal status are successfully processed, the data skew scheduling center notifies each upstream Task of the latest Task-data shard-key mapping relationship and resumes data distribution based on the new mapping relationship.

[0095] This completes the online data skew adjustment for this time period.

[0096] The online data skew adjustment method for stream computing jobs provided by this invention adopts a dynamic data shard-key mapping relationship. During runtime, the mapping between data shards and tasks is not adjusted, but only the mapping of keys is adjusted, which can provide online data skew adjustment. Moreover, the allocation relationship between data shards and tasks can be adjusted without restarting the job, which can further improve the efficiency of stream computing.

[0097] Based on the content of the foregoing embodiment, as an optional embodiment, said controlling all said upstream Tasks to implement data distribution for the target Task according to the mapping relationship comprises: in the case that the upstream Task determines that the Key to be distributed is in the data shard-Key mapping table, determining the data shard corresponding to the Key to be distributed according to the data shard-Key mapping table, and distributing the Key to be distributed to said data shard; in the case that the upstream Task determines that the Key to be distributed is not in the data shard-Key mapping table, calculating the data shard number corresponding to the Key to be distributed by using a data shard-Key mapping function; finding the downstream Task number corresponding to the data shard number from the Task-data shard mapping relationship according to the data shard number, so as to complete distribution of the Key to be distributed.

[0098] Before the present invention analyzes whether data skew occurs in the process of a stream computing job, the present invention further comprises, based on Figure 2 the shown stream computing cluster configuration, implementing distribution of the stream computing job, which comprises the following content:

[0099] designing a relationship among Tasks, data shards and Keys, and performing data distribution after completing the foregoing design.

[0100] As shown in Figure 2 , a Task is an executor of data distribution, a data shard is a bucket for Keys, and a three-level mapping relationship is adopted among Tasks, data shards and Keys. The design of the relationship among Tasks, data shard and Keys mainly comprises: allocating data shards to each Task, and distributing all Keys to each data shard respectively.

[0101] (1) Allocating data shards for each Task

[0102] Figure 7 is a schematic diagram of data distribution using Task-data shard provided by the present invention. As shown in Figure 7 , since the number of data shards is much larger than the number of Tasks, the present invention adopts a Hash algorithm to allocate data shards for each Task, that is, one Task is associated with multiple data shards, and one data shard comprises multiple Keys. The upstream Task determines which downstream Task the data is distributed to at runtime according to the mapping relationship between downstream Tasks and data shards.

[0103] It should be noted that the Task-data shard mapping table mentioned in the foregoing embodiment may use a structure of <Task number, data shard number> to store the mapping relationship between Tasks and data shards.

[0104] (2) Allocating all Keys to each data shard respectively

[0105] Since the number of Keys can reach hundreds of millions or even billions, if a record is used to store the mapping relationship between each Key and a data shard, a large amount of memory will be occupied, which is not practical in actual production applications. Therefore, in the present invention, a Key-data shard mapping table and a data shard-Key mapping function are used to determine the data shard to which a Key belongs.

[0106] 1) Data shard-Key mapping function

[0107] By means of a hash function, Keys are evenly distributed into data shards, and the number of the data shard to which a Key belongs is calculated, which comprises:

[0108] Data shard number = hash(Key) % Partition Number;

[0109] wherein, Partition Number is the number of shards, % is a remainder operator in programming languages, and hash(Key) is a hash representation of Key.

[0110] 2) Data shard-Key mapping table

[0111] Figure 8 is a schematic diagram of the data shard-Key mapping table provided by the present invention, as shown in Figure 8 , the table records the mapping relationship <Key, data shard number> between moved Keys and data shards of the system according to operating status. After the data shards are allocated, data skew occurs during operation, Keys will be dynamically moved between data shards, and the mapping relationship between moved Keys and data shards is recorded as a special case.

[0112] Data streams are distributed according to data shards, and the mapping relationship between data shards and Tasks is determined. When a data stream arrives, the data shard to which the data stream belongs is calculated according to the Key in the data, so that which downstream Task the data should be distributed to can be determined.

[0113] Figure 9 is a schematic diagram of the distribution decision process provided by the present invention, as shown in Figure 9 , in the data distribution process, a Task first determines whether a Key exists in the data shard-Key mapping table; if yes, the number of the data shard recorded therein is used; if not, the corresponding data shard number is calculated by using the data shard-Key mapping function, then the corresponding downstream Task number is found in the Task-data shard mapping relationship according to the data shard number, and the data is sent to the corresponding Task.

[0114] Furthermore, considering the cost of maintaining the mapping table, when moving a key, if the mapping relationship between the key being moved and the data shard is exactly the same as the calculation result of the data shard-key mapping function, then there is no need to record it separately in the data shard-key mapping table. The mapping function can be used directly, which can avoid the expansion of the data shard-key mapping table and reduce the maintenance cost of the mapping table.

[0115] Figure 10 This is a schematic diagram of the online data skew adjustment device for stream computing jobs provided by the present invention, as shown below. Figure 10 As shown, it mainly includes: a data acquisition unit 11, a data tilt calculation unit 12, a deviation state analysis unit 13, a data tilt adjustment unit 14, and a data distribution unit 15, wherein:

[0116] Data Acquisition Form 11 is mainly used to obtain statistical information on the amount of data of each computing thread Task within a time period;

[0117] The data skewness calculation sheet 12 is mainly used to determine the data skewness of each data fragment in the target task based on the data volume statistics when the overall data volume statistics of any target task are unbalanced.

[0118] The deviation state analysis unit 13 is mainly used to analyze the data quantity deviation state of Key in any data segment when the data skewness of any data segment is greater than the skewness threshold.

[0119] The data skew adjustment unit 14 is mainly used to adjust the data shard to which the Key belongs in any data shard according to the data volume deviation state of the Key, and add the mapping relationship between the adjusted Key and the corresponding data shard to the data shard-Key mapping table;

[0120] The data distribution unit 15 is mainly used to control all upstream tasks of the target task and, in conjunction with the data sharding-key mapping table, perform data distribution to the target task.

[0121] The online data skew adjustment device for stream computing jobs provided by this invention is based on data sharding. According to data volume statistics, it dynamically optimizes data skew by changing the mapping relationship between the key and the data shard, realizes online migration of the key state, and ensures that related data are distributed to the same task, effectively improving the speed of stream computing.

[0122] It should be noted that the online data skew adjustment device for streaming computing jobs provided in this embodiment of the invention can be based on the online data skew adjustment method for streaming computing jobs described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0123] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 11 As shown, the electronic device may include: a processor 1110, a communications interface 1120, a memory 1130, and a communications bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other through the communications bus 1140. Processor 1110 can call logic instructions in memory 1130 to execute an online data skew adjustment method for stream computing jobs. This method includes: acquiring data volume statistics for each computing thread Task within a time period; determining the data skewness of each data shard in the target Task based on the data volume statistics when the overall data volume statistics of any target Task are unbalanced; analyzing the data volume deviation state of the Key in any data shard when the data skewness of any data shard is greater than a skew threshold; adjusting the data shard to which the Key belongs in any data shard based on the data volume deviation state of the Key, and adding the mapping relationship between the adjusted Key and the corresponding data shard to a data shard-Key mapping table; and controlling all upstream Tasks of the target Task to perform data distribution to the target Task in conjunction with the data shard-Key mapping table.

[0124] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the online data skew adjustment method for stream computing jobs provided by the above methods, the method comprising: acquiring data volume statistics of each computing thread Task within a time period; determining the data skewness of each data shard in the target Task based on the data volume statistics when the overall data volume statistics of any target Task are unbalanced; analyzing the data volume deviation state of the Key in the any data shard when the data skewness of any data shard is greater than a skew threshold; adjusting the data shard to which the Key belongs in the any data shard based on the data volume deviation state of the Key, and adding the mapping relationship between the adjusted Key and the corresponding data shard to a data shard-Key mapping table; controlling all upstream Tasks of the target Task to perform data distribution to the target Task in conjunction with the data shard-Key mapping table.

[0126] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an online data skew adjustment method for stream computing jobs provided in the above embodiments. The method includes: acquiring data volume statistics of each computing thread Task within a time period; determining the data skewness of each data shard in the target Task based on the data volume statistics when the overall data volume statistics of any target Task are unbalanced; analyzing the data volume deviation state of the Key in the any data shard when the data skewness of any data shard is greater than a skew threshold; adjusting the data shard to which the Key belongs in the any data shard based on the data volume deviation state of the Key, and adding the mapping relationship between the adjusted Key and the corresponding data shard to a data shard-Key mapping table; and controlling all upstream Tasks of the target Task to perform data distribution to the target Task in conjunction with the data shard-Key mapping table.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for online data skew adjustment of a stream computing job, characterized in that, include: Obtain statistical information on the amount of data for each computation thread Task within a given time period; When the overall data volume statistics of any target task are unbalanced, the data skewness of each data shard in the target task is determined based on the data volume statistics. If the data skewness of any data shard is greater than the skewness threshold, analyze the data volume deviation status of the Key in that data shard. Based on the data volume deviation status of the Key, adjust the data shard to which the Key belongs in any data shard, and add the mapping relationship between the adjusted Key and the corresponding data shard to the data shard-Key mapping table; Control all upstream tasks of the target task, and in conjunction with the data sharding-key mapping table, perform data distribution to the target task; The step of adjusting the data shard to which the Key belongs in any data shard according to the data volume deviation status of the Key includes: If the data volume deviation of the Key is balanced, the Key with the smallest data volume in any data shard will be adjusted to other data shards in the target Task. When the data volume deviation of the Key is unbalanced, the data shard with the largest data volume and the data shard with the smallest data volume in the target Task are determined, and at least one Key in the data shard with the largest data volume is adjusted to the data shard with the smallest data volume.

2. The online data skew adjustment method for stream computing jobs according to claim 1, characterized in that, The data volume statistics include: the total data volume of each data shard and the total data volume of each key within each time period.

3. The online data skew adjustment method for stream computing jobs according to claim 1, characterized in that, The process of adding the mapping relationship between the key to be adjusted and the corresponding data shard to the data shard-key mapping table includes: For any adjusted Key, if the mapping relationship between the adjusted Key and the corresponding data shard is inconsistent with the calculation result of the mapping relationship function, then the mapping relationship between the Key and the corresponding data shard is added to the data shard-Key mapping table.

4. The online data skew adjustment method for stream computing jobs according to claim 1, characterized in that, Controlling all upstream tasks of the target task, and in conjunction with the shard-key mapping table, performing data distribution to the target task, including: A first notification is sent to all upstream tasks based on the data sharding-key mapping relationship to control all upstream tasks to suspend data distribution; Send a second notification to all downstream Tasks where the target Task is located to determine the state corresponding to the Key that needs to be migrated and the location of the state; After all the downstream Tasks respond to the second notification and migrate the state corresponding to the Key, update the mapping relationship between the Task, data shard, and Key, and send the mapping relationship to all the upstream Tasks; Control all upstream Tasks to distribute data to the target Task according to the mapping relationship.

5. The online data skew adjustment method for stream computing jobs according to claim 4, characterized in that, The control of all upstream Tasks to distribute data to the target Task according to the mapping relationship includes: If the upstream Task determines that the key to be distributed is in the data shard-key mapping table, then the data shard corresponding to the key to be distributed is determined according to the data shard-key mapping table, and the key to be distributed is assigned to the data shard. If the upstream Task determines that the Key to be distributed is not in the data shard-Key mapping table, the data shard number corresponding to the Key to be distributed is calculated using the data shard-Key mapping function; based on the data shard number, the downstream Task number corresponding to the data shard number is found from the Task-data shard mapping relationship to complete the allocation of the Key to be distributed.

6. The online data skew adjustment method for stream computing jobs according to claim 1, characterized in that, Before obtaining the data volume statistics for each computation thread Task within a time period, the following is also included: Based on the hash algorithm, data shards are allocated to each Task, and a Task-data shard mapping table is constructed. Each upstream Task distributes the corresponding stream computing job data to the corresponding downstream Task based on the mapping relationship between its downstream Task and the data shard; A hash function is used to determine the data shard number to which each key belongs, so that all keys corresponding to the stream computing job data can be evenly distributed to each data shard.

7. An online data skew adjustment device for stream computing jobs, characterized in that, include: The data acquisition unit is used to obtain statistical information on the amount of data of each computing thread Task within a time period; The data skewness calculation unit is used to determine the data skewness of each data fragment in the target task based on the data volume statistics when the overall data volume statistics of any target task are unbalanced. The deviation status analysis unit is used to analyze the data quantity deviation status of the Key in any data segment when the data skewness of any data segment is greater than the skewness threshold. The data skew adjustment method is used to adjust the data shard to which the Key belongs in any data shard according to the data volume deviation state of the Key, and add the mapping relationship between the adjusted Key and the corresponding data shard to the data shard-Key mapping table; The step of adjusting the data shard to which the Key belongs in any data shard according to the data volume deviation status of the Key includes: If the data volume deviation of the Key is balanced, the Key with the smallest data volume in any data shard will be adjusted to other data shards in the target Task. When the data volume deviation of the Key is unbalanced, determine the data shard with the largest data volume and the data shard with the smallest data volume in the target Task, and adjust at least one Key in the data shard with the largest data volume to the data shard with the smallest data volume. The data distribution unit is used to control all upstream tasks of the target task and, in conjunction with the data sharding-key mapping table, perform data distribution to the target task.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the online data skew adjustment method for the stream computing job as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the online data skew adjustment method for the stream computing job as described in any one of claims 1 to 6.

Images