Data partitioning method and device and computer equipment

Abstract

The invention relates to a data partitioning method and device and computer equipment. The method comprises the steps of obtaining a to-be-partitioned data set output by a current calculation task in a mapping stage; according to the current task scene features of the current calculation task in the data shuffling stage, determining the current partition number of the current calculation task in the data shuffling stage; configuring the number of data partitions of the current partition number; and according to the current partition number and the data identifier of each piece of to-be-partitioned data in the to-be-partitioned data set, determining a data partition to which each piece of to-be-partitioned data belongs, and distributing each piece of to-be-partitioned data to the corresponding data partition for processing. By adopting the method, the execution efficiency of the current calculation task can be improved, and calculation resources are saved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data partitioning method, apparatus, and computer equipment. Background Technology

[0002] In today's digital age, the scale and complexity of data are exploding. Enterprises and organizations need to process massive amounts of data to extract valuable information to support decision-making, business optimization, and innovation. Big data processing technologies have therefore become crucial. Distributed computing frameworks can effectively improve data processing efficiency and are widely used in large-scale data processing tasks.

[0003] In distributed computing framework-based computing tasks, the data shuffle phase is a specific stage between the map and reduce phases. During the shuffle phase, the output data from the map phase needs to be copied between different nodes in the computing cluster, based on the number of partitions, to achieve data partitioning. The number of partitions determines the data distribution, the parallelism of subsequent data processing, resource usage, etc., directly affecting the computational efficiency and computing resources used in the shuffle phase.

[0004] Currently, before the Shuffle stage, the number of partitions is usually set by technicians based on experience. However, accurately setting the number of partitions based solely on experience may lead to uneven data distribution and insufficient parallelism if the number of partitions is not set reasonably, resulting in low execution efficiency of the computing task. Summary of the Invention

[0005] Therefore, it is necessary to provide a data partitioning method, apparatus, and computer device that can improve the execution efficiency of computing tasks, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a data partitioning method, including:

[0007] Obtain the dataset to be partitioned, output by the current computation task during the mapping phase;

[0008] Based on the current task scenario characteristics of the current computing task in the data shuffling stage, determine the current number of partitions for the current computing task in the data shuffling stage.

[0009] Configure the current number of data partitions;

[0010] Based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, the data partition to which each piece of data to be partitioned belongs is determined, and each piece of data to be partitioned is assigned to the corresponding data partition for processing.

[0011] In one embodiment, determining the current number of partitions for the current computing task in the data shredding stage based on the current task scenario characteristics of the current computing task in the data shredding task includes:

[0012] Based on the current task scenario characteristics of the current computing task in the data shuffling stage, and the historical task scenario characteristics of each historical computing task in the data shuffling stage, the matching results between the current computing task and each of the historical computing tasks are determined.

[0013] Based on the matching results, the current number of partitions for the current computing task in the data shuffling stage is determined.

[0014] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result includes:

[0015] If the matching result indicates the existence of a first computing task, the current number of partitions of the current computing task in the data shuffling stage is determined based on the number of historical partitions corresponding to the first computing task; wherein, the first computing task is a historical computing task that matches the current computing task.

[0016] If the matching result is that there is no match with the first computing task, the current task scenario features are input into the partitioning model to obtain the current number of partitions for the current computing task in the data shuffling stage.

[0017] In one embodiment, the method further includes:

[0018] For each historical computing task under the characteristics of each historical task scenario, based on the task execution status of each historical computing task in the data shuffling stage, a second computing task that meets the preset execution conditions is selected from each of the historical computing tasks; and...

[0019] Based on the historical task scenario features and the number of historical partitions corresponding to the second computing task, a training sample corresponding to the historical task scenario features is constructed.

[0020] The initial model is iteratively trained using training samples corresponding to the features of each historical task scenario to obtain the partitioning model.

[0021] In one embodiment, determining the matching result between the current computing task and each of the historical computing tasks based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage includes:

[0022] The current task scenario features of the current computing task in the data shuffling stage are matched with the historical task scenario features of each historical computing task in the data shuffling stage.

[0023] When the current task scenario features do not perfectly match the features of each of the historical task scenarios, the current task scenario features are mapped to obtain a current feature vector; and,

[0024] Map each of the historical task scene features to obtain a historical feature vector for each of the historical task scene features.

[0025] Determine the vector similarity between the current feature vector and each of the historical feature vectors, and determine the matching result between the current computing task and each of the historical computing tasks based on the highest vector similarity among the vector similarities.

[0026] In one embodiment, determining the matching result between the current computing task and each of the historical computing tasks based on the highest vector similarity among the vector similarities includes:

[0027] If the highest vector similarity among the vector similarities is greater than the similarity threshold, the matching result between the current computing task and each of the historical computing tasks is determined to be a first computing task that matches the current computing task; wherein, the first computing task is the historical computing task corresponding to the highest vector similarity.

[0028] If the highest vector similarity is not greater than the similarity threshold, the matching result is determined to be that the first computation task does not exist.

[0029] In one embodiment, the current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

[0030] In one embodiment, the data partition has a corresponding partition identifier, and both the data identifier and the partition identifier are of numeric type;

[0031] The step of determining the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned includes:

[0032] For each piece of data to be partitioned in the dataset to be partitioned, a remainder is obtained by performing a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned.

[0033] Select the partition identifier that is equal to the remainder from the partition identifiers corresponding to each data partition;

[0034] The data partition corresponding to the selected partition identifier is taken as the data partition to which the data to be partitioned belongs.

[0035] Secondly, this application also provides a data partitioning device, comprising:

[0036] The acquisition module is used to acquire the dataset to be partitioned, which is output by the current computing task during the mapping phase.

[0037] The determination module is used to determine the current number of partitions of the current computing task in the data shuffling stage based on the current task scenario characteristics of the current computing task in the data shuffling stage.

[0038] The configuration module is used to configure the current number of data partitions.

[0039] The allocation module is used to determine the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, and to allocate each piece of data to the corresponding data partition for processing.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the various method embodiments provided in the first aspect above.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the various method embodiments provided in the first aspect above.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the various method embodiments provided in the first aspect above.

[0043] The aforementioned data partitioning method, apparatus, and computer equipment determine the current number of partitions based on the current task scenario characteristics of the current computing task in the data shuffling stage, configure the current number of data partitions, and determine the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned output in the mapping stage. The data to be partitioned is then assigned to the corresponding data partition for processing. By accurately determining the appropriate number of partitions for the current computing task based on its current task scenario characteristics in real time, data partitioning based on the current number of partitions can improve resource utilization during the data partitioning process. Furthermore, a reasonable number of partitions can improve the parallelism of subsequent data processing, increase data processing efficiency, and ultimately improve the task execution efficiency of the computing task. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a diagram illustrating the application environment of the data partitioning method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a data partitioning method in one embodiment;

[0047] Figure 3 This is a flowchart illustrating the steps for determining the current number of partitions in one embodiment;

[0048] Figure 4 This is a flowchart illustrating the steps for determining a matching result in one embodiment;

[0049] Figure 5 This is a flowchart illustrating the training steps of a partitioning model in one embodiment;

[0050] Figure 6 This is a flowchart illustrating the steps for determining a data partition in one embodiment;

[0051] Figure 7 This is a flowchart illustrating the data partitioning method in another embodiment;

[0052] Figure 8 This is a structural block diagram of a data partitioning device in one embodiment;

[0053] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0056] In today's digital age, the scale and complexity of data are exploding. Enterprises and organizations need to process massive amounts of data to extract valuable information to support decision-making, business optimization, and innovation. Big data processing technologies have therefore become crucial. Distributed computing frameworks can effectively improve data processing efficiency and are widely used in large-scale data processing tasks.

[0057] In distributed computing framework-based computing tasks, the data shuffle phase is a specific stage between the map and reduce phases. During the shuffle phase, the output data from the map phase needs to be copied between different nodes in the computing cluster, based on the number of partitions, to achieve data partitioning. The number of partitions determines the data distribution, the parallelism of subsequent data processing, resource usage, etc., directly affecting the computational efficiency and computing resources used in the shuffle phase.

[0058] Currently, before the Shuffle stage, the number of partitions is usually set by technicians based on experience. However, accurately setting the number of partitions based solely on experience may lead to uneven data distribution and insufficient parallelism if the number of partitions is not set reasonably, resulting in low execution efficiency of the computing task.

[0059] The data partitioning method provided in this application embodiment can be applied to, for example... Figure 1The distributed computing cluster shown includes mapping nodes, data shuffling nodes, and aggregation nodes, which communicate with each other via a network. The data shuffling node obtains the dataset to be partitioned from the mapping phase output of the current computing task (i.e., the output of the mapping node). Based on the current task scenario characteristics during the data shuffling phase, it determines the current number of partitions for the current computing task, configures the current number of data partitions, and determines the data partition to which each piece of data belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset. It then assigns each piece of data to the corresponding data partition so that the aggregation node can process the data in the data partition. The mapping nodes, data shuffling nodes, and aggregation nodes can all be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services.

[0060] In one exemplary embodiment, such as Figure 2 As shown, a data partitioning method is provided. Taking the application of this method to the partitioner in the Shuffle stage as an example, it is illustrated as follows:

[0061] S201: Obtain the dataset to be partitioned, which is output by the current computation task in the mapping phase.

[0062] The current computing task can be a distributed computing task executed on a distributed computing framework (such as Spark), the mapping stage can be the Map stage, and the dataset to be partitioned needs to be copied between different nodes in the cluster during the data shuffle stage to distribute the data to be partitioned to different data partitions.

[0063] S202: Based on the current task scenario characteristics of the current computing task in the data shuffling stage, determine the current number of partitions for the current computing task in the data shuffling stage.

[0064] The current task scenario features represent the scenario information of the current computing task during the data shuffling phase. Optionally, the current task scenario features may include at least one of the following: cluster resource features of the current computing task, data features of the dataset to be partitioned, and task configuration features of the current computing task during the data shuffling phase. Cluster resource features represent the resource usage of the cluster, such as the number of remaining Central Processing Unit (CPU) cores, available memory size, and number of cluster nodes. Data features represent the data distribution of the dataset to be partitioned, such as the amount of data in the dataset, the number of data entries to be partitioned, and the distribution of data keys. Task configuration features represent the configuration parameters of the current computing task during the data shuffling phase, such as the shuffle operator type (e.g., reduce By Key type, join type) and the default number of partitions.

[0065] Based on the characteristics of the current task scenario, the number of partitions applicable to the current computing task can be obtained by effectively combining the actual characteristics of the cluster.

[0066] S203: Configure the number of data partitions in the current partition.

[0067] For example, if the current number of partitions is 100, then configure 100 data partitions.

[0068] S204: Based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, determine the data partition to which each piece of data to be partitioned belongs, and assign each piece of data to the corresponding data partition for processing.

[0069] The data identifier for the data to be partitioned is generated based on the processing logic of the map function in the mapping phase. The specific identifier can be set according to actual needs; for example, it can be a field from the input data, or it can be obtained through data transformation or calculation operations on the input data. The data type of the identifier can be string, numeric, etc., without specific limitations. When determining the data partition to which each piece of data belongs, the data to be partitioned can be evenly distributed across the data partitions, or data with the same identifier can be assigned to the same data partition, or the similarity between the partition identifier and the data identifier can be used to determine the data partition to which each piece of data belongs, without specific limitations. After the data to be partitioned is assigned to the corresponding data partition, the data in each data partition will be processed by a dedicated reduce node.

[0070] In the above method, the number of partitions is determined based on the current task scenario characteristics of the current computing task in the data shuffling stage. This number of data partitions is then configured. Based on the current number of partitions and the data identifier of each piece of data to be partitioned output in the mapping stage, the data partition to which each piece of data belongs is determined, and the data to be partitioned is assigned to the corresponding data partition for processing. By accurately determining the appropriate number of partitions for the current computing task in real time based on its current task scenario characteristics, data partitioning based on this number can improve resource utilization during the data partitioning process. Furthermore, a reasonable number of partitions can enhance the parallelism of subsequent data processing, improve data processing efficiency, and ultimately improve the task execution efficiency of the computing task.

[0071] In one exemplary embodiment, such as Figure 3 As shown, the steps for determining the current number of partitions are further refined, including:

[0072] S301: Based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage, determine the matching results between the current computing task and each historical computing task.

[0073] The matching results between the current computing task and each historical computing task are used to characterize whether there are any historical computing tasks that match the current computing task. It can be understood that matching the current computing task with historical computing tasks is actually matching the current task scenario features with the historical task scenario features. Optionally, in the matching process, matching can be performed based on the similarity between the current task scenario features and the historical task scenario features to obtain the matching results.

[0074] S302: Based on the matching results, determine the current number of partitions in the data shuffling phase of the current computing task.

[0075] Specifically, if there are historical computing tasks that match the current computing task, the number of historical partitions in the matching historical computing tasks can be used to determine the current number of partitions. For example, the number of historical partitions can be directly used as the current number of partitions, or the current number of partitions can be obtained by taking a value near the number of historical partitions. If there are no historical computing tasks that match the current computing task, the current number of partitions can be determined based on the characteristics of the current task scenario.

[0076] In the above embodiments, by matching the current computing task with each historical computing task, the current number of partitions can be obtained by combining the historical number of partitions of the historical computing tasks, ensuring the rationality of the current number of partitions, thereby improving task execution efficiency.

[0077] In one exemplary embodiment, such as Figure 4 As shown, the steps for determining the matching result in S301 are further refined, including:

[0078] S401: Match the current task scenario features of the current computing task in the data shuffling stage with the historical task scenario features of each historical computing task in the data shuffling stage.

[0079] The process involves matching the features of the current task scenario with those of historical task scenarios. This comparison assesses the similarity of the feature data contained in the current and historical task scenario features. Optionally, the current task scenario features include a first current parameter and a second current parameter, while the historical task scenario features include a first historical parameter and a second historical parameter. If the first current parameter and the first historical parameter are identical, and the error between the second current parameter and the second historical parameter is within the acceptable error range, the current task scenario features are determined to be a complete match with the historical task scenario features. Otherwise, the current task scenario features are determined to be a partial match with the historical task scenario features. Specifically, the first current parameter and the first historical parameter may include the number of cluster nodes for the corresponding computation task, the distribution of data identifiers for the data to be partitioned, and the type of shuffle operator. The second current parameter and the second historical parameter may include the number of remaining CPU cores in the cluster for the corresponding computation task, the available memory size of the cluster, the amount of data in the dataset to be partitioned, and the number of data entries to be partitioned in the dataset to be partitioned. The error range can be set according to the actual parameters and is not specifically limited here.

[0080] Optionally, if there are historical task scenario features that completely match the current task scenario features, the matching result is determined to be that there is a historical computing task that matches the current computing task.

[0081] S402: When the features of the current task scenario do not completely match the features of each historical task scenario, map the features of the current task scenario to obtain the current feature vector; and map each historical task scenario feature to obtain the historical feature vector of each historical task scenario feature.

[0082] Optionally, the current task scene features can be mapped to a pre-trained natural language processing network to obtain a current feature vector, and the historical task scene features can be mapped to obtain a historical feature vector.

[0083] S403: Determine the vector similarity between the current feature vector and each historical feature vector, and determine the matching result between the current computing task and each historical computing task based on the highest vector similarity among all vector similarities.

[0084] The vector similarity between the current feature vector and the historical feature vector can be cosine similarity. For example, the vector similarity 1 between the current feature vector and historical feature vector 1 is 0.5, the vector similarity 2 between the current feature vector and historical feature vector 2 is 0.9, and the vector similarity 3 between the current feature vector and historical feature vector 3 is 0.2. The highest vector similarity is 0.9, which means that the historical computation task corresponding to the highest vector similarity matches the current computation task. Alternatively, a similarity threshold can be used to further filter the highest similarity to determine whether the historical computation task corresponding to the highest similarity matches the current computation task.

[0085] In the above embodiments, by filtering historical computing tasks based on the vector similarity between the current feature vector and each historical feature vector, the matching between the current computing task and the historical computing task can be accurately identified, thereby improving the matching accuracy.

[0086] In an exemplary embodiment, the step of determining the matching result in S403 is further refined, including:

[0087] If the highest vector similarity among all vector similarities is greater than the similarity threshold, the matching result between the current computing task and each historical computing task is determined as the existence of a first computing task that matches the current computing task; if the highest vector similarity is not greater than the similarity threshold, the matching result is determined as the existence of a first computing task.

[0088] The similarity threshold can be set based on experience, multiple trials, and actual needs. The first calculation task is the historical calculation task corresponding to the highest vector similarity. Taking historical feature vector 1, historical feature vector 2, and historical feature vector 3 as an example, the highest vector similarity is 0.9, and the similarity threshold is 0.85. Therefore, the first calculation task is historical calculation task 2 corresponding to historical feature vector 2. If the highest vector similarity is not greater than the similarity threshold, it is considered that each historical calculation task has a low matching degree with the current calculation task, and the matching result is determined to be that there is no first calculation task.

[0089] In the above embodiments, further filtering of historical computing tasks using a similarity threshold can ensure the matching between the first computing task and the current computing task and improve the accuracy of the matching results.

[0090] In one exemplary embodiment, the step of determining the current number of partitions based on the matching results is further refined, including:

[0091] If the matching result indicates the existence of a first computing task, the current number of partitions for the current computing task in the data shuffling stage is determined based on the number of historical partitions corresponding to the first computing task. If the matching result indicates the absence of a first computing task, the current task scenario features are input into the partitioning model to obtain the current number of partitions for the current computing task in the data shuffling stage.

[0092] The first computation task is a historical computation task that matches the current computation task; that is, the historical computation task with the highest similarity score (greater than the similarity threshold) mentioned above, or a historical computation task that perfectly matches the current computation task. Optionally, the number of historical partitions of the first computation task can be used as the number of current partitions. The partitioning model is a trained neural network model used to determine the number of partitions. The current task scene features are input into the partitioning model, and the partitioning model outputs the current number of partitions based on the current task scene features.

[0093] In the above embodiments, determining the current number of partitions in the data shuffling stage of the current computing task through a trained partitioning model can improve the accuracy of the current number of partitions, thereby improving the task execution efficiency of the current computing task.

[0094] In one exemplary embodiment, such as Figure 5 As shown, the partitioning model is trained through the following steps:

[0095] S501: For each historical computing task under each type of historical task scenario feature, based on the task execution status of each historical computing task in the data shuffling stage, select a second computing task that meets the preset execution conditions from each historical computing task; and construct training samples corresponding to the historical task scenario features based on the historical task scenario features and the number of historical partitions corresponding to the second computing task.

[0096] It is understandable that each historical computing task has corresponding historical task scenario features. Therefore, each historical computing task can be classified according to the corresponding historical task scenario features. Optionally, the historical task scenario features can be clustered to obtain the historical task scenario features of each category. Furthermore, the similarity between each historical task scenario feature and other historical task scenario features can be determined separately. Historical task scenario features with similarity greater than a preset threshold can be classified into one category. The preset threshold can be set according to actual needs.

[0097] In the process of constructing training samples, for each type of historical task scenario feature, the historical task scenario feature is used as sample data. There may be multiple historical computing tasks under the historical task scenario feature, so it is necessary to filter each historical computing task and use the number of historical partitions of the second computing task as the label of the sample data, so as to construct the training samples corresponding to the historical task scenario feature through the sample data and the label.

[0098] When filtering historical computing tasks, the filtering is based on the task execution status of the historical computing tasks in the data shuffling stage. The task execution status can include the execution time of the historical computing tasks in the Shuffle stage, the average processing time of each Reducer in the Reduce stage, the cluster resource idle rate (such as CPU idle rate), and the data skew occurrence rate of each data partition. If the task execution status of a historical computing task meets the following conditions: the execution time in the Shuffle stage is ≤ the execution time threshold, the cluster CPU utilization ( / or memory utilization) is ≥ the utilization threshold, and the data skew occurrence rate is less than the occurrence rate threshold, then the number of historical partitions corresponding to the historical computing task can be used as a label. The execution time threshold, utilization threshold, and occurrence rate threshold can all be set based on experience, multiple experiments, and actual needs. For example, the average execution time of each historical computing task under the characteristics of the historical task scenario can be determined, and 80% of the average execution time can be used as the execution time threshold.

[0099] In addition, if multiple historical computing tasks meet the above conditions, the number of historical partitions of the historical computing task with the shortest execution time will be used as the label.

[0100] S502: Using training samples corresponding to the features of each historical task scenario, the initial model is iteratively trained to obtain the partitioned model.

[0101] The training sample set consists of training samples corresponding to the features of each historical task scenario. The initial model is iteratively trained to obtain the partition model. Optionally, after obtaining the partition model, each time the current task scenario features are input into the partition model to obtain the current number of partitions, the task execution status of the current computing task in the Shuffle stage can be collected. Based on the current task scenario features, the current number of partitions, and the task execution status, training samples can be reconstructed, and the partition model can be retrained to improve model performance.

[0102] In the above embodiments, the accuracy of training data is improved by filtering historical computing tasks under the characteristics of historical task scenarios, ensuring the performance of the partition model, and the partition model is obtained by continuously iterating based on production data. The partition model dynamically provides the number of partitions in the Shuffle stage according to the cluster status. As task data accumulates, the model provides increasingly accurate predictions, giving full play to the capabilities of the distributed computing engine.

[0103] In one exemplary embodiment, the data partition has a corresponding partition identifier, and both the data identifier and the partition identifier are numeric; for example... Figure 6 As shown, the steps for determining data partitions are further detailed, including:

[0104] S601: For each piece of data to be partitioned in the dataset to be partitioned, perform a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned, and obtain the remainder.

[0105] For example, if the data identifier of the data to be partitioned is 100 and the current number of partitions is 15, then the remainder is 10.

[0106] S602: Select the partition identifier that is equal to the remainder from the partition identifiers corresponding to each data partition.

[0107] The partition identifiers for each data partition are encoded starting from 0. If the current number of partitions is 20, the partition identifiers for the data partitions are encoded from 0 to 19. For example, if the remainder is 10, then the partition identifier with the same remainder is 10.

[0108] S603: The data partition corresponding to the selected partition identifier is used as the data partition to which the data to be partitioned belongs.

[0109] In the above embodiments, by allocating the data to be partitioned according to the data identifier and partition identifier, the data skew problem can be effectively avoided, the task execution efficiency can be improved, and long-tail tasks can be optimized.

[0110] Based on the above embodiments, in an exemplary embodiment, before the Shuffle phase of the current computing task is executed, it is first determined whether there is a historical computing task that completely matches the current computing task based on the current task scenario features of the current computing task and the historical task scenario features of each historical computing task. If there is a completely matching historical computing task, the number of partitions of the completely matching historical computing task is taken as the current number of partitions. If there is no matching historical computing task, the cosine similarity between the current task scenario features and the historical task scenario features is calculated. If the highest cosine similarity is greater than the similarity threshold, the number of partitions of the historical computing task corresponding to the highest cosine similarity is taken as the current number of partitions. Otherwise, the current task scenario features are input into the partitioning model to obtain the current number of partitions.

[0111] Based on the above embodiments, in an exemplary embodiment, such as Figure 7 As shown, the data partitioning method in this application embodiment may include the following steps:

[0112] S701: Obtain the dataset to be partitioned, output by the current computation task during the mapping phase.

[0113] S702: Match the current task scenario features of the current computing task in the data shuffling stage with the historical task scenario features of each historical computing task in the data shuffling stage.

[0114] The current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

[0115] S703: When the features of the current task scenario do not completely match the features of each historical task scenario, map the features of the current task scenario to obtain the current feature vector; and map each historical task scenario feature to obtain the historical feature vector of each historical task scenario feature; determine the vector similarity between the current feature vector and each historical feature vector.

[0116] S704: If the highest similarity among all vector similarities is greater than the similarity threshold, determine that the matching result between the current computing task and each historical computing task is the first computing task that matches the current computing task; determine the number of current partitions of the current computing task in the data shuffling stage based on the number of historical partitions corresponding to the first computing task.

[0117] Among them, the first calculation task is the historical calculation task corresponding to the highest vector similarity;

[0118] S705: If the highest vector similarity is not greater than the similarity threshold, determine that the matching result is that there is no first computation task; input the current task scenario features into the partitioning model to obtain the current number of partitions for the current computation task in the data shuffling stage.

[0119] S706: Configure the number of data partitions in the current partition.

[0120] S707: For each piece of data to be partitioned in the dataset to be partitioned, perform a remainder operation on the current number of partitions based on the data identifier of the data to be partitioned; select the partition identifier that is equal to the remainder from the partition identifiers corresponding to each data partition; and take the data partition corresponding to the selected partition identifier as the data partition to which the data to be partitioned belongs.

[0121] S708: Assign each piece of data to be partitioned to the corresponding data partition for processing.

[0122] The specific implementation methods of S701-S708 are the same as those in the above method embodiments, and will not be repeated here.

[0123] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0124] Based on the same inventive concept, this application also provides a data partitioning apparatus for implementing the data partitioning method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data partitioning apparatus embodiments provided below can be found in the limitations of the data partitioning method described above, and will not be repeated here.

[0125] In one exemplary embodiment, such as Figure 8 As shown, a data partitioning device is provided, including: an acquisition module 801, a determination module 802, a configuration module 803, and an allocation module 804, wherein:

[0126] The acquisition module 801 is used to acquire the dataset to be partitioned output by the current computing task in the mapping phase;

[0127] The determination module 802 is used to determine the current number of partitions of the current computing task in the data shuffling stage based on the current task scenario characteristics of the current computing task in the data shuffling stage.

[0128] Configuration module 803 is used to configure the current number of data partitions;

[0129] The allocation module 804 is used to determine the data partition to which each piece of data belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, and to allocate each piece of data to the corresponding data partition for processing.

[0130] In one exemplary embodiment, the determining module 802 includes:

[0131] The matching unit is used to determine the matching result between the current computing task and each historical computing task based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage.

[0132] The determination unit is used to determine the current number of partitions in the data shuffling stage of the current computing task based on the matching results.

[0133] In one exemplary embodiment, the determining unit is specifically used for:

[0134] If the matching result indicates the existence of a first computing task, the current number of partitions for the current computing task in the data shuffling stage is determined based on the number of historical partitions corresponding to the first computing task; wherein, the first computing task is a historical computing task that matches the current computing task.

[0135] If the matching result does not match the first computation task, the current task scenario features are input into the partitioning model to obtain the current number of partitions for the current computation task in the data shuffling stage.

[0136] In one exemplary embodiment, the determining unit is further configured to:

[0137] For each historical computation task under the characteristics of each historical task scenario, based on the task execution status of each historical computation task in the data shuffling stage, a second computation task that meets the preset execution conditions is selected from each historical computation task; and,

[0138] Based on the characteristics of historical task scenarios and the number of historical partitions corresponding to the second computation task, training samples corresponding to the characteristics of historical task scenarios are constructed.

[0139] The initial model is iteratively trained using training samples corresponding to the features of each historical task scenario to obtain the partitioned model.

[0140] In one exemplary embodiment, the matching unit is specifically used for:

[0141] The current task scenario characteristics of the current computing task in the data washing stage are matched with the historical task scenario characteristics of each historical computing task in the data washing stage.

[0142] When the features of the current task scenario do not perfectly match the features of all historical task scenarios, the features of the current task scenario are mapped to obtain the current feature vector; and,

[0143] Map the features of each historical task scenario to obtain the historical feature vector of each historical task scenario feature.

[0144] Determine the vector similarity between the current feature vector and each historical feature vector, and determine the matching result between the current computing task and each historical computing task based on the highest vector similarity among all vector similarities.

[0145] In one exemplary embodiment, the matching unit is specifically used for:

[0146] If the highest vector similarity among all vector similarities is greater than the similarity threshold, the matching result between the current computing task and each historical computing task is determined as the first computing task that matches the current computing task; where the first computing task is the historical computing task corresponding to the highest vector similarity.

[0147] If the highest vector similarity is not greater than the similarity threshold, the matching result is determined to be that there is no first computation task.

[0148] In an exemplary embodiment, the current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

[0149] In one exemplary embodiment, the data partition has a corresponding partition identifier, and both the data identifier and the partition identifier are of numeric type;

[0150] The allocation module 804 is specifically used for:

[0151] For each piece of data to be partitioned in the dataset to be partitioned, perform a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned, and obtain the remainder;

[0152] Select the partition identifier that is equal to the remainder from the partition identifiers corresponding to each data partition;

[0153] The data partition corresponding to the selected partition identifier is used as the data partition to which the data to be partitioned belongs.

[0154] Each module in the aforementioned data partitioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0155] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores information such as the number of historical partitions and characteristics of historical task scenarios. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a data partitioning method.

[0156] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to: acquire the dataset to be partitioned output by the current computing task in the mapping phase; determine the current number of partitions in the data shuffling phase of the current computing task based on the current task scenario characteristics of the current computing task in the data shuffling phase; configure the current number of data partitions; determine the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, and assign each piece of data to be partitioned to the corresponding data partition for processing.

[0158] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the current task scenario characteristics of the current computing task in the data shuffling stage includes: determining the matching result between the current computing task and each historical computing task based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage; and determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result.

[0159] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result includes: if the matching result indicates the existence of a first computing task, determining the current number of partitions for the current computing task in the data shuffling stage based on the historical number of partitions corresponding to the first computing task; wherein the first computing task is a historical computing task that matches the current computing task; if the matching result indicates the absence of a first computing task, inputting the current task scenario features into the partitioning model to obtain the current number of partitions for the current computing task in the data shuffling stage.

[0160] In one embodiment, the method further includes: for each historical computing task under each type of historical task scenario feature, selecting a second computing task that meets the preset execution conditions from each historical computing task based on the task execution status of each historical computing task in the data shuffling stage; and constructing training samples corresponding to the historical task scenario features based on the historical task scenario features and the number of historical partitions corresponding to the second computing task; and using the training samples corresponding to each historical task scenario feature to iteratively train the initial model to obtain a partition model.

[0161] In one embodiment, the matching result between the current computing task and each historical computing task is determined based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage. This includes: matching the current task scenario characteristics of the current computing task in the data shuffling stage with the historical task scenario characteristics of each historical computing task in the data shuffling stage; mapping the current task scenario characteristics to obtain a current feature vector when the current task scenario characteristics do not completely match the historical task scenario characteristics; mapping each historical task scenario characteristic to obtain a historical feature vector for each historical task scenario characteristic; determining the vector similarity between the current feature vector and each historical feature vector; and determining the matching result between the current computing task and each historical computing task based on the highest vector similarity among the vector similarities.

[0162] In one embodiment, determining the matching result between the current computing task and each historical computing task based on the highest vector similarity among all vector similarities includes: if the highest vector similarity among all vector similarities is greater than a similarity threshold, determining that the matching result between the current computing task and each historical computing task is that there is a first computing task matching the current computing task; wherein, the first computing task is the historical computing task corresponding to the highest vector similarity; if the highest vector similarity is not greater than the similarity threshold, determining that the matching result is that there is no first computing task.

[0163] In one embodiment, the current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

[0164] In one embodiment, the data partitions have corresponding partition identifiers, and both the data identifiers and partition identifiers are of numeric type. Determining the data partition to which each piece of data belongs, based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, includes: for each piece of data to be partitioned in the dataset to be partitioned, performing a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned, obtaining a remainder; selecting the partition identifier from the partition identifiers corresponding to each data partition that is equal to the remainder; and taking the data partition corresponding to the selected partition identifier as the data partition to which the data to be partitioned belongs.

[0165] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following: obtaining the dataset to be partitioned output by the current computing task in the mapping phase; determining the current number of partitions in the data shuffling phase of the current computing task based on the current task scenario characteristics of the current computing task in the data shuffling phase; configuring the current number of data partitions; determining the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, and assigning each piece of data to be partitioned to the corresponding data partition for processing.

[0166] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the current task scenario characteristics of the current computing task in the data shuffling stage includes: determining the matching result between the current computing task and each historical computing task based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage; and determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result.

[0167] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result includes: if the matching result indicates the existence of a first computing task, determining the current number of partitions for the current computing task in the data shuffling stage based on the historical number of partitions corresponding to the first computing task; wherein the first computing task is a historical computing task that matches the current computing task; if the matching result indicates the absence of a first computing task, inputting the current task scenario features into the partitioning model to obtain the current number of partitions for the current computing task in the data shuffling stage.

[0168] In one embodiment, the method further includes: for each historical computing task under each type of historical task scenario feature, selecting a second computing task that meets the preset execution conditions from each historical computing task based on the task execution status of each historical computing task in the data shuffling stage; and constructing training samples corresponding to the historical task scenario features based on the historical task scenario features and the number of historical partitions corresponding to the second computing task; and using the training samples corresponding to each historical task scenario feature to iteratively train the initial model to obtain a partition model.

[0169] In one embodiment, the matching result between the current computing task and each historical computing task is determined based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage. This includes: matching the current task scenario characteristics of the current computing task in the data shuffling stage with the historical task scenario characteristics of each historical computing task in the data shuffling stage; mapping the current task scenario characteristics to obtain a current feature vector when the current task scenario characteristics do not completely match the historical task scenario characteristics; mapping each historical task scenario characteristic to obtain a historical feature vector for each historical task scenario characteristic; determining the vector similarity between the current feature vector and each historical feature vector; and determining the matching result between the current computing task and each historical computing task based on the highest vector similarity among the vector similarities.

[0170] In one embodiment, determining the matching result between the current computing task and each historical computing task based on the highest vector similarity among all vector similarities includes: if the highest vector similarity among all vector similarities is greater than a similarity threshold, determining that the matching result between the current computing task and each historical computing task is that there is a first computing task matching the current computing task; wherein, the first computing task is the historical computing task corresponding to the highest vector similarity; if the highest vector similarity is not greater than the similarity threshold, determining that the matching result is that there is no first computing task.

[0171] In one embodiment, the current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

[0172] In one embodiment, the data partitions have corresponding partition identifiers, and both the data identifiers and partition identifiers are of numeric type. Determining the data partition to which each piece of data belongs, based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, includes: for each piece of data to be partitioned in the dataset to be partitioned, performing a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned, obtaining a remainder; selecting the partition identifier from the partition identifiers corresponding to each data partition that is equal to the remainder; and taking the data partition corresponding to the selected partition identifier as the data partition to which the data to be partitioned belongs.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following: acquiring the dataset to be partitioned output by the current computing task in the mapping phase; determining the current number of partitions in the data shuffling phase of the current computing task based on the current task scenario characteristics of the current computing task in the data shuffling phase; configuring the current number of data partitions; determining the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, and assigning each piece of data to be partitioned to the corresponding data partition for processing.

[0174] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the current task scenario characteristics of the current computing task in the data shuffling stage includes: determining the matching result between the current computing task and each historical computing task based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage; and determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result.

[0175] In one embodiment, determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result includes: if the matching result indicates the existence of a first computing task, determining the current number of partitions for the current computing task in the data shuffling stage based on the historical number of partitions corresponding to the first computing task; wherein the first computing task is a historical computing task that matches the current computing task; if the matching result indicates the absence of a first computing task, inputting the current task scenario features into the partitioning model to obtain the current number of partitions for the current computing task in the data shuffling stage.

[0176] In one embodiment, the method further includes: for each historical computing task under each type of historical task scenario feature, selecting a second computing task that meets the preset execution conditions from each historical computing task based on the task execution status of each historical computing task in the data shuffling stage; and constructing training samples corresponding to the historical task scenario features based on the historical task scenario features and the number of historical partitions corresponding to the second computing task; and using the training samples corresponding to each historical task scenario feature to iteratively train the initial model to obtain a partition model.

[0177] In one embodiment, the matching result between the current computing task and each historical computing task is determined based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage. This includes: matching the current task scenario characteristics of the current computing task in the data shuffling stage with the historical task scenario characteristics of each historical computing task in the data shuffling stage; mapping the current task scenario characteristics to obtain a current feature vector when the current task scenario characteristics do not completely match the historical task scenario characteristics; mapping each historical task scenario characteristic to obtain a historical feature vector for each historical task scenario characteristic; determining the vector similarity between the current feature vector and each historical feature vector; and determining the matching result between the current computing task and each historical computing task based on the highest vector similarity among the vector similarities.

[0178] In one embodiment, determining the matching result between the current computing task and each historical computing task based on the highest vector similarity among all vector similarities includes: if the highest vector similarity among all vector similarities is greater than a similarity threshold, determining that the matching result between the current computing task and each historical computing task is that there is a first computing task matching the current computing task; wherein, the first computing task is the historical computing task corresponding to the highest vector similarity; if the highest vector similarity is not greater than the similarity threshold, determining that the matching result is that there is no first computing task.

[0179] In one embodiment, the current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

[0180] In one embodiment, the data partitions have corresponding partition identifiers, and both the data identifiers and partition identifiers are of numeric type. Determining the data partition to which each piece of data belongs, based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, includes: for each piece of data to be partitioned in the dataset to be partitioned, performing a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned, obtaining a remainder; selecting the partition identifier from the partition identifiers corresponding to each data partition that is equal to the remainder; and taking the data partition corresponding to the selected partition identifier as the data partition to which the data to be partitioned belongs.

[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0184] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data partitioning method, characterized in that, The method includes: Obtain the dataset to be partitioned, output by the current computation task during the mapping phase; Based on the current task scenario characteristics of the current computing task in the data shuffling stage, determine the current number of partitions for the current computing task in the data shuffling stage. Configure the current number of data partitions; Based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, the data partition to which each piece of data to be partitioned belongs is determined, and each piece of data to be partitioned is assigned to the corresponding data partition for processing.

2. The method according to claim 1, characterized in that, The step of determining the current number of partitions for the current computing task in the data shredding stage based on the current task scenario characteristics of the current computing task in the data shredding task includes: Based on the current task scenario characteristics of the current computing task in the data shuffling stage, and the historical task scenario characteristics of each historical computing task in the data shuffling stage, the matching results between the current computing task and each of the historical computing tasks are determined. Based on the matching results, the current number of partitions for the current computing task in the data shuffling stage is determined.

3. The method according to claim 2, characterized in that, Determining the current number of partitions for the current computing task in the data shuffling stage based on the matching result includes: If the matching result indicates the existence of a first computing task, the current number of partitions of the current computing task in the data shuffling stage is determined based on the number of historical partitions corresponding to the first computing task; wherein, the first computing task is a historical computing task that matches the current computing task. If the matching result is that there is no match with the first computing task, the current task scenario features are input into the partitioning model to obtain the current number of partitions for the current computing task in the data shuffling stage.

4. The method according to claim 3, characterized in that, The method further includes: For each historical computing task under the characteristics of each historical task scenario, based on the task execution status of each historical computing task in the data shuffling stage, a second computing task that meets the preset execution conditions is selected from each of the historical computing tasks; and... Based on the historical task scenario features and the number of historical partitions corresponding to the second computing task, a training sample corresponding to the historical task scenario features is constructed. The initial model is iteratively trained using training samples corresponding to the features of each historical task scenario to obtain the partitioning model.

5. The method according to claim 2, characterized in that, The step of determining the matching results between the current computing task and each of the historical computing tasks in the data shuffling stage, based on the current task scenario characteristics of the current computing task in the data shuffling stage and the historical task scenario characteristics of each historical computing task in the data shuffling stage, includes: The current task scenario features of the current computing task in the data shuffling stage are matched with the historical task scenario features of each historical computing task in the data shuffling stage. When the current task scenario features do not perfectly match the features of each of the historical task scenarios, the current task scenario features are mapped to obtain a current feature vector; and, Map each of the historical task scene features to obtain a historical feature vector for each of the historical task scene features. Determine the vector similarity between the current feature vector and each of the historical feature vectors, and determine the matching result between the current computing task and each of the historical computing tasks based on the highest vector similarity among the vector similarities.

6. The method according to claim 5, characterized in that, The step of determining the matching result between the current computing task and each of the historical computing tasks based on the highest vector similarity among the vector similarities includes: If the highest vector similarity among the vector similarities is greater than the similarity threshold, the matching result between the current computing task and each of the historical computing tasks is determined to be a first computing task that matches the current computing task; wherein, the first computing task is the historical computing task corresponding to the highest vector similarity. If the highest vector similarity is not greater than the similarity threshold, the matching result is determined to be that the first computation task does not exist.

7. The method according to any one of claims 1 to 6, characterized in that, The current task scenario features include at least one of the following: the cluster resource features of the current computing task, the data features of the dataset to be partitioned, and the task configuration features of the current computing task in the data shuffling stage.

8. The method according to any one of claims 1 to 6, characterized in that, The data partition has a corresponding partition identifier, and both the data identifier and the partition identifier are of numeric type; The step of determining the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned includes: For each piece of data to be partitioned in the dataset to be partitioned, a remainder is obtained by performing a modulo operation on the current number of partitions based on the data identifier of the data to be partitioned. Select the partition identifier that is equal to the remainder from the partition identifiers corresponding to each data partition; The data partition corresponding to the selected partition identifier is taken as the data partition to which the data to be partitioned belongs.

9. A data partitioning device, characterized in that, The device includes: The acquisition module is used to acquire the dataset to be partitioned, which is output by the current computing task during the mapping phase. The determination module is used to determine the current number of partitions of the current computing task in the data shuffling stage based on the current task scenario characteristics of the current computing task in the data shuffling stage. The configuration module is used to configure the current number of data partitions. The allocation module is used to determine the data partition to which each piece of data to be partitioned belongs based on the current number of partitions and the data identifier of each piece of data to be partitioned in the dataset to be partitioned, and to allocate each piece of data to the corresponding data partition for processing.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images