Parallelism degree determination method in distributed operation, electronic equipment and storage medium
By obtaining the data characteristics and candidate parallelism list of distributed jobs, using AI model and jump grid search algorithm to optimize parallelism, the problem of parallelism tuning in distributed computing systems is solved, computing efficiency and stability are improved, and the parallelism tuning is achieved. High efficiency and low overhead parallelism settings.
Patent Information
- Application Number
- CN202310516599.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-08
AI Technical Summary
In distributed computing systems, it is difficult for the prior art to effectively tune the degree of parallelism, resulting in the impact of computing efficiency and stability, especially in the case of large-scale cluster deployment.
By obtaining the data characteristics of distributed jobs, obtaining candidate parallelism lists, and determining the target parallelism based on these characteristics and the execution time of candidate specification phases, using AI model and jump grid search algorithm to optimize the parallelism setting to ensure the distributed system's Performance improvement.
It realizes finding the better parallelism in less time, improves the computing efficiency and stability of the distributed system, reduces the overhead of parallelism tuning, and has no intrusion, low overhead, short time and high accuracy.
Smart Images

Figure CN118916153A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method for determining parallelism in a distributed job, an electronic device, and a storage medium. Background Art
[0002] MapReduce is a distributed parallel computing model that mainly includes three core stages: map, shuffle, and reduce. It has been widely used in mainstream distributed computing systems such as Hadoop and Spark. The setting of parallelism (i.e., the number of tasks in each stage of MapReduce) will directly affect the disk and network overhead in the distributed operation process, thereby affecting the computing efficiency and stability of the entire distributed computing system. Therefore, setting a reasonable parallelism is a key issue in improving the performance of distributed systems. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a method for determining the degree of parallelism in a distributed job, an electronic device, and a storage medium, which can obtain a reasonable degree of parallelism of the distributed job and improve the system performance of the distributed system.
[0004] To solve the above technical problems, the embodiments of the present application are implemented through the following aspects.
[0005] In the first aspect, an embodiment of the present application provides a method for determining parallelism in a distributed job, including: obtaining data features in job data corresponding to the distributed job; obtaining each candidate parallelism in a candidate parallelism list, and determining a candidate specification phase execution time based on the candidate parallelism and the data features; and determining a target parallelism corresponding to the distributed job from the candidate parallelism list based on the candidate specification phase execution time.
[0006] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and computer executable instructions stored in the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of executing the method described in the first aspect.
[0007] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are implemented.
[0008] In an embodiment of the present application, when determining the parallelism in a distributed job, data features in job data corresponding to the distributed job are obtained; each candidate parallelism in a candidate parallelism list is obtained, and a candidate specification phase execution time is determined based on the candidate parallelism and the data features; and a target parallelism corresponding to the distributed job is determined from the candidate parallelism list based on the candidate specification phase execution time, thereby being able to obtain a reasonable parallelism for the distributed job and improve the system performance of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0010] Figure 1 A flow chart of a method for determining the degree of parallelism in a distributed job provided in an embodiment of the present application is shown.
[0011] Figure 2 A schematic diagram showing the model training process provided in an embodiment of the present application is shown.
[0012] Figure 3 A schematic diagram of the structure of a device for determining parallelism in a distributed job provided in an embodiment of the present application is shown.
[0013] Figure 4 A schematic diagram of the hardware structure of an electronic device for executing the method for determining the degree of parallelism in a distributed job provided in an embodiment of the present application. DETAILED DESCRIPTION
[0014] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0015] It should be noted that in the three stages of the map-reduce protocol, the parallelism of the map stage is generally determined by the distributed system based on the data block of the input storage system, and usually does not need to be set manually; the shuffle is responsible for the data transmission process between the map stage and the reduce stage, and its parallelism is the parallelism of the reduce stage, which does not need to be set separately; while the parallelism of the reduce stage needs to be manually set by the user based on his own experience. Therefore, the existing related parallelism tuning technology is mainly aimed at the parallelism tuning of the reduce stage.
[0016] As the scale of distributed clusters continues to expand, reasonable parallelism settings can fully utilize cluster resources and improve computing efficiency while ensuring the stability of distributed systems, especially in large-scale cluster deployments.
[0017] In this case, Figure 1 A flow chart of a method 100 for determining the degree of parallelism in a distributed job provided in an embodiment of the present application is shown, and the method can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed in a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.
[0018] Optionally, in the present application, the terminal device or server device may adopt a distributed big data computing framework such as Hadoop MapReduce, computing engine (Spark), etc. when optimizing job parallelism for distributed jobs.
[0019] In addition, considering that the terminal device or server device needs to determine the parallelism in the distributed job (also called the reduced parallelism), the terminal device or server device may have the ability to receive job data of the distributed job, receive the distributed job to be optimized, etc.
[0020] Based on this, Figure 1 As shown, the method 100 may include but is not limited to the following steps.
[0021] S110, obtaining data features in job data corresponding to the distributed job.
[0022] Among them, the data characteristics may include at least one of the reduction operation operator type, input data scale, mapping parallelism, and reduction parallelism. The aforementioned reduction operation operator type, input data scale, mapping parallelism, and reduction parallelism can effectively describe the execution process of the distributed computing job, such as how many map tasks there are in total, how much data each map task processes, etc.
[0023] In one implementation, when the job data is obtained, the data format may be a JSON string, and the required job data may be parsed and extracted from the JSON string.
[0024] S120, obtaining each candidate degree of parallelism in the candidate degree of parallelism list, and determining a candidate reduction phase execution time according to the candidate degree of parallelism and the data feature.
[0025] In this embodiment, the execution time of the reduction phase is used as the main optimization target to determine the target parallelism. In other words, the relationship between the parallelism and the execution time of the reduction phase is used to tune the parallelism.
[0026] Of course, there may be many ways to determine the execution time of the candidate protocol stage, such as using the candidate parallelism and the data features as inputs to a preconfigured or trained artificial intelligence (AI) model to predict the execution time of the candidate protocol stage, etc., which are not limited here.
[0027] S130: Determine a target parallelism corresponding to the distributed job from the candidate parallelism list according to the candidate reduction stage execution time.
[0028] In this embodiment, when determining the parallelism in a distributed job, at least one of the reduction operation operator type, input data scale, mapping parallelism, reduction parallelism, etc. is used as a data feature, and the reduction stage execution time is used as a data label to determine the target parallelism, thereby achieving parallelism tuning. This can accurately represent the intrinsic execution process of the distributed job, facilitate reasonable and effective prediction of job performance, and ensure the accuracy of the parallelism determination results and the system performance of the distributed job system.
[0029] Based on this, as a possible implementation method, when using the data features to characterize the distributed job, in order to better characterize the intrinsic execution characteristics of the distributed job, different characterization methods can be used for different data types in this embodiment to abstractly characterize the distributed job; at the same time, it can also solve the problem of difficulty in fine-grained abstract representation of the distributed job execution process existing in related technologies.
[0030] For example, after obtaining the data features, the data features can be first vectorized according to the data type of the data features (i.e., the data features can be characterized into vector form), such as converting the data features into feature vectors according to a first rule; wherein the first rule includes at least one of the following (a)-(c).
[0031] (a) When the data feature is an enumeration type value, the data feature is converted into the feature vector according to a one-hot encoding rule.
[0032] For example, assuming that the reduction operator is an enumeration type value, and there are 10 reduction operators in total, then the reduction operator repartitioning (repartition) can be characterized as "[1,0,0,0,0,0,0,0,0,0]", the reduction operator grouping (GroupBy) can be characterized as "[0,1,0,0,0,0,0,0,0,0,0]", and so on.
[0033] (b) When the data feature is a numerical value, the data feature is converted into the feature vector according to the second rule, and the second rule includes splitting the data feature according to the numerical digits, and retaining the first two non-zero digits in the split numerical digits based on the rounding logic, and ignoring the non-zero digits in other numerical digits after the first two non-zero digits, thereby utilizing the characteristic that the distributed system is not sensitive to small changes in numerical values such as parallelism, and the rounding logic can effectively reduce the search space, which is beneficial to the subsequent model prediction and search process. . For example, for data exceeding 100, the tens and hundreds digits are retained, and the influence of the ones digit is ignored. For data exceeding 1,000, the thousands and hundreds digits are retained, and the influence of the ones digit and tens digit is ignored, and so on. Wherein, the value range of each numerical digit is 0 to 9.
[0034] For example, assuming that the reduced parallelism is "289", then it is "290" after characterization according to the second rule mentioned above. This is because the distributed system is not sensitive to small changes in values such as parallelism, and the rounding logic can effectively reduce the search space, which is beneficial to the subsequent model prediction and search process.
[0035] It is worth noting that, in this embodiment, the mapping parallelism, reduction parallelism, etc. are vectorized according to numerical type values.
[0036] (c) When the data feature is a numerical value including a unit, any unit in the data feature is selected as a reference unit, and the data feature is converted into the feature vector based on the reference unit and the second rule.
[0037] For example, assuming that the input data size is in GB, the data volume value is characterized according to the second rule.
[0038] It is worth noting that, in this embodiment, the input data size, the execution time of the reduction phase, etc. can be vectorized according to the rules described in (c) above.
[0039] Furthermore, in a possible implementation, the target parallelism is determined for each candidate parallelism in the candidate parallelism list, which can be implemented in a variety of different ways. For example, in this embodiment, the implementation process may include the following processes (1)-(2).
[0040] (1) Traversing the candidate parallelism in the candidate parallelism list, wherein the candidate parallelism list is determined according to the candidate parallelism range and the first interval. For example, assuming that the candidate parallelism range is [10, 100] and the first interval is 10, then the candidate parallelism in the determined candidate parallelism list includes {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}.
[0041] (2) When the search duration does not reach the first threshold and a candidate parallelism satisfying the first condition is found, stop searching and use the candidate parallelism satisfying the first condition as the target parallelism of the distributed job.
[0042] The first condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement.
[0043] Optionally, the process of determining the first requirement includes: determining an improvement effect based on the execution time of the candidate specification phase and the execution time of the benchmark specification phase, and determining whether the first requirement is met or not based on the improvement effect; if the improvement effect is greater than a second threshold, it is determined that the first requirement is met; otherwise, it is determined that the first requirement is not met.
[0044] In this embodiment, the improvement effect can be characterized by the improvement ratio and improvement duration of the execution time of the specification phase. For example, assuming that the execution time of the benchmark specification phase is 100s and the execution time of the candidate specification phase is 80s, then the improvement effect is (100-80) / 100=20%, or the improvement effect is 100-80=20. This embodiment does not impose any limitation on this.
[0045] It is worth noting that the first threshold, the second threshold, and the third threshold involved in this embodiment can be configured by the user based on the interactive interface, or can be generated in real time according to the execution time of the historical protocol stage, etc., and there is no limitation here.
[0046] In addition, the aforementioned benchmark reduction phase execution time can also be referred to as a baseline value, which can be configured by the user based on an interactive interface, or determined based on the data features and the benchmark parallelism. For example, when determined based on the data features and the benchmark parallelism (or called initial parallelism), the data features and the benchmark parallelism can be input into the target prediction model to predict the benchmark reduction phase execution time. It can be understood that the benchmark parallelism can be configured by the user based on an interactive interface, or it can be a specified parallelism in a candidate parallelism list, such as the first parallelism in the candidate parallelism list, the minimum parallelism, etc., and is not limited here.
[0047] Among them, the target prediction model can be obtained by training AI models such as random forest, neural network, linear regression, regression prediction model (XG Boost) based on the job data of distributed jobs, so that it has the ability to predict the execution time in the benchmark specification stage.
[0048] Furthermore, for a distributed job system, considering that the performance of a distributed job will first increase and then decrease as the reduced parallelism increases, therefore, in the process of searching for the reduced parallelism (i.e., the target parallelism), the optimal value and the suboptimal value of the parallelism are often in adjacent positions. Based on this, in this embodiment, as a possible implementation method, if when determining the target parallelism based on the above (1)-(2), there is a situation where the search time does not reach the first threshold but the candidate parallelism list has been traversed and there is no candidate parallelism that meets the first condition, then the candidate parallelism list can be re-determined by narrowing the search interval and expanding the search range to search for the target parallelism, so as to ensure that a better candidate reduced parallelism is found within a limited search time.
[0049] For example, in this embodiment, when the search duration does not reach the first threshold but the candidate parallelism list has been traversed and the candidate parallelism that meets the first condition has not been found, the candidate parallelism list can be re-determined according to the candidate parallelism range and the second interval, and the second interval is smaller than the first interval; the candidate parallelisms in the re-determined candidate parallelism list are traversed again; when the search duration does not reach the first threshold and the candidate parallelism that meets the first condition is found, the search is stopped and the candidate parallelism that meets the first condition is used as the target parallelism of the distributed job.
[0050] Exemplarily, assuming that the candidate parallelism range is [10, 100] and the first interval is 10, then the candidate parallelisms in the determined candidate parallelism list include {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}, if the search time does not reach the first threshold, but the candidate parallelism list {10, 20, 30, 40, 50, 60, 70, 80, 90, 100} has been traversed, and no candidate parallelism satisfying the first condition is found In the case of parallelism, the candidate parallelism list can be re-determined based on the candidate parallelism range of [10, 100] and the second interval of 5, that is, {10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100}. Then, compared with the candidate parallelism lists determined based on the first interval and the second interval, the candidate parallelism list determined based on the second interval narrows the search interval to expand the search range.
[0051] Of course, if the candidate parallelism that meets the first condition is still not found based on the re-determined candidate parallelism list, then, if the search time does not reach the first threshold, the search interval can be shortened again, such as the third interval 4, the fourth interval 2, the fifth interval 1, etc., and the candidate parallelism list can be determined to re-search the target parallelism, and this process can be repeated until the target parallelism that meets the first condition is found and / or the search time reaches the first threshold.
[0052] In addition, in the aforementioned process of searching for the target parallelism by continuously narrowing the search interval and expanding the search range, if the search duration reaches the first threshold but the candidate parallelism that meets the first condition is not found, a prompt message can be returned. If the candidate parallelism that meets the first condition is not found, the candidate parallelism that meets the second condition during the search process can also be used as the target parallelism of the distributed job; wherein the second condition includes that the execution time of the candidate reduction stage determined based on the candidate parallelism and the data characteristics meets the second requirement.
[0053] Among them, similar to the determination of the aforementioned first requirement, the determination process of the second requirement may also include: determining the improvement effect according to the execution time of the candidate specification phase and the execution time of the benchmark specification phase, and when the improvement effect is greater than a third threshold, determining that the second requirement is met, otherwise, determining that the second requirement is not met; wherein the benchmark specification phase execution time is determined according to the data characteristics and the benchmark parallelism, and the third threshold is less than the second threshold, that is, when the candidate parallelism that meets the first condition is not searched, the requirement for the improvement effect can be reduced. For example, assuming that the second threshold corresponding to the improvement effect of the first condition is 20%, then the second threshold corresponding to the improvement effect of the second condition can be 10%.
[0054] Alternatively, the process of determining the second requirement may also include: determining the improvement effect corresponding to each candidate parallelism, and taking the candidate parallelism pair with the best improvement effect among the multiple candidate parallelisms traversed during the search process as the target parallelism.
[0055] It can be understood that the process of searching for target parallel symbols by continuously narrowing the search interval and expanding the search range can also be called a jumping grid search algorithm, which makes full use of the changing relationship between the specification parallelism and the job performance (i.e., the execution time of the specification stage), so that the entire parallelism tuning process has the advantages of non-intrusion, low overhead, short time consumption and high precision.
[0056] Of course, in the aforementioned search process for the target parallelism, there may be multiple ways to determine "determining the candidate reduction phase execution time based on the candidate parallelism and the data features". For example, similar to the acquisition of the aforementioned benchmark reduction phase execution time, it may be acquired based on a target prediction model, such as inputting the feature vector corresponding to the candidate parallelism and the data features into the target prediction model to obtain the candidate reduction phase execution time.
[0057] Among them, before the data features are input into the target prediction model, the data features can be vectorized according to the aforementioned first rule to better characterize the intrinsic execution characteristics of distributed jobs, and then the vectorized data features and the candidate parallelism are input into the target prediction model. The process of vectorizing the data features will not be repeated here.
[0058] In addition, the target prediction model is trained based on a first training data set, wherein the first training data set includes feature data and label data in historical job data corresponding to the distributed job, and the label data is the execution time of the protocol phase corresponding to the distributed job.
[0059] In this embodiment, the target prediction model may be obtained by training the electronic device itself, or by training other devices other than the electronic device itself, and there is no limitation here.
[0060] The following takes the random forest model as the base model for training the target prediction model and the process of the electronic device itself training the target prediction model as an example to introduce the training process of the target prediction model. Figure 2 As shown, the training process may include the following steps S210-S240.
[0061] S210, construct or obtain a random forest model.
[0062] Among them, the random forest model takes the feature vector of distributed job data as input and the execution time of the job specification phase as output. The impurity measurement indicator is the square mean error. There is no limit on the maximum depth of the decision tree and the minimum number of node samples in the random forest model, and the sampling method is the self-service sampling method.
[0063] S220, obtaining feature data and label data in historical job data corresponding to the distributed job.
[0064] For example, historical job data in the form of JSON strings is collected through Spark's Representational State Transfer (REST) application programming interface (API), and feature data (such as reduction operator type, input data scale, mapping parallelism, etc.) and label data (such as reduction parallelism, etc.) are extracted from the historical job data through data preprocessing. The feature data can effectively describe the execution process of distributed jobs, such as how many map tasks there are in total, how much data each map task processes, etc. The label data corresponds to the main optimization goal.
[0065] S230: performing vectorization processing on the acquired feature data and label data based on the first rule.
[0066] It can be understood that the processing process can refer to the aforementioned description of the first rule, which will not be repeated here.
[0067] S240, input the vectorized feature vector and label vector into the random forest model for model training. In the iterative model training process of the guided focusing algorithm, a different sampling set is generated for each decision tree by a self-service sampling method, and then the base learner decision tree is trained based on the generated different sampling sets. The decision tree uses the squared mean error to find the feature column in the sampling set that can minimize the data impurity, and continuously splits and grows downward until it cannot be split, and finally the output of all decision trees is averaged as the output value of the entire random forest model.
[0068] Among them, the random forest model (i.e., the target prediction model) obtained after training in S240 has learned the relationship between the data features of distributed jobs and the label data, and has the ability to fit the label data to predict job performance (such as the execution time of the reduction phase). For example, it takes the data features of distributed jobs as input and the average output of all base learners as the output value, i.e., the predicted execution time of the reduction phase.
[0069] Of course, the aforementioned benchmark protocol phase execution time and the candidate protocol phase execution time can both be predicted based on the trained target prediction model.
[0070] Furthermore, in order to enable the trained target prediction model (such as a random forest model) to adapt to the ever-changing actual production environment, during the operation of the distributed system, the job data of newly completed distributed jobs in the distributed job cluster (such as job data of untuned distributed jobs, job data of previously tuned distributed jobs, etc.) can be collected in real time, and the collected job data of newly completed distributed jobs can be used to perform model training again on the trained model using the model training process provided in the aforementioned S210-S240, such as incremental training.
[0071] For example, taking the target prediction model as a random forest model, the collected job data of the newly completed distributed jobs can be used as incremental data, and the incremental training data set (or second training data set) is determined based on the incremental data. Then, a new decision tree is added to the trained random forest model, and the incremental training data set is used to train the newly added decision tree, such as first generating different sampling sets for each newly added decision tree through the self-service sampling method, and then training the newly added base learner decision tree based on the generated different sampling sets, using the squared mean error to find the feature column in the sampling set that can minimize the data impurity, and continuously split and grow downward until it cannot be split. After the incremental training is completed, the new model is used to replace the original model. In this way, the target prediction model can be adapted to different actual production environments, and the stability of the model fitting effect can also be determined.
[0072] The second training data set includes feature data and label data in the job data corresponding to the newly added distributed job, and the label data is the execution time of the protocol phase corresponding to the newly added distributed job. Of course, when using the newly added training data set for incremental training of the model, the newly added training data set can be subjected to feature vector processing according to the first rule, and this embodiment will not be repeated here.
[0073] Based on the relevant description in the aforementioned method embodiment 100, the parallelism determination process in the distributed job provided by the present application is further exemplarily described below in combination with examples, as follows.
[0074] Among them, the predicted reduction phase execution time is used as the performance measurement indicator, and the jump grid search algorithm is adopted. Under the condition of a given search time upper limit (such as the first threshold), the initial reduction parallelism (i.e., the benchmark parallelism) and the obtained feature vector are first determined as the input of the target prediction model (such as the random forest model), and the benchmark predicted reduction phase execution time is obtained as the baseline value. Then, according to the candidate parallelism range and the set initial value interval (such as the first interval), a candidate parallelism list is given, and each candidate parallelism is traversed and tried in a loop. The candidate parallelism and the feature vector are input into the trained target prediction model to obtain the predicted candidate reduction phase execution time. If the candidate phase execution time is greater than the baseline job performance improvement effect than the second threshold, or the search time exceeds the first threshold, the search process is exited, otherwise the value interval is reduced to half of the original value (such as the second threshold), and the candidate parallelism list is continued to be traversed and tried. Finally, the candidate reduction parallelism value with the best performance in the search process is used as the final recommended reduction parallelism, i.e., the target parallelism.
[0075] Based on this, the following takes the random forest model as an example to further introduce its implementation process. The content is as follows.
[0076] S301, initializing the skip grid search algorithm, for example, assigning initial values to the search time upper limit (such as a first threshold), the value interval (such as a first interval), the performance improvement threshold (such as a second threshold), etc.
[0077] S302, input the feature vector and the benchmark parallelism corresponding to the distributed job to be tuned into the trained random forest model, and predict the execution time of the benchmark reduction phase, that is, the baseline value.
[0078] S303: Determine a candidate parallelism list according to the input candidate parallelism range and a set value interval, such as a first interval.
[0079] S304, traverse the candidate parallelism list in turn to determine whether there is a target candidate parallelism and whether the search duration exceeds the first threshold. If it does not exceed the first threshold and there is a target candidate parallelism, execute S307; if it does not exceed the first threshold and there is no target candidate parallelism, execute S305; if it times out, execute S307.
[0080] It is worth noting that when executing S304, you can first determine whether the search time exceeds the first threshold, and then determine whether there is a target candidate parallelism, or you can first determine whether there is a target candidate parallelism, and then determine whether the search time exceeds the first threshold. This embodiment does not limit this.
[0081] S305, determine whether the traversal of the current candidate parallelism list is complete, if yes, proceed to S306, if not, proceed to S307.
[0082] S306, reducing the current value interval to half of the original value (such as the second interval), redetermining the candidate parallelism list, executing step 304 again based on the redetermined candidate parallelism list, and repeating S304-S306 until the search duration exceeds the first threshold or the target parallelism is searched.
[0083] S307: The searched target parallelism is used as the final recommended specification parallelism and fed back to the job to be tuned.
[0084] Alternatively, if the target parallelism is not found when the search time reaches the first threshold, the candidate parallelism with the best performance improvement among multiple candidate parallelisms in the search process can be used as the final recommended specification parallelism and fed back to the job to be tuned.
[0085] It is worth noting that the implementation process of the aforementioned steps provided in this example can refer to the relevant description in the aforementioned embodiment of the method for determining the degree of parallelism in distributed jobs, and achieve the same or corresponding technical effects. To avoid repetition, it will not be repeated here.
[0086] Furthermore, the parallelism determination process in the distributed job provided by the application may include but is not limited to the aforementioned steps, and may include more or fewer steps than the aforementioned steps, which is not limited here.
[0087] Based on the foregoing description, it can be seen that the parallelism determination solution in distributed jobs provided in this application has the following technical effects.
[0088] (1) It can reasonably abstractly represent distributed jobs in actual production environments. Combined with the fact that the random forest model has low training and storage overhead and can be used to tune multiple distributed jobs after only one training, the lightweight random forest model is used to learn the relationship between the data characteristics of distributed jobs and the execution time of the job specification phase, and to predict the execution time of the job specification phase, which can reduce the overhead of the parallelism tuning process.
[0089] (2) This application uses the changing relationship between the specified parallelism and the job performance to perform parallelism tuning, which can effectively narrow the search range to find a better specified parallelism in a shorter time, thereby improving the efficiency of parallelism tuning and making the tuning effect more stable.
[0090] (3) The parallelism tuning process (i.e., the process of determining the target parallelism) provided by the present application is located before the execution of the distributed job, and does not cause any impact on the original job execution process. In addition, the training and storage overhead of the random forest model is relatively small, and the jumping grid search algorithm can find a better solution without exceeding the set upper limit of the tuning time. Therefore, the entire tuning process can be non-invasive, low-overhead, short-time, and high-precision.
[0091] (4) This application can be applied to various distributed computing frameworks based on the MapReduce parallel computing model, such as Hadoop MapReduce and Spark, and can adaptively select the parallelism of the reduce side to achieve the purpose of improving the job performance by optimizing the parallelism of the reduce side. The whole process is non-intrusive, low-cost, short-time and high-precision.
[0092] Figure 4 A structural schematic diagram of a device 300 for determining parallelism in a distributed job provided in an embodiment of the present application is shown, and the device 300 includes: a data acquisition module 310, used to obtain data features in job data corresponding to the distributed job; a determination module 320, used to obtain each candidate parallelism in a candidate parallelism list, and determine a candidate specification phase execution time according to the candidate parallelism and the data features; and used to determine the target parallelism corresponding to the distributed job from the candidate parallelism list according to the candidate specification phase execution time.
[0093] Optionally, the determination module 320 obtains each candidate parallelism in the candidate parallelism list, determines the candidate reduction phase execution time according to the candidate parallelism and the data characteristics, and determines the target parallelism corresponding to the distributed job from the candidate parallelism list according to the candidate reduction phase execution time, including: traversing the candidate parallelism in the candidate parallelism list, wherein the candidate parallelism list is determined according to the candidate parallelism range and the first interval; when the search time does not reach the first threshold and the candidate parallelism that meets the first condition is searched, stop the search and use the candidate parallelism that meets the first condition as the target parallelism of the distributed job; wherein the first condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement.
[0094] Optionally, the determination module 320 obtains each candidate parallelism in the candidate parallelism list, determines the candidate reduction phase execution time according to the candidate parallelism and the data characteristics, and determines the target parallelism corresponding to the distributed job from the candidate parallelism list according to the candidate reduction phase execution time. It also includes: when the search time does not reach the first threshold but the candidate parallelism list has been traversed and the candidate parallelism that meets the first condition has not been searched, redetermine the candidate parallelism list according to the candidate parallelism range and the second interval, and the second interval is smaller than the first interval; traverse the candidate parallelism in the redetermined candidate parallelism list again; when the search time does not reach the first threshold and the candidate parallelism that meets the first condition is searched, stop the search and use the candidate parallelism that meets the first condition as the target parallelism of the distributed job.
[0095] Optionally, the determination module 320 obtains each candidate parallelism in the candidate parallelism list, determines the candidate reduction phase execution time according to the candidate parallelism and the data characteristics, and determines the target parallelism corresponding to the distributed job from the candidate parallelism list according to the candidate reduction phase execution time, and also includes: when the search time reaches the first threshold but the candidate parallelism that satisfies the first condition is not found, the candidate parallelism that satisfies the second condition during the search process is used as the target parallelism of the distributed job; wherein the second condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics satisfies the second requirement.
[0096] Optionally, the determination module 320 determines that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement, including: determining the improvement effect according to the candidate reduction phase execution time and the benchmark reduction phase execution time; when the improvement effect is greater than a second threshold, determining that the first requirement is met; wherein the benchmark reduction phase execution time is determined according to the data characteristics and the benchmark parallelism.
[0097] Optionally, the determination module 320 determines the candidate reduction phase execution time based on the candidate parallelism and the data features, including: inputting the feature vectors corresponding to the candidate parallelism and the data features into a target prediction model to obtain the candidate reduction phase execution time; wherein the target prediction model is trained based on a first training data set, the first training data set includes feature data and label data in historical job data corresponding to the distributed job, and the label data is the reduction phase execution time corresponding to the distributed job.
[0098] Optionally, the target prediction model is a random forest model, and the determination module 320 is also used to determine a second training data set based on the job data corresponding to a newly added distributed job relative to the distributed job; and use the second training data set to perform incremental training on a newly added decision tree in the random forest; wherein the newly added decision tree corresponds to the newly added distributed job, and the second training data set includes feature data and label data in the job data corresponding to the newly added distributed job, and the label data is the execution time of the protocol phase corresponding to the newly added distributed job.
[0099] Optionally, the data acquisition module is also used to convert the data feature into a feature vector according to a first rule; wherein the first rule includes at least one of the following: when the data feature is an enumeration type value, the data feature is converted into the feature vector according to a one-hot encoding rule; when the data feature is a numerical type value, the data feature is converted into the feature vector according to a second rule, the second rule including performing numerical bit splitting on the data feature according to the numerical bit, and retaining the first two non-zero digits in the split numerical bits based on rounding logic, and ignoring the non-zero digits in other numerical bits after the first two non-zero digits; when the data feature is a numerical value including a unit, any unit in the data feature is selected as a base unit, and the data feature is converted into the feature vector based on the base unit and the second rule.
[0100] Optionally, the data characteristics include at least one of reduction operation operator type, input data scale, mapping parallelism, and reduction parallelism.
[0101] The device 300 provided in the embodiment of the present application can execute the various methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be repeated here.
[0102] Figure 4 A schematic diagram of the hardware structure of an electronic device that implements the embodiment of the present application is shown. Referring to the figure, at the hardware level, the electronic device includes a processor, and optionally, an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.
[0103] The processor, the network interface and the memory may be interconnected through an internal bus, which may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0104] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0105] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for locating the specified user at the logical level. The processor executes the program stored in the memory and is specifically used to execute: Figure 1-Figure 3 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.
[0106] The above application Figure 1-Figure 2The method disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0107] The electronic device can also execute the methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.
[0108] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0109] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable medium stores one or more programs, and when the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1-Figure 3 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.
[0110] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0111] Furthermore, an embodiment of the present application also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the following process is implemented: Figure 1-Figure 3 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.
[0112] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0113] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0116] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
Claims
1. A method for determining the degree of parallelism in a distributed job, characterized in that: include: Obtain data features in the job data corresponding to the distributed job; Obtain each candidate parallelism in the candidate parallelism list, and determine the candidate reduction phase execution time according to the candidate parallelism and the data characteristics; The target parallelism corresponding to the distributed job is determined from the candidate parallelism list according to the candidate reduction stage execution time.
2. The method according to claim 1, characterized in that The obtaining of each candidate degree of parallelism in the candidate degree of parallelism list, determining a candidate specification phase execution time according to the candidate degree of parallelism and the data feature, and determining a target degree of parallelism corresponding to the distributed job from the candidate degree of parallelism list according to the candidate degree of parallelism phase execution time, includes: Traversing the candidate parallelism in the candidate parallelism list, wherein the candidate parallelism list is determined according to the candidate parallelism range and the first interval; If the search duration does not reach the first threshold but a candidate parallelism satisfying the first condition is found, stop searching and use the candidate parallelism satisfying the first condition as the target parallelism corresponding to the distributed job; The first condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement.
3. The method according to claim 2, characterized in that The obtaining of each candidate degree of parallelism in the candidate degree of parallelism list, determining a candidate specification phase execution time according to the candidate degrees of parallelism and the data feature, and determining a target degree of parallelism corresponding to the distributed job from the candidate degree of parallelism list according to the candidate degree of parallelism phase execution time, further includes: When the search duration does not reach the first threshold, but the candidate parallelism list has been traversed and no candidate parallelism satisfying the first condition is found, the candidate parallelism list is re-determined according to the candidate parallelism range and the second interval, and the second interval is smaller than the first interval; Traversing the candidate parallelism in the re-determined candidate parallelism list again; If the search duration does not reach the first threshold and a candidate parallelism satisfying the first condition is found, the search is stopped and the candidate parallelism satisfying the first condition is used as the target parallelism of the distributed job.
4. The method according to claim 2, characterized in that Obtaining each candidate parallelism in the candidate parallelism list, determining a candidate specification phase execution time according to the candidate parallelism and the data feature, and determining a target parallelism corresponding to the distributed job from the candidate parallelism list according to the candidate specification phase execution time, further comprising: When the search duration reaches the first threshold but the candidate parallelism satisfying the first condition is not found, taking the candidate parallelism satisfying the second condition during the search as the target parallelism of the distributed job; The second condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the second requirement.
5. The method according to claim 2, characterized in that The candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement, including: Determine the improvement effect according to the candidate specification phase execution time and the benchmark specification phase execution time; When the improvement effect is greater than a second threshold, determining that the first requirement is met; The execution time of the benchmark reduction phase is determined according to the data characteristics and the benchmark parallelism.
6. The method according to claim 5, characterized in that The step of determining the candidate reduction phase execution time according to the candidate parallelism and the data characteristics includes: Inputting the candidate parallelism and the feature vector corresponding to the data feature into a target prediction model to obtain the candidate reduction stage execution time; Among them, the target prediction model is trained based on a first training data set, and the first training data set includes feature data and label data in historical job data corresponding to the distributed job, and the label data is the execution time of the protocol phase corresponding to the distributed job.
7. The method according to claim 6, characterized in that The target prediction model is a random forest model, and the method further includes: When a new distributed job relative to the distributed job is acquired, determining a second training data set according to job data corresponding to the new distributed job; The second training data set is used to perform incremental training on the newly added decision tree in the random forest; wherein the newly added decision tree corresponds to the newly added distributed job, and the second training data set includes feature data and label data in the job data corresponding to the newly added distributed job, and the label data is the execution time of the protocol phase corresponding to the newly added distributed job.
8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Converting the data features into feature vectors according to a first rule; The first rule includes at least one of the following: In the case where the data feature is an enumeration type value, converting the data feature into the feature vector according to a one-hot encoding rule; In the case where the data feature is a numerical type value, converting the data feature into the feature vector according to a second rule, wherein the second rule includes performing numerical bit splitting on the data feature according to numerical bits, and retaining the first two non-zero digits of the split numerical bits based on rounding logic, and ignoring non-zero digits in other numerical bits after the first two non-zero digits; In the case that the data feature is a numerical value including a unit, any unit in the data feature is selected as a reference unit, and the data feature is converted into the feature vector based on the reference unit and the second rule.
9. The method according to any one of claims 1 to 7, characterized in that The data characteristics include at least one of the reduction operation operator type, input data scale, mapping parallelism, and reduction parallelism.
10. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the method of any one of claims 1 to 9.
11. A computer-readable medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to perform the steps of the method according to any one of claims 1 to 9.
Citation Information
Cited By
Parallelism determination method in distributed operation, and electronic device and storage medium
EP4708046A1
