Parallelism determination method in distributed operation, and electronic device and storage medium
By acquiring the data characteristics of distributed jobs, using artificial intelligence models to predict the execution time of the reduction phase, and combining a jump grid search algorithm to optimize the parallelism setting, the problem of low computational efficiency caused by unreasonable parallelism settings in existing technologies is solved, and the performance of efficient and stable distributed systems is improved.
Patent Information
- Application Number
- PCT/CN2024/086526
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-04-08
- Publication Date
- 2024-11-14
AI Technical Summary
Existing technologies struggle to fully utilize cluster resources while ensuring system stability when setting the parallelism of distributed jobs, resulting in low computational efficiency, especially in large-scale cluster deployments.
By acquiring the data characteristics of distributed jobs, using artificial intelligence models to predict the execution time of the reduction phase, and combining a jump grid search algorithm to optimize the parallelism setting, a random forest model is used to train the relationship between parallelism and execution time to determine the target parallelism.
It improves the computational efficiency and stability of distributed systems, reduces the overhead of parallelism tuning, and enables the finding of optimal parallelism settings within a limited time, with a non-intrusive and highly accurate process.
Smart Images

Figure CN2024086526_14112024_PF_FP_ABST
Abstract
Description
Method for determining parallelism in distributed operations, electronic device, and storage medium
[0001] Cross-references
[0002] This application claims priority to a Chinese patent application filed with the Patent Office of China on May 8, 2023, with application number 202310516599.3 and invention name “Method for determining parallelism in distributed operations, electronic device and storage medium”. The entire contents of the application are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of computer technology, and in particular to a method for determining parallelism in distributed operations, an electronic device, and a storage medium. Background Art
[0004] MapReduce is a distributed parallel computing model consisting of three core phases: map, shuffle, and reduce. It is widely used in mainstream distributed computing systems such as Hadoop and Spark. The degree of parallelism (i.e., the number of tasks in each MapReduce phase) directly affects the disk and network overhead of distributed operations, thereby impacting the efficiency and stability of the entire distributed computing system. Therefore, setting an appropriate degree of parallelism is crucial for improving the performance of distributed systems.
[0005] Summary of the Invention
[0006] The purpose of the embodiments of the present application is to provide a method for determining the parallelism in a distributed job, an electronic device, and a storage medium, which can obtain a reasonable parallelism of the distributed job and improve the system performance of the distributed system.
[0007] To solve the above technical problems, the embodiments of the present application are implemented through the following aspects.
[0008] In the first aspect, an embodiment of the present application provides a method for determining parallelism in a distributed job, including: obtaining data features in job data corresponding to the distributed job; obtaining each candidate parallelism in a candidate parallelism list, and determining the candidate specification phase execution time based on the candidate parallelism and the data features; and determining the target parallelism corresponding to the distributed job from the candidate parallelism list based on the candidate specification phase execution time.
[0009] In a second aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and computer-executable instructions stored on the memory and executable on the processor, wherein the computer-executable instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0010] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the method described in the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0012] FIG1 is a flow chart showing a method for determining the degree of parallelism in a distributed job according to an embodiment of the present application.
[0013] FIG2 shows a schematic diagram of the model training process provided in an embodiment of the present application.
[0014] FIG3 shows a schematic structural diagram of a device for determining parallelism in a distributed job provided in an embodiment of the present application.
[0015] FIG4 is a schematic diagram of the hardware structure of an electronic device that executes the method for determining the degree of parallelism in a distributed job provided in an embodiment of the present application. DETAILED DESCRIPTION
[0016] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.
[0017] It's important to note that in the three phases of the MapReduce protocol, the parallelism of the map phase is generally determined by the distributed system based on the data partitioning of the input storage system and generally does not require manual configuration. The shuffle phase transfers data between the map and reduce phases, and its parallelism is the same as the reduce phase parallelism, which does not require separate configuration. The reduce phase parallelism, on the other hand, must be manually configured by the user based on their own experience. Therefore, existing parallelism tuning techniques primarily focus on reducing phase parallelism.
[0018] As the scale of distributed clusters continues to expand, reasonable parallelism settings can fully utilize cluster resources and improve computing efficiency while ensuring the stability of distributed systems, especially in large-scale cluster deployments.
[0019] In this case, FIG1 shows a flow chart of a method 100 for determining the degree of parallelism in a distributed job provided by an embodiment of the present application. The method can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed on the terminal device or the server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.
[0020] In some embodiments, in the present application, the terminal device or server device may use a distributed big data computing framework such as Hadoop MapReduce, computing engine (Spark), etc. when tuning the job parallelism for distributed jobs.
[0021] In addition, considering that the terminal device or server device needs to determine the parallelism in the distributed job (also known as the stipulated parallelism), the terminal device or server device may have the ability to receive job data of the distributed job, receive the distributed job to be optimized, etc.
[0022] Based on this, as shown in FIG1 , the method 100 may include but is not limited to the following steps S110 to S130 .
[0023] In step S110 , data features in job data corresponding to the distributed job are obtained.
[0024] Among them, the data features may include at least one of the reduction operator type, input data scale, mapping parallelism, and reduction parallelism. The aforementioned reduction operator type, input data scale, mapping parallelism, and reduction parallelism can effectively describe the execution process of the distributed computing job, such as how many map tasks there are in total, how much data each map task processes, etc.
[0025] In one implementation, when the job data is obtained, the data format may be a JSON string, and the required job data may be parsed and extracted from the JSON string.
[0026] In step S120, each candidate degree of parallelism in the candidate degree of parallelism list is obtained, and a candidate reduction phase execution time is determined according to the candidate degree of parallelism and the data characteristics.
[0027] In this embodiment, the execution time of the reduction phase is used as the main optimization target to determine the target parallelism. In other words, this embodiment uses the relationship between the parallelism and the execution time of the reduction phase to adjust the parallelism.
[0028] In this embodiment, there are multiple ways to determine the execution time of the candidate protocol stage, such as using the candidate parallelism and the data features as inputs to a preconfigured or trained artificial intelligence (AI) model to predict the execution time of the candidate protocol stage, etc., which are not limited here.
[0029] In step S130 , a target parallelism corresponding to the distributed job is determined from the candidate parallelism list according to the candidate reduction phase execution time.
[0030] In this embodiment, when determining the parallelism of a distributed job, at least one of the reduction operator type, input data size, mapping parallelism, and reduction parallelism is used as a data feature, and the reduction phase execution time is used as a data tag to determine the target parallelism, thereby achieving parallelism tuning. This accurately represents the internal execution process of the distributed job, facilitates reasonable and effective prediction of job performance, and ensures the accuracy of the parallelism determination results and the system performance of the distributed job system.
[0031] Based on this, as a possible implementation method, when using the data features to characterize the distributed job, in order to better characterize the inherent execution characteristics of the distributed job, different characterization methods can be used for different data types in this embodiment to abstractly characterize the distributed job; at the same time, it can also solve the problem of difficulty in fine-grained abstract representation of the distributed job execution process existing in related technologies.
[0032] For example, after obtaining the data features, the data features may be vectorized (i.e., the data features may be transformed into vector form) according to the data type of the data features. For example, the data features may be transformed into feature vectors according to a first rule. The first rule includes at least one of the following (a) to (c).
[0033] (a) When the data feature is an enumeration type value, the data feature is converted into the feature vector according to the one-hot encoding rule.
[0034] For example, assuming that the reduction operator is an enumeration type value and there are 10 reduction operators, then the reduction operator repartitioning can be characterized as "[1,0,0,0,0,0,0,0,0,0]", the reduction operator grouping (GroupBy) can be characterized as "[0,1,0,0,0,0,0,0,0,0]", and so on.
[0035] (b) In the case where the data feature is a numerical type value, the data feature is converted into the feature vector according to the second rule, and the second rule includes splitting the data feature according to the numerical digits, and retaining the first two non-zero digits in the split numerical digits based on the rounding logic, and ignoring the non-zero digits in other numerical digits after the first two non-zero digits. This makes use of the fact that the distributed system is not sensitive to small changes in numerical values such as parallelism, and the rounding logic can effectively reduce the search space, which is beneficial to the subsequent model prediction and search process. For example, for data over 100, the tens digit and hundreds digit are retained, and the influence of the ones digit is ignored. For data over 1,000, the thousands digit and hundreds digit are retained, and the influence of the ones digit and tens digit is ignored, and so on. The value range of each numerical digit is 0 to 9.
[0036] For example, assuming the reduced degree of parallelism is "289", then after characterization according to the second rule mentioned above, it is "290". This is done because distributed systems are not sensitive to small changes in values such as parallelism. Rounding logic can effectively reduce the search space, which is beneficial to subsequent model prediction and search processes.
[0037] In this embodiment, the mapping parallelism, reduction parallelism, etc. are vectorized according to numerical type values.
[0038] (c) When the data feature is a numerical value including a unit, any unit in the data feature is selected as a base unit, and the data feature is converted into the feature vector based on the base unit and the second rule.
[0039] For example, assuming that the input data size is in GB, the data volume value is characterized according to the second rule.
[0040] In this embodiment, the input data size, the execution time of the reduction phase, etc. can be vectorized according to the rules described in (c) above.
[0041] In one possible implementation, the target parallelism is determined for each candidate parallelism in the candidate parallelism list. This can be achieved in a variety of different ways. For example, in this embodiment, the implementation process may include the following processes (1)-(2).
[0042] (1) Traversing the candidate parallelisms in the candidate parallelism list, wherein the candidate parallelism list is determined based on a candidate parallelism range and a first interval. For example, assuming the candidate parallelism range is [10, 100] and the first interval is 10, then the candidate parallelisms in the determined candidate parallelism list include {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}.
[0043] (2) If the search duration does not reach the first threshold and a candidate parallelism that meets the first condition is found, stop searching and use the candidate parallelism that meets the first condition as the target parallelism of the distributed job.
[0044] The first condition includes that the candidate specification phase execution time determined according to the candidate parallelism and the data characteristics meets a first requirement.
[0045] In some embodiments, the process of determining the first requirement may include but is not limited to: determining the improvement effect based on the execution time of the candidate specification stage and the execution time of the baseline specification stage, and determining whether the first requirement is met or not based on the improvement effect, for example, if the improvement effect is greater than a second threshold, determining that the first requirement is met; if the improvement effect is not greater than the second threshold, determining that the first requirement is not met.
[0046] In this embodiment, the improvement effect can be characterized by the improvement ratio and improvement duration of the execution time of the specification phase. For example, assuming that the execution time of the baseline specification phase is 100s and the execution time of the candidate specification phase is 80s, then the improvement effect is (100-80) / 100=20%, or the improvement effect is 100-80=20. This embodiment does not impose any restrictions on this.
[0047] The first threshold, the second threshold, and the third threshold involved in this embodiment can be configured by the user based on the interactive interface, or can be generated in real time based on the execution time of the historical protocol stage, etc., and there is no limitation here.
[0048] In addition, the aforementioned benchmark reduction phase execution time can also be referred to as a baseline value, which can be configured by the user based on an interactive interface, or determined based on the data features and the benchmark parallelism. For example, when determined based on the data features and the benchmark parallelism (or referred to as the initial parallelism), the data features and the benchmark parallelism can be input into the target prediction model to predict the benchmark reduction phase execution time. The benchmark parallelism can be configured by the user based on an interactive interface, or it can be a specified parallelism in a candidate parallelism list, such as the first parallelism in the candidate parallelism list, the minimum parallelism, etc., and is not limited here.
[0049] In this embodiment, the target prediction model can be obtained by training AI models such as random forest, neural network, linear regression, regression prediction model (XG Boost) based on the job data of distributed jobs, so that it has the ability to predict the execution time of the benchmark specification stage.
[0050] For distributed job systems, considering that the performance of distributed jobs will first increase and then decrease as the specified parallelism increases, therefore, in the process of searching for the specified parallelism (i.e., the target parallelism), the optimal value and the suboptimal value of the parallelism are often in adjacent positions. Based on this, in this embodiment, as a possible implementation method, if when determining the target parallelism based on the above (1)-(2), there is a situation where the search time does not reach the first threshold but the candidate parallelism list has been traversed and there is no candidate parallelism that meets the first condition, then the candidate parallelism list can be re-determined by narrowing the search interval and expanding the search range to search for the target parallelism, so as to ensure that a better candidate specified parallelism is found within a limited search time.
[0051] For example, in this embodiment, when the search duration does not reach the first threshold, but the candidate parallelism list has been traversed and the candidate parallelism that meets the first condition has not been found, the candidate parallelism list can be re-determined based on the candidate parallelism range and the second interval, and the second interval is smaller than the first interval; the candidate parallelisms in the re-determined candidate parallelism list are traversed again; when the search duration does not reach the first threshold and the candidate parallelism that meets the first condition is found, the search is stopped and the candidate parallelism that meets the first condition is used as the target parallelism of the distributed job.
[0052] For example, assuming that the candidate parallelism range is [10, 100] and the first interval is 10, then the candidate parallelisms in the determined candidate parallelism list include {10, 20, 30, 40, 50, 60, 70, 80, 90, 100}. If the search time does not reach the first threshold, but the candidate parallelism list {10, 20, 30, 40, 50, 60, 70, 80, 90, 100} has been traversed, and no candidate that meets the first condition is found, In the case of parallelism, the candidate parallelism list can be re-determined based on the candidate parallelism range of [10, 100] and the second interval of 5, that is, {10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100}. Then, compared with the candidate parallelism lists determined based on the first interval and the second interval, the candidate parallelism list determined based on the second interval narrows the search interval to expand the search range.
[0053] In some embodiments, if the candidate parallelism that meets the first condition is still not found based on the re-determined candidate parallelism list, then the search interval can be shortened again if the search time does not reach the first threshold, such as the third interval 4, the fourth interval 2, the fifth interval 1, etc. to determine the candidate parallelism list to re-search the target parallelism, and this process is repeated until the target parallelism that meets the first condition is found and / or the search time reaches the first threshold.
[0054] In addition, in the aforementioned search process for the target parallelism by continuously narrowing the search interval and expanding the search range, if the search duration reaches the first threshold but the candidate parallelism that meets the first condition is not found, a prompt message can be returned. If the candidate parallelism that meets the first condition is not found, the candidate parallelism that meets the second condition during the search process can also be used as the target parallelism of the distributed job; wherein, the second condition includes that the execution time of the candidate specification stage determined based on the candidate parallelism and the data characteristics meets the second requirement.
[0055] Among them, similar to the determination of the aforementioned first requirement, the determination process of the second requirement may also include: determining the improvement effect based on the execution time of the candidate specification phase and the execution time of the baseline specification phase, and when the improvement effect is greater than a third threshold, determining that the second requirement is met, otherwise, determining that the second requirement is not met; wherein, the baseline specification phase execution time is determined based on the data characteristics and the baseline parallelism, and the third threshold is less than the second threshold, that is, when the candidate parallelism that meets the first condition is not found, the requirement for the improvement effect can be lowered. For example, assuming that the second threshold corresponding to the improvement effect of the first condition is 20%, then the second threshold corresponding to the improvement effect of the second condition can be 10%.
[0056] Alternatively, the process of determining the second requirement may also include: determining the improvement effect corresponding to each candidate parallelism, and taking the candidate parallelism pair with the best improvement effect among the multiple candidate parallelisms traversed during the search process as the target parallelism.
[0057] In this embodiment, the process of searching for the target parallelism by continuously narrowing the search interval and expanding the search range can also be called a jumping grid search algorithm, which fully utilizes the changing relationship between the specification parallelism and the job performance (i.e., the execution time of the specification stage), so that the entire parallelism tuning process has the advantages of being non-invasive, low-overhead, short in time, and high in precision.
[0058] In the aforementioned search process for the target parallelism, there can be multiple ways to determine "determining the candidate protocol stage execution time based on the candidate parallelism and the data features". For example, similar to the acquisition of the aforementioned benchmark protocol stage execution time, it can be acquired based on a target prediction model, such as inputting the feature vector corresponding to the candidate parallelism and the data features into the target prediction model to obtain the candidate protocol stage execution time.
[0059] Among them, before inputting the data features into the target prediction model, the data features can be vectorized according to the aforementioned first rule to better characterize the intrinsic execution characteristics of the distributed job, and then the vectorized data features and the candidate parallelism are input into the target prediction model. The process of vectorizing the data features will not be repeated here.
[0060] In addition, the target prediction model is trained based on a first training data set, which includes feature data and label data in historical job data corresponding to the distributed job, and the label data is the execution time of the protocol phase corresponding to the distributed job.
[0061] In this embodiment, the target prediction model can be obtained by training the electronic device itself, or by training other devices other than the electronic device itself, and there is no limitation here.
[0062] The following introduces the training process of the target prediction model by taking the random forest model as the base model for training the target prediction model and the process of the electronic device itself training the target prediction model as an example. For example, as shown in Figure 2, the training process may include but is not limited to the following steps S210-step S240.
[0063] In step S210, a random forest model is constructed or obtained.
[0064] The random forest model uses the feature vector of distributed job data as input and the execution time of the job specification phase as output. The impurity measurement indicator is the squared mean error. There are no restrictions on the maximum depth of the decision tree and the minimum number of node samples in the random forest model, and the sampling method is the self-service sampling method.
[0065] In step S220 , feature data and label data in historical job data corresponding to the distributed job are obtained.
[0066] For example, historical job data in the form of JSON strings is collected through Spark's Representational State Transfer (REST) application programming interface (API). Feature data (such as reduction operator type, input data size, mapping parallelism, etc.) and label data (such as reduction parallelism as feature data) are extracted from the historical job data through data preprocessing. The feature data can effectively describe the execution process of the distributed job, such as the total number of map tasks and the amount of data processed by each map task. The label data corresponds to the main optimization target.
[0067] In step S230 , vectorization processing is performed on the acquired feature data and label data based on the first rule.
[0068] When performing quantization processing, the processing process can refer to the aforementioned description of the first rule, which will not be repeated here.
[0069] In step S240, the vectorized feature vector and label vector are input into the random forest model for model training. During the iterative model training process of the guided focusing algorithm, a bootstrap sampling method is used to generate different sampling sets for each decision tree. The base learner decision trees are then trained based on the generated different sampling sets. The decision trees use the squared mean squared error to identify the feature columns in the sampling set that minimize data impurity. The trees continue to split and grow until they become unsplittable. Finally, the outputs of all decision trees are averaged as the output of the entire random forest model.
[0070] The random forest model (i.e., the target prediction model) obtained after training in step S240 has learned the relationship between the data features of the distributed job and the label data, and has the ability to fit the label data to predict the job performance (such as the execution time of the protocol phase). For example, it uses the data features of the distributed job as input and the average output of all base learners as the output value, that is, the predicted execution time of the protocol phase.
[0071] The aforementioned benchmark protocol phase execution time and the candidate protocol phase execution time can both be predicted based on the trained target prediction model.
[0072] Furthermore, in order to enable the trained target prediction model (such as a random forest model) to adapt to the ever-changing actual production environment, during the operation of the distributed system, the job data of newly completed distributed jobs in the distributed job cluster (such as job data of untuned distributed jobs, job data of previously tuned distributed jobs, etc.) can be collected in real time, and the collected job data of newly completed distributed jobs can be used to perform model training on the trained model again using the model training process provided in the aforementioned steps S210-S240, such as incremental training.
[0073] For example, taking the target prediction model as a random forest model, the collected job data of the newly completed distributed jobs can be used as incremental data, and the incremental training data set (or second training data set) is determined based on the incremental data. Then, a new decision tree is added to the trained random forest model, and the incremental training data set is used to train the newly added decision tree. For example, a different sampling set is first generated for each newly added decision tree through a self-service sampling method, and then the newly added base learner decision tree is trained based on the generated different sampling sets. The squared mean error is used to find the feature column in the sampling set that can minimize the data impurity, and the model is continuously split and grown downward until it cannot be split. After the incremental training is completed, the new model is used to replace the original model. In this way, the target prediction model can be adapted to different actual production environments, and the stability of the model fitting effect can also be determined.
[0074] The second training dataset includes feature data and label data from the job data corresponding to the newly added distributed job, and the label data is the execution time of the protocol phase corresponding to the newly added distributed job. In this embodiment, when incrementally training the model using the newly added training dataset, the newly added training dataset can be subjected to feature-based vector processing according to the first rule. This embodiment is not further described here.
[0075] Based on the relevant description in the aforementioned method embodiment 100, the parallelism determination process in the distributed job provided by this application is further exemplarily described below with reference to examples, as follows.
[0076] The predicted reduction phase execution time is used as a performance metric. A skip-grid search algorithm is employed. Given a given search time upper limit (e.g., a first threshold), the initial reduction parallelism (i.e., baseline parallelism) and the obtained feature vector are first input into a target prediction model (e.g., a random forest model). This yields a baseline predicted reduction phase execution time, which serves as the baseline value. Next, a list of candidate parallelisms is generated based on the candidate parallelism range and a set initial value spacing (e.g., a first interval). Each candidate parallelism is then iterated through and tried. The candidate parallelism and feature vector are input into the trained target prediction model to obtain the predicted candidate reduction phase execution time. If the candidate phase execution time improves the baseline job performance by more than a second threshold, or if the search time exceeds the first threshold, the search process is terminated. Otherwise, the value spacing is reduced to half of the original value (e.g., a second threshold), and the candidate parallelism list is re-tried. Finally, the best-performing candidate parallelism value is selected as the final recommended parallelism, i.e., the target parallelism.
[0077] Based on this, the following takes the random forest model as an example of the target prediction model and further introduces its implementation process in combination with steps S301 to S307. The example content is as follows.
[0078] In step S301, the skip grid search algorithm is initialized, for example, initial values are assigned to the search time upper limit (such as the first threshold), the value interval (such as the first interval), the performance improvement threshold (such as the second threshold), etc.
[0079] In step S302, the feature vector and the benchmark parallelism corresponding to the distributed job to be tuned are input into the trained random forest model to predict the execution time of the benchmark reduction phase, that is, the baseline value.
[0080] In step S303 , a candidate parallelism list is determined according to the input candidate parallelism range and a set value interval, such as a first interval.
[0081] In step S304, the candidate parallelism list is traversed in sequence to determine whether there is a target candidate parallelism and whether the search duration exceeds a first threshold. If it does not exceed the first threshold and there is a target candidate parallelism, step S307 is executed; if it does not exceed the first threshold and there is no target candidate parallelism, step S305 is executed. If it times out, step S307 is executed.
[0082] When executing step S304, you can first determine whether the search time exceeds the first threshold, and then determine whether there is target candidate parallelism, or you can first determine whether there is target candidate parallelism, and then determine whether the search time exceeds the first threshold. This embodiment does not limit this.
[0083] In step S305, it is determined whether the current candidate parallelism list has been traversed. If so, the process proceeds to step S306; if not, the process proceeds to step S307.
[0084] In step S306, the current value interval is reduced to half of the original value (such as the second interval), the candidate parallelism list is re-determined, step 304 is executed again based on the re-determined candidate parallelism list, and steps S304-S306 are repeated until the search time exceeds the first threshold or the target parallelism is searched.
[0085] Step S307: The searched target parallelism is used as the final recommended reduced parallelism and fed back to the job to be tuned.
[0086] Alternatively, if the target parallelism is not found when the search time reaches the first threshold, the candidate parallelism with the best performance improvement among multiple candidate parallelisms in the search process can be used as the final recommended specification parallelism and fed back to the job to be tuned.
[0087] The implementation process of the aforementioned steps provided in this example can refer to the relevant description in the aforementioned embodiment of the method for determining the degree of parallelism in distributed jobs, and achieve the same or corresponding technical effects. To avoid repetition, they will not be described here.
[0088] Furthermore, the parallelism determination process in the distributed job provided by the application may include but is not limited to the aforementioned steps, and may include more or fewer steps than the aforementioned steps, which is not limited here.
[0089] Based on the foregoing description, it can be seen that the parallelism determination solution in distributed jobs provided in this application has the following technical effects.
[0090] (1) It can reasonably abstractly represent distributed jobs in actual production environments. Combined with the fact that the random forest model has low training and storage overhead and can be trained once to tune multiple distributed jobs, the lightweight random forest model learns the relationship between the data characteristics of distributed jobs and the execution time of the job specification phase, and predicts the execution time of the job specification phase, which can reduce the overhead of the parallelism tuning process.
[0091] (2) This application uses the changing relationship between the specified parallelism and the job performance to perform parallelism tuning, which can effectively narrow the search range to find a better specified parallelism in a shorter time, thereby improving the parallelism tuning efficiency and making the tuning effect more stable.
[0092] (3) The parallelism tuning process provided by this application (i.e., the process of determining the target parallelism) is located before the execution of the distributed job, and does not cause any impact on the original job execution process. In addition, the training and storage overhead of the random forest model is relatively small, and the jumping grid search algorithm can find a better solution without exceeding the set tuning time limit. This makes the entire tuning process non-invasive, low-overhead, short-time, and high-precision.
[0093] (4) This application can be applied to various distributed computing frameworks based on the MapReduce parallel computing model, such as Hadoop MapReduce and Spark, and can adaptively select the parallelism of the reduce side to achieve the purpose of improving job performance by optimizing the parallelism of the reduce side. The entire process is non-invasive, low-overhead, short-time, and high-precision.
[0094] Figure 4 shows a structural schematic diagram of a parallelism determination device 300 in a distributed job provided in an embodiment of the present application. The device 300 includes: a data acquisition module 310, used to obtain data features in job data corresponding to the distributed job; a determination module 320, used to obtain each candidate parallelism in a candidate parallelism list, and determine the candidate specification phase execution time based on the candidate parallelism and the data features; and used to determine the target parallelism corresponding to the distributed job from the candidate parallelism list based on the candidate specification phase execution time.
[0095] In some embodiments, the determination module 320 obtains each candidate parallelism in the candidate parallelism list, determines the candidate reduction phase execution time based on the candidate parallelism and the data characteristics, and determines the target parallelism corresponding to the distributed job from the candidate parallelism list based on the candidate reduction phase execution time, including: traversing the candidate parallelism in the candidate parallelism list, wherein the candidate parallelism list is determined based on the candidate parallelism range and the first interval; when the search time does not reach the first threshold and the candidate parallelism that meets the first condition is searched, stop searching and use the candidate parallelism that meets the first condition as the target parallelism of the distributed job; wherein the first condition includes that the candidate reduction phase execution time determined based on the candidate parallelism and the data characteristics meets the first requirement.
[0096] In some embodiments, the determination module 320 obtains each candidate parallelism in the candidate parallelism list, determines the candidate reduction phase execution time based on the candidate parallelism and the data characteristics, and determines the target parallelism corresponding to the distributed job from the candidate parallelism list based on the candidate reduction phase execution time. It also includes: when the search time does not reach the first threshold but the candidate parallelism list has been traversed and the candidate parallelism that meets the first condition has not been searched, the candidate parallelism list is redetermined based on the candidate parallelism range and the second interval, and the second interval is smaller than the first interval; the candidate parallelism in the redetermined candidate parallelism list is traversed again; when the search time does not reach the first threshold and the candidate parallelism that meets the first condition is searched, the search is stopped and the candidate parallelism that meets the first condition is used as the target parallelism of the distributed job.
[0097] In some embodiments, the determination module 320 obtains each candidate parallelism in the candidate parallelism list, determines the candidate reduction phase execution time based on the candidate parallelism and the data characteristics, and determines the target parallelism corresponding to the distributed job from the candidate parallelism list based on the candidate reduction phase execution time. It also includes: when the search time reaches the first threshold but the candidate parallelism that meets the first condition is not found, the candidate parallelism that meets the second condition during the search process is used as the target parallelism of the distributed job; wherein the second condition includes that the candidate reduction phase execution time determined based on the candidate parallelism and the data characteristics meets the second requirement.
[0098] In some embodiments, the determination module 320 determines whether the candidate specification phase execution time determined based on the candidate parallelism and the data characteristics meets the first requirement, including: determining the improvement effect based on the candidate specification phase execution time and the baseline specification phase execution time; when the improvement effect is greater than a second threshold, determining that the first requirement is met; wherein, the baseline specification phase execution time is determined based on the data characteristics and the baseline parallelism.
[0099] In some embodiments, the determination module 320 determines the candidate reduction phase execution time based on the candidate parallelism and the data features, including: inputting the feature vectors corresponding to the candidate parallelism and the data features into a target prediction model to obtain the candidate reduction phase execution time; wherein, the target prediction model is trained based on a first training data set, and the first training data set includes feature data and label data in historical job data corresponding to the distributed job, and the label data is the reduction phase execution time corresponding to the distributed job.
[0100] In some embodiments, the target prediction model is a random forest model, and the determination module 320 is further used to determine a second training data set based on the job data corresponding to the newly added distributed job relative to the distributed job; and use the second training data set to perform incremental training on the newly added decision tree in the random forest; wherein the newly added decision tree corresponds to the newly added distributed job, and the second training data set includes feature data and label data in the job data corresponding to the newly added distributed job, and the label data is the execution time of the protocol phase corresponding to the newly added distributed job.
[0101] In some embodiments, the data acquisition module is further used to convert the data feature into a feature vector according to a first rule; wherein the first rule includes at least one of the following: when the data feature is an enumeration type value, the data feature is converted into the feature vector according to a one-hot encoding rule; when the data feature is a numerical type value, the data feature is converted into the feature vector according to a second rule, the second rule including splitting the data feature according to the numerical bit, and retaining the first two non-zero digits in the split numerical bit based on rounding logic, and ignoring the non-zero digits in other numerical bits after the first two non-zero digits; when the data feature is a numerical value including a unit, any unit in the data feature is selected as a base unit, and the data feature is converted into the feature vector based on the base unit and the second rule.
[0102] In some embodiments, the data characteristics include at least one of the reduction operator type, input data size, mapping parallelism, and reduction parallelism.
[0103] The device 300 provided in the embodiment of the present application can execute the various methods described in the above method embodiments and realize the functions and beneficial effects of the various methods described in the above method embodiments, which will not be repeated here.
[0104] FIG4 shows a schematic diagram of the hardware structure of an electronic device that implements an embodiment of the present application. Referring to the figure, at the hardware level, the electronic device includes a processor, and in some embodiments, an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. In addition, the electronic device may also include hardware required for other services.
[0105] The processor, network interface, and memory can be interconnected via an internal bus, such as an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. These buses can be classified as address buses, data buses, and control buses. For ease of illustration, the figure uses only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.
[0106] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.
[0107] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, logically forming a device for locating a specific user. The processor executes the program stored in the memory and is specifically configured to perform the methods disclosed in the embodiments shown in Figures 1-3 and to achieve the functions and benefits of the various methods described in the preceding method embodiments, which will not be further described here.
[0108] The methods disclosed in the embodiments shown in Figures 1 and 2 of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.
[0109] The electronic device can also execute the methods described in the above method embodiments and realize the functions and beneficial effects of the methods described in the above method embodiments, which will not be described in detail here.
[0110] In some embodiments, in addition to software implementation, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0111] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the method disclosed in the embodiments shown in Figures 1 to 3 and realizes the functions and beneficial effects of the various methods described in the previous method embodiments, which will not be repeated here.
[0112] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0113] Furthermore, an embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the following process is implemented: the method disclosed in the embodiments shown in Figures 1 to 3 and the functions and beneficial effects of the various methods described in the previous method embodiments are implemented, which will not be repeated here.
[0114] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
[0115] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.
[0117] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0118] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.
Claims
1. A method for determining the degree of parallelism in a distributed job, comprising: Obtain data features in job data corresponding to distributed jobs; Obtain each candidate parallelism in the candidate parallelism list, and determine the candidate reduction phase execution time according to the candidate parallelism and the data characteristics; The target parallelism corresponding to the distributed job is determined from the candidate parallelism list according to the candidate reduction stage execution time.
2. The method according to claim 1, wherein obtaining each candidate degree of parallelism in the candidate degree of parallelism list, determining a candidate specification phase execution time according to the candidate degree of parallelism and the data feature, and determining a target degree of parallelism corresponding to the distributed job from the candidate degree of parallelism list according to the candidate degree of parallelism phase execution time comprises: Traversing the candidate parallelism in the candidate parallelism list, wherein the candidate parallelism list is determined according to the candidate parallelism range and the first interval; If the search duration does not reach the first threshold but a candidate parallelism satisfying the first condition is found, stop searching and use the candidate parallelism satisfying the first condition as the target parallelism corresponding to the distributed job; The first condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement.
3. The method according to claim 2, wherein the step of obtaining each candidate degree of parallelism in the candidate degree of parallelism list, determining a candidate reduction phase execution time according to the candidate degree of parallelism and the data feature, and determining a target degree of parallelism corresponding to the distributed job from the candidate degree of parallelism list according to the candidate degree of parallelism phase execution time, further comprises: When the search duration does not reach the first threshold, but the candidate parallelism list has been traversed and no candidate parallelism satisfying the first condition is found, the candidate parallelism list is re-determined according to the candidate parallelism range and the second interval, and the second interval is smaller than the first interval; Traversing the candidate parallelism in the re-determined candidate parallelism list again; If the search duration does not reach the first threshold and a candidate parallelism satisfying the first condition is found, the search is stopped and the candidate parallelism satisfying the first condition is used as the target parallelism of the distributed job.
4. The method according to claim 2, wherein the step of obtaining each candidate degree of parallelism in the candidate degree of parallelism list, determining a candidate reduction phase execution time according to the candidate degree of parallelism and the data feature, and determining a target degree of parallelism corresponding to the distributed job from the candidate degree of parallelism list according to the candidate degree of parallelism phase execution time, further comprises: When the search duration reaches the first threshold but the candidate parallelism satisfying the first condition is not found, taking the candidate parallelism satisfying the second condition during the search as the target parallelism of the distributed job; The second condition includes that the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the second requirement.
5. The method according to claim 2, wherein the candidate reduction phase execution time determined according to the candidate parallelism and the data characteristics meets the first requirement, comprising: Determine the improvement effect according to the candidate specification phase execution time and the benchmark specification phase execution time; When the improvement effect is greater than a second threshold, determining that the first requirement is met; The execution time of the benchmark reduction phase is determined according to the data characteristics and the benchmark parallelism.
6. The method according to claim 5, wherein determining the candidate reduction phase execution time according to the candidate parallelism and the data characteristics comprises: Inputting the candidate parallelism and the feature vector corresponding to the data feature into a target prediction model to obtain the candidate reduction stage execution time; The target prediction model is trained based on a first training data set, wherein the first training data set includes feature data in the historical job data corresponding to the distributed job. and label data, where the label data is the execution time of the protocol phase corresponding to the distributed job.
7. The method according to claim 6, wherein the target prediction model is a random forest model, and the method further comprises: When a new distributed job relative to the distributed job is acquired, determining a second training data set according to job data corresponding to the new distributed job; The second training data set is used to perform incremental training on the newly added decision tree in the random forest; wherein the newly added decision tree corresponds to the newly added distributed job, and the second training data set includes feature data and label data in the job data corresponding to the newly added distributed job, and the label data is the execution time of the protocol phase corresponding to the newly added distributed job.
8. The method according to any one of claims 1 to 7, further comprising: Converting the data features into feature vectors according to a first rule; The first rule includes at least one of the following: In the case where the data feature is an enumeration type value, converting the data feature into the feature vector according to a one-hot encoding rule; In the case where the data feature is a numerical type value, converting the data feature into the feature vector according to a second rule, wherein the second rule includes performing numerical bit splitting on the data feature according to numerical bits, and retaining the first two non-zero digits of the split numerical bits based on rounding logic, and ignoring non-zero digits in other numerical bits after the first two non-zero digits; In the case that the data feature is a numerical value including a unit, any unit in the data feature is selected as a reference unit, and the data feature is converted into the feature vector based on the reference unit and the second rule.
9. The method according to any one of claims 1 to 7, wherein the data characteristics include at least one of the reduction operation operator type, input data scale, mapping parallelism, and reduction parallelism.
10. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the method of any one of claims 1 to 9.
11. A computer-readable medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to perform the steps of the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Machine learning method for Map Reduce task resource allocation parameters
CN103593323A
Configuration method and apparatus for related parameters of MapReduce application
CN105302536A
Dynamic resource allocation method and device, storage medium and equipment
CN110389842A
Method and device for predicting platform operation execution time and electronic equipment
CN112286990A
