Method, device, equipment and medium for automatically adjusting data tilt optimization factor

Abstract

The present invention discloses a method, device, equipment and medium for automatically adjusting a data skew optimization factor. The method includes: determining the type of task whose processing speed per unit data volume exceeds a preset speed range; optimizing the speed of the task according to the type of task; judging whether the processing speed per unit data volume of the task after optimization is within the preset speed range; if the processing speed per unit data volume of the task after optimization is within the preset speed range, recording the process parameters of the speed optimization and ending the adjustment method; if the processing speed per unit data volume of the task after optimization is outside the preset speed range, calculating the optimization factor according to the memory data of the cluster server; and re-optimizing the cluster server according to the relative relationship between the optimization factor and the preset optimization value. The present application scheme realizes the automatic adjustment of the data skew optimization factor, reduces the workload of developers and improves the reliability of data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, equipment and medium for automatically adjusting a data skew optimization factor. Background Art

[0002] The MapReduce model is often used when analyzing big data. Data skew during the Reduce phase of MapReduce is common. If the amount of data corresponding to a primary key is particularly large, this can cause node congestion or even crashes, significantly impacting data processing progress.

[0003] The existing method for dealing with data skew is to pay attention to the execution status of tasks every day during the processing process. When data skew causes task execution failure or task duration exceeds the threshold, the task operation log is checked one by one to identify the specific problem.

[0004] Existing methods for dealing with data skew require developers to monitor and troubleshoot for a long time, which is labor-intensive and time-consuming. Summary of the Invention

[0005] The present invention provides a method, device, equipment and medium for automatically adjusting a data tilt optimization factor, so as to realize automatic adjustment of the data tilt optimization factor, reduce the workload of developers and improve the reliability of data processing.

[0006] According to one aspect of the present invention, a method for automatically adjusting a data skew optimization factor is provided, the method comprising:

[0007] Determine the type of task whose processing speed per unit data volume exceeds the preset speed range;

[0008] Optimizing the speed of the task according to the type of the task;

[0009] Determining whether the processing speed of the task per unit data volume after optimization is within the preset speed range;

[0010] If the processing speed of the task per unit data volume after optimization is within the preset speed range, recording the process parameters of the speed optimization and ending the adjustment method;

[0011] If the processing speed of the task per unit data volume after optimization is outside the preset speed range, calculating an optimization factor based on memory data of a cluster server, wherein the cluster server is used to implement the Reduce phase of the MapReduce model;

[0012] The cluster server is optimized again according to the relative relationship between the optimization factor and the preset optimization value.

[0013] According to another aspect of the present invention, there is provided a device for automatically adjusting a data tilt optimization factor, the device comprising: a task type determination module, a speed optimization module, a processing speed determination module, a process parameter recording module, an optimization factor calculation module, and a re-optimization module;

[0014] The task type determination module is used to determine the type of task whose processing speed per unit data volume exceeds a preset speed range;

[0015] The speed optimization module is used to optimize the speed of the task according to the type of the task;

[0016] The processing speed determination module is used to determine whether the processing speed of the task per unit data volume after optimization is within the preset speed range;

[0017] The process parameter recording module is configured to record the process parameters of the speed optimization and terminate the adjustment method if the processing speed of the task per unit data volume after optimization is within the preset speed range;

[0018] The optimization factor calculation module is configured to calculate the optimization factor based on memory data of a cluster server if the processing speed of the task per unit data volume after optimization is outside the preset speed range, wherein the cluster server is configured to implement the Reduce phase of the MapReduce model;

[0019] The re-optimization module is used to re-optimize the cluster server according to the relative relationship between the optimization factor and the preset optimization value.

[0020] According to another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for automatically adjusting the data skew optimization factor described in any of the aforementioned items.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any of the aforementioned methods for automatically adjusting the data skew optimization factor when executed.

[0022] The technical solution of the embodiment of the present invention implements different speed optimization methods according to the different task types in the Reduce stage of the MapReduce model, and judges whether the expected optimization effect is achieved according to the processing speed of the unit data volume after optimization. If not, the optimization factor can be calculated according to the memory occupancy ratio of the cluster server, and the further optimization plan of the cluster server is further determined according to the size of the optimization factor, thereby realizing speed optimization of the tasks in the Reduce stage, implementing the optimization plan in advance before data skew occurs, and implementing further optimization according to the optimization factor, thereby realizing automatic adjustment of the data skew optimization factor, reducing the workload of developers, and improving the reliability of data processing.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of the implementation process of the MapReduce model in the background technology;

[0026] Figure 2 A flowchart of a method for automatically adjusting a data skew optimization factor provided by an embodiment of the present invention;

[0027] Figure 3 A flowchart of another method for automatically adjusting a data skew optimization factor provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of a second random prefix optimization process provided by an embodiment of the present invention;

[0029] Figure 5 A flow chart diagram of a sampling statistics method provided by an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of a process of first random prefix optimization or first prefix range expansion optimization provided by an embodiment of the present invention;

[0031] Figure 7 A schematic diagram comparing the processes of a JION operation and a MAP JION operation provided in an embodiment of the present invention;

[0032] Figure 8 A schematic diagram of the structure of a device for automatically adjusting a data skew optimization factor provided by an embodiment of the present invention;

[0033] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] As described in the background technology, the MapReduce model is often used when analyzing big data in the existing technology. Figure 1As shown, the MapReduce model is a programming model for parallel computing of large-scale data sets. The MapReduce model divides large amounts of data into blocks, including data with the same primary key, which may be divided into different data blocks. The data is then read in blocks during the Map phase, and the data in each block is classified according to the primary key. The classification results determine which Reduce process the data is assigned to. In the Reduce phase, all data with the same primary key is moved locally and processed again, ultimately generating the result data. Data skew is a common problem in the Reduce phase of the MapReduce model. The inventors have discovered that when performing aggregation or JOIN operations on data with the same primary key, the data must be assigned to a task on a specific node for processing. If the amount of data corresponding to a certain primary key is particularly large, the node corresponding to that primary key will become blocked or even crash. In a parallel computing job, the progress of the entire job is determined by the task with the longest running time. When data skew occurs, the entire job will run very slowly, and even memory overflow exceptions may occur. To improve data processing speed, traditional data skew optimization methods require constant monitoring of the execution of tasks in the Reduce phase. When data skew causes task execution failure or task duration exceeds a threshold, developers are required to review the task's running logs one by one to troubleshoot specific issues. This method is labor-intensive and time-consuming.

[0037] Based on the above reasons, the present invention proposes a method for automatically adjusting a data skew optimization factor. Figure 2 A flowchart of a method for automatically adjusting a data tilt optimization factor provided by an embodiment of the present invention, referring to Figure 2 ,The automatic adjustment methods for data tilt optimization factors include:

[0038] S101: Determine the type of task whose processing speed per unit data volume exceeds a preset speed range.

[0039] Specifically, the Reduce phase of the MapReduce model includes two task types, namely aggregation operations and JOIN operations. The types of various tasks can be divided according to the task type of the Reduce phase. Compare whether the processing speed of the unit data volume during the implementation of the task exceeds the preset speed range. If so, it can be determined that the processing speed of the task is abnormal and the task type of the task is determined. If the processing speed of the unit data volume during the implementation of the task is within the preset speed range, it can be determined that the processing speed of the task is normal. The way to determine the task type is to judge based on the label or ID set when the task is created. For example, the preset speed range can be greater than or equal to 10M / s. If the processing speed of the unit data volume of a task is less than 10M / s, it is determined that the processing speed of the task is lower than the normal value, and data skew is likely to occur. At this time, it is necessary to further determine the task type based on basic information such as the label or ID of the task.

[0040] S102: Optimize the speed of the task according to the type of the task.

[0041] Specifically, an aggregation operation aggregates data with the same primary key into a single dataset. A join operation maps data from two datasets into corresponding data. The specific optimization methods for these two types of tasks depend on the task type. Aggregation operations can use random prefix optimization or prefix range expansion optimization. Join operations can use task replacement optimization, capacity expansion optimization, capacity range expansion optimization, random prefix optimization, or prefix range expansion optimization.

[0042] S103: Determine whether the processing speed of the task per unit data volume after optimization is within a preset speed range.

[0043] Specifically, after speed optimization is complete, a check is performed to determine whether the optimized processing speed per unit of data is within a preset speed range. If so, the task's processing speed has reached the normal range after speed optimization, and data processing can proceed normally. Otherwise, the task's processing speed still has not reached the normal range after speed optimization, and there is still the possibility of data skew, requiring further speed optimization.

[0044] S104: If the processing speed of the task per unit data volume after optimization is within the preset speed range, record the process parameters of the speed optimization and end the adjustment method.

[0045] Specifically, the process parameters for speed optimization are the recorded data of the operation content and test results during the speed optimization of the task in step S102. The process parameters for speed optimization can include parameters related to the optimization process, such as the operation name, operation number, judgment result, and execution time. If it is, it means that after the speed optimization, the processing speed of the task has reached the normal range and data processing can proceed normally. Recording the process parameters for speed optimization can facilitate the adjustment of optimization factors during subsequent data processing, facilitating maintenance work for developers.

[0046] S105: If the processing speed of the task per unit data volume after optimization is outside the preset speed range, calculate the optimization factor according to the memory data of the cluster server.

[0047] Among them, the cluster server is used to implement the Reduce phase of the MapReduce model.

[0048] Specifically, if the processing speed of the task per unit data volume after optimization is outside the preset speed range, it means that after speed optimization, the processing speed of the task still does not reach the normal range, and there is still the possibility of data skew, and further speed optimization is needed. The cluster server is a plurality of servers that implement the Reduce phase of the MapReduce model, which can realize parallel processing of tasks. The memory data of the cluster server can be the occupancy ratio of the running memory of all servers in the cluster server, which can reflect the ratio of the amount of data processed in parallel by the cluster server at the same time to the maximum running memory of the cluster server. The optimization factor is positively correlated with the occupancy ratio of the memory data of the cluster server. For example, if the total running memory of the cluster server is 4000T and the occupied running memory is 3000T, the optimization factor is 3000 divided by 4000, which is equal to 75%.

[0049] S106: Optimize the cluster server again according to the relative relationship between the optimization factor and the preset optimization value.

[0050] Specifically, the preset optimization value can be a value set by the developer based on actual needs and experimental data. When the optimization factor is greater than the preset optimization value, the total data processing efficiency of the cluster server is significantly reduced or the cluster server experiences phenomena such as heating and freezing. At this time, the occupancy rate of the cluster server's running memory is close to 100%, which is not suitable for further increasing the optimization factor. Exemplarily, the preset optimization value can be 95%. Determine whether the optimization factor is greater than or equal to the preset optimization value. If so, it indicates that the occupancy rate of the cluster server's running memory is close to 100%, which is not suitable for further increasing the optimization factor. At this time, the developer can be prompted to intervene and correct it or to expand the capacity. Otherwise, it indicates that the occupancy rate of the cluster server's running memory is low, and the optimization factor can be further increased. At this time, return to step S102 to further optimize the task speed.

[0051] The method for automatically adjusting the data skew optimization factor provided in this embodiment implements different speed optimization methods according to different task types in the Reduce phase of the MapReduce model, and determines whether the expected optimization effect is achieved based on the processing speed of the unit data volume after optimization. If not, the optimization factor can be calculated based on the memory usage ratio of the cluster server, and further optimization schemes for the cluster server can be determined based on the size of the optimization factor. This achieves speed optimization of tasks in the Reduce phase, implements the optimization scheme in advance before data skew occurs, and implements further optimization based on the optimization factor. This not only achieves automatic adjustment of the data skew optimization factor, reduces the workload of developers, but also improves the reliability of data processing.

[0052] Figure 3 A flowchart of another method for automatically adjusting a data skew optimization factor provided by an embodiment of the present invention is provided. Figure 4 A schematic diagram of a second random prefix optimization process provided by an embodiment of the present invention, referring to Figure 3 ,The automatic adjustment methods for data tilt optimization factors include:

[0053] S201. Record the time consumed in the Reduce phase, calculate the processing speed per unit data volume, and establish a time series model.

[0054] Specifically, when implementing the MapReduce model to process big data, the time consumption data for each task is recorded in real time after completion. This data can be recorded based on the task ID, task description, task execution time, operation type, operation description, and operation duration. Operation types can include speed-optimized operations. Furthermore, a time series model is constructed based on the recorded data. The time series model includes at least the task ID, task execution time, and operation type.

[0055] S202: Determine, based on the time series model, tasks whose processing speed per unit data volume exceeds a preset speed range.

[0056] Specifically, the processing speed per unit of data for each task is calculated based on the task execution time and the task data processing volume in the time series model. This speed is then compared with a preset speed range, which can be greater than 10M / s. Tasks with a speed per unit of data exceeding the preset speed range are considered to have a processing speed below the normal range, potentially exhibiting data skew and requiring speed optimization. If the processing speed per unit of data falls within the preset speed range, the task processing speed is within the normal range, and the method can be terminated directly.

[0057] S203: Determine the type of task whose processing speed per unit data volume exceeds a preset speed range.

[0058] Specifically, step S203 is identical to step S101 and will not be described again here.

[0059] S204: If the task is a JOIN operation, perform a first speed optimization operation based on the size of the data set in the task and the number of primary keys that cause data skew.

[0060] Specifically, the first speed optimization operation includes at least one of the following: task replacement optimization, capacity expansion optimization, capacity range extension optimization, first random prefix optimization, and first prefix range extension optimization. The optimization operation can be selected based on the dataset size and historical optimization records. Task replacement optimization involves replacing a task with a JOIN operation and a MAP JOIN operation. When implementing capacity expansion optimization, a dataset is first selected for a JOIN operation. Each data item in the dataset is mapped into multiple data items. Each of the multiple data items is prefixed with a random number between 0 and n. Each data item in the other dataset is then prefixed with a random number within n. Finally, the two datasets are joined and the prefix removed. Capacity expansion optimization, based on the already implemented capacity expansion optimization for the dataset corresponding to the task, increases the capacity expansion range by increasing the prefix range (i.e., increasing the value of n) to increase the data separation within the dataset. The first random prefix optimization involves assigning a random number between 0 and k to each data item in the dataset and performing a scatter operation before performing a JOIN operation to improve data separation. The first prefix range expansion optimization is to expand the range of random numbers (ie, increase the value of k) on the basis of the first random prefix optimization implemented on the data set.

[0061] S205: If the task is an aggregation operation, perform a second speed optimization operation based on the historical optimization status of the task.

[0062] Specifically, the second speed optimization operation includes at least one of the following: a second random prefix optimization and a second prefix range expansion optimization, and the optimization operation can be selected based on historical optimization records. Figure 4 For example, the second random prefix optimization involves adding a random prefix to the primary key and then breaking up the keywords. The keywords are then divided into multiple groups, followed by a first local aggregation. The prefix of each keyword is then removed, and all keywords are then globally aggregated. The second prefix range expansion optimization involves further expanding the range of the random prefix in the second random prefix optimization.

[0063] Performing a second speed optimization operation based on the task's historical optimization status includes: S2051: Determining whether the aggregation operation has already been subjected to the second random prefix optimization based on the historical optimization status; S2052: If so, performing the second prefix range expansion optimization on the data corresponding to the aggregation operation; S2053: Otherwise, performing the second random prefix optimization on the data corresponding to the aggregation operation.

[0064] S206: Determine whether the processing speed of the task per unit data volume after optimization is within a preset speed range.

[0065] Specifically, after the speed optimization operation is completed, the Reduce phase can be continued, and the time series model can be continuously updated to determine the optimized processing speed per unit data volume based on the time series model, and determine whether the optimized processing speed per unit data volume is within a preset speed range. The preset speed range is the same as that in step S202.

[0066] S207: If the processing speed of the task per unit data volume after optimization is within the preset speed range, record the process parameters of the speed optimization and end the adjustment method.

[0067] S208: If the processing speed of the task per unit data volume after optimization is outside the preset speed range, calculate the optimization factor according to the memory data of the cluster server.

[0068] Specifically, the contents of step S207 and step S208 are the same as those of step S104 and step S105, respectively, and are not repeated here.

[0069] S209: Determine whether the optimization factor is less than a preset optimization value.

[0070] Specifically, the optimization factor may be the amount of memory occupied by the cluster server. The preset optimization value is a preset value set by a developer, and may be 98% by way of example.

[0071] S210: If the optimization factor is less than the preset optimization value, return to the operation of executing the type of task of determining the processing speed of the unit data volume within the preset speed range.

[0072] Specifically, if the optimization factor is less than the preset optimization value, it indicates that the cluster server still has a certain amount of free running memory available, and the amount of data for parallel calculation can be further increased. At this time, return to step S203 to re-determine the type of task whose processing speed per unit data volume is within the preset speed range.

[0073] S211. If the optimization factor is greater than or equal to the preset optimization value, the developer is prompted to expand the cluster server and the adjustment method ends.

[0074] Specifically, if the optimization factor is greater than or equal to the preset optimization value, it indicates that the cluster server's running memory is close to full capacity and the amount of data for parallel computing cannot be further increased. At this time, it is necessary to prompt the developer to expand the cluster server capacity and end the adjustment method. The prompt method can be a warning sound or a flashing light.

[0075] The method for automatically adjusting the data skew optimization factor provided in this embodiment records the time consumed by the task in the Reduce phase and calculates the processing speed per unit data volume after each task is executed, establishes a time series model, and determines the processing speed per unit data volume of the task based on the time series model. When the data processing speed is lower than or exceeds the preset speed range, the data processing flow is automatically optimized, and the optimization factor parameters are adjusted to optimize the data processing speed and performance of the cluster, thereby achieving automatic adjustment of the data skew optimization factor and automatically generating a complete optimization record after the optimization is completed. When the operation optimization is invalid and the cluster needs to be expanded, an alarm prompt is issued in a timely manner, and a record is kept, so that developers or operation and maintenance personnel can promptly discover data skew problems and take corresponding measures, further improving the automation and reliability of the optimization factor adjustment method.

[0076] Figure 5 A flow chart diagram of a sampling statistics method provided by an embodiment of the present invention, Figure 6 A schematic diagram of a process of first random prefix optimization or first prefix range expansion optimization provided by an embodiment of the present invention, Figure 7 A schematic diagram comparing the processes of a JION operation and a MAP JION operation provided in an embodiment of the present invention. Figure 4 Optionally, in step S204, if the task is a JOIN operation, a first speed optimization operation is performed based on the size of the data set in the task and the number of primary keys that cause data skew, including:

[0077] S301: Determine whether the data set corresponding to the JOIN operation exceeds a preset data volume.

[0078] Specifically, the preset data volume is the maximum threshold allowed for broadcasting. It is determined whether the sizes of the two data sets corresponding to the JOIN operation exceed the maximum threshold allowed for broadcasting, and whether task replacement optimization can be adopted can be determined based on the determination result.

[0079] S302: If yes, perform capacity expansion optimization, capacity expansion range extension optimization, first random prefix optimization, or first prefix range extension optimization according to the number of primary keys in the data set and the historical optimization status of the task.

[0080] Specifically, if yes, it indicates that the task is not suitable for task replacement optimization. Figure 5 First, S401, perform sampling statistics on the data set where the JOIN operation is implemented to determine the number of primary keys that cause data skew, wherein the sampling statistics is to perform random sampling on all the data in the data set, and the sampling ratio can be set to 10%, and then sort the number of times the primary key appears in the sampled data, so that one or more primary keys that cause data skew can be found based on the number of primary key appearances. Secondly, S402, determine whether the number of primary keys that cause data skew is greater than the preset value. S403, if the number of primary keys is greater than the preset value, implement the first random prefix optimization or the first prefix range expansion optimization based on historical optimization conditions. S404, if the number of primary keys is less than or equal to the preset value, determine whether the data set has implemented capacity expansion optimization based on historical optimization conditions. S405, if so, implement capacity expansion range expansion optimization on the data set. S406, otherwise implement capacity expansion optimization on the data set.

[0081] Among them, S403, if the number of primary keys is greater than the preset value, implement the first random prefix optimization or the first prefix range expansion optimization according to the historical optimization situation, including: S501, filter out the data in the data set that causes data skew, wherein the data that causes data skew is the data containing the primary key that causes data skew. S502, determine whether the data in the data set has been subjected to the first random prefix optimization according to the historical optimization situation. S503, if so, implement the first prefix range expansion optimization on the data in the data set that causes data skew. S504, otherwise, implement the first random prefix optimization on the data in the data set that causes data skew. S505, perform a JOIN operation on the optimized data, remove the prefix and merge it with ordinary data, wherein ordinary data is the data in the data set that does not cause data skew when the JOIN operation is performed. Exemplarily, the first random prefix optimization or the first prefix range expansion optimization can be combined with Figure 6 .

[0082] S303: Otherwise, broadcast the smaller data set and then perform task replacement optimization.

[0083] Specifically, the task replacement optimization includes broadcasting the smaller of the two data sets that implement the JION operation, and then replacing the task from the JION operation to the MAP JION operation. If the size of the data set does not exceed the preset data volume, the smaller data set is broadcasted, and JOIN is not used to connect the two data sets, because the ordinary JOIN operation will trigger the Shuffle process. Once the Shuffle is triggered, the data with the same primary key will be pulled into the same task for processing. Using the MAP JOIN operation to obtain the data in the smaller data set from the broadcast variable for the connection operation will not trigger the Shuffle, which can avoid the occurrence of data skew. For example, the process comparison of the JION operation and the MAP JION operation can be referred to. Figure 7 .

[0084] The method for automatically adjusting the data skew optimization factor provided in this embodiment can adaptively perform speed optimization operations on tasks according to the task type, data set size and historical optimization conditions, thereby optimizing the data processing flow and further improving the reliability of automatic adjustment of the optimization factor.

[0085] An embodiment of the present invention also provides a device for automatically adjusting a data skew optimization factor. Figure 8 A schematic diagram of a data tilt optimization factor automatic adjustment device provided by an embodiment of the present invention, referring to Figure 8 The data skew optimization factor automatic adjustment device 800 includes: a task type determination module 801, a speed optimization module 802, a processing speed determination module 803, a process parameter recording module 804, an optimization factor calculation module 805, and a re-optimization module 806. The task type determination module 801 is used to determine the type of task whose processing speed per unit data volume exceeds a preset speed range; the speed optimization module 802 is used to optimize the speed of the task according to the task type; the processing speed determination module 803 is used to determine whether the processing speed per unit data volume of the task after optimization is within the preset speed range; the process parameter recording module 804 is used to record the speed optimization process parameters and terminate the adjustment method if the processing speed per unit data volume of the task after optimization is within the preset speed range; the optimization factor calculation module 805 is used to calculate the optimization factor based on the memory data of the cluster server if the processing speed per unit data volume of the task after optimization is outside the preset speed range, wherein the cluster server is used to implement the Reduce phase of the MapReduce model. The re-optimization module 806 is used to re-optimize the cluster server based on the relative relationship between the optimization factor and the preset optimization value.

[0086] An embodiment of the present invention further provides an electronic device, Figure 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, referring to Figure 9 The electronic device 900 includes: at least one processor 901; and a memory 902 in communication with the at least one processor 901; wherein the memory 902 stores a computer program that can be executed by the at least one processor 901, and the computer program is executed by the at least one processor 901 so that the at least one processor 901 can execute any of the aforementioned methods for automatically adjusting the data skew optimization factor.

[0087] The electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 900 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein. The processor 901 can be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors 901 that run machine learning model algorithms, digital signal processors 901 (DSPs), and any appropriate processor 901, controllers, microcontrollers, etc.

[0088] An embodiment of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the aforementioned method for automatically adjusting any data skew optimization factor when executed.

[0089] The data skew optimization factor automatic adjustment method, device, equipment and medium provided by the present invention implement different speed optimization methods according to different task types in the Reduce stage of the MapReduce model, and judge whether the expected optimization effect is achieved according to the processing speed of the unit data volume after optimization. If not, the optimization factor can be calculated according to the memory occupancy ratio of the cluster server, and the further optimization plan of the cluster server can be further determined according to the size of the optimization factor, thereby realizing speed optimization of tasks in the Reduce stage, implementing the optimization plan in advance before data skew occurs, and implementing further optimization according to the optimization factor, thereby realizing automatic adjustment of the data skew optimization factor, reducing the workload of developers, and improving the reliability of data processing.

[0090] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0094] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0095] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0096] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0097] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for automatically adjusting a data tilt optimization factor, characterized in that: include: Determine the type of task whose processing speed per unit data volume exceeds the preset speed range; Performing speed optimization on the task according to the type of the task, specifically including, if the task is a JOIN operation, performing a first speed optimization operation based on the size of the data set in the task and the number of primary keys that cause data skew, wherein the first speed optimization operation includes at least one of the following: task replacement optimization, capacity expansion optimization, capacity expansion range extension optimization, first random prefix optimization, and first prefix range extension optimization; if the task is an aggregation operation, performing a second speed optimization operation based on a historical optimization situation of the task, wherein the second speed optimization operation includes at least one of the following: second random prefix optimization and second prefix range extension optimization; Determining whether the processing speed of the task per unit data volume after optimization is within the preset speed range; If the processing speed of the task per unit data volume after optimization is within the preset speed range, recording the process parameters of the speed optimization and ending the adjustment method; If the processing speed of the task per unit data volume after optimization is outside the preset speed range, calculating an optimization factor based on memory data of a cluster server, wherein the cluster server is used to implement the Reduce phase of the MapReduce model; The cluster server is optimized again according to the relative relationship between the optimization factor and the preset optimization value.

2. The method for automatically adjusting the data tilt optimization factor according to claim 1, characterized in that: Further optimizing the cluster server according to the relative relationship between the optimization factor and the preset optimization value includes: Determining whether the optimization factor is less than the preset optimization value; If the optimization factor is less than the preset optimization value, returning to the operation of executing the type of task of determining that the processing speed of the unit data amount is within the preset speed range; If the optimization factor is greater than or equal to the preset optimization value, the developer is prompted to expand the capacity of the cluster server and the adjustment method is terminated.

3. The method for automatically adjusting the data tilt optimization factor according to claim 1, characterized in that: Before determining the type of task whose unit data processing speed exceeds the preset speed range, it also includes: Record the time consumed in the Reduce phase, calculate the processing speed of the unit data volume, and establish a time series model; Determine, according to the time series model, tasks in which the processing speed of a unit data volume exceeds a preset speed range.

4. The method for automatically adjusting the data tilt optimization factor according to claim 1, characterized in that: A first speed optimization operation is performed based on the size of the dataset in the task and the number of primary keys that cause data skew, including: Determine whether the data set corresponding to the JOIN operation exceeds the preset data volume; If so, performing the capacity expansion optimization, the capacity expansion range extension optimization, the first random prefix optimization, or the first prefix range extension optimization according to the number of primary keys of the data set and the historical optimization status of the task; Otherwise, the smaller of the two data sets subjected to the JOIN operation is broadcasted, and then the task replacement optimization is performed, wherein the task replacement optimization includes replacing the task from the JOIN operation to the MAP JOIN operation.

5. The method for automatically adjusting the data tilt optimization factor according to claim 4, characterized in that: Implementing the capacity expansion optimization, the capacity expansion range extension optimization, the first random prefix optimization, or the first prefix range extension optimization according to the number of primary keys of the data set and the historical optimization status of the task includes: Performing sampling statistics on the data set subjected to the JOIN operation to determine the number of primary keys that cause data skew; Determine whether the number of primary keys causing data skew is greater than a preset value; If the number of primary keys is greater than the preset value, performing the first random prefix optimization or the first prefix range expansion optimization according to the historical optimization situation; If the number of primary keys is less than or equal to the preset value, determining whether the data set has implemented the capacity expansion optimization based on the historical optimization situation; If yes, performing the capacity expansion optimization on the data set; Otherwise, the capacity expansion optimization is performed on the data set.

6. The method for automatically adjusting the data tilt optimization factor according to claim 5, characterized in that: Implementing the first random prefix optimization or the first prefix range expansion optimization according to the historical optimization situation includes: Filter out data that causes data skew in the data set; Determining whether the data in the data set has been subjected to the first random prefix optimization according to the historical optimization situation; If so, performing the first prefix range expansion optimization on the data in the data set that causes data skew; Otherwise, performing a first random prefix optimization on the data in the data set that causes data skew; The JOIN operation is performed on the optimized data, prefixes are removed, and the data is merged with normal data, wherein the normal data is data in the data set in which the JOIN operation is performed that will not cause data skew.

7. The method for automatically adjusting the data tilt optimization factor according to claim 1, characterized in that: Performing a second speed optimization operation based on the historical optimization status of the task includes: Determining whether the aggregation operation has implemented the second random prefix optimization according to the historical optimization situation; If so, performing the second prefix range expansion optimization on the data corresponding to the aggregation operation; Otherwise, the second random prefix optimization is performed on the data corresponding to the aggregation operation.

8. A data tilt optimization factor automatic adjustment device, characterized in that: include: A task type determination module, used to determine the type of task whose processing speed per unit data volume exceeds a preset speed range; a speed optimization module, configured to optimize the speed of the task according to the type of the task, and specifically to, if the task is a JOIN operation, perform a first speed optimization operation based on the size of the data set in the task and the number of primary keys that cause data skew, wherein the first speed optimization operation includes at least one of the following: task replacement optimization, capacity expansion optimization, capacity expansion range extension optimization, first random prefix optimization, and first prefix range extension optimization; and if the task is an aggregation operation, perform a second speed optimization operation based on the historical optimization status of the task, wherein the second speed optimization operation includes at least one of the following: second random prefix optimization and second prefix range extension optimization; A processing speed determination module, configured to determine whether the processing speed of the task per unit data volume after optimization is within the preset speed range; A process parameter recording module, configured to record the process parameters of the speed optimization and terminate the adjustment method if the processing speed of the task per unit data volume after optimization is within the preset speed range; an optimization factor calculation module, configured to calculate an optimization factor based on memory data of a cluster server if the processing speed of the task per unit data volume after optimization is outside the preset speed range, wherein the cluster server is used to implement the Reduce phase of the MapReduce model; The re-optimization module is used to re-optimize the cluster server according to the relative relationship between the optimization factor and the preset optimization value.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for automatically adjusting the data skew optimization factor according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for automatically adjusting the data skew optimization factor according to any one of claims 1 to 7 when executed.

Images