Data skew detection method, apparatus, and related device

Abstract

The application provides a data skew detection method, device and related equipment applied to a big data processing system. The method comprises the following steps: the big data processing system splits a user's big data computing task into multiple subtasks, and runs the multiple subtasks, then sorts multiple running durations corresponding to the multiple subtasks from large to small to obtain a sorting list; and then determines a difference between adjacent first running duration and second running duration in the sorting list, and when the difference is greater than a threshold, it is determined that the first subtask corresponding to the first running duration and the subtask arranged before the first subtask in the sorting list are subtasks with data skew, wherein the first running duration is greater than the second running duration. Through the above method, the subtask with data skew in the computing task can be detected and prompted, so that the user can optimize the computing task, and the running duration of the computing task is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data skew detection method and device and related equipment. BACKGROUND

[0002] In the field of big data, a large number of request queries and data aggregation will occur every day, and in such large-scale data processing, the application of the divide-and-conquer idea is very extensive, such as Hadoop, Spark and other platforms. By applying the divide-and-conquer idea, a computing task can be divided into multiple subtasks, and then the running results of the multiple subtasks are aggregated to obtain the result of the computing task. When the computing task is run by applying the divide-and-conquer idea, the running time of the computing task is determined by the slowest subtask. Generally, data skew is the main reason for long task running time, and if the subtask with data skew can be determined, the running time of the entire computing task can be effectively shortened by optimizing the slow subtask.

[0003] The current big data processing system can run the computing task and provide the computing result to the user, but cannot determine which subtask is the subtask with data skew, and therefore, how to determine the subtask with data skew in the multiple subtasks is a problem to be solved. SUMMARY

[0004] The present application discloses a data skew detection method, device and related equipment, which can detect and prompt the subtask with data skew in the computing task, so as to enable the user to optimize the computing task.

[0005] In a first aspect, an embodiment of the present application provides a data skew detection method, which comprises: a big data processing system splits a big data computing task uploaded by a user through a user terminal into multiple subtasks, and runs the multiple subtasks, and then sorts multiple running time lengths corresponding to the multiple subtasks from large to small to obtain a sorting list, wherein each subtask corresponds to a running time length; and then determines the difference between a first running time length and a second running time length in the sorting list, and when the difference is greater than a threshold value, determines that a first subtask corresponding to the first running time length and a subtask arranged before the first subtask in the sorting list are subtasks with data skew, wherein the first running time length is greater than the second running time length.

[0006] The data skew detection method described above determines the subtask with data skew by obtaining and sorting the runtime of each subtask in the computing task and determining the subtask with data skew according to the difference between the runtime of two subtasks. When the difference between the larger runtime and the smaller runtime of the two adjacent runtimes in the sorted list is greater than the threshold, it indicates that the difference between the runtimes of the two subtasks is large, and the subtask with the longer runtime is the subtask with data skew, and the subtasks with runtimes greater than or equal to the runtime of the subtask with data skew are also the subtasks with data skew. The method described above can determine the subtasks with data skew in the multiple subtasks. When the running time of the computing task is too long and the user needs to optimize the subtasks, the big data processing system can provide the user with the subtasks with data skew, so that the user can optimize the subtasks with data skew.

[0007] In a specific implementation, the first runtime is any one of the runtimes in the sorted list. By taking the runtime of each subtask in the multiple subtasks as the first runtime, the method described above can detect all the subtasks with data skew in the multiple subtasks.

[0008] In a specific implementation, the big data processing system can also send prompt information to the user, which informs the user that the first subtask and the subtasks arranged before the first subtask in the sorted list are the subtasks with data skew, so that the user can optimize the subtasks with data skew. The big data processing system can automatically send the prompt information to the user terminal to remind the user after determining the subtasks with data skew, or send the prompt information to the user terminal when the user queries the trial running result of the computing task.

[0009] In a specific implementation, the big data processing system performs parallel computing on the multiple subtasks when performing the trial running on the multiple subtasks.

[0010] In a specific implementation, before splitting the big data computing task of the user into multiple subtasks, the big data processing system further includes: receiving the big data computing task uploaded by the user.

[0011] In a second aspect, the embodiments of the present application provide a data skew detection device applied to a big data processing system, which includes:

[0012] The computing module is configured to split the big data computing task of the user into multiple subtasks and perform trial running on the multiple subtasks.

[0013] The obtaining module is configured to obtain the runtime of each subtask in the multiple subtasks.

[0014] The processing module is configured to sort the multiple running times from large to small to obtain a sorted list, wherein the multiple running times are running times corresponding to the multiple subtasks, and each subtask corresponds to a running time.

[0015] The difference between the first running time and the second running time is obtained, and when the difference is greater than a threshold, it is determined that the first subtask corresponding to the first running time and the subtasks arranged before the first subtask in the sorted list are subtasks with data skew, wherein the first running time is greater than the second running time.

[0016] In a specific implementation, the first running time is any one of the running times in the sorted list.

[0017] In a specific implementation, the apparatus further includes a communication module configured to send prompt information to a user terminal of a user, the prompt information being used to inform the user that the first subtask and the subtasks arranged before the first subtask in the sorted list are subtasks with data skew.

[0018] In a specific implementation, the big data processing system uses parallel computing for the multiple subtasks when the multiple subtasks are being tested.

[0019] In a specific implementation, the communication module is further configured to receive a big data computing task uploaded by the user terminal.

[0020] In a third aspect, an embodiment of the present application provides a computing device, including a processor and a memory, the memory being configured to store instructions, and the processor being configured to run the instructions, when the processor runs the instructions, running the method in the first aspect or any specific implementation manner of the first aspect.

[0021] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing instructions, when the instructions are run on a computing device, running the method in the first aspect or any specific implementation manner of the first aspect. The implementation manners provided in the above aspects of the present application can be further combined to provide more implementation manners.

[0022] The implementation manners provided in the above aspects of the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a schematic diagram of a cloud big data processing system provided by an embodiment of the present application;

[0025] Figure 2 is a structural schematic diagram of a big data processing system provided by an embodiment of the present application;

[0026] Figure 3 is an interaction schematic diagram of a data skew detection method provided by an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of a data skew detection device provided by an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] Mass data analysis is a main application scenario in the field of big data, and big data processing systems such as Hadoop and Spark can fully utilize the computing and storage capabilities of clusters to complete the analysis and processing of mass data.

[0030] Based on different deployment modes, big data processing systems can be divided into local big data processing systems and cloud big data processing systems. The local big data processing system refers to a big data processing system deployed on a user device directly controlled by a user or a device (for example, a local computer cluster) geographically close to the user. Due to the problems of high cost, long cycle and difficult maintenance of the user self-deploying the above-mentioned big data processing system, cloud service providers can provide big data processing cloud services for users, that is, the above-mentioned cloud big data processing system. The cloud big data processing system has at least a part of components deployed in a cloud platform. It should be noted that the cloud platform can be a cloud platform of a central cloud, a cloud platform of an edge cloud, or a cloud platform including a central cloud and an edge cloud, which is not limited in the embodiments of the present application. Moreover, when the cloud platform is a cloud platform including a central cloud and an edge cloud, the big data processing system can be partially deployed in the cloud platform of the edge cloud and partially deployed in the cloud platform of the central cloud.

[0031] As shown in Figure 1 Figure 1 ​is a deployment schematic diagram of cloud-based big data processing provided by an embodiment of the present application. The big data processing system is deployed in a cloud platform. A large number of underlying resources provided by a cloud service provider are deployed in the cloud platform. For example, computing resources, storage resources, network resources, and the like are deployed in the cloud platform 1, and the computing resources can be a large number of computing devices (for example, servers). The big data processing system can utilize the underlying resources deployed in the cloud platform to implement the data processing and data skew detection methods provided by the embodiments of the present application.

[0032] It should be understood that the big data processing system can be a hardware system or a software system deployed on a hardware device and / or a virtual machine. The big data processing system can also be implemented by a combination of hardware and software. When the big data processing system is a hardware system, it can be a collection of computing devices. When the big data processing system is a software system, the big data processing system can be independently deployed on a server or a virtual machine in the cloud platform, or the big data processing system can be distributedly deployed on multiple servers in the cloud platform, or the big data processing system can be distributedly deployed on multiple virtual machines in the cloud platform, or the big data processing system can be distributedly deployed on servers and virtual machines in the cloud platform.

[0033] As shown in Figure 1 , the big data processing system can be abstracted by the cloud service provider into a data processing cloud service in the cloud platform to provide the user. After the user purchases the data processing cloud service in the cloud platform, the cloud platform can utilize the big data processing system deployed in the cloud platform to provide the data processing cloud service to the user. For example, the cloud platform can provide a graphical user interface to the user who purchases the cloud service, display the graphical user interface on the user terminal of the user, and the user performs data analysis and processing in the graphical user interface. The way to purchase the data processing cloud service can include pre-recharge and settlement according to the actual use of the final resources, or settlement according to the time of using the cloud service or according to the function or resources of the purchased cloud service.

[0034] In the field of big data, a large number of request queries and data aggregation computing tasks occur every day. When processing computing tasks, big data processing systems such as Hadoop usually divide a computing task into multiple subtasks, and then aggregate the execution results of the multiple subtasks to obtain the result of the computing task. As shown in Figure 2 , the big data processing system can be deployed in the cloud platform to provide the data processing cloud service to the user. The user can use the data processing cloud service to perform data analysis and processing in the cloud platform. Figure 2is a schematic diagram of a big data processing system performing a computing task provided by an embodiment of the present application. Taking Hadoop as an example, Hadoop includes components such as Hive, yet another resource negotiator (YARN), and hadoop distributed filesystem (HDFS). Hive is a data warehouse tool based on Hadoop, which can provide a structured query language (SQL) query function and convert SQL statements into MapReduce tasks, that is, after a user submits a computing task to a cloud platform through a user terminal, the big data processing system splits the computing task into multiple subtasks through Hive; YARN is a general resource management system that can provide unified resource management and scheduling for upper-layer applications, and after the resources scheduled by YARN complete each subtask, the execution results of each subtask are stored in HDFS for user query and analysis.

[0035] In the method described above, the running time of the computing task is determined by the slowest subtask. Generally, data skew will cause the execution time of the task to become longer, and if the subtask with data skew can be determined, the running time of the entire computing task can be effectively shortened by optimizing the execution of the slow subtask. Figure 2

[0036] However, the current big data processing system can provide the user with the execution result of the computing task, but the computing result only includes the creation time, execution statement, and running time of the computing task, and when the running time of the computing task is too long, the user needs to optimize the computing task, but the user cannot determine which subtask has data skew and then optimize the subtask with data skew. Therefore, how to determine the subtask with data skew in multiple subtasks is a problem to be solved.

[0037] To solve the above problem, an embodiment of the present application provides a data skew detection method, which is applied to the above local big data processing system or cloud big data processing system. As shown in Figure 3 Figure 3 is an interaction diagram of the data skew detection method provided by an embodiment of the present application, and the data skew detection method includes S301 to S306.

[0038] S301, the user terminal acquires a big data computing task set by the user and sends the big data computing task to the big data processing system.

[0039] ​​The big data processing system can provide a user interface to the user terminal, the user interface including an edit window and an execution control, the user being able to set a big data computing task in the user interface displayed by the user terminal, for example, inputting an SQL statement in the edit window. After the user sets the big data computing task, the user triggers the big data computing task by triggering (for example, clicking with a mouse) the execution control, and the user terminal sends the big data computing task to the big data processing system in response to the triggering operation of the user.

[0040] S302, the big data processing system receives the big data computing task and splits the big data computing task into a plurality of subtasks.

[0041] After receiving the big data computing task, the big data processing system splits the big data computing task into a plurality of subtasks in order to make full use of the cluster computing capability of the big data processing system, so that the big data processing system schedules different resources to process each subtask and reduces the processing time. For example, as described above, if the big data processing system is Hadoop, the Hive component in Hadoop converts the SQL statement into a MapReduce task. Figure 2

[0042] S303, the big data processing system runs the plurality of subtasks and obtains a running time corresponding to each subtask.

[0043] The big data processing system schedules different resources to run the plurality of subtasks in parallel, and after running each subtask, the big data processing system records a running result corresponding to each subtask, the running result including a task identifier of the subtask, an execution statement, and a running time of running the subtask.

[0044] S304, the big data processing system sorts the plurality of running times from large to small to obtain a sorted list.

[0045] After the big data processing system runs the plurality of subtasks and obtains a running time corresponding to each subtask, the big data processing system sorts the plurality of running times corresponding to the plurality of subtasks from large to small to obtain a sorted list, the sorted list including a task identifier of each subtask and a running time of each subtask. For example, the computing task includes five subtasks in total, and the running time of each subtask is 100 milliseconds (ms), 95 ms, 101 ms, 94 ms, and 97 ms. After sorting the five subtasks from long to short according to the running time, the order of the running time in the sorted list is 101 ms, 100 ms, 97 ms, 95 ms, and 94 ms.

[0046] ​S305, the big data processing system acquires a difference between the first running time and the second running time adjacent in the sorting list, and when the difference is greater than a threshold, determines that the first subtask corresponding to the first running time and the subtasks arranged before the first subtask in the sorting list are subtasks with data skew.

[0047] The first running time is greater than the second running time, and the first running time is any one running time in the sorting list except the last one. After the big data processing system sorts the running time of the above subtasks from large to small, when it determines that there are subtasks with data skew, it only needs to acquire the first running time and the second running time adjacent in the sorting list according to the sorting of the running time in the sorting list, and then calculate the difference between the first running time and the second running time. If the difference between the first running time and the second running time is less than or equal to the threshold, it is determined that the subtask corresponding to the first running time does not have data skew. If the difference between the first running time and the second running time is greater than the threshold, it is determined that the first subtask represented by the first task identifier is a subtask with data skew, and the subtasks represented by the task identifiers arranged before the first task identifier in the sorting list are also subtasks with data skew, that is, in the computing task to which the first subtask belongs, the subtasks with running time greater than or equal to the first running time are all subtasks with data skew.

[0048] The big data processing system sequentially takes each running time in the sorting list as the first running time, and takes the running time adjacent to the first running time in the sorting list and less than the first running time as the second running time. According to the above method, it can be determined that the subtasks with data skew in the above plurality of subtasks.

[0049] For example, the computing task includes five sub-tasks in total, and the running time of each sub-task is 100 ms, 95 ms, 101 ms, 94 ms and 97 ms respectively, the threshold is 2 ms, and after the five sub-tasks are sorted in descending order of running time, the order of the running time in the sorted list is 101 ms, 100 ms, 97 ms, 95 ms and 94 ms. The computing device first takes 101 ms as the first running time and 100 ms as the second running time, and since the difference between the first running time (101 ms) and the second running time (100 ms) is 1 ms, which is less than the threshold 2 ms, the sub-task corresponding to 101 ms is not a sub-task with data skew. After determining that the sub-task corresponding to the first running time (i.e. 101 ms) does not have data skew, 100 ms is taken as the first running time and 97 ms is taken as the second running time, and since the difference between the first running time (100 ms) and the second running time (97 ms) is 3 ms, which is greater than the threshold 2 ms, it is determined that the sub-task corresponding to 100 ms is a sub-task with data skew, and it is determined that the sub-task corresponding to the running time 101 ms is a sub-task with data skew. After determining that the sub-task corresponding to the first running time (100 ms) has data skew, 97 ms is taken as the first running time and 95 ms is taken as the second running time, and it is determined whether the sub-task corresponding to the first running time (i.e. 97 ms) has data skew, and so on, until the running time corresponding to the last two sub-tasks in the sorted list is taken as the first running time and the second running time respectively.

[0050] It should be understood that the big data processing system can also sort a plurality of running times corresponding to a plurality of sub-tasks in ascending order, and if the plurality of running times are sorted in ascending order, in S305 described above, when the difference between the first running time and the second running time is greater than the threshold, it is determined that the first sub-task corresponding to the first running time and the sub-tasks arranged after the first sub-task in the sorted list are sub-tasks with data skew.

[0051] S306, the big data processing system sends prompt information to the user terminal.

[0052] The prompt information is used to inform the user that the first subtask and the subtask arranged before the first subtask in the ranking list are subtasks with data skew, so that the user optimizes and adjusts the subtasks to reduce the execution time of the computing task and improve the efficiency of big data processing. The prompt information includes the task identifier of the subtask with data skew, the corresponding SQL statement, and the running time. It should be understood that the big data processing system can automatically send the prompt information to the user terminal to remind the user after determining the subtask with data skew, or send the prompt information to the user terminal when the user queries the trial running result of the computing task. For example, the user interface further includes a running result control. The user queries the running result of the computing task by triggering the running result control. In response to the triggering operation of the user, the user terminal sends a query request to the big data processing system. The big data processing system sends the corresponding execution statement, running time, and detection result of each subtask to the user terminal. The detection result indicates whether the corresponding subtask is a subtask with data skew. The detection result can be considered as the prompt information. The user terminal displays the running result on the user interface after receiving the running result, and highlights the subtask with data skew indicated by the detection result, such as displaying it in a font different from other colors.

[0053] In a possible implementation, after obtaining the plurality of task running times corresponding to the plurality of subtasks, the big data processing system can further not sort the plurality of task running times. After obtaining the running times corresponding to the plurality of subtasks, the big data processing system obtains a first running time and a second running time. The first running time is greater than the second running time, and the second running time is the largest one of the running times smaller than the first running time. For example, the computing task includes five subtasks in total, and the running times of the subtasks are 100 ms, 95 ms, 101 ms, 93 ms, and 97 ms, respectively. If the first running time is 100 ms, the second running time is 97 ms. Then, the difference between the first running time and the second running time is calculated. If the difference between the first running time and the second running time is less than or equal to a threshold value, it is determined that the subtask corresponding to the first running time does not have data skew. If the difference between the first running time and the second running time is greater than the threshold value, it is determined that the subtask corresponding to the first running time is a subtask with data skew, and the subtasks with running times greater than or equal to the first running time are all subtasks with data skew.

[0054] For example, the total computing task includes five subtasks, and the runtime of each subtask is 100 ms, 95 ms, 101 ms, 93 ms, and 97 ms, respectively, and the threshold is 2 ms. When the first runtime obtained by the big data processing system is 101 ms, the second runtime is 100 ms, the difference between the first runtime and the second runtime is 1 ms, which is less than the threshold 2 ms, and the big data processing system determines that the subtask corresponding to 101 ms is not a subtask with data skew. If the first runtime obtained by the big data processing system is 100 ms, the second runtime is 97 ms, and the difference between the first runtime and the second runtime is 3 ms, which is greater than the threshold 2 ms, and the big data processing system determines that the subtask corresponding to 100 ms has data skew, and the subtask corresponding to 101 ms has data skew.

[0055] The big data processing system traverses the runtimes corresponding to all subtasks in turn according to the above method until all runtimes corresponding to all subtasks are traversed, and determines the subtasks with data skew in all subtasks. That is, the big data processing system obtains the runtime corresponding to each subtask as the first runtime in turn, then selects the maximum runtime in the runtimes less than the first runtime as the second runtime, if the difference between the first runtime and the second runtime is less than or equal to the threshold, the big data processing system determines that the subtask corresponding to the first runtime and the subtask corresponding to the second runtime do not have data skew. If the difference between the first runtime and the second runtime is greater than the threshold, it is determined that the first subtask corresponding to the first runtime is a subtask with data skew, and in the computing task to which the first subtask belongs, the subtasks with runtimes greater than or equal to the first runtime are all subtasks with data skew. Until all runtimes corresponding to all subtasks are taken as the first runtime, the subtasks with data skew in all subtasks can be determined.

[0056] It should be understood that after the big data processing system determines that a subtask is a subtask with data skew, the runtime corresponding to the subtask is no longer taken as the first runtime to perform the above operation, regardless of whether the runtime corresponding to the subtask is taken as the first runtime. For example, if the computing device takes 100 ms as the first runtime for the first time, the second runtime is 97 ms, and the difference between the first runtime and the second runtime is 3 ms, which is greater than the threshold 2 ms, the computing device determines that the subtask corresponding to 100 ms has data skew, and the subtask corresponding to 101 ms has data skew. Then, when the computing device traverses the runtimes corresponding to the subtasks, 101 ms is no longer taken as the first runtime.

[0057] By acquiring the running time of each subtask in the computing task and sorting, the subtask with data skew is determined according to the difference of the running time of two subtasks. When the difference between the larger running time and the smaller running time of the two adjacent running time in the sorting is greater than a threshold, it is indicated that the running time of the two subtasks is greatly different, and the subtask with larger running time is the subtask with data skew, and the subtask with running time greater than or equal to the subtask is also the subtask with data skew. Through the above method, the subtask with data skew in the plurality of subtasks can be determined, and when the running time of the computing task is too long and the user needs to optimize the subtask, the big data processing system can provide the subtask with data skew to the user, so that the user optimizes the subtask.

[0058] It should be noted that, for the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, and those skilled in the art should also know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the present application.

[0059] Those skilled in the art can think of other reasonable step combinations according to the above description, which also belong to the protection scope of the present application. Secondly, those skilled in the art should be familiar with the fact that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the present application.

[0060] The above describes the data method provided by the embodiments of the present application, and the following will be combined with Figures 1 to 3 The data skew detection device and the computing device provided by the embodiments of the present application are described. Figure 4 With Figure 5 , the data skew detection device and the computing device provided by the embodiments of the present application are described.

[0061] Figure 4 is a structural schematic diagram of a data skew detection device provided by the embodiments of the present application, which is used in Figure 1 the big data processing system shown in the figure, the data skew detection device 400 includes a communication module 410, a computing module 420, an acquisition module 430 and a processing module 440, wherein

[0062] The communication module 410 is used for receiving the big data computing task uploaded by the user through the user terminal.

[0063] The computing module 420 is used for splitting the above-mentioned big data computing task into a plurality of subtasks; and running the plurality of subtasks. Specifically, the method for the computing module 420 to perform the above operation can refer to the related operations in S302 and S303.

[0064] The acquisition module 430 is configured to acquire a running time length corresponding to each of the plurality of subtasks.

[0065] The processing module 440 is configured to sort the plurality of running time lengths in descending order to obtain a sorted list.

[0066] The processing module 440 is configured to acquire a difference between adjacent first and second running time lengths in the sorted list, and determine that a first subtask corresponding to the first running time length and a subtask arranged before the first subtask in the sorted list are subtasks with data skew when the difference is greater than a threshold, where the first running time length is greater than the second running time length. The method for determining the subtasks with data skew by the processing module 440 can refer to the related descriptions in S304 and S305.

[0067] The communication module 410 is further configured to send prompt information to a user terminal, where the prompt information is used to inform the user that the first subtask and the subtask arranged before the first subtask in the sorted list are subtasks with data skew, so that the user optimizes and adjusts the subtasks to reduce the execution time length of the computing task and improve the big data processing efficiency.

[0068] It should be understood that the data skew detection apparatus 400 of the embodiments of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. When the data skew detection method shown in the above method embodiments is implemented by software, the data skew detection apparatus 400 and each module thereof can also be software modules. Figure 3

[0069] Specifically, the operation of the data skew detection apparatus 400 for implementing data skew detection can refer to the related operations of the big data processing system for implementing data skew detection in the above method embodiments, and will not be described in detail here.

[0070] Since each module in the big data processing system provided by the embodiments of the present application can be distributedly deployed on a plurality of computers in the same environment or different environments, the embodiments of the present application further provide a computing device, which can be a computer system including one or more computers 500 in some cases.​Figure 5 As shown in FIG. 5, each computer 500 includes a memory 501, a processor 502, a communication interface 503, and a bus 504. The memory 501, the processor 502, and the communication interface 503 are communicatively connected to each other through the bus 504.

[0071] The memory 501 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 501 can store a program, and when the program stored in the memory 501 is executed by the processor 502, the processor 502 and the communication interface 503 are configured to execute the database design method provided in the embodiments of the present application. The memory 501 can also include software modules and data required by other running processes such as an operating system. The operating system can be LINUX TM , UNIX TM , WINDOWS TM , etc.

[0072] The processor 502 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0073] The processor 502 can also be an integrated circuit chip having a processing capability of signals. In the implementation process, part or all of the functions of the database design system of the present application can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 502. The processor 502 described above can also be a general processor, a digital signal processor (Digital Signal Processing, DSP), an application specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware code processor execution, or executed by a combination of hardware and software modules in the code processor. The software module can be located in a random memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 501, and the processor 502 reads the information in the memory 501, and combines the hardware to complete part of the function of the database design system of the present application.

[0074] The communication interface 503 uses a transceiver module such as but not limited to a transceiver to realize the communication between the computer 500 and other devices or communication networks.

[0075] The bus 504 can include a path for transmitting information between various components (for example, the memory 501, the processor 502, the communication interface 503) of the computer 500.

[0076] Each computer 500 is connected to each other through a communication network. Each computer 500 is used for part of the function of the database design system. Any computer 500 can be a computer (for example: server) in the cloud platform, or a computer in the edge data center.

[0077] The description of the flow corresponding to each of the above figures has its own emphasis, and the part not described in detail in a certain flow can refer to the related description of other flows.

[0078] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of computer program product in whole or in part. The computer program product for providing database design cloud service includes one or more computer instructions, which, when loaded and executed on a computer, generates the database design system according to the embodiments of the present application in whole or in part. Figure 3The method flow or function.

[0079] The embodiment of the present application further provides a non-transient computer storage medium, which stores instructions, and when the instructions are run on a processor, the method steps in the above method embodiment can be realized, and the processor of the computer storage medium can refer to the specific operation of the above method embodiment for the specific implementation of the above method steps, and details are not described herein.

[0080] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0081] The above embodiments can be realized all or partially by software, hardware, firmware or other any combination. When realized by software, the above embodiments can be realized all or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the above flow or function described in the embodiments of the present application is all or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like containing one or more available medium sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium or a semiconductor medium, and the semiconductor medium can be a solid state disk.

[0082] The above is only a specific implementation of the present application. Those skilled in the art can think of changes or replacements based on the specific implementation provided by the present application, which should be covered within the protection scope of the present application.

Claims

1. A data tilt detection method, characterized in that, The method, applied to big data processing systems, includes: Break down the user's big data computing tasks into multiple sub-tasks; Run the multiple subtasks in a trial run and obtain the runtime of each subtask. Sort the multiple runtimes from largest to smallest to obtain a sorted list; Obtain the difference between adjacent first runtime and second runtime in the sorted list. When the difference is greater than a threshold, determine that the first subtask corresponding to the first runtime and the subtask arranged before the first subtask in the sorted list are subtasks with data skew, wherein the first runtime is greater than the second runtime.

2. The method according to claim 1, characterized in that, The first runtime is any runtime in the sorted list.

3. The method according to claim 1 or 2, characterized in that, The method further includes: A prompt message is sent to the user's terminal, the prompt message being used to notify the user that the first subtask and the subtasks arranged before the first subtask in the sorting list are subtasks with data skew.

4. The method according to any one of claims 1 to 3, characterized in that, When the big data processing system is running the multiple sub-tasks in trial operation, it performs parallel computation on the multiple sub-tasks.

5. The method according to claim 3, characterized in that, Before breaking down the user's big data computing task into multiple sub-tasks, the method further includes: Receive big data computing tasks uploaded by the user terminal.

6. A data tilt detection device, characterized in that, The device, used in a big data processing system, includes: The computing module is used to break down the user's big data computing task into multiple sub-tasks and to test run the multiple sub-tasks. The acquisition module is used to acquire the runtime of each subtask among the plurality of subtasks; The processing module is used to sort multiple runtimes from largest to smallest to obtain a sorted list; Obtain the difference between adjacent first runtime and second runtime in the sorted list. When the difference is greater than a threshold, determine that the first subtask corresponding to the first runtime and the subtask arranged before the first subtask in the sorted list are subtasks with data skew, wherein the first runtime is greater than the second runtime.

7. The apparatus according to claim 6, characterized in that, The first runtime is any runtime in the sorted list.

8. The apparatus according to claim 6 or 7, characterized in that, The device further includes: The communication module is used to send a prompt message to the user's user terminal. The prompt message is used to notify the user that the first subtask and the subtasks arranged before the first subtask in the sorting list are subtasks with data skew.

9. The apparatus according to any one of claims 6 to 8, characterized in that, When the big data processing system is running the multiple sub-tasks in trial operation, it performs parallel computation on the multiple sub-tasks.

10. The apparatus according to claim 8, characterized in that, The communication module is also used to receive big data computing tasks uploaded by the user terminal.

11. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store instructions, and the processor being used to execute the instructions, wherein when the processor executes the instructions, it performs the method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images