Task arrangement method and device for big data platform
By obtaining the running time and memory of the big data platform tasks, and automatic task arrangement is carried out based on the preset coordinate system, the low performance problems caused by manual arrangement are solved, and resource utilization and computing performance are improved.
Patent Information
- Application Number
- CN202411105362.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, multiple computing tasks need to be manually arranged, resulting in low performance of big data platforms.
By obtaining the running time and running memory of multiple target tasks, the task arrangement is arranged based on the preset coordinate system, and the arrangement scheme with the smallest resource occupancy is obtained to achieve automatic arrangement of target tasks.
It improves the resource utilization and performance of the big data platform, realizes automatic arrangement of target tasks, and avoids low performance caused by manual intervention.
Smart Images

Figure CN119003172A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of big data technology, and in particular to a task scheduling method and device for a big data platform. Background Art
[0002] A big data platform is usually a computing environment that integrates various tools, technologies, and infrastructure to process, store, and analyze large-scale data sets. The use of a big data platform usually involves the operation of multiple computing tasks. In order to realize the resource utilization of the big data platform, multiple computing tasks need to be arranged.
[0003] However, the arrangement of multiple computing tasks cannot be completed automatically at present. As the number of computing tasks increases, multiple computing tasks need to be arranged manually, resulting in low performance of the big data platform. Summary of the invention
[0004] An embodiment of the present application provides a task scheduling method for a big data platform to solve the technical problem in the prior art that multiple computing tasks need to be manually scheduled, resulting in low performance of the big data platform; an embodiment of the present application also provides a task scheduling device for a big data platform.
[0005] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:
[0006] In a first aspect, a task arrangement method for a big data platform is provided, comprising:
[0007] Get the running time and running memory of multiple target tasks;
[0008] Based on the running time and the running memory, the plurality of target tasks are arranged in a preset coordinate system to obtain a plurality of arrangement schemes; the preset coordinate system is configured to characterize the running resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of the plurality of target tasks in the preset coordinate system;
[0009] Based on the starting coordinates, the running time and the running memory, obtaining the occupied resources of each of the arrangement schemes;
[0010] The target tasks are arranged based on the arrangement scheme that occupies the least resources.
[0011] In combination with the first aspect, based on the running time and the running memory, the plurality of target tasks are arranged in a preset coordinate system to obtain a plurality of arrangement schemes, including:
[0012] Determine a first target task and a second target task among the plurality of target tasks; the first target task has a first running time and a first running memory, and the second target task has a second running time and a second running memory;
[0013] Determine a first starting coordinate of any of the target tasks;
[0014] Based on the first starting coordinates, the first running time, the first running memory, the second running time, and the second running memory, obtaining the second starting coordinates of the second target task;
[0015] Determine the second target task as the new first target task, and determine the new second target task from the remaining multiple target tasks, and return to the step of determining the first starting coordinate of any one of the target tasks until the second starting coordinate of the last target task is obtained;
[0016] A plurality of the first starting coordinates and a plurality of the second starting coordinates are jointly determined as the arrangement scheme.
[0017] In combination with the first aspect, based on the first starting coordinates, the first running time, the first running memory, the second running time, and the second running memory, obtaining the second starting coordinates of the second target task includes:
[0018] Obtaining the relative position of the second target task with respect to the first target task;
[0019] Based on the relative position, the first starting coordinates, the first running time, the first running memory, the second running time and the second running memory, the second starting coordinates of the second target task are acquired.
[0020] In combination with the first aspect, the first starting coordinate and the second starting coordinate are obtained based on the following model:
[0021]
[0022]
[0023] Wherein, x1 and y1 are the first starting coordinates, x2 and y2 are the second starting coordinates, H1 is the first running time, R1 is the first running memory, H2 is the second running time, R2 is the second running memory, i is the target task, R i For the running memory.
[0024] In combination with the first aspect, based on the running time and the running memory, the plurality of target tasks are arranged in a preset coordinate system to obtain a plurality of arrangement schemes, further comprising:
[0025] Traversing the starting coordinates;
[0026] In response to the plurality of target tasks being independent of each other, the arrangement scheme is obtained.
[0027] In combination with the first aspect, the starting coordinates are traversed based on the following model:
[0028]
[0029] Among them, x i 、x j ,y i and j is the starting coordinate, H i and H j is the running time, R i and R j For the running memory.
[0030] In combination with the first aspect, the running time and the running memory are obtained based on the following steps:
[0031] Obtaining multiple historical running durations and multiple historical running memories of the target task within a preset historical period;
[0032] Determine an average value of a plurality of the historical running times as the running time;
[0033] An average value of the plurality of historical running memories is determined as the running memory.
[0034] In combination with the first aspect, before arranging the target tasks based on the arrangement scheme that occupies the least resources, the method further includes:
[0035] Obtain the minimum operating resources of the big data platform,
[0036] In response to the occupied resources being greater than or equal to the minimum operating resources, the arrangement scheme with the minimum occupied resources is acquired.
[0037] In combination with the first aspect, the minimum operating resource is obtained based on the following formula:
[0038]
[0039] Among them, R min is the minimum operating resource, i is the target task, H i is the running time, R i For the running memory.
[0040] In a second aspect, a task arrangement device for a big data platform is provided, comprising:
[0041] A first acquisition module is configured to acquire the running time and running memory of multiple target tasks;
[0042] A second acquisition module is configured to arrange the plurality of target tasks in a preset coordinate system based on the running time and the running memory to obtain a plurality of arrangement schemes; the preset coordinate system is configured to represent the running resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of the plurality of target tasks in the preset coordinate system;
[0043] A third acquisition module is configured to acquire occupied resources of each of the arrangement schemes based on the starting coordinates, the running time and the running memory;
[0044] The arrangement module is configured to arrange the target tasks based on the arrangement scheme that occupies the least resources.
[0045] One of the above technical solutions has the following advantages or beneficial effects:
[0046] Compared with the prior art, the task arrangement method for a big data platform provided by the present application includes: obtaining the running time and running memory of multiple target tasks; arranging the multiple target tasks in a preset coordinate system based on the running time and running memory, and obtaining multiple arrangement schemes; the preset coordinate system is configured to characterize the running resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of the multiple target tasks in the preset coordinate system; based on the starting coordinates, running time and running memory, the occupied resources of each arrangement scheme are obtained; and the target tasks are arranged based on the arrangement scheme that occupies the least resources. In this way, the running resources of the big data platform are characterized by the preset coordinate system, and then arranged in the preset coordinate system based on the running time and running memory of each target task, so as to realize the automatic arrangement of the target tasks and improve the resource utilization and performance of the big data platform.
[0047] The task scheduling device for a big data platform provided in the present application has all the technical features and beneficial effects of the above-mentioned task scheduling method for a big data platform, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.
[0049] Figure 1 A flowchart of a task arrangement method for a big data platform provided in an embodiment of the present application;
[0050] Figure 2 A schematic diagram of the structure of a task scheduling device for a big data platform provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the present application, it is necessary to understand that the orientation or position relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present application. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0052] The specific implementation methods of the present application are described below by way of examples:
[0053] See also Figure 1 , Figure 1 The flowchart of the task arrangement method for a big data platform provided by the embodiment of the present application is illustrated. The present application provides a task arrangement method for a big data platform, including the following steps 101 to 104.
[0054] Step 101, obtain the running time and running memory of multiple target tasks. Specifically, the target task of the present application refers to the computing task to be run in the big data platform, and each target task has its own running time and running memory. In an embodiment of the present application, each target task can be run separately in an environment with sufficient resources to evaluate the running time and running memory of each target task. At this time, sufficient resources refer to an environment with sufficient running memory, and the target task does not need to run at a speed limit. For example, the entire big data platform only runs one target task to obtain the running time and running memory of the target task.
[0055] In addition, in order to save the operating resources of the big data platform, in one embodiment, the running time and the running memory can also be obtained based on the following steps: obtaining multiple historical running times and multiple historical running memories of the target task within a preset historical period; determining the average value of the multiple historical running times as the running time; determining the average value of the multiple historical running memories as the running memory. In this way, by obtaining multiple historical running times and multiple historical running memories of the target task within a preset historical period, and calculating the average value as the running time and running memory, it helps to avoid over-allocation of resources and save the operating resources of the big data platform. Secondly, in the big data platform environment, the nature and data volume of the target task may change. By continuously collecting historical data to update the average value of the running time and running memory, dynamic resource adjustment can be achieved, so that the big data platform has better scalability and adaptability, and can cope with changing needs and workloads.
[0056] Step 102, based on the running time and the running memory, multiple target tasks are arranged in a preset coordinate system to obtain multiple arrangement schemes; the preset coordinate system is configured to represent the running resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of multiple target tasks in the preset coordinate system. Specifically, the running resources of the big data platform include time resources and memory resources. This application represents the running resources of the big data platform as a first rectangle in the preset coordinate system. The length of the first rectangle is located on the horizontal axis of the preset coordinate system, representing the time resources; the width of the first rectangle is located on the vertical axis of the preset coordinate system, representing the memory resources. Similarly, the target task is also represented as a second rectangle with the running time as the length and the running memory as the width. The arrangement scheme of this application is the arrangement scheme of multiple second rectangles in the first rectangle. It can be understood that when the area of the first rectangle is the smallest and the total area of the multiple second rectangles accounts for the largest proportion of the area of the first rectangle, the resource utilization rate of the big data platform is the highest, and the arrangement scheme at this time is the optimal arrangement scheme. Under the premise that the length and width of each second rectangle have been determined, the arrangement scheme only needs to give the starting coordinates of each second rectangle. In this application, the starting coordinates refer to the coordinates of the second rectangle closest to the origin of the preset coordinate system, with (x i ,y i ). In the embodiment of the present application, the first rectangle is located in the first quadrant of the preset coordinate system. The duration resource of the big data platform is 24 hours, that is, 86400 seconds. The upper limit of the memory resource is the sum of multiple running memories. Therefore, the mathematical model of the big data platform can be expressed as:
[0057]
[0058] Where x is the horizontal coordinate of the first rectangle, y is the vertical coordinate of the first rectangle, i is the target task, R iis the running memory, and N is the number of target tasks.
[0059] In one embodiment, step 102 arranges multiple target tasks in a preset coordinate system based on the running time and the running memory to obtain multiple arrangement schemes, including the following steps 201 to 205.
[0060] Step 201, determine the first target task and the second target task among multiple target tasks; the first target task has a first running time and a first running memory, and the second target task has a second running time and a second running memory; step 202, determine the first starting coordinates of any target task; step 203, based on the first starting coordinates, the first running time, the first running memory, the second running time and the second running memory, obtain the second starting coordinates of the second target task; step 204, determine the second target task as a new first target task, and determine a new second target task from the remaining multiple target tasks, return to step 202 to determine the first starting coordinates of any target task, until the second starting coordinates of the last target task are obtained; step 205, multiple first starting coordinates and multiple second starting coordinates are jointly determined as an arrangement scheme. Specifically, by determining the first target task and the second target task, the second starting coordinates of the second target task can be determined according to the first starting coordinates of the first target task. It should be noted that the first target task and the second target task here are only the order of obtaining the starting coordinates, that is, obtaining the second starting coordinates according to the first starting coordinates does not mean that the first target task will be arranged first and then the second target task in the final arrangement scheme. In this way, by gradually updating the first target task and the second target task, the first starting coordinate and the second starting coordinate are continuously obtained until all target tasks are updated, so as to obtain the starting coordinates corresponding to all target tasks and realize the arrangement of multiple target tasks.
[0061] In one embodiment, step 203, based on the first starting coordinates, the first running time, the first running memory, the second running time, and the second running memory, obtains the second starting coordinates of the second target task, including: obtaining the relative position of the second target task and the first target task; based on the relative position, the first starting coordinates, the first running time, the first running memory, the second running time, and the second running memory, obtains the second starting coordinates of the second target task. In the embodiment of the present application, the relative position includes the second target task being located on the upper side of the first target task, or the second target task being located on the lower side of the first target task, or the second target task being located on the left side of the first target task, or the second target task being located on the right side of the first target task.
[0062] Specifically, after obtaining the first starting coordinates through model (1) and model (2), if the second target task is located above the first target task, the second starting coordinates are obtained through the following model (6) and model (7):
[0063]
[0064] If the second target task is located below the first target task, the second starting coordinates are obtained by the following model (8) and model (9):
[0065]
[0066] If the second target task is located on the left side of the first target task, the second starting coordinates are obtained by the following model (10) and model (11):
[0067]
[0068] If the second target task is located on the right side of the first target task, the second starting coordinates are obtained by the following model (12) and model (13):
[0069]
[0070] In models (6) to (13), x1 and y1 are the first starting coordinates, x2 and y2 are the second starting coordinates, R1 is the first running memory, H2 is the second running time, R2 is the second running memory, i is the target task, R i For running memory.
[0071] In one embodiment, the first starting coordinate and the second starting coordinate are obtained based on the following models (1) to (4):
[0072]
[0073]
[0074] Among them, x1 and y1 are the first starting coordinates, x2 and y2 are the second starting coordinates, H1 is the first running time, R1 is the first running memory, H2 is the second running time, R2 is the second running memory, i is the target task, R i For running memory.
[0075] In one embodiment, step 102, based on the running time and the running memory, arranges multiple target tasks in a preset coordinate system to obtain multiple arrangement schemes, and also includes: traversing the starting coordinates; in response to the multiple target tasks being independent of each other, obtaining an arrangement scheme. It is understandable that in the process of arranging the target tasks, there may be a second rectangular intersection, that is, the problem of cross-utilization of resources on the big data platform, and the intersecting target tasks will interfere with each other. Therefore, it is necessary to spread the starting coordinates and verify whether the multiple target tasks are independent of each other based on the running time and the running memory. If there is a second rectangular intersection, rearrange it. If the multiple target tasks are independent of each other, obtain the corresponding arrangement scheme and proceed to the subsequent steps.
[0076] In one embodiment, the starting coordinates are traversed based on the following model (5):
[0077]
[0078] Among them, x i 、x j ,y i and j is the starting coordinate, H i and H j is the running time, R i and R j For running memory.
[0079] Step 103, based on the starting coordinates, running time and running memory, obtain the occupied resources of each arrangement scheme. Specifically, the occupied resources of each arrangement scheme are R 总 =y max , that is, the sum of the ordinates of multiple first rectangles in the preset coordinate system. If the ordinates of two second rectangles partially overlap, the overlapping part only needs to be calculated once. In this way, the application presents the occupied resources in a visual way through the second rectangles, which can more intuitively show the resource usage of the big data platform.
[0080] In one embodiment, before arranging the target tasks based on the arrangement scheme with the smallest occupied resources in step 104, the task arrangement method for the big data platform also includes: obtaining the minimum operating resources of the big data platform, and in response to the occupied resources being greater than or equal to the minimum operating resources, obtaining the arrangement scheme with the smallest occupied resources. Specifically, since the minimum operating resources are the operating memory of all target tasks occupying the entire available space of the first rectangle, but in fact there will inevitably be gaps between the second rectangles, therefore, the occupied resources finally calculated should be greater than or equal to the minimum operating resources.
[0081] In one embodiment, the minimum operating resources are obtained based on the following formula (14):
[0082]
[0083] Among them, R min is the minimum operating resource, i is the target task, H i is the running time, R i For running memory.
[0084] Step 104, arrange the target tasks based on the arrangement scheme that occupies the least resources. Specifically, by comparing the occupied resources of all arrangement schemes, the scheme that occupies the least resources is selected as the optimal arrangement scheme and the target tasks are arranged. At this time, the resources of the big data platform required to run all the target tasks are the least, and the resource utilization rate of the big data platform is the highest. And through the task arrangement method of this application, when the target tasks increase, the arrangement of the target tasks can be automatically realized without periodic rearrangement, so as to improve the resource utilization efficiency of the big data platform and improve the computing performance of the big data platform.
[0085] The task arrangement method for a big data platform provided in the present application includes: obtaining the running time and running memory of multiple target tasks; arranging the multiple target tasks in a preset coordinate system based on the running time and running memory, and obtaining multiple arrangement schemes; the preset coordinate system is configured to characterize the running resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of the multiple target tasks in the preset coordinate system; based on the starting coordinates, the running time and the running memory, the occupied resources of each arrangement scheme are obtained; and the target tasks are arranged based on the arrangement scheme with the smallest occupied resources. In this way, the running resources of the big data platform are characterized by the preset coordinate system, and then arranged in the preset coordinate system based on the running time and running memory of each target task, so as to realize the automatic arrangement of the target tasks and improve the resource utilization and performance of the big data platform.
[0086] Accordingly, see Figure 2 , Figure 2 The structural schematic diagram of the task arrangement device of the big data platform provided by the embodiment of the present application is illustrated. The embodiment of the present application provides a task arrangement device for a big data platform, including a first acquisition module, a second acquisition module, a third acquisition module and an arrangement module, wherein: the first acquisition module is configured to acquire the running time and running memory of multiple target tasks; the second acquisition module is configured to arrange the multiple target tasks in a preset coordinate system based on the running time and running memory, and acquire multiple arrangement schemes; the preset coordinate system is configured to characterize the running resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of the multiple target tasks in the preset coordinate system; the third acquisition module is configured to acquire the occupied resources of each arrangement scheme based on the starting coordinates, running time and running memory; the arrangement module is configured to arrange the target tasks based on the arrangement scheme that occupies the least resources.
[0087] The task scheduling device for a big data platform provided in the present application has all the technical features and beneficial effects of the above-mentioned task scheduling method for a big data platform, which will not be repeated here.
[0088] The above is a detailed introduction to the task scheduling method and device for a big data platform provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the technical solution and its core idea of the present application. Ordinary technical personnel in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A task arrangement method for a big data platform, characterized in that: include: Get the running time and running memory of multiple target tasks; Arrange the plurality of target tasks in a preset coordinate system based on the running time and the running memory to obtain a plurality of arrangement schemes; The preset coordinate system is configured to represent the operating resources of the big data platform; the arrangement scheme is configured to include the starting coordinates of the plurality of target tasks in the preset coordinate system; Based on the starting coordinates, the running time and the running memory, obtaining the occupied resources of each of the arrangement schemes; The target tasks are arranged based on the arrangement scheme that occupies the least resources.
2. The task arrangement method for a big data platform according to claim 1, characterized in that: Arrange the plurality of target tasks in a preset coordinate system based on the running time and the running memory to obtain a plurality of arrangement schemes, including: Determine a first target task and a second target task among the plurality of target tasks; the first target task has a first running time and a first running memory, and the second target task has a second running time and a second running memory; Determine a first starting coordinate of any of the target tasks; Based on the first starting coordinates, the first running time, the first running memory, the second running time, and the second running memory, obtaining the second starting coordinates of the second target task; Determine the second target task as the new first target task, and determine the new second target task from the remaining multiple target tasks, and return to the step of determining the first starting coordinate of any one of the target tasks until the second starting coordinate of the last target task is obtained; A plurality of the first starting coordinates and a plurality of the second starting coordinates are jointly determined as the arrangement scheme.
3. The task arrangement method for a big data platform according to claim 2, characterized in that: Acquiring a second starting coordinate of the second target task based on the first starting coordinate, the first running time, the first running memory, the second running time, and the second running memory includes: Obtaining the relative position of the second target task with respect to the first target task; Based on the relative position, the first starting coordinates, the first running time, the first running memory, the second running time and the second running memory, the second starting coordinates of the second target task are acquired.
4. The task arrangement method for a big data platform according to claim 2, characterized in that: The first starting coordinate and the second starting coordinate are obtained based on the following model: Wherein, x1 and y1 are the first starting coordinates, x2 and y2 are the second starting coordinates, H1 is the first running time, R1 is the first running memory, H2 is the second running time, R2 is the second running memory, i is the target task, R i For the running memory.
5. The task arrangement method for a big data platform according to claim 2, characterized in that: Arranging the plurality of target tasks in a preset coordinate system based on the running time and the running memory to obtain a plurality of arrangement schemes further includes: Traversing the starting coordinates; In response to the plurality of target tasks being independent of each other, the arrangement scheme is obtained.
6. The task arrangement method for a big data platform according to claim 5, characterized in that: The starting coordinates are traversed based on the following model: Among them, x i 、x j ,y i and j is the starting coordinate, H i and H j is the running time, R i and R j For the running memory.
7. The task arrangement method for a big data platform according to claim 1, characterized in that: The running time and the running memory are obtained based on the following steps: Obtaining multiple historical running durations and multiple historical running memories of the target task within a preset historical period; Determine an average value of a plurality of the historical running times as the running time; An average value of the plurality of historical running memories is determined as the running memory.
8. The task arrangement method for a big data platform according to claim 1, characterized in that: Before arranging the target tasks based on the arrangement scheme that occupies the least resources, the method further includes: Obtain the minimum operating resources of the big data platform, In response to the occupied resources being greater than or equal to the minimum operating resources, the arrangement scheme with the minimum occupied resources is acquired.
9. The task arrangement method for a big data platform according to claim 8, characterized in that: The minimum operating resources are obtained based on the following formula: Among them, R min is the minimum operating resource, i is the target task, H i is the running time, R i For the running memory.
10. A task arrangement device for a big data platform, characterized in that: include: A first acquisition module is configured to acquire the running time and running memory of multiple target tasks; A second acquisition module is configured to arrange the plurality of target tasks in a preset coordinate system based on the running time and the running memory to obtain a plurality of arrangement schemes; The preset coordinate system is configured to represent the operating resources of the big data platform; The arrangement scheme is configured to include a plurality of starting coordinates of the target tasks in the preset coordinate system; A third acquisition module is configured to acquire occupied resources of each of the arrangement schemes based on the starting coordinates, the running time and the running memory; The arrangement module is configured to arrange the target tasks based on the arrangement scheme that occupies the least resources.
