A method and device for adaptive deployment based on big data tasks and electronic equipment

Abstract

The application discloses a self-adaptive deployment method and device based on a big data task and electronic equipment, the method comprises the following steps: starting a first task in a task set, and allocating device resources to the first task according to a resource allocation strategy; when it is determined that the first task completely normally operates, running the first task, and starting a second task in the task set, and then allocating device resources to the second task according to the resource allocation strategy; when it is determined that the device resources are allocated completely, if the first task partially normally operates, closing the part of the first task that cannot normally operate, and generating a corresponding alarm report. Based on the above method, the device resources can be fully utilized through automatic operation and maintenance under limited device resources, and the problem of waste of device resources in the current big data task deployment process can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a self-adaptive deployment method and device based on a big data task and electronic equipment. BACKGROUND

[0002] The deployment of a big data task is a complex process. At present, the implementation and deployment of a big data task is generally as follows: first, the data magnitude of a user is obtained, such as 100w / s of TPS (Transaction Processing Systems); then, the performance parameters of device resources, such as device computing capacity, are obtained through testing, so as to provide a deployment scheme of the big data task for the user. For example, three devices with XXX memory and XXX CPU (Central Processing Unit) configuration are required, which meet the deployment performance requirements of the big data task of the user.

[0003] However, the above method has the problem of waste of device resources in actual operation. For example, if it is obtained through testing that 256g of memory is allocated to the big data task in the environment of three devices with 128g of memory (3*128=384g of memory), and the highest data amount that can be calculated is 500w / s, then the resources of the big data task are configured according to this standard. However, if the data amount of the user does not reach 500w / s or the cluster performance of the user is higher, in other words, in this case, the big data task does not need to be allocated so many resources, which causes the waste of device resources. SUMMARY

[0004] The present application provides a self-adaptive deployment method and device based on a big data task and electronic equipment, which realize the full use of device resources through automatic operation and maintenance under limited device resources, so as to solve the problem of waste of device resources in the current deployment process of a big data task.

[0005] In a first aspect, the present application provides a method for monitoring network asset changes, which comprises the following steps:

[0006] starting a first task in a task set and allocating device resources to the first task according to a resource allocation strategy, wherein the resource allocation strategy represents a strategy of allocating the device resources to tasks in the task set according to a preset proportion;

[0007] if the first task runs completely normally, running the first task and starting a second task in the task set, and allocating the device resources to the second task according to the resource allocation strategy;

[0008] If the first task part runs normally after the device resources are allocated, the part of the first task that does not run normally is closed, and a corresponding alarm report is generated.

[0009] Through the above method, the device resources are fully utilized through automatic operation and maintenance under limited device resources.

[0010] In a possible design, before a first task in the start task set is started and device resources are allocated to the first task according to a resource allocation strategy, the method further includes the following steps.

[0011] A plurality of big data tasks carrying priority identifiers are obtained, wherein each big data task includes a plurality of computing tasks carrying sub-priority identifiers, and the priority identifier represents the order of task starting.

[0012] The big data tasks are sorted according to the priority identifiers to obtain a first sorting result.

[0013] The computing tasks in each big data task are sorted according to the sub-priority identifiers to obtain a second sorting result.

[0014] A task set sorted according to priorities is obtained according to the first sorting result and the second sorting result.

[0015] Through the above method, the device resources are fully utilized through automatic operation and maintenance under limited device resources.

[0016] In a possible design, the step of allocating device resources to the first task according to a resource allocation strategy includes the following steps.

[0017] A preset proportion of device resources is allocated to the first task, wherein the first task includes a plurality of computing tasks.

[0018] If the currently allocated device resources are insufficient to support the first task to run normally, and the device resources are not allocated completely, the device resources are continuously allocated to the first task.

[0019] If the currently allocated device resources are sufficient to support the first task to run normally, the resource allocation to the first task is completed.

[0020] If the device resources are allocated completely, and part of the computing tasks in the first task run normally, the resource allocation to the part of the computing tasks in the first task is completed.

[0021] Through the above method, the device resources are fully utilized through automatic operation and maintenance under limited device resources.

[0022] In a possible design, after the first task is completely normally operated, the first task is run, and a second task in the task set is started, and device resources are allocated to the second task according to the resource allocation strategy, and the method further includes:

[0023] determining whether the device resources currently allocated to the second task are sufficient to support the second task to be completely normally operated;

[0024] if yes, the second task is run, and a next task of the second task is started, and device resources are allocated to the next task of the second task according to the resource allocation strategy;

[0025] if no, a part of the second task that cannot be normally operated is closed, and a corresponding alarm report is generated.

[0026] By using the method, device resources are fully utilized in automatic operation and maintenance.

[0027] In a possible design, after the device resources are allocated, if the first task is partially normally operated, the method further includes:

[0028] if the first task cannot be normally operated, the first task is closed, and a corresponding alarm report is generated.

[0029] By using the method, the process of automatic operation and maintenance is implemented.

[0030] In a second aspect, the present application provides a device for adaptive deployment based on a big data task, and the device includes:

[0031] a distribution module configured to start a first task in a task set, and allocate device resources to the first task according to a resource allocation strategy, wherein the resource allocation strategy represents a strategy of allocating the device resources to tasks in the task set according to a preset proportion;

[0032] a running module configured to, if the first task is completely normally operated, run the first task, and start a second task in the task set, and allocate the device resources to the second task according to the resource allocation strategy;

[0033] a generation module configured to, after the device resources are allocated, if the first task is partially normally operated, close a part of the first task that cannot be normally operated, and generate a corresponding alarm report.

[0034] In a possible design, before the distribution module, the apparatus further includes an acquisition module, configured to acquire a plurality of big data tasks carrying priority identifiers, where each big data task includes a plurality of computing tasks carrying sub-priority identifiers, and the priority identifier represents the order of task starting; sort the big data tasks according to the priority identifiers to obtain a first sorting result; sort the computing tasks in each big data task according to the sub-priority identifiers to obtain a second sorting result; and obtain a task set sorted according to priorities according to the first sorting result and the second sorting result.

[0035] In a possible design, the distribution module is specifically configured to distribute a preset proportion of device resources to the first task, where the first task includes a plurality of computing tasks; if the currently distributed device resources are insufficient to support normal running of the first task, and the device resources are not fully distributed, continue to distribute the device resources to the first task; if the currently distributed device resources are sufficient to support normal running of the first task, complete resource distribution to the first task; and if the device resources are fully distributed, and part of the computing tasks in the first task are running normally, complete resource distribution to the part of the computing tasks in the first task.

[0036] In a possible design, after the running module, the apparatus further includes a judgment module, configured to judge whether the currently distributed device resources are sufficient to support complete normal running of the second task; if yes, run the second task, start a next task of the second task, and distribute device resources to the next task of the second task according to the resource distribution strategy; and if no, close part of the second task that cannot run normally, and generate a corresponding alarm report.

[0037] In a possible design, the generation module is specifically configured to, if the first task cannot run normally after the device resources are fully distributed, close the first task, and generate a corresponding alarm report.

[0038] In a third aspect, the present application provides an electronic device, which includes:

[0039] a memory configured to store a computer program;

[0040] a processor configured to execute the computer program stored in the memory, so as to implement the method steps of detecting an object with an abnormal motion state.

[0041] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method steps of detecting an object with an abnormal motion state.

[0042] The above-mentioned aspects of the second to fourth aspects and the technical effects that can be achieved by the aspects will be described in detail with reference to the above-mentioned technical effects that can be achieved by the first aspect or the various possible schemes in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a self-adaptive deployment method based on a big data task provided in the present application is shown in the figure;

[0044] Figure 2 A schematic diagram of a device for self-adaptive deployment based on a big data task provided in the present application is shown in the figure;

[0045] Figure 3 A schematic diagram of the structure of an electronic device provided in the present application is shown in the figure. DETAILED DESCRIPTION

[0046] In order to facilitate those skilled in the art to better understand the present application, first, the terms involved in the present application are explained as follows:

[0047] 1、TPS: understood as the number of transactions processed by the server per second 100w / s can be understood as, the server needs to process 1 million data per second.

[0048] 2、Distributed task: a program that needs multiple servers to cooperate to complete, generally applied to big data use scenarios.

[0049] 3、Allocation of device resources: generally, when a big data task is running, all server resources will be managed first, each server has 100g of memory, so three servers have 300g of memory, when deploying a big data task, if 120g of memory is needed to run, it is not necessary to care about which machine the 120g of memory comes from, this part will be automatically managed by the big data component. In the embodiment of the present application, only the obtained device resources, such as memory resources, CPU resources, etc. need to be concerned.

[0050] 4、Back pressure: in the process of big data stream processing, data flows through each computing node like water flow, if the processing speed of a certain node is slower than that of other nodes upstream, it will cause congestion, just like a water pipe is blocked, then the bottleneck of the processing capacity of the upstream computing node depends on the processing capacity of the downstream node, if the speed of the upstream data processing node is greater than that of the downstream, that is, the downstream node limits the performance of the upstream node, this situation is usually called back pressure.

[0051] The embodiment of the present application provides a self-adaptive deployment method, device and electronic equipment based on a big data task, which solves the problem of waste of device resources in the deployment process of a big data task.

[0052] According to the method provided in the embodiments of the present application, different from the method of the prior art which needs a technician to evaluate project scale data in advance, then determine the cluster scale or equipment resources, and then complete project deployment, thereby causing waste of equipment resources. The present application first determines a certain number of machines, i.e., determines fixed equipment resources, and in this limited execution environment, through dynamic adjustment of execution task automation, the equipment resources are fully utilized, thereby solving the problem of waste of equipment resources in the process of deployment of big data tasks.

[0053] The method provided by the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0054] Referring to Figure 1 The embodiments of the present application provide a self-adaptive deployment method based on big data tasks, and the specific process is as follows:

[0055] Step 101: Obtain a task set sorted according to priority;

[0056] In the embodiments of the present application, the task set includes multiple big data tasks sorted according to priority, and multiple computing tasks sorted according to priority in each big data task.

[0057] Specifically, the user can add a priority identifier to the big data task according to his own needs, and the priority identifier is mainly used to define the order of task execution, and then the priority identifier is sorted, i.e., the big data task with high importance is executed first.

[0058] In addition, each big data task can be further subdivided into computing tasks, and the user can add a sub-priority identifier to each computing task in each big data task according to his own needs. Then, according to the sub-priority identifier, the computing tasks in each big data task are sorted, i.e., for the same big data task, the computing task with high importance is executed first.

[0059] For example, for the big data task of "traffic analysis system", it is further subdivided into two computing tasks of two computing dimensions: A1 dimension is to count the access volume of a network address per hour; and A2 dimension is to count the top ten network addresses in terms of access times in an hour.

[0060] As a user, if more attention is paid to the calculation task of the A2 dimension, the priority of the calculation task of the A2 dimension is set to be higher than that of other dimensions; if more attention is paid to the calculation task of the A1 dimension, the priority of the calculation task of the A1 dimension is set to be higher than that of other dimensions. Although the calculation tasks of the two dimensions are both calculated in the same big data task, when the device resources are insufficient to support the simultaneous running of the two calculation tasks, the calculation task of the lower-priority dimension can be selected to be closed to release the device resources to ensure the normal running of the calculation task of the higher-priority dimension.

[0061] It is worth noting that the above-mentioned manner of adding priority identifiers is only one possible case of obtaining a priority-ordered task set, and other cases are not specifically described herein.

[0062] Step 102: Start the first task in the task set, and allocate device resources to the first task according to the resource allocation strategy;

[0063] After obtaining the task set and the fixed device resources, the first task with the highest priority in the task set is started first, and all the calculation tasks in the first task are started at the same time. Then, according to the resource allocation strategy of allocating device resources from low to high, device resources are allocated to the first task.

[0064] For example, the priority ordering of the task set is obtained, and it is assumed that two 16g memory servers are used as a cluster, i.e., the fixed allocable device resources are 32g memory. When there are two tasks in the task set, the priority of the first task A is set to be higher than that of the second task B, and A includes two-dimension calculation tasks, the priority of the first calculation task A1 in A is set to be higher than that of the second calculation task A2, B includes two-dimension calculation tasks, and the priority of the first calculation task B1 in B is set to be higher than that of the second calculation task B2. That is, the relationship between the tasks is: A(A1>A2)>B(B1>B2).

[0065] Based on the above assumption, A is started first, and A1 and A2 are started by default. According to the preset proportion, lower device resources are allocated to the tasks started this time by default, for example, 10g memory is allocated, and at the same time, the running state of the above-mentioned tasks is continuously monitored within a preset time period.

[0066] Step 103: Determine whether the first task is completely normally running;

[0067] After starting and monitoring the first task, the running state of the first task is monitored according to the preset time, and it is determined whether the first task is completely normally running:

[0068] If yes, step 104 is performed; if no, step 105 is performed.

[0069] Specifically, if the upstream and downstream task processing speeds are not obviously inconsistent within a preset time period, it is determined that the task is completely normally running; if the upstream and downstream task processing speeds are obviously inconsistent within the preset time, it is determined that the task is not completely normally running.

[0070] It is worth noting that whether the upstream and downstream task processing speeds are obviously consistent can be defined according to specific businesses, for example, back pressure, database query speed is too slow, and the like are obviously inconsistent conditions, and other determination methods are not specifically described herein.

[0071] Step 104: running the first task, and starting a second task in the task set, and allocating the device resources to the second task according to the resource allocation strategy;

[0072] When the first task is completely normally running, the first task and all computing tasks under the first task are run, and a second task with a priority only lower than the first task in the task set is started, and the second task is taken as the currently started task. Then, the device resources are allocated to the second task according to the resource allocation strategy of allocating the device resources from low to high.

[0073] For example, based on the assumption in the step 102 example, after determining that A is completely normally running, B is started, and B is allocated 10g of memory by default, and the state of B is monitored, and the subsequent execution content is the same as that after A is started, and is not specifically described herein.

[0074] Step 105: determining whether the device resources are allocated completely;

[0075] Specifically, based on the assumption in the step 102 example, there are 32g of memory in total, if 32g of memory has been allocated to A, then the device resources are allocated completely.

[0076] According to the determination rule, whether the fixed device resources are allocated completely is determined:

[0077] If yes, step 106 is executed;

[0078] If no, the device resources are continuously allocated to the first task, for example, based on the assumption in the step 102 example, 2g of memory is continuously allocated to the first task, and then step 103 is executed.

[0079] It is worth noting that in the embodiment of the present application, the specific manner of continuously allocating the device resources to the first task can be restarting the first task after being closed, and allocating more device resources to the first task than the last time, for example, 10g of memory is allocated to the first task last time, and 12g of memory is allocated to the first task this time.

[0080] Step 106: Close the part of the first task that cannot run normally, and generate a corresponding alarm report.

[0081] If the currently available device resources are all allocated to the first task, but the first task still cannot run completely normally, then according to the priority order of the plurality of computing tasks in the first task, the computing tasks in the first task are closed in order of priority from low to high, until the first task after closing part of the computing tasks can run normally, and a corresponding alarm report is generated.

[0082] Specifically, based on the assumption in the step 102 example, if 32g of memory is allocated to A for running, but A still cannot run completely normally, then A2 is closed, and whether A containing only A1 can run normally is monitored: if it can run normally, then A1 in A is run, that is, the computing task of A1 dimension is supported, and a corresponding report is generated; if it cannot run normally, then A1 is closed, that is, A will not start, and a corresponding alarm report is generated.

[0083] It should be noted that in the above embodiments, the first task is taken as an example, and the starting and running modes of other tasks in the task set are the same as those of the first task.

[0084] In this way, the starting of the big data tasks in the task set in order of priority from high to low, the allocation of device resources to the current big data task in the manner of allocation of device resources from low to high, and a plurality of situations are described in detail.

[0085] 1. When the current big data task can run completely normally, the current task is run, and the next big data task is started in order of priority from high to low;

[0086] 2. When the current big data task cannot run completely normally, the current resource allocation strategy is adjusted, and then monitoring is continued until the current big data task can run normally or the current device resources are allocated completely;

[0087] 3. When the current device resources are allocated completely and the current big data task cannot run completely normally, in the current big data task, the computing tasks are closed in order of priority from low to high, and after each time a computing task is closed, it is determined whether the current big data task runs normally, until it is determined that the current big data task can run normally, the closing operation is stopped, the current big data task is started, and a corresponding alarm report is generated.

[0088] Through the method provided by the embodiment of the application, the resources of the server are fully utilized through automatic operation and maintenance under limited servers. Different from the method of the prior art, which needs a technician to evaluate the project scale and data volume in advance, and then determines the cluster scale, i.e., the device resources, to complete the project deployment, the application first determines a certain number of machines, i.e., fixed device resources, and under the limited execution environment, the computing power resources of the server are fully utilized through automatic dynamic adjustment of the execution tasks, so as to solve the problem of waste of device resources in the process of deployment of a big data task.

[0089] Based on the same inventive concept, the application further provides an apparatus for adaptive deployment of a big data task, to solve the problem of waste of device resources in the process of deployment of a big data task by fully utilizing device resources through automatic operation and maintenance under limited device resources, as shown in Figure 2 The apparatus comprises:

[0090] The allocation module 201 starts a first task in the task set and allocates device resources to the first task according to a resource allocation strategy, wherein the resource allocation strategy represents a strategy of allocating the device resources to the tasks in the task set according to a preset proportion.

[0091] The running module 202 runs the first task and starts a second task in the task set if the first task is completely normally run, and allocates the device resources to the second task according to the resource allocation strategy.

[0092] The generation module 203 closes the part of the first task that cannot normally run and generates a corresponding alarm report if the first task is partially normally run after the device resources are allocated.

[0093] In a possible design, before the allocation module 201, the apparatus is further configured to acquire a plurality of big data tasks carrying priority identifiers, wherein each big data task comprises a plurality of computing tasks carrying sub-priority identifiers, the priority identifier represents the order of task starting, sort the big data tasks according to the priority identifier to obtain a first sorting result, sort the computing tasks in each big data task according to the sub-priority identifier to obtain a second sorting result, and obtain a task set sorted according to the priority according to the first sorting result and the second sorting result.

[0094] In one possible design, the allocation module 201 is specifically used to allocate a preset proportion of device resources to the first task, wherein the first task includes multiple computing tasks; if the currently allocated device resources are insufficient to support the normal operation of the first task, and the device resources have not been fully allocated, then the device resources are continuously allocated to the first task; if the currently allocated device resources are sufficient to support the normal operation of the first task, then the resource allocation for the first task is completed; if the currently allocated device resources are fully allocated, and some computing tasks in the first task are operating normally, then the resource allocation for some computing tasks in the first task is completed.

[0095] In one possible design, after the running module 202, it is further configured to determine whether the device resources currently allocated to the second task are sufficient to support the second task to run completely normally; if so, the second task is run and the next task of the second task is started, and device resources are allocated to the next task of the second task according to the resource allocation strategy; if not, the part of the second task that cannot run normally is shut down and a corresponding alarm report is generated.

[0096] In one possible design, the generation module 203 is specifically used to shut down the first task and generate a corresponding alarm report if the first task cannot run normally after the device resources have been allocated.

[0097] Based on the above-mentioned device, it is possible to make full use of equipment resources through automatic operation and maintenance under limited equipment resources, thereby solving the problem of equipment resource waste in the current big data task deployment process.

[0098] Based on the same inventive concept, this application also provides an electronic device that can realize the function of the aforementioned adaptive deployment device based on big data tasks. (Refer to...) Figure 3 The electronic device includes:

[0099] At least one processor 301 and a memory 302 connected to at least one processor 301. In this embodiment, the specific connection medium between the processor 301 and the memory 302 is not limited. Figure 3 The example shown is the connection between processor 301 and memory 302 via bus 300. Bus 300 is... Figure 3 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Bus 300 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 3 The term 301 is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 301 can also be called a controller; there is no restriction on the name.

[0100] In the embodiments of the present application, the memory 302 stores instructions executable by the at least one processor 301, and the at least one processor 301 can execute the adaptive deployment method of the big data task based on the instructions stored in the memory 302. The processor 301 can implement the functions of various modules of the apparatus shown in the embodiments of the present application. Figure 3

[0101] The processor 301 is the control center of the apparatus, and can connect various parts of the control device through various interfaces and lines, and perform overall monitoring of the apparatus by running or executing the instructions stored in the memory 302 and calling the data stored in the memory 302, thereby processing various functions and data of the apparatus.

[0102] In a possible design, the processor 301 can include one or more processing units, and the processor 301 can integrate an application processor and a modem processor, where the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the modem processor can also not be integrated into the processor 301. In some embodiments, the processor 301 and the memory 302 can be implemented on the same chip, and in some embodiments, they can also be implemented on separate chips respectively.

[0103] The processor 301 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the adaptive deployment method of the big data task based on the embodiments of the present application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0104] ​The memory 302, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 302 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 302 is any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 302 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used to store program instructions and / or data.

[0105] By designing and programming the processor 301, the code corresponding to the adaptive deployment method based on a big data task introduced in the foregoing embodiments can be fixed into the chip, so that the chip can execute the adaptive deployment method based on a big data task at runtime. Figure 1 The steps of the adaptive deployment method based on a big data task of the embodiments shown are known to those skilled in the art as how to design and program the processor 301, which will not be described here.

[0106] Based on the same inventive concept, the embodiments of the present application also provide a storage medium storing computer instructions, when the computer instructions run on a computer, the computer instructions make the computer execute the adaptive deployment method based on a big data task discussed above.

[0107] In some possible implementations, various aspects of the adaptive deployment method based on a big data task provided by the present application can also be implemented in the form of a program product, which includes program code, when the program product runs on a device, the program code is used to make the control device execute the steps in the adaptive deployment method based on a big data task according to various exemplary embodiments of the present application described above in the specification.

[0108] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one

[0109] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0110] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0111] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0112] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the apparatus and methods disclosed herein, equivalents and substitutions thereof could be made by one of ordinary skill in the art without departing from the spirit and scope of the disclosure. Any and all such modifications and variations are intended to be included herein within the scope of the present application and expressed as a part thereof.

Claims

1. A method for adaptive deployment based on big data tasks, characterized in that, The method comprises: obtaining a plurality of big data tasks carrying priority identifiers, wherein each big data task comprises a plurality of computing tasks carrying sub-priority identifiers, and the priority identifier represents the order of task starting; sorting the big data tasks according to the priority identifiers to obtain a first sorting result; sorting the computing tasks in each big data task according to the sub-priority identifiers to obtain a second sorting result; obtaining a task set sorted according to priorities according to the first sorting result and the second sorting result; starting a first task in the task set and allocating device resources to the first task according to a resource allocation strategy, wherein the resource allocation strategy represents a strategy of allocating the device resources to tasks in the task set according to a preset proportion; if the first task completely normally operates, running the first task and starting a second task in the task set, and allocating the device resources to the second task according to the resource allocation strategy; if the first task partially normally operates after the device resources are allocated, shutting down a part of the first task that cannot normally operate and generating a corresponding alarm report.

2. The method of claim 1, wherein, The allocation of the device resources to the first task according to the resource allocation strategy comprises: allocating a preset proportion of the device resources to the first task, wherein the first task comprises a plurality of computing tasks; if the currently allocated device resources are insufficient to support the first task to normally operate and the device resources are not allocated completely, continuing to allocate the device resources to the first task; if the currently allocated device resources are sufficient to support the first task to normally operate, completing the resource allocation to the first task; if the device resources are allocated completely and the first task cannot completely normally operate, shutting down the computing tasks with the lowest priorities in the first task one by one according to the second sorting result until the first task normally operates, and completing the resource allocation to the first task.

3. The method of claim 1, wherein, After the allocation of the device resources to the second task according to the resource allocation strategy, the method further comprises: judging whether the currently allocated device resources to the second task are sufficient to support the second task to completely normally operate; if yes, running the second task and starting a next task of the second task, and allocating device resources to the next task of the second task according to the resource allocation strategy; if no, shutting down the computing tasks with the lowest priorities in the second task one by one according to the second sorting result until the second task completely normally operates or the computing tasks in the second task are all shut down, and generating a corresponding alarm report.

4. The method of claim 1, wherein, After the shutting down of the part of the first task that cannot normally operate and the generation of the corresponding alarm report, the method further comprises: If the first task cannot run normally after the device resources are allocated, the first task is closed, and a corresponding alarm report is generated.

5. An apparatus for adaptive deployment based on big data tasks, the apparatus comprising: The device comprises: The allocation module acquires a plurality of big data tasks carrying priority identifiers, wherein each big data task comprises a plurality of computing tasks carrying sub-priority identifiers, the priority identifier represents the order of task starting, the big data tasks are sorted according to the priority identifier to obtain a first sorting result, the computing tasks in each big data task are sorted according to the sub-priority identifier to obtain a second sorting result, a task set sorted according to priority is obtained according to the first sorting result and the second sorting result, a first task in the task set is started, and device resources are allocated to the first task according to a resource allocation strategy, wherein the resource allocation strategy represents a strategy of allocating the device resources to the tasks in the task set according to a preset proportion. The running module runs the first task if the first task runs completely normally, starts a second task in the task set, and allocates the device resources to the second task according to the resource allocation strategy. The generation module closes the part of the first task that cannot run normally if the first task partially runs normally after the device resources are allocated, and generates a corresponding alarm report.

6. The apparatus of claim 5, wherein, The allocation module is specifically configured to allocate a preset proportion of device resources to the first task, wherein the first task comprises a plurality of computing tasks, if the currently allocated device resources are insufficient to support the first task to run normally, and the device resources are not allocated completely, the device resources are continuously allocated to the first task, if the currently allocated device resources are sufficient to support the first task to run normally, the resource allocation to the first task is completed, and if the device resources are allocated completely, and the first task cannot run completely normally, the computing task with the lowest priority in the first task is closed one by one according to the second sorting result until the first task runs normally, and the resource allocation to the first task is completed.

7. An electronic device, comprising: The device comprises: The memory is used to store computer programs. The processor is used to execute the computer programs stored in the memory to realize the method in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is executed by the processor to realize the method in any one of claims 1-4.

Images