Task processing method, electronic equipment, storage medium and computer program product

Abstract

The embodiment of the invention provides a task processing method, electronic equipment, a storage medium and a computer program product, and relates to the technical field of computers. Comprising the steps that a target host and a target auxiliary node on the target host are determined from the auxiliary nodes on the basis of a deployment mode of the auxiliary nodes on the at least one host, and the deployment mode is determined according to the number of tasks to be processed; and distributing a to-be-processed task to the target auxiliary node, so that the target auxiliary node executes the to-be-processed task. According to the technical scheme, the deployment mode of the auxiliary nodes can be flexibly determined according to the number of the to-be-processed tasks, so that system resources can be reasonably utilized, resource waste can be avoided while the service requirements of task processing are met, and the task processing cost is saved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a task processing method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] Multi-concurrent application systems have been increasingly applied in enterprises, for example, a big data processing system is a multi-concurrent application system. With the development of Internet technology, data volume presents exponential growth, and big data processing becomes an important means for many enterprises to mine potential value of massive data, thereby improving enterprise operation ability and reducing enterprise operation cost. In related technologies, a big data task scheduling system is usually used as a processing tool for big data processing, for example, all tasks can be distributed to a Hadoop cluster, and the Hadoop cluster is relied on for big data processing.

[0003] The deployment mode of nodes in the multi-concurrent application system in related technologies is usually fixed in multiple hosts, which is easy to cause waste of resources (for example, hardware resources). SUMMARY

[0004] The purpose of the embodiments of the present disclosure is to provide a task processing method, an electronic device, a storage medium and a computer program product.

[0005] To solve the above technical problems, the embodiments of the present disclosure are implemented through the following aspects.

[0006] According to a first aspect of the embodiments of the present disclosure, a task processing method is provided, applied to a master node, the master node being connected to multiple slave nodes on at least one host, one host corresponding to at least one slave node, the method comprising: determining a target host and a target slave node on the target host from the multiple slave nodes based on a deployment mode of the multiple slave nodes on the at least one host, the deployment mode being determined according to a number of tasks to be processed; and assigning a task to be processed to the target slave node, so that the target slave node executes the task to be processed.

[0007] According to a second aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and wherein the processor is configured to execute the steps of the task processing method of the first aspect.

[0008] According to a third aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the computer readable storage medium storing one or more programs, the one or more programs, when executed by an electronic device comprising multiple application programs, causing the electronic device to execute the steps of the task processing method of the first aspect.

[0009] According to a fourth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises a computer program, and the computer program, when executed by a processor, implements the steps of the method for task processing of the first aspect.

[0010] One of the above technical solutions has the following advantages or beneficial effects: based on the deployment mode of the plurality of secondary nodes on the at least one host, a target host and a target secondary node on the target host are determined from the plurality of secondary nodes; and a to-be-processed task is allocated to the target secondary node, so that the target secondary node executes the to-be-processed task. The deployment mode of the plurality of secondary nodes can be flexibly determined according to the number of to-be-processed tasks, so that system resources can be reasonably utilized, the business demand for task processing can be met, and resource waste can be avoided, thereby saving task processing cost.

[0011] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

[0012] Other features and advantages of the present disclosure will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 A flowchart of a method for task processing provided by the embodiments of the present disclosure is shown;

[0015] Figure 2 A schematic diagram of a deployment mode provided by the embodiments of the present disclosure is shown;

[0016] Figure 3 Another flowchart of a method for task processing provided by the embodiments of the present disclosure is shown;

[0017] Figure 4 Another flowchart of a method for task processing provided by the embodiments of the present disclosure is shown;

[0018] Figure 5 Another flowchart of a method for task processing provided by the embodiments of the present disclosure is shown;

[0019] Figure 6 Another flowchart of a method for task processing provided by the embodiments of the present disclosure is shown;

[0020] Figure 7 Another flowchart showing the method for task processing provided by the embodiments of the present disclosure is shown.

[0021] Figure 8 Another flowchart showing the method for task processing provided by the embodiments of the present disclosure is shown.

[0022] Figure 9 A system block diagram of the task processing system provided by the embodiments of the present disclosure is shown.

[0023] Figure 10 A hardware structure schematic diagram of the electronic device for performing the method for task processing provided by the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] In order to enable persons skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by persons skilled in the art without creative labor should fall within the protection scope of the present disclosure.

[0025] First, the terms involved in the present application are described.

[0026] Host: configured with a processor and a corresponding operating system, capable of processing corresponding tasks through one or more nodes running, one or more nodes on the same host can run in parallel, and one or more nodes can directly interact with information and interact with the nodes running on other hosts.

[0027] Node: also known as process, is the basic unit of resource allocation of the system, and is an instance of the running program. One process can run one or more threads, i.e. thread is an independent unit that can be executed within a process. According to the role of the node in task processing, it can be divided into master node and auxiliary node.

[0028] Deployment mode: the deployment mode can include the number of hosts required by the node when deployed on the host, and the specific distribution of multiple nodes on at least one host. In the inventive concept of the present disclosure, for different task processing needs, the deployment mode can be different.

[0029] Figure 1 A flowchart showing the method for task processing provided by the embodiments of the present disclosure is shown, applied to a master node, the master node is connected to multiple auxiliary nodes on at least one host, and one host corresponds to at least one auxiliary node, such as Figure 1As shown, the method can include the following steps:

[0030] In step S101, a target host and a target secondary node on the target host are determined from the plurality of secondary nodes based on a deployment mode of the plurality of secondary nodes on at least one host.

[0031] The deployment mode is determined according to the number of tasks to be processed.

[0032] In some embodiments, the deployment mode can specifically include a single-host multi-node mode, a multi-host mode, or a hybrid mode.

[0033] Figure 2 A schematic diagram of the deployment mode provided by the embodiments of the present disclosure is shown as follows. Figure 2 As shown in (a), in the case where the deployment mode is the single-host multi-node mode, the master node and at least one secondary node are deployed on the same host (host A). As shown in (b), in the case where the deployment mode is the multi-host mode, the master node and each secondary node can be respectively deployed on different hosts. It can be understood that in the multi-host mode, part of the secondary nodes in the plurality of secondary nodes can also be flexibly deployed on the same host according to needs, for example, secondary node 1 and secondary node 2 are deployed on host B, and secondary node 3 and secondary node 4 are deployed on host C, which is not limited by the present disclosure. As shown in (c), in the case where the deployment mode is the hybrid mode, the master node can be deployed on the same host (host A) as part of the secondary nodes, and some secondary nodes for processing critical tasks can be deployed on an independent host (host B), i.e., the master node and the secondary node for processing critical tasks can be deployed on different hosts. Figure 2 Figure 2

[0034] With the above flexible deployment mode, different task processing needs can be met. For example, enterprise A has a small number of tasks to process, and can select the single-host multi-node mode, so as to save the occupation of hardware resources as much as possible. Enterprise B has a large number of tasks to process, and can select the multi-host mode for deployment, so as to improve the performance of task processing. Enterprise C has a small number of tasks to process, but has individual critical task processing, and can select the hybrid mode to deploy the critical task processing on an independent host, so as to save the occupation of hardware resources as much as possible while guaranteeing the reliability and processing performance of the critical task processing.

[0035] In some embodiments, the target secondary node is a secondary node with the smallest node load determined by the master node according to node load; or the target secondary node is a specified secondary node corresponding to the task to be processed.

[0036] ​​In some possible implementation manners, in a case where the to-be-processed task (for example, the to-be-processed task is a user-specified key task) has a pre-specified specified secondary node, the primary node can take the specified secondary node as the target secondary node, and in a case where the to-be-processed task does not have a pre-specified specified secondary node, the primary node can take a secondary node with minimum node load as the target secondary node according to node loads of the plurality of secondary nodes.

[0037] After the primary node determines the target secondary node, the target host where the target secondary node is located can also be determined at the same time, so that the primary node allocates the to-be-processed task to the target secondary node through a connection between the primary node and the target secondary node on the target host.

[0038] In step S102, the to-be-processed task is allocated to the target secondary node, so that the target secondary node executes the to-be-processed task.

[0039] After the primary node determines the target host and the target secondary node on the target host, the primary node can allocate the to-be-processed task to the target secondary node through a connection between the primary node and the target secondary node on the target host, so that the target secondary node executes the to-be-processed task.

[0040] With the technical solution described above, the deployment mode of the plurality of secondary nodes can be flexibly determined according to the number of to-be-processed tasks, so that system resources can be reasonably utilized, the business demand for task processing can be met, and waste of resources and task processing cost can be saved.

[0041] Figure 3 Another flowchart of a method for task processing provided by an embodiment of the present disclosure is shown in FIG. 3. Figure 3 As shown in FIG. 3, before determining the target host and the target secondary node on the target host from the plurality of secondary nodes based on the deployment mode of the plurality of secondary nodes on the at least one host, the method can further include the following steps.

[0042] In step S103, the plurality of secondary nodes are deployed according to the determined deployment mode of the plurality of secondary nodes.

[0043] In some embodiments, the deployment mode of the plurality of secondary nodes can be determined in any of the following manners.

[0044] Manner 1: determining the deployment mode of the plurality of secondary nodes according to the number of to-be-processed tasks and system resources of the at least one host.

[0045] The system resources of the host can be the number of cores of a CPU of the host.

[0046] For example, the number of cores of the single host CPU is M, when the number of tasks to be processed is less than or equal to a preset multiple of M, it can be determined that the deployment mode is the single host multi-node mode, when the number of tasks to be processed is greater than the preset multiple of M, it can be determined that the deployment mode is the multi-host mode, and preferably, the preset multiple can be 3.

[0047] Mode 2, determining the deployment mode of the plurality of secondary nodes according to the number of tasks to be processed, the system resources of the at least one host and the task configuration of the tasks to be processed.

[0048] The task configuration of the tasks to be processed can include whether there is a specified secondary node corresponding to the task, and in the case that the task configuration of at least one task to be processed includes a specified secondary node corresponding to the task, it can be determined that the deployment mode is the hybrid mode.

[0049] After determining the deployment mode, the host corresponding to each secondary node can be determined, including the specified host corresponding to the specified secondary node and / or the host corresponding to the ordinary secondary node.

[0050] It can be understood that the system resources of the host can also be other possible bottleneck resources, such as memory resources. Those skilled in the art can combine the above embodiments based on different resources to obtain the corresponding conditions for determining the deployment mode, and the present disclosure will not expand on this.

[0051] Figure 4 Another flowchart of the method for processing tasks provided by the embodiments of the present disclosure is shown in FIG. 7. Figure 4 As shown in FIG. 7, the plurality of secondary nodes can be deployed according to the determined deployment mode of the plurality of secondary nodes by the following method, and the master node and the secondary node in the present embodiment exchange information through JSCH (Java Secure Channel, Java Secure Channel).

[0052] In step 100, the deployment mode of the plurality of secondary nodes is read.

[0053] According to different deployment modes, different deployment methods can be adopted, in the case of the single host multi-node mode, steps 101-102 are executed, in the case of the multi-host mode, steps 103-106 are executed, and in the case of the hybrid mode, steps 107-111 are executed.

[0054] In step 101, the IP, port, installation directory and other information of the master node are read for local installation and deployment.

[0055] In step 102, the IP, port, installation directory and other information of each secondary node are read in sequence, and the secondary nodes are installed and deployed locally through Runtime.exec and started.

[0056] In step 103, the IP, port, installation directory and the like of the master node are read for local installation deployment.

[0057] In step 104, the IP, port, installation directory and the like of each secondary node are read in turn.

[0058] In step 105, the required software package is pushed to the host corresponding to the secondary node through the Sftp pipe of JSCH for deployment.

[0059] In step 106, the start script of the secondary node is executed through the SSH pipe of JSCH to start the secondary node.

[0060] In step 107, the IP, port, installation directory and the like of the master node are read for local installation deployment.

[0061] In step 108, the IP, port, installation directory and the like of each secondary node are read in turn.

[0062] In step 109, in the case where the secondary node is in a non-remote mode (the host corresponding to the secondary node is the same as the host corresponding to the master node), local installation deployment is performed through Runtime.exec and the secondary node is started.

[0063] In step 110, in the case where the secondary node is in a remote mode (for example, the secondary node is a designated secondary node, and the host corresponding to the secondary node is different from the host corresponding to the master node), the required software package is pushed to the host corresponding to the secondary node through the Sftp pipe of JSCH for deployment.

[0064] In step 111, the start script of the secondary node is executed through the SSH pipe of JSCH to start the secondary node.

[0065] With the above technical solution, the deployment mode of the plurality of secondary nodes can be determined flexibly according to the number of tasks to be processed in combination with the system resources of the host, or according to the number of tasks to be processed, the system resources of the host and the task configuration, so that the system resources can be utilized reasonably, the business demand for task processing can be met, and the waste of resources can be avoided, thereby saving the cost of task processing.

[0066] In some embodiments, the node load can be determined by at least one of thread occupancy, memory occupancy and CPU occupancy of the secondary node.

[0067] Taking thread occupancy and memory occupancy as examples for determining the node load, the node load can be determined by Formula One as follows.

[0068] W i =α×T i +β×M i(Formula One)

[0069] wherein, W i is the node load of the i-th secondary node, is the thread occupancy of the i-th secondary node, T active is the number of active threads of the i-th secondary node, T max is the maximum number of threads of the i-th secondary node. is the memory occupancy of the i-th secondary node, M use is the used memory of the i-th secondary node, M total is the total memory of the i-th secondary node. α and β are weight coefficients of the thread occupancy and the memory occupancy respectively, α+β=1, and the values of α and β are both in the range of [0, 1].

[0070] In some possible implementation manners, the master node can obtain the heartbeat information reported by the secondary nodes in a timing manner through the connection between the master node and each secondary node, the heartbeat information for example including the node information (such as IP, port, node identifier, total memory, maximum number of threads) and the number of active threads, used memory and the like of the secondary nodes, and the node load corresponding to each secondary node is determined according to the heartbeat information corresponding to each secondary node respectively and Formula One, the secondary node with the minimum node load is taken as the target secondary node, so as to realize the node load balancing among the multiple secondary nodes. The master node can collect the heartbeat information of the secondary nodes by calling the RMI (Remote Method Invocation) of each secondary node.

[0071] By using the technical solution, the node load of the secondary nodes can be determined in multiple ways, and the secondary node with the minimum node load is selected as the target secondary node, which is beneficial to realize the node load balancing among the multiple secondary nodes and further improve the performance of task processing.

[0072] Figure 5 Another flowchart of a method for task processing provided by an embodiment of the present disclosure is shown, the master node is connected with a license server, as shown in Figure 5 The method further includes the following steps.

[0073] In step S104, at least one local license of the task is obtained from the license control table, and the server license corresponding to the local license is obtained from the license server.

[0074] In step S105, the start or stop of the task is controlled according to the local license and the server license.

[0075] In some embodiments, after the master node assigns one or more tasks to the corresponding target secondary nodes, the master node can also synchronize the state of the license from the license server and compare it with the state of the license in the local license control table, so as to flexibly control the start and stop of the tasks according to the local license and the server license corresponding to the tasks.

[0076] In some embodiments, one set of license (including one server license and one local license) can control the start and stop of multiple tasks, for example, the server license 1a and the local license 1b can control the start and stop of tasks 1-3. One task can also be controlled by multiple sets of license, for example, the start and stop of task 1 are controlled by the server license 1a, the local license 1b, the server license 2a and the local license 2b. Among them, 1a and 1b are a set of license, and 2a and 2b are another set of license.

[0077] In some embodiments, the start or stop of the task can be controlled according to the local license and the server license in the following ways.

[0078] Method 1: In the case where the number of local licenses is one and the state of the local license is different from that of the server license, the state of the local license is updated according to the state of the server license, and the start or stop of the task is controlled according to the updated state of the local license.

[0079] Method 2: In the case where the number of local licenses is multiple and there is at least one first local license whose state is inconsistent with that of the corresponding first server license, the state of the first local license is updated according to the state of the corresponding first server license, and the start or stop of the task is controlled according to the updated state of the local license.

[0080] Figure 6 Another flowchart for illustrating the method of task processing provided by the embodiments of the present disclosure is shown in FIG. 6. Figure 6 As shown in FIG. 6, the start or stop of the task can be controlled according to the local license and the server license in the following ways.

[0081] In step 200, in the case where the state of the local license and the state of the server license are both open, the task is already in the start state, and at this time, no processing can be performed.

[0082] In step 201, in the case that the server License is open, but the state of the local License is closed, the state of the local License can be updated to open.

[0083] It can be understood that the server License can be changed in response to the setting operation of the user, or can be changed automatically in response to the satisfaction of the preset condition.

[0084] The master node can periodically determine whether the server License and the local License are consistent, and in the case that the server License and the local License are inconsistent, the state of the local License can be updated to open, so that the server License and the local License remain consistent.

[0085] In step 202, the master node further determines whether the task is controlled by other Licenses according to the License control table, and in the case that the task is not controlled by other Licenses, or the task is controlled by other Licenses, but all the other Licenses controlling the task are open, the state of the task is changed from closed to open.

[0086] In step 203, in the case that the task is controlled by other Licenses, and there is at least one closed License in the other Licenses controlling the task, the task has not reached the starting condition, and the starting is not performed.

[0087] For example, the task A is controlled by the server License 1a and the local License 1b in the License control table, and the Licenses 1a and 1b are both in the closed state, and the task A is also in the unstarted state.

[0088] The state of the server License can be changed to the open state in response to the setting operation of the user or in response to the satisfaction of the preset starting condition, and the master node can obtain the state change of the License 1a by periodic interaction with the License server. Since the state of the local License 1b is closed at this time, the master node can execute step 201 to update the state of the local License 1b to open. Since the task A is not controlled by other Licenses, the master node can execute step 202 to change the state of the task A from closed to open. For example, the master node can add the task A to the quartz scheduler for scheduling to start the task A.

[0089] For another example, task B is controlled by server License 2a and local License 2b in the license control table, and is also controlled by server License 3a and local License 3b in the license control table, and License 2a, License 2b, License 3a and License 3b are all in the closed state, and task A is also in the unstarted state.

[0090] The state of server License 2a can be changed to the open state in response to a setting operation of a user or in response to a preset starting condition being met, and the master node can obtain the state change of License 2a to the open state through periodic interaction with the license server. Since the state of local License 2b is closed at this time, the master node can execute step 201 to update the state of local License 2b to open. Since task B is also controlled by other licenses, and the state of the corresponding local License 3b is still not closed, the master node will not start task B at this time, and similarly, after the state of server License 3a is changed to the open state, the master node can obtain the state change of License 3a to the open state through periodic interaction with the license server, and the master node can execute step 201 to update the state of local License 3b to open. Although task B is controlled by two licenses at this time, since the states of local Licenses 2b and 3b are both open, the master node can execute step 202 to change the state of task B from closed to open. For example, the master node can add task B to the quartz scheduler for scheduling to start task B.

[0091] In step 204, in the case where the states of the local license and the server license are both closed, the task is already in the closed state, and at this time, no processing can be performed.

[0092] In step 205, in the case where the license of the server is closed, but the state of the local license is open, the state of the local license can be updated to closed.

[0093] In step 206, the master node further determines whether the task is controlled by other licenses according to the license control table, and in the case where the task is not controlled by other licenses, or the task is controlled by other licenses, but the other licenses controlling the task are all closed, the state of the task is changed from open to closed.

[0094] For example, the master node can remove the task from the quartz scheduler to close the task.

[0095] In step 207, if the task is controlled by other License, and there is at least one enabled License in the other local License controlling the task, the task has not reached the closing condition, and closing is not performed.

[0096] The steps 204-207 described above are steps of closing the task corresponding to the steps 200-203, and the disclosure will not be expanded here.

[0097] With the technical solution described above, since one set of License can control the start and stop of one or more tasks, and one task can be controlled by one or more sets of License, the start and stop of a task or a batch of tasks can be controlled through License, and flexible management of task start or stop is achieved.

[0098] Figure 7 Another flowchart of a method for processing a task provided by an embodiment of the disclosure is shown, which is applied to a target secondary node connected to a master node, the master node is connected to multiple secondary nodes on at least one host, and one host corresponds to at least one secondary node, as shown in the figure, the method comprises the following steps. Figure 7

[0099] In step S201, a task to be processed is received from the master node.

[0100] The target secondary node is a secondary node on a target host determined by the master node from multiple secondary nodes based on a deployment mode of the multiple secondary nodes on the at least one host.

[0101] In some embodiments, the deployment mode is determined according to the number of tasks to be processed. The deployment mode includes a single-host multi-node mode, a multi-host mode, or a hybrid mode.

[0102] In some embodiments, the target secondary node is a secondary node with the smallest node load determined by the master node according to node load; or the target secondary node is a specified secondary node corresponding to the task to be processed.

[0103] The detailed meaning of the deployment mode, the method for determining the deployment mode by the master node, and the steps for determining the target secondary node by the master node can be referred to the corresponding embodiments of the master node, and will not be expanded here.

[0104] In step S202, the task to be processed is executed.

[0105] The target secondary node can receive the task assigned by the master node and execute it, and can also store the execution result.

[0106] ​According to the technical solution, the deployment mode of the multiple secondary nodes can be flexibly determined according to the number of the to-be-processed tasks, so that the system resources can be reasonably utilized, the business requirement of task processing can be met, and the waste of resources can be avoided, thereby saving the cost of task processing.

[0107] In some embodiments, the to-be-processed task includes a task type, the task type includes a guarantee task or a common task, the target secondary node includes a guarantee task thread pool and a common task thread pool, and the resource acquisition priority of the task in the guarantee task thread pool is higher than the resource acquisition priority of the task in the common task thread pool. Step S202 can specifically include: distributing the to-be-processed task into the guarantee task thread pool or the common task thread pool according to the task type for execution.

[0108] It can be understood that the resource acquisition priority of the task in the guarantee task thread pool can be higher than the resource acquisition priority of the task in the common task thread pool in a plurality of possible manners, for example, a certain amount of guarantee resources can be reserved for the task in the guarantee task thread pool, the task in the common task thread pool cannot acquire the guarantee resources, and the task in the guarantee task thread pool can preempt the resources occupied by the task in the common task thread pool within a certain proportion. The present disclosure does not limit this.

[0109] In some embodiments, at least one task in the guarantee task thread pool is transferred to the common task thread pool for execution when a preset condition is met.

[0110] The preset condition can include:

[0111] The number of threads in the guarantee task thread pool is greater than or equal to a first quantity threshold, or the thread occupancy rate in the guarantee task thread pool is greater than or equal to a first proportion threshold; and

[0112] The number of threads in the common task thread pool is less than or equal to a second quantity threshold, and / or the thread occupancy rate in the common task thread pool is less than or equal to a second proportion threshold.

[0113] Figure 8 Another flowchart of a method for task processing provided by an embodiment of the present disclosure is shown in FIG. 3. Figure 8 As shown in FIG. 3, the method can include the following steps.

[0114] In step 301, when the task type of the to-be-processed task is a guarantee task, the to-be-processed task can be distributed to the guarantee task thread pool for execution.

[0115] In step 302, when the task type of the to-be-processed task is a common task, the to-be-processed task can be distributed to the common task thread pool for execution.

[0116] In step 303, at least one task in the guarantee task thread pool is transferred to the common task thread pool for execution when a preset condition (for example, guarantee task thread pool full) is met.

[0117] In step 304, the tasks in the guarantee task thread pool and the common task thread pool are instantiated, and the corresponding task instances are executed.

[0118] For example, one or more types of tasks of four types of service types, such as Shell, FTP / SFTP, stored procedures, and Java, can be supported, and other service type tasks can be added by those skilled in the art without creative labor, which are within the protection scope of the present disclosure.

[0119] Figure 9 A system block diagram of the task processing system provided by the embodiments of the present disclosure is shown, as shown in the figure, the system includes a master node 4 and at least one secondary node 5, the master node 4 can include a timing scheduling module 41, a task processing module 42, a task allocation module 43, a heartbeat management module 44, a License management module 45 and an alarm management module 46. Figure 9

[0120] In some possible implementation manners, the timing scheduling module 41 can be based on a quartz framework to realize on-time scheduling of tasks.

[0121] The task management module 42 can add, edit, delete and query tasks through multiple ways such as responding to user operations on the interface, RestFul API and SQL scripts.

[0122] The task allocation module 43 can determine a target host and a target secondary node on the target host from multiple secondary nodes based on a deployment mode of the multiple secondary nodes on at least one host; and allocate a to-be-processed task to the target secondary node, so that the target secondary node executes the to-be-processed task.

[0123] The heartbeat management module 44 can collect and manage heartbeat information reported by the secondary node.

[0124] The License management module 45 can synchronize the License state of the task at a regular time, so as to realize control on the start and stop states of the task.

[0125] The alarm management module 46 can receive and process abnormal alarm information reported by the secondary node task execution.

[0126] The secondary node 5 can include a task execution module 51, a heartbeat reporting module 52 and an alarm reporting 53.

[0127] ​The task execution module 51 receives a to-be-processed task from the master node and executes the to-be-processed task.

[0128] The heartbeat reporting module 52 reports heartbeat information in response to a query event of the master node or a timing.

[0129] The alarm management module 53 reports exception information of a task execution failure to the master node.

[0130] It can be understood that the task allocation module 43 and the task execution module 51 in the task processing system are core modules, and the remaining modules can be appropriately simplified based on needs, and the technical solutions obtained thereby also belong to the protection scope of the present application. For example, the alarm management module 46 can not be arranged in the master node 4, and the alarm reporting module 53 can not be arranged in the auxiliary node 5. The master node 4 can not be connected to the License server, and the technical solution of flexibly controlling the starting or stopping of the task in the embodiment of the present application can not be adopted.

[0131] Figure 10 A hardware structure schematic diagram of an electronic device for executing the embodiments of the present disclosure is shown as follows. Figure 10 As shown in the figure, at the hardware level, the electronic device includes at least one processor, and optionally includes an internal bus, a network interface, and a memory. The memory can include a memory such as a random-access memory (RAM), and can also include a non-volatile memory such as at least one disk memory. Of course, the electronic device can also include other hardware required by a business.

[0132] The processor, the network interface, and the memory can be connected to each other through the internal bus. The internal bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0133] The memory stores a program. Specifically, the program can include program code including at least one computer operation instruction. The memory can include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0134] The at least one processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, and logically forms the device for positioning the target user. The at least one processor executes the program stored in the memory, and specifically executes: the method disclosed in the embodiment of the first aspect and realizes the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0135] The method disclosed in the embodiment of the first aspect of the present disclosure can be applied to at least one processor or implemented by at least one processor. The processor can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the at least one processor or the instruction in the form of software. The processor mentioned above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present disclosure can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0136] The electronic device can also execute the methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0137] Of course, in addition to the software implementation, the electronic device of the present disclosure does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc. That is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0138] The embodiment of the present disclosure further provides a computer readable storage medium, which stores one or more programs. The one or more programs, when executed by at least one processor, implement the method disclosed in the embodiment of the first aspect and achieve the functions and advantages of the methods described in the foregoing method embodiments, which will not be repeated here.

[0139] The computer readable storage medium includes a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0140] Further, the embodiment of the present disclosure further provides a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium. The computer program includes program instructions, which, when executed by a computer, implement the following processes: the method disclosed in the embodiment of the first aspect and achieve the functions and advantages of the methods described in the foregoing method embodiments, which will not be repeated here.

[0141] The embodiment of the present application provides a computer program product, which includes a computer program. The computer program, when executed by a processor, implements each process of the method disclosed in the embodiment of the first aspect and achieves the same technical effects. To avoid repetition, details will not be repeated here.

[0142] In summary, the above only describes the preferred embodiments of the present disclosure, and does not limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0143] The system, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0144] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0145] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0146] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

Claims

1. A task processing method applied to a master node, wherein the master node connects to multiple slave nodes on at least one host, and one host corresponds to at least one slave node, the method comprising: Based on the deployment mode of the plurality of auxiliary nodes on the at least one host, a target host and a target auxiliary node on the target host are determined from the plurality of auxiliary nodes, wherein the deployment mode is determined according to the number of tasks to be processed; The task to be processed is assigned to the target auxiliary node so that the target auxiliary node can execute the task to be processed.

2. The method according to claim 1, wherein the deployment mode includes a single-host multi-node mode, a multi-host mode, or a hybrid mode, and the method further includes: The multiple auxiliary nodes are deployed according to the determined deployment mode of the multiple auxiliary nodes.

3. The method according to claim 2, further comprising: The deployment mode of the multiple auxiliary nodes is determined based on the number of tasks to be processed and the system resources of the at least one host. or, The deployment mode of the multiple auxiliary nodes is determined based on the number of tasks to be processed, the system resources of the at least one host, and the task configuration of the tasks to be processed.

4. The method according to claim 1, wherein the target auxiliary node is the auxiliary node with the lowest node load determined by the master node based on node load; or, The target auxiliary node is a pre-specified auxiliary node corresponding to the task to be processed.

5. The method according to any one of claims 1 to 4, wherein the master node is connected to the license server, and the method further comprises: Obtain at least one local license for the task from the license control table, and obtain the server license corresponding to the local license from the license server; The task is started or stopped based on the local license and the server license.

6. The method according to claim 5, wherein controlling the start or stop of the task based on the local license and the server license comprises: If there is only one local license, and the status of the local license differs from that of the server license, the status of the local license is updated according to the status of the server license, and the start or stop of the task is controlled based on the updated status of the local license; or, If there are multiple local licenses and at least one first local license has a state that is inconsistent with the state of the corresponding first server license, the state of the first local license is updated according to the state of the corresponding first server license, and the start or stop of the task is controlled according to the updated state of the local license.

7. A task processing method applied to a target auxiliary node, the target auxiliary node being connected to a master node, the master node being connected to multiple auxiliary nodes on at least one host, one host corresponding to at least one auxiliary node, the method comprising: The master node receives tasks to be processed. The target auxiliary node is an auxiliary node on the target host determined from among the multiple auxiliary nodes based on the deployment mode of the multiple auxiliary nodes on the at least one host. The deployment mode is determined according to the number of tasks to be processed. Execute the task to be processed.

8. The method according to claim 7, wherein the deployment mode includes a single-host multi-node mode, a multi-host mode, or a hybrid mode.

9. The method according to claim 8, wherein the target auxiliary node is the auxiliary node with the lowest node load determined by the master node based on node load; or, The target auxiliary node is a pre-specified auxiliary node corresponding to the task to be processed.

10. The method according to any one of claims 7 to 9, wherein the task to be processed includes a task type, the task type includes a guaranteed task or a normal task, the target auxiliary node includes a guaranteed task thread pool and a normal task thread pool, the resource acquisition priority of tasks in the guaranteed task thread pool is higher than the resource acquisition priority of tasks in the normal task thread pool, and the execution of the task to be processed includes: The task to be processed is assigned to either the guaranteed task thread pool or the normal task thread pool for execution based on the task type.

11. The method according to claim 10, further comprising: Under certain preset conditions, at least one task in the guaranteed task thread pool will be transferred to the ordinary task thread pool for execution. These preset conditions include: The number of threads in the guaranteed task thread pool is greater than or equal to a first quantity threshold, or the thread occupancy rate of the guaranteed task thread pool is greater than or equal to a first proportion threshold; and, The number of threads in the ordinary task thread pool is less than or equal to the second quantity threshold, and / or the thread occupancy rate of the ordinary task thread pool is less than or equal to the second proportion threshold.

12. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the task processing method as described in any one of claims 1 to 6, or the computer program, when executed by the processor, implements the task processing method as described in any one of claims 7 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the task processing method as described in any one of claims 1 to 6, or, when executed by a processor, implements the task processing method as described in any one of claims 7 to 11.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the task processing method according to any one of claims 1-11.

Images