Lightweight parallel scheduling management method and system

Abstract

The invention relates to the technical field of computers, in particular to a lightweight parallel scheduling management method and system. The method comprises the following steps: receiving a connection request and a parallel computing task request of a computing request node, analyzing the parallel computing task request, generating single computing task data, sequentially and circularly processing computing tasks in a message queue according to priorities from high to low, and generating specific requirements corresponding to the computing tasks; according to each specific requirement, obtaining a calculation execution node of which the calculation node state is an idle state, and preferentially allocating a target calculation execution node of which the idle resource is in the first position; connecting a target calculation execution node, and sending calculation task data to the target calculation execution node; receiving calculation result data returned by the target calculation execution node, and pushing the calculation result data to the calculation request node in a message subscription mode; and receiving a next parallel computing task request, and repeating the steps until the parallel computing task request is executed, thereby realizing parallel computing scheduling of the business application.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a lightweight parallel scheduling management method and system. Background Art

[0002] With the development of technologies such as big data and cloud computing, the ability to flexibly configure computing, storage, network, IO and other resources has become stronger and stronger, especially in the field of localization. The widespread application of technologies such as big data and container cloud has improved the scheduling ability of local basic computing resources, making the allocation of computing resources more reasonable, and effectively supporting the resource needs of various business applications. However, with the increasing business applications, technologies such as container cloud have high requirements for basic hardware facilities, complex system operation and maintenance, and high costs. It is difficult to meet the needs of projects with low investment, low basic computing power, and small space. How to reasonably and fully utilize limited configuration resources to meet the parallel computing scheduling needs of multiple business applications.

[0003] In view of this, how to provide a lightweight parallel scheduling management method to better solve the business parallel scheduling management in resource-limited scenarios has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application provide a lightweight parallel scheduling management method applied to a parallel scheduling management server, a lightweight parallel scheduling management method applied to a physical computing node, a lightweight parallel scheduling management method applied to a computing request node, a lightweight parallel scheduling management device applied to a parallel scheduling management server, a lightweight parallel scheduling management device applied to a physical computing node, a lightweight parallel scheduling management device applied to a computing request node, a lightweight parallel scheduling system, an electronic device and a computer storage medium, which are used to solve business parallel scheduling management problems in resource-limited scenarios.

[0005] In a first aspect of an embodiment of the present application, a lightweight parallel scheduling management method is provided, which is applied to a parallel scheduling management server, wherein the parallel scheduling management service is deployed on any physical computing node, including:

[0006] Establishing a first parallel computing service monitor, wherein the first parallel computing service monitor is used to receive computing node information reported by the computing execution node, and update and record the node status in real time according to the computing node information, wherein the computing node information carries the status information reported by the computing execution node and the node resource usage information;

[0007] Receive the connection request from the computing request node and record the message subscription type;

[0008] Receive the parallel computing task request sent by the computing request node, parse the parallel computing task request according to the parallel scheduling message format, generate single computing task data, and put the generated single computing task data into message processing queues of different levels according to the priority, cyclically process the computing tasks in the message queue from high to low priority, and generate specific requirements corresponding to each computing task;

[0009] According to the specific requirements of each computing task, the computing execution nodes whose computing node status is idle are obtained, and the computing execution nodes are sorted according to the idle amount of resources, and the target computing execution node with the first idle resource is preferentially allocated;

[0010] Connecting to the target computing execution node, sending computing task data to the target computing execution node, and receiving the computing status returned by the target computing execution node, and updating and recording the computing node status according to the computing status, wherein the computing status includes an executing state and an idle state;

[0011] Receive the calculation result data returned by the target calculation execution node, and push the calculation result data to the calculation request node through a message subscription method;

[0012] Receive the next parallel computing task request and repeat the above steps until the parallel computing task request is completed, close the service monitoring, and release the connection with the computing execution node.

[0013] In a second aspect of an embodiment of the present application, a lightweight parallel scheduling management device is provided, which is applied to a parallel scheduling management server, wherein the parallel scheduling management service is deployed on any physical computing node, including:

[0014] A first establishing module is configured to establish a first parallel computing service monitor, wherein the first parallel computing service monitor is used to receive computing node information reported by a computing execution node, and update and record the node status in real time according to the computing node information, wherein the computing node information carries status information reported by the computing execution node and node resource usage information;

[0015] A first receiving module is configured to receive a connection request from a computing request node and record a message subscription type;

[0016] A generation module is configured to receive a parallel computing task request sent by a computing request node, parse the parallel computing task request according to a parallel scheduling message format, generate single computing task data, and put the generated single computing task data into message processing queues of different levels according to priority, cyclically process computing tasks in the message queues in descending order according to priority, and generate specific requirements corresponding to each computing task;

[0017] The allocation module is configured to obtain the computing execution nodes whose computing node status is idle according to the specific requirements corresponding to each computing task, and prioritize the target computing execution node with the first idle resource according to the idle amount of computing execution nodes;

[0018] An update module is configured to connect to the target computing execution node, send parallel computing task data to the target computing execution node, receive the computing status returned by the target computing execution node, and update and record the computing node status according to the computing status, wherein the computing status includes an executing state and an idle state;

[0019] A push module is configured to receive the calculation result data returned by the target calculation execution node, and push the calculation result data to the calculation request node through a message subscription method;

[0020] The second receiving module is configured to receive the next parallel computing task request, repeat the above steps until the parallel computing task request is executed, close the service monitoring, and release the connection with the computing execution node.

[0021] In a third aspect of an embodiment of the present application, a lightweight parallel scheduling management method is provided, which is applied to a physical computing node, wherein the physical computing node carries at least one computing execution node, including:

[0022] Establishing a second parallel computing service monitor, wherein the second parallel computing service monitor is used to receive and process connection requests and computing task requests of the parallel scheduling management service;

[0023] Set a timed heartbeat to regularly report status information to the parallel scheduling management service, where the status information carries status information reported by the computing execution node and node resource usage information;

[0024] Receive computing tasks sent by the parallel scheduling management service and update the computing execution node status to executing;

[0025] Perform calculations on the computing task request data according to the actual processing logic of the business application, generate calculation results, send the calculation results to the parallel scheduling management service, and update the computing execution node status to idle;

[0026] Receive the next computing task sent by the parallel scheduling management service and execute the computing task to completion.

[0027] In a fourth aspect of an embodiment of the present application, a lightweight parallel scheduling management device is provided, which is applied to a physical computing node end, wherein the physical computing node carries at least one computing execution node, including:

[0028] A second establishing module is configured to establish a second parallel computing service monitor, wherein the second parallel computing service monitor is used to receive and process connection requests and computing task requests of the parallel scheduling management service;

[0029] A reporting module is configured to set a timed heartbeat and regularly report status information to the parallel scheduling management service, wherein the status information carries status information reported by the computing execution node and node resource usage information;

[0030] A third receiving module is configured to receive the computing task sent by the parallel scheduling management service, and update the computing execution node state to being executed;

[0031] The computing module is configured to perform computing on the computing task request data according to the actual processing logic of the business application, generate computing results, send the computing results to the parallel scheduling management service, and update the computing execution node status to idle;

[0032] The fourth receiving module is configured to receive the next computing task sent by the parallel scheduling management service and execute the computing task to complete.

[0033] In a fifth aspect of an embodiment of the present application, a lightweight parallel scheduling management method is provided, which is applied to a computing request node, including:

[0034] In response to a computing request node being connected to a parallel scheduling management service, recording a first message subscription type of interaction between the computing request node and the parallel scheduling management service, and receiving first interaction data according to the first message subscription type;

[0035] Based on the first interaction data, sending a parallel computing task to the parallel scheduling management service;

[0036] Receive the calculation results returned by the parallel scheduling management service, and process the calculation results according to the actual logic of the business application;

[0037] Send the next parallel computing task to the parallel scheduling management service until the parallel computing task is completed.

[0038] In a sixth aspect of an embodiment of the present application, a lightweight parallel scheduling management device is provided, which is applied to a computing request node, including:

[0039] a recording module configured to record, in response to a computing request node connecting to a parallel scheduling management service, a first message subscription type of interaction between the computing request node and the parallel scheduling management service, and receive first interaction data according to the first message subscription type;

[0040] A first sending module, configured to send a parallel computing task to the parallel scheduling management service based on the first interaction data;

[0041] A processing module, configured to receive the calculation results returned by the parallel scheduling management service, and process the calculation results according to the actual logic of the business application;

[0042] The second sending module is configured to send the next parallel computing task to the parallel scheduling management service until the parallel computing task is executed.

[0043] In a seventh aspect of the embodiments of the present application, a lightweight parallel scheduling system is provided, including a parallel scheduling management service and a parallel scheduling interface, wherein:

[0044] The parallel scheduling management service includes computing execution node status management, computing task management, task scheduling strategy management and parallel scheduling log management, which is used to obtain and manage the status of business application parallel computing execution nodes in real time; receive and decompose parallel computing request tasks, send computing tasks to parallel computing execution nodes according to task scheduling strategies, and receive computing results;

[0045] The parallel scheduling interface includes a computing execution node interface and a computing request node interface, which are used to establish a message subscription-based method for receiving and sending computing data.

[0046] According to an eighth aspect of an embodiment of the present application, a computing device is provided, including:

[0047] Memory and processor;

[0048] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned lightweight parallel scheduling management method are implemented.

[0049] According to a ninth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned lightweight parallel scheduling management method are implemented.

[0050] The application of a lightweight parallel scheduling method provided by this application can help business applications perform parallel scheduling calculations under limited computing resources. First, this application has low requirements for basic hardware, does not occupy additional computing nodes, and can be deployed together with business applications; then, this application not only retains the traditional business single direct connection mode, but also supports parallel computing processing, with flexible deployment and strong scalability; secondly, this application does not associate specific business data during parallel scheduling processing, supports various types of business access for parallel scheduling management, and has strong business function scalability; finally, this application realizes unified management of computing node resources and parallel scheduling development interface, which makes development and operation and maintenance very simple, greatly reducing the development and operation costs.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0053] Figure 1 A schematic diagram of the structure of a lightweight parallel scheduling management system provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the structure of a lightweight parallel scheduling management architecture provided in an embodiment of the present application;

[0055] Figure 3 A flowchart of a lightweight parallel scheduling management method applied to a parallel scheduling management server provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of a lightweight parallel scheduling management device applied to a parallel scheduling management server provided in an embodiment of the present application;

[0057] Figure 5 A flowchart of a lightweight parallel scheduling management method applied to a physical computing node provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the structure of a lightweight parallel scheduling management device applied to a physical computing node end provided in an embodiment of the present application;

[0059] Figure 7 A flowchart of a lightweight parallel scheduling management method applied to a computing request node provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of a lightweight parallel scheduling management device applied to a physical computing node end provided in an embodiment of the present application;

[0061] Fig. 9 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0063] See also Figure 1 , Figure 1 A schematic diagram of a lightweight parallel scheduling management system provided in an embodiment of the present application. Without adding additional computing resources, the embodiment of the present application fully and reasonably utilizes the configured limited resources to achieve parallel computing scheduling of business applications, and has the characteristics of flexible deployment and simple operation and maintenance.

[0064] like Figure 1 As shown, the lightweight parallel scheduling system includes two parts: a parallel scheduling management service 101 and a parallel scheduling interface 102 .

[0065] Specifically, the parallel scheduling management service 101 includes four parts: computing execution node status management 1011, computing task management 1012, task scheduling strategy management 1013 and parallel scheduling log management 1014, which are used to obtain and manage the status of parallel computing execution nodes of business applications in real time; receive and decompose parallel computing request tasks, send computing tasks to parallel computing execution nodes according to task scheduling strategies, and receive computing results; it should be noted that the parallel scheduling management service 101 does not deeply parse the content of parallel computing request data and is decoupled from specific business applications.

[0066] The parallel scheduling interface 102 includes a computing execution node interface 1021 and a computing request node interface 1022, which are used to establish a message subscription-based method for receiving and sending computing data. It should be noted that the main features of the parallel scheduling interface 102 are: establishing a message subscription-based method for receiving and sending computing data, supporting both parallel computing and the original single node direct connection computing mode (computing request nodes directly connected to business computing execution nodes), and supporting the specific implementation of parallel computing for actual business applications.

[0067] More specifically, the computing execution node state management 1011 is used for computing execution nodes to actively report state information to the parallel scheduling management service 101, and the parallel scheduling management service 101 summarizes the states of each computing execution node and senses the states of the computing execution nodes. That is, the computing execution node actively reports online heartbeats, node resource usage (including CPU, memory, disk, etc.), and computing execution node state changes (idle, executing) to the parallel scheduling management service 101 through an interface, and the parallel scheduling management service 101 summarizes and manages the states of each business computing execution node, and timely senses the load state and heartbeat state of the computing execution node for use by the task scheduling strategy and query by the business application client.

[0068] The computing task management 1012 is used to receive the parallel computing task request sent by the computing request node, parse the parallel computing task request according to the parallel scheduling message format, generate a single computing task data, and put the generated single computing task data into message processing queues of different levels according to the priority, and process the computing tasks in the message queue in a cyclic manner from high to low according to the priority, and generate specific requirements corresponding to each computing task; connect to the target computing execution node, send the parallel computing task data to the target computing execution node, and receive the computing status returned by the target computing execution node, and update the computing execution node status according to the computing status; receive the computing result data returned by the target computing execution node, and push the computing result data to the computing request node through the message subscription method. That is, on the one hand, a parallel scheduling message format is designed, which is characterized by not being specifically associated with any business computing data, being able to carry various types of business data, supporting both parallel computing task data and ordinary business data; on the other hand, task parallel computing requests are received, computing tasks are decomposed and generated, and are placed in the corresponding task computing queue according to task priority. The task computing queue is processed cyclically through the task scheduling strategy, and the computing tasks are sent to the corresponding computing execution nodes. The computing results are received and pushed to the computing request nodes; in addition, when interacting with the computing execution nodes, the execution node status (idle, executing) is perceived in a timely manner; a timeout window is set for the computing task, and when it expires, "no available computing execution node" or "computing wait timeout" can be returned.

[0069] The task scheduling strategy management 1013 is used to comprehensively analyze the status information of the computing execution nodes and perform comprehensive scheduling of parallel computing tasks. That is, comprehensively analyze the computing execution node status (idle, executing), the resource usage of the computing node (including CPU, memory, disk, etc.), the maximum threshold value of the computing node resources available, and the demand of the computing task (priority, resource usage estimate, timeout time, idle or all execution nodes), perform comprehensive scheduling of parallel computing tasks, and give priority to scheduling idle and resource-sufficient computing execution nodes to perform calculations, so as to avoid the situation where the computing fails when the application resources cannot be met during the execution of business computing. If the computing task scheduling fails, the specific scheduling failure reason is returned ("no available computing execution node", "computing execution node resources are not satisfied", "computing waiting timeout", etc.).

[0070] The parallel scheduling log management 1014 is used to record the state change information of the computing execution node in real time; record the interaction data between the computing request node and the parallel scheduling management service 101, and between the parallel scheduling management service 101 and the computing execution node. That is, record the state change of the computing execution node, including online / offline, idle / executing, and resource usage; record the computing task data sent and received between the computing request node and the parallel scheduling management service 101, and between the parallel scheduling management service 101 and the computing execution node.

[0071] More specifically, the computing execution node interface 1021 is used to provide business application computing service establishment and shutdown, as well as timed heartbeats; receive and send computing task data and data transceiving status; and change node status. That is, it provides computing service establishment and shutdown, timed heartbeats (including node status and node resource usage); receive and send computing task data and data transceiving status; and actively change node status (idle / executing). The interface automatically completes the acquisition of the node's CPU, memory, disk, and network load conditions.

[0072] The computing request node interface 1022 is used to provide connection and disconnection with the parallel scheduling management service 101, message subscription, receiving and sending computing task data and data sending and receiving status functions.

[0073] The technical architecture of the lightweight parallel scheduling management method provided in the embodiment of the present application is as follows: Figure 2 As shown, Figure 2 A schematic diagram of a lightweight parallel scheduling management architecture provided for an embodiment of the present application. Physical computing nodes and physical request nodes, whose corresponding entities are servers (which can be racks, blades, etc.), computers (which can be desktops, notebooks, etc.) and other physical hardware or virtual (which can be cloud servers) devices. It should be noted that a physical request / computing node has complete and independent CPU, memory, hard disk and other supporting computing resources.

[0074] like Figure 2 As shown in the figure, the business application software is deployed on N (≥1) physical computing nodes, and one physical computing node runs one business application instance of a certain type (the execution process and operating environment of the business application instance are consistent), that is, ≤N business application instances of this type are run at the same time; one physical computing node can deploy other business applications at the same time, such as XX business application, YY business application, etc.; each type of business only deploys one computing execution instance (computing execution node) on each physical computing node. Various business applications are reasonably deployed according to the actual parallel scheduling requirements and the resource availability of the physical computing nodes. When N=1, it is a minimalist deployment mode, and can still be scheduled through the parallel scheduling management service, which is convenient for the later expansion of computing resources.

[0075] The business application software implemented through the parallel scheduling interface supports both the parallel scheduling mode, which schedules parallel computing tasks through the parallel scheduling management service, and the original single direct connection mode, that is, the computing request node software is directly connected to the computing execution node software, and the parallel scheduling management service does not need to be deployed in this case. This design architecture facilitates the flexible configuration (reduction and expansion) of computing resources without adjusting the business application software.

[0076] See also Figure 3 , Figure 3 The flowchart of a lightweight parallel scheduling management method applied to a parallel scheduling management server provided in an embodiment of the present application is shown in FIG. It should be noted that the parallel scheduling management service can be deployed on any physical computing node. Figure 3 As shown, the specific steps include the following steps.

[0077] Step S302: Establish a first parallel computing service monitor, wherein the first parallel computing service monitor is used to receive computing node information reported by the computing execution node, and update and record the node status in real time according to the computing node information, wherein the computing node information carries the status information reported by the computing execution node and the node resource usage information.

[0078] In an embodiment of the present application, the node status is updated and recorded in real time based on the computing node information, including: in response to not receiving the computing node information within a preset time interval, determining that the node status is an offline state; in response to receiving the reported status information again, determining that the node status is an online state, and repeating the offline determination at the preset time interval.

[0079] In actual applications, the parallel scheduling management service establishes a parallel computing service monitor, receives the status and resource usage reported by the business application computing execution nodes, updates and records the node status in real time, and determines that the node is offline if the status is not received within the preset time interval. When the status information is received again, the node is determined to be online, and the offline determination is repeated at the preset time interval.

[0080] Step S304: Receive a connection request from a computing request node and record the message subscription type. It should be noted that the computing request node in the embodiment of the present application is generally a computing request end of the business application or other business applications.

[0081] Step S306: Receive the parallel computing task request sent by the computing request node, parse the parallel computing task request according to the parallel scheduling message format, generate a single computing task data, and put the generated single computing task data into message processing queues of different levels according to the priority, and process the computing tasks in the message queue in a loop from high to low according to the priority, and generate specific requirements corresponding to each computing task.

[0082] In an embodiment of the present application, the method of sorting by the idle amount of computing execution node resources and giving priority to allocating target computing execution nodes with the first idle resources includes: in the process of loop processing, when the computing task timeout reaches the preset time, in response to the absence of any computing execution node in the idle state or the executing state, an indication of no available computing execution node is returned; in response to the existence of a computing execution node in the idle state and the available resources of the computing execution node do not meet the estimated resource usage, an indication of unsatisfied computing execution node resources is returned; in response to the absence of a computing execution node in the idle state but the existence of a computing execution node in the executing state, a computing wait timeout indication is returned.

[0083] In actual applications, the computing tasks in the message queue are processed in a loop from high to low priority. According to the specific requirements of the taken computing tasks, the idle / all computing execution nodes are obtained. When the task requires idle nodes, they are sorted according to the idle amount of computing execution node resources, and the computing execution nodes with the most idle resources are allocated first. During the loop processing, when the computing task timeout expires, if there is no computing execution node, "No available computing execution node" is returned; if there is a computing execution node, but the available resources of the computing execution node are insufficient, "Computing execution node resources are not satisfied" is returned, and "Computing wait timeout" is returned in other cases.

[0084] Step S308: According to the specific requirements corresponding to each computing task, the computing execution nodes whose computing node status is idle are obtained, and the computing execution nodes are sorted according to the idle amount of resources, and the target computing execution node with the first idle resource is preferentially allocated.

[0085] Step S310: connect to the target computing execution node, send computing task data to the target computing execution node, and receive the computing status returned by the target computing execution node, and update and record the computing node status according to the computing status, wherein the computing status includes the executing state and the idle state.

[0086] In actual applications, connect to the assigned computing execution node (only connect when initially assigned), send computing task data, receive the computing status (executing, idle, etc.) returned by the computing execution node, and update and record the node status.

[0087] Step S312: receiving the calculation result data returned by the target calculation execution node, and pushing the calculation result data to the calculation request node through a message subscription method.

[0088] Step S314: Receive the next parallel computing task request, repeat the above steps until the parallel computing task request is executed, close the service monitoring, and release the connection with the computing execution node.

[0089] In actual applications, the process loops through step S304 to connect and subscribe new computing request nodes; the process loops through step S306 to generate computing tasks; the process loops through steps S308 to S312 to schedule the execution of computing tasks and push computing result data.

[0090] When the service ends, close the service monitor and release the connection with the computing execution node.

[0091] Corresponding to the above method embodiment, this specification also provides an embodiment of a lightweight parallel scheduling management device applied to a parallel scheduling management server. Figure 4 The structure diagram of a lightweight parallel scheduling management device applied to a parallel scheduling management server provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device comprises:

[0092] The first establishing module 402 is configured to establish a first parallel computing service monitor, wherein the first parallel computing service monitor is used to receive computing node information reported by the computing execution node, and update and record the node status in real time according to the computing node information, wherein the computing node information carries the status information reported by the computing execution node and the node resource usage information;

[0093] A first receiving module 404 is configured to receive a connection request from a computing requesting node and record a message subscription type;

[0094] The generation module 406 is configured to receive the parallel computing task request sent by the computing request node, parse the parallel computing task request according to the parallel scheduling message format, generate single computing task data, and put the generated single computing task data into message processing queues of different levels according to the priority, cyclically process the computing tasks in the message queue in order from high to low priority, and generate specific requirements corresponding to each computing task;

[0095] The allocation module 408 is configured to obtain the computing execution nodes whose computing node status is idle according to the specific requirements corresponding to each computing task, and prioritize the target computing execution node with the first idle resource according to the idle amount of computing execution nodes;

[0096] The updating module 410 is configured to connect to the target computing execution node, send the parallel computing task data to the target computing execution node, receive the computing status returned by the target computing execution node, and update and record the computing node status according to the computing status, wherein the computing status includes an executing state and an idle state;

[0097] The push module 412 is configured to receive the calculation result data returned by the target calculation execution node, and push the calculation result data to the calculation request node through a message subscription method;

[0098] The second receiving module 414 is configured to receive the next parallel computing task request, repeat the above steps until the parallel computing task request is executed, close the service monitoring, and release the connection with the computing execution node.

[0099] In an optional embodiment, the first establishing module 402 is further configured to:

[0100] In response to not receiving status information reported by the computing execution node within a preset time interval, determining that the node status is an offline status;

[0101] In response to receiving the computing node information again, determining that the node state is online, repeating the offline determination at a preset time interval

[0102] In an optional embodiment, the allocation module 408 is further configured to:

[0103] During the loop processing, when the computing task timeout reaches the preset time, in response to the absence of any computing execution node whose computing node status is idle or in execution, an indication of no available computing execution node is returned;

[0104] In response to the presence of a computing execution node whose computing node state is an idle state, and the available resources of the computing execution node do not meet the estimated resource usage, returning an indication that the computing execution node resources are not met;

[0105] In response to the fact that there is no computing execution node whose computing node status is the idle state, but there is a computing execution node whose computing node status is the executing state, a computing wait timeout indication is returned.

[0106] See also Figure 5 , Figure 5A flowchart of a lightweight parallel scheduling management method applied to a physical computing node provided in an embodiment of the present application. It should be noted that the physical computing node carries at least one computing execution node, wherein the computing execution node (business application computing execution part, generally in the form of a server) establishes a computing service monitor through a parallel scheduling interface, sets a timed heartbeat, processes the received computing data, actively changes the node status to executing during computing, and sends the computing results. Figure 5 As shown, the specific steps include the following steps.

[0107] Step S502: Establish a second parallel computing service monitor, wherein the second parallel computing service monitor is used to receive and process connection requests and computing task requests of the parallel scheduling management service.

[0108] Step S504: Set a timed heartbeat to regularly report status information to the parallel scheduling management service, wherein the status information carries status information reported by the computing execution node and node resource usage information. In actual applications, the computing execution node sets a timed heartbeat to regularly report the computing status (idle, executing) and node resource usage (including CPU, memory, disk, etc.) to the parallel scheduling management service.

[0109] Step S506: receiving the computing task sent by the parallel scheduling management service, and updating the computing execution node status to being executed.

[0110] Step S508: perform calculations on the computing task request data according to the actual processing logic of the business application, generate calculation results, send the calculation results to the parallel scheduling management service, and update the computing execution node status to idle.

[0111] Step S510: receiving the next computing task sent by the parallel scheduling management service, executing the computing task and completing it, that is, waiting for the next computing task, and looping through steps S506 to S508.

[0112] Corresponding to the above method embodiment, this specification also provides an embodiment of a lightweight parallel scheduling management device applied to a physical computing node end, Figure 6 The present invention provides a schematic diagram of a lightweight parallel scheduling management device for a physical computing node. Figure 6 As shown, the device comprises:

[0113] A second establishing module 602 is configured to establish a second parallel computing service monitor, wherein the second parallel computing service monitor is used to receive and process connection requests and computing task requests of a parallel scheduling management service;

[0114] The reporting module 604 is configured to set a timed heartbeat and regularly report status information to the parallel scheduling management service, wherein the status information carries status information reported by the computing node and node resource usage information;

[0115] The third receiving module 606 is configured to receive the computing task sent by the parallel scheduling management service, and update the computing execution node state to being executed;

[0116] The computing module 608 is configured to perform computing on the computing task request data according to the actual processing logic of the business application, generate computing results, send the computing results to the parallel scheduling management service, and update the computing execution node status to idle;

[0117] The fourth receiving module 610 is configured to receive the next computing task sent by the parallel scheduling management service and execute the computing task to complete.

[0118] See also Figure 7 , Figure 7 A flowchart of a lightweight parallel scheduling management method applied to a computing request node provided in an embodiment of the present application. It should be noted that the computing request node (the computing request part of this business application, generally in the form of a client, or the computing request part of other business applications, generally in the form of a server or client) connects to the parallel scheduling management service through the parallel scheduling interface, subscribes to the received message type, sends parallel computing task requests, and receives and processes the computing results. Figure 7 As shown, the specific steps include the following steps.

[0119] Step S702: In response to the computing request node being connected to the parallel scheduling management service, a first message subscription type of interaction between the computing request node and the parallel scheduling management service is recorded, and first interaction data is received according to the first message subscription type. It should be noted that the computing request node here can be connected to the parallel management service or directly connected to the computing execution node, that is, a single direct connection mode.

[0120] Step S704: Based on the first interaction data, send a parallel computing task to the parallel scheduling management service.

[0121] Step S706: receiving the calculation result returned by the parallel scheduling management service, and processing the calculation result according to the actual logic of the business application.

[0122] Step S708: Send the next parallel computing task to the parallel scheduling management service until the parallel computing task is completed.

[0123] In another embodiment, the method further includes: in response to a direct connection between a computing request node and a computing execution node, recording a second message subscription type of direct interaction between the computing request node and the computing execution node, and receiving second interaction data according to the second message subscription type; based on the second interaction data, sending a computing task to the computing execution node; receiving a computing result returned by the computing execution node, and processing the computing result according to the actual logic of the business application; and sending the next computing task to the computing execution node until the computing task is completed.

[0124] Corresponding to the above method embodiment, this specification also provides an embodiment of a lightweight parallel scheduling management device applied to a computing node end, Figure 8 The present invention provides a schematic diagram of a lightweight parallel scheduling management device for a physical computing node. Figure 8 As shown, the device comprises:

[0125] The recording module 802 is configured to record a first message subscription type of interaction between the computing request node and the parallel scheduling management service in response to the computing request node being connected to the parallel scheduling management service, and receive first interaction data according to the first message subscription type;

[0126] A first sending module 804 is configured to send a parallel computing task to the parallel scheduling management service based on the first interaction data;

[0127] The processing module 806 is configured to receive the calculation result returned by the parallel scheduling management service and process the calculation result according to the actual logic of the business application;

[0128] The second sending module 808 is configured to send the next parallel computing task to the parallel scheduling management service until the parallel computing task is executed.

[0129] In an optional embodiment, the device further includes:

[0130] In response to a computation requesting node being directly connected to a computation execution node, recording a second message subscription type for direct interaction between the computation requesting node and the computation execution node, and receiving second interaction data according to the second message subscription type;

[0131] Sending a computing task to the computing execution node based on the second interaction data;

[0132] Receive the calculation results returned by the calculation execution node, and process the calculation results according to the actual logic of the business application;

[0133] The next computing task is sent to the computing execution node until the computing task is completed.

[0134] The lightweight task parallel scheduling management method proposed in this application can help business applications perform parallel scheduling calculations under limited computing resources. The embodiment of this application has low requirements for basic hardware, does not occupy additional computing nodes, and can be deployed together with business applications; it not only retains the traditional business single direct connection mode, but also supports parallel computing processing, flexible deployment, and strong scalability; it does not associate specific business data during parallel scheduling processing, supports various types of business access for parallel scheduling management, and has strong business function scalability; it realizes unified management of computing node resources and parallel scheduling development interface, which makes development and operation and maintenance very simple, greatly reducing the development and operation costs.

[0135] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the lightweight parallel scheduling management device, since it is basically similar to the lightweight parallel scheduling management method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the lightweight parallel scheduling management method embodiment.

[0136] Fig. 9 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0137] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0138] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0139] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0140] The processor 920 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned lightweight parallel scheduling management method.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the lightweight parallel scheduling management method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the lightweight parallel scheduling management method embodiment.

[0142] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned lightweight parallel scheduling management method.

[0143] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the lightweight parallel scheduling management method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the lightweight parallel scheduling management method embodiment.

[0144] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned lightweight parallel scheduling management method.

[0145] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the lightweight parallel scheduling management method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the lightweight parallel scheduling management method embodiment.

[0146] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0147] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0148] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0149] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A lightweight parallel scheduling management method, applied to a parallel scheduling management server, wherein the parallel scheduling management service is deployed on any physical computing node, characterized in that: include: Establishing a first parallel computing service monitor, wherein the first parallel computing service monitor is used to receive computing node information reported by the computing execution node, and update and record the node status in real time according to the computing node information, wherein the computing node information carries the status information reported by the computing execution node and the node resource usage information; Receive the connection request from the computing request node and record the message subscription type; Receive the parallel computing task request sent by the computing request node, parse the parallel computing task request according to the parallel scheduling message format, generate single computing task data, and put the generated single computing task data into message processing queues of different levels according to the priority, cyclically process the computing tasks in the message queue from high to low priority, and generate specific requirements corresponding to each computing task; According to the specific requirements of each computing task, the computing execution nodes whose computing node status is idle are obtained, and the computing execution nodes are sorted according to the idle amount of resources, and the target computing execution node with the first idle resource is preferentially allocated; Connecting to the target computing execution node, sending computing task data to the target computing execution node, and receiving the computing status returned by the target computing execution node, and updating and recording the computing node status according to the computing status, wherein the computing status includes an executing state and an idle state; Receive the calculation result data returned by the target calculation execution node, and push the calculation result data to the calculation request node through a message subscription method; Receive the next parallel computing task request and repeat the above steps until the parallel computing task request is completed, close the service monitoring, and release the connection with the computing execution node.

2. The method according to claim 1, characterized in that The updating and recording of node status in real time according to the computing node information includes: In response to not receiving status information reported by the computing execution node within a preset time interval, determining that the node status is an offline status; In response to receiving the reported status information again, determining that the node status is an online status, and repeating the offline determination at a preset time interval.

3. The method according to claim 1, characterized in that The step of sorting the idle amount of computing execution nodes and preferentially allocating the target computing execution node with the first idle resource comprises: During the loop processing, when the computing task timeout reaches the preset time, in response to the absence of any computing execution node whose computing node status is idle or in execution, an indication of no available computing execution node is returned; In response to the presence of a computing execution node whose computing node state is an idle state, and the available resources of the computing execution node do not meet the estimated resource usage, returning an indication that the computing execution node resources are not met; In response to the fact that there is no computing execution node whose computing node status is the idle state, and there is a computing execution node whose computing node status is the executing state, a computing wait timeout indication is returned.

4. A lightweight parallel scheduling management method, applied to a physical computing node, wherein the physical computing node carries at least one computing execution node, characterized in that: include: Establishing a second parallel computing service monitor, wherein the second parallel computing service monitor is used to receive and process connection requests and computing task requests of the parallel scheduling management service; Set a timed heartbeat to regularly report status information to the parallel scheduling management service, where the status information carries status information reported by the computing execution node and node resource usage information; Receive computing tasks sent by the parallel scheduling management service and update the computing execution node status to executing; Perform calculations on the computing task request data according to the actual processing logic of the business application, generate calculation results, send the calculation results to the parallel scheduling management service, and update the computing execution node status to idle; Receive the next computing task sent by the parallel scheduling management service and execute the computing task to completion.

5. A lightweight parallel scheduling management method, applied to a computing request node, characterized in that: include: In response to a computing request node being connected to a parallel scheduling management service, recording a first message subscription type of interaction between the computing request node and the parallel scheduling management service, and receiving first interaction data according to the first message subscription type; Based on the first interaction data, sending a parallel computing task to the parallel scheduling management service; Receive the calculation results returned by the parallel scheduling management service, and process the calculation results according to the actual logic of the business application; Send the next parallel computing task to the parallel scheduling management service until the parallel computing task is completed.

6. The method according to claim 5, characterized in that The method further comprises: In response to a computation requesting node being directly connected to a computation execution node, recording a second message subscription type for direct interaction between the computation requesting node and the computation execution node, and receiving second interaction data according to the second message subscription type; Sending a computing task to the computing execution node based on the second interaction data; Receive the calculation results returned by the calculation execution node, and process the calculation results according to the actual logic of the business application; The next computing task is sent to the computing execution node until the computing task is completed.

7. A lightweight parallel scheduling system, characterized in that: Includes parallel scheduling management services and parallel scheduling interfaces, among which, The parallel scheduling management service includes computing execution node status management, computing task management, task scheduling strategy management and parallel scheduling log management, which is used to obtain and manage the status of business application parallel computing execution nodes in real time; receive and decompose parallel computing request tasks, send computing tasks to parallel computing execution nodes according to task scheduling strategies, and receive computing results; The parallel scheduling interface includes a computing execution node interface and a computing request node interface, which are used to establish a message subscription-based method for receiving and sending computing data.

8. The lightweight parallel scheduling system according to claim 7, characterized in that: The computing execution node status management is used for the computing execution node to actively report status information to the parallel scheduling management service, and the parallel scheduling management service summarizes the status of each computing execution node and perceives the status of the computing execution node; The computing task management is used to receive the parallel computing task request sent by the computing request node, parse the parallel computing task request according to the parallel scheduling message format, generate a single computing task data, and put the generated single computing task data into message processing queues of different levels according to the priority, and cyclically process the computing tasks in the message queue in order from high to low priority, and generate specific requirements corresponding to each computing task; connect to the target computing execution node, send the computing task data to the target computing execution node, and receive the computing status returned by the target computing execution node, and update the computing node status according to the computing status; Receive the calculation result data returned by the target calculation execution node, and push the calculation result data to the calculation request node through a message subscription method; The task scheduling strategy management is used to comprehensively analyze the status information of the computing execution nodes and perform comprehensive scheduling of parallel computing tasks; The parallel scheduling log management is used to record the status change information of the computing execution node in real time; record the interaction data between the computing request node and the parallel scheduling management service, and between the parallel scheduling management service and the computing execution node.

9. The lightweight parallel scheduling system according to claim 7, characterized in that: The computing execution node interface is used to provide business application computing execution service establishment and shutdown, as well as timed heartbeats; receive and send computing task data and data sending and receiving status; Change node status; The computing request node interface is used to provide connection and disconnection with the parallel scheduling management service, message subscription, receiving and sending computing task data and data sending and receiving status functions.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images