A hybrid scheduling method for privacy computing tasks

Abstract

The present invention relates to the field of privacy computing technology, and specifically discloses a hybrid scheduling method applied to privacy computing tasks, including the following steps: step s1: the user initiates a task, and the task is transferred to the scheduling service inside the node; step s2: the task is transferred to the task parser TaskParser of the scheduling service; step s3: the task parser TaskParser parses the task and calls the task verifier TaskValidator to verify the task content; step s4: if the task verification fails, the task process is directly terminated. The present invention can realize the intelligent use of different scheduling engines for scheduling according to different application scenarios and different resource usage conditions through the use of different scheduling engines combined with scheduling processes, and use system resources in the largest and most reasonable way to perform related scheduling work; and at the same time, it can report resource usage in real time.

Description

Technical Field

[0001] The present invention relates to the field of privacy computing technology, and specifically to a hybrid scheduling method applied to privacy computing tasks. Background Art

[0002] With the rapid development of artificial intelligence, the traditional financial industry has begun to embrace cutting-edge machine learning algorithms, such as XGBoost and deep learning. In order to further improve the model effect, it is often necessary to carry out cross-institutional or cross-departmental data cooperation while protecting data privacy and security. Privacy computing technology has emerged. At present, there are some machine learning algorithms that can protect data privacy, but due to their weak interpretability, they are still difficult to implement in financial scenarios. XGBoost and SHAP in federated learning; XGBoost is a boosting tree algorithm that processes low-dimensional structured data with high speed and accuracy. Federated learning is a framework for privacy computing, which comes from distributed machine learning. It can realize joint machine learning by sharing model parameters without the original data leaving the domain. However, since model parameters may leak the model and infer the original data, most federated learning currently uses cryptography-related technologies to share encrypted parameters, but still requires a trusted third party as the central computing node. SHAP stands for SHapleyAdditive exPlanation, where the concept of Shapley value comes from game theory and is used to solve the problem of distribution equilibrium in cooperative games. SHAP can be used in machine learning to explain complex models and calculate the contribution of each feature to model prediction. Federated learning introduces it to calculate the contribution of each participant.

[0003] In summary, in order to solve the problem that the existing technology has weak interpretability and is still difficult to implement in financial scenarios, we propose a hybrid scheduling method for privacy computing tasks. Summary of the invention

[0004] The purpose of the present invention is to provide a hybrid scheduling method for privacy computing tasks. By using different scheduling engines in combination with scheduling processes, different scheduling engines can be used intelligently for scheduling according to different application scenarios and different resource usage conditions, so as to maximize and most rationally use system resources and perform related scheduling work; and at the same time, resource usage can be reported in real time.

[0005] To achieve the above object, the present invention provides the following technical solution: a hybrid scheduling method applied to privacy computing tasks, comprising the following steps:

[0006] Step S1: The user initiates a task, which is transferred to the scheduling service within the node;

[0007] Step S2: The task is transferred to the task parser TaskParser of the scheduling service;

[0008] Step S3: The task parser TaskParser parses the task and calls the task validator TaskValidator to verify the task content;

[0009] Step S4: If the task verification fails, the task process is terminated directly;

[0010] Step S5: The tasks that have passed the verification are handed over to the task manager TaskManager for management and scheduling;

[0011] Step S6: Determine whether to schedule a task from the queue or directly schedule the current task according to the task status in the TaskQueue;

[0012] Step S7: Submit the task to Scheduler for scheduling;

[0013] Step S8: Scheduler calls ResourceManager to obtain available resource information;

[0014] Step S9: ResourceManager uses the available resource table entry of ResourceTable to return to Scheduler for scheduling; Scheduler makes a task scheduling strategy based on the resource and task information and returns it to TaskManager;

[0015] Step S10: TaskManager makes a decision on subsequent operations for the task decision;

[0016] Step S11: If no scheduling strategy is made, the task will be added to the task waiting queue TaskQueue;

[0017] Step S12: If a decision is made on resources, VRF election calculation resources need to be performed based on the decision;

[0018] Step S13: Distribute the task message and the selected computing resources to the relevant participants and computing providers for consensus voting;

[0019] Step S14: The ConsensusEngine of each node performs task message consensus;

[0020] Step S15: If no consensus is reached on the task, the task continues to be accumulated in the task waiting queue TaskQueue and waits for the next scheduling;

[0021] Step S16: Once a consensus is reached on the task, the task will be triggered and started by the TaskManager of the scheduling service of each node;

[0022] Step S17: The TaskManager of the scheduling service notifies the TaskEngine of the computing service to start executing the task context;

[0023] Step S18: TaskEngine of the computing service starts executing the task context TaskContext to execute the task;

[0024] Step S19: The TaskEngine of the computing service feeds back the task execution result to the TaskManager of the scheduling service;

[0025] Step S20: The TaskManager of the scheduling service calls the storage module to store the result of the task.

[0026] As a preferred solution of the present invention, in each computing service in the present method, there is a background task monitoring process TaskMonitor Daemon to monitor the actual status of task resources, mainly monitoring TaskContext.

[0027] As a preferred solution of the present invention, in this method, there is a background task monitoring process TaskMonitor Daemon in the computing service, which will report the actual situation of task resource usage to the reporting engine ReportEngine of the computing service in a timely manner to report the actual situation of local resources.

[0028] As a preferred solution of the present invention, in this method, the reporting engine ReportEngine of the computing service further reports the resource usage of the task to the ResourceManager of the scheduling service in real time, and the ResourceManager updates the local ResourceTable table entry of the scheduling service.

[0029] As a preferred solution of the present invention, in this method, the Scheduler includes a StarveFIFO Scheduler, a Capacity Scheduler and a Fair Scheduler.

[0030] As a preferred solution of the present invention, in the design of StarveFIFO Scheduler, two queues are used to accumulate queued tasks, and a concept such as hunger value is set to determine the priority of task dequeuing.

[0031] As a preferred solution of the present invention, the detailed method for determining the task priority is as follows: a new task is submitted to TaskManager, and TaskManager calls StarveFIFO Scheduler to make a scheduling decision.

[0032] As a preferred solution of the present invention, in this method, the two queues inside StarveFIFO Schedule are respectively a normal task accumulation queue and a starvation task accumulation queue, and then each task waiting to be executed will carry a starvation value, and when the starvation value is greater than the minimum starvation threshold, the task will be executed preferentially.

[0033] As a preferred solution of the present invention, the task priority execution is divided into three situations as follows:

[0034] When a new task arrives, StarveFIFO Scheduler will first check whether the hungry task accumulation queue is empty. If it is not empty, the new task will be increased by a hunger value and added to the normal task accumulation queue, and a task with the largest hunger value will be popped out from the hungry task accumulation queue for execution; if the hungry task accumulation queue is empty, but the normal task accumulation queue is not empty, the new task will be increased by a hunger value and added to the normal task accumulation queue, and a head task will be popped out from the normal task accumulation queue for execution; if both queues are empty, the newly arrived task will be scheduled directly.

[0035] As a preferred solution of the present invention, in resource allocation during task scheduling, each node can customize its own minimum usable resource unit in resource management.

[0036] Compared with the prior art, the beneficial effects of the present invention are: the present invention can realize intelligent use of different scheduling engines for scheduling according to different application scenarios and different resource usage conditions through the use of different scheduling engines combined with scheduling processes, maximize and most rationally use system resources, and perform related scheduling work; and at the same time, it can report resource usage in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0038] Figure 1 It is a topological diagram of the method flow of the present invention;

[0039] Figure 2 The SchedulerEngine architecture composition diagram of the present invention;

[0040] Figure 3It is a schematic diagram of the working principle of SchedulerEngine and other modules and services of the present invention;

[0041] Figure 4 The Scheduler scheduling flow chart of the present invention;

[0042] Figure 5 This is a schematic diagram of resource allocation during task scheduling of the present invention. DETAILED DESCRIPTION

[0043] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] The present invention provides a hybrid scheduling method applied to privacy computing tasks, comprising the following steps:

[0045] Step S1: The user initiates a task, which is transferred to the scheduling service within the node;

[0046] Step S2: The task is transferred to the task parser TaskParser of the scheduling service;

[0047] Step S3: The task parser TaskParser parses the task and calls the task validator TaskValidator to verify the task content;

[0048] Step S4: If the task verification fails, the task process is terminated directly;

[0049] Step S5: The tasks that have passed the verification are handed over to the task manager TaskManager for management and scheduling;

[0050] Step S6: Determine whether to schedule a task from the queue or directly schedule the current task according to the task status in the TaskQueue;

[0051] Step S7: Submit the task to Scheduler for scheduling;

[0052] Step S8: Scheduler calls ResourceManager to obtain available resource information;

[0053] Step S9: ResourceManager uses the available resource table entry of ResourceTable to return to Scheduler for scheduling; Scheduler makes a task scheduling strategy based on the resource and task information and returns it to TaskManager;

[0054] Step S10: TaskManager makes a decision on subsequent operations for the task decision;

[0055] Step S11: If no scheduling strategy is made, the task will be added to the task waiting queue TaskQueue;

[0056] Step S12: If a decision is made on resources, VRF election calculation resources need to be performed based on the decision;

[0057] Step S13: Distribute the task message and the selected computing resources to the relevant participants and computing providers for consensus voting;

[0058] Step S14: The ConsensusEngine of each node performs task message consensus;

[0059] Step S15: If no consensus is reached on the task, the task continues to be accumulated in the task waiting queue TaskQueue and waits for the next scheduling;

[0060] Step S16: Once a consensus is reached on the task, the task will be triggered and started by the TaskManager of the scheduling service of each node;

[0061] Step S17: The TaskManager of the scheduling service notifies the TaskEngine of the computing service to start executing the task context;

[0062] Step S18: TaskEngine of the computing service starts executing the task context TaskContext to execute the task;

[0063] Step S19: The TaskEngine of the computing service feeds back the task execution result to the TaskManager of the scheduling service;

[0064] Step S20: The TaskManager of the scheduling service calls the storage module to store the result of the task;

[0065] The Scheduler Engine of this solution consists of TaskValidator, TaskParser, Scheduler, ResourceManager and TaskManager.

[0066] TaskParser: Task parser, responsible for parsing tasks.

[0067] TaskValidator: Task validator, responsible for validating the task.

[0068] ResourceManager: Resource manager, responsible for managing the internal resources of the node and counting the external resources of the node, and also managing the real-time service resource usage reported by the ReportEngine of the computing service. It allocates resources according to the tasks provided by the TaskManager.

[0069] TaskManager: Task manager, responsible for scheduling tasks, including initiating task consensus, issuing tasks, counting tasks, calling task storage, sorting tasks, queuing tasks, etc. It is used in conjunction with ResourceManager and Scheduler.

[0070] Scheduler: Task scheduler, there are three implementations in the design concept, and it can also support user-defined implementation in the future. The current design only has StarveFIFO Scheduler implementation. It mainly makes scheduling strategies for task scheduling.

[0071] For further information, see Figure 1 At A, in each computing service in this method, there is a background task monitoring process TaskMonitor Daemon to monitor the actual status of task resources, mainly monitoring TaskContex.

[0072] For further information, see Figure 1 At B, in this method, there is a background task monitoring process TaskMonitor Daemon in the computing service, which will report the actual situation of task resource usage to the reporting engine ReportEngine of the computing service in a timely manner to report the actual situation of local resources.

[0073] For further information, see Figure 1 At C, the reporting engine ReportEngine of the computing service in this method further reports the actual resource usage of the task to the ResourceManager of the scheduling service, and the ResourceManager updates the local ResourceTable entry of the scheduling service.

[0074] For further information, see Figure 4In this method, Scheduler includes the StarveFIFOScheduler, the Capacity Scheduler, and the Fair Scheduler. In the design of StarveFIFOScheduler, two queues are used to accumulate queued tasks, and a concept of hunger value is set to determine the priority of task dequeue. The detailed method for determining task priority is as follows: a new task is submitted to TaskManager, and TaskManager calls StarveFIFO Scheduler to make a scheduling decision. In this method, the two queues inside StarveFIFO Schedule are the normal task accumulation queue and the hunger task accumulation queue, and then each task waiting to be executed will carry a hunger value. When the hunger value is greater than the minimum hunger threshold, the task will be executed first. Task priority execution is divided into three situations as follows: When a new task arrives, StarveFIFO Scheduler will first check whether the hungry task accumulation queue is empty. If it is not empty, the new task will be increased by a hunger value and added to the normal task accumulation queue, and a task with the largest hunger value will be popped out from the hungry task accumulation queue for execution; if the hungry task accumulation queue is empty, but the normal task accumulation queue is not empty, the new task will be increased by a hunger value and added to the normal task accumulation queue, and a head task will be popped out from the normal task accumulation queue for execution; if both queues are empty, the newly arrived task will be scheduled directly.

[0075] For further information, see Figure 5 In the resource allocation during task scheduling, each node can customize its own minimum available resource unit in resource management. Here we use slot identification. As shown in the figure, a node divides the total resources of its internal computing service [memory: 32GB; kernel: 16cpu] into 16 slots, each of which represents [memory: 2GB; kernel: 1cpu]. Note that slots allow decimal points. If the remaining resource after resource division is less than one slot, it can be expressed as 0.x slots. At this time, the size is [memory: 7GB; kernel: 2cpu]. At this time, after calculation, the remaining resources of the node are 8 slots, and the slots required for the task are 4. The resources of the node will allocate the remaining 4 slots.

[0076] In summary, please refer to Figure 1-4The working principle of SchedulerEngine and other modules and services is as follows: The client's Task is submitted to the scheduling service, and the TaskParser of the Scheduler Engine of the scheduling service will first parse the Task, and TaskValidator will be called during the process to verify the task. The tasks that pass the verification will be transferred to TaskManager for management. TaskManager will schedule and distribute the tasks according to the scheduling strategy of Scheduler. When scheduling the Task, it will evaluate and calculate the resource usage of the Task. First, the resources given in the Task parameters and the node-defined minimum available unit of resources slot will be allocated to the Task and assembled into a Task distribution message. The Task distribution message forwards the Task to the TaskEngine of the corresponding computing node or data node through the rpc interface. When the computing node and the data node receive the computing task or data sharding task, they start TaskContext to execute the task. The computing node or data node monitors the actual usage of the Task through TaskMonitor Daemon, and reports the actual resource usage of each Task to the ResourceManager of the scheduling service through ReportEngine for real-time resource monitoring and management.

[0077] Exemplarily, the processor takes out instructions one by one from the memory, analyzes the instructions, and then completes the corresponding operations according to the instruction requirements, generating a series of control commands, so that the various parts of the computer automatically, continuously and coordinately move to become an organic whole, realize the input of programs, the input of data, and the calculation and output of results. The arithmetic operations or logical operations generated in this process are all completed by the operator; the memory includes a read-only memory (ROM), which is used to store computer programs, and a protection device is provided outside the memory.

[0078] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0079] Those skilled in the art will understand that the description of the above service equipment is merely an example and does not constitute a limitation on the terminal equipment. It may include more or fewer components than described above, or a combination of certain components, or different components, for example, it may include input and output devices, network access devices, buses, etc.

[0080] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire user terminal.

[0081] The memory can be used to store computer programs and / or modules. The processor can realize various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an information collection template display function, a product information release function, etc.); the data storage area can store data created according to the use of the berth status display system (such as product information collection templates corresponding to different product types, product information that different product providers need to release, etc.). In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0082] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the modules / units in the above-mentioned embodiment system, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can realize the functions of the above-mentioned various system embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0083] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0084] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A hybrid scheduling method for privacy computing tasks, characterized in that: The steps include: Step S1: The user initiates a task, which is transferred to the scheduling service within the node; Step S2: The task is transferred to the task parser TaskParser of the scheduling service; Step S3: The task parser TaskParser parses the task and calls the task validator TaskValidator to verify the task content; Step S4: If the task verification fails, the task process is terminated directly; Step S5: The tasks that have passed the verification are handed over to the task manager TaskManager for management and scheduling; Step S6: Determine whether to schedule a task from the queue or directly schedule the current task according to the task status in the TaskQueue; Step S7: Submit the task to Scheduler for scheduling; Step S8: Scheduler calls ResourceManager to obtain available resource information; Step S9: ResourceManager uses the available resource table entry of ResourceTable to return to Scheduler for scheduling; Scheduler makes a task scheduling strategy based on the resource and task information and returns it to TaskManager; Step S10: TaskManager makes a decision on subsequent operations for the task decision; Step S11: If no scheduling strategy is made, the task will be added to the task waiting queue TaskQueue; Step S12: If a decision is made on resources, VRF election calculation resources need to be performed based on the decision; Step S13: Distribute the task message and the selected computing resources to the relevant participants and computing providers for consensus voting; Step S14: The ConsensusEngine of each node performs task message consensus; Step S15: If no consensus is reached on the task, the task continues to be accumulated in the task waiting queue TaskQueue and waits for the next scheduling; Step S16: Once a consensus is reached on the task, the task will be triggered and started by the TaskManager of the scheduling service of each node; Step S17: The TaskManager of the scheduling service notifies the TaskEngine of the computing service to start executing the task context; Step S18: TaskEngine of the computing service starts executing the task context TaskContext to execute the task; Step S19: The TaskEngine of the computing service feeds back the task execution result to the TaskManager of the scheduling service; Step S20: The TaskManager of the scheduling service calls the storage module to store the result of the task.

2. A hybrid scheduling method for privacy computing tasks according to claim 1, characterized in that: In this method, each computing service has a background task monitoring process TaskMonitor Daemon to monitor the actual status of task resources, mainly monitoring TaskContex.

3. A hybrid scheduling method for privacy computing tasks according to claim 1, characterized in that: In this method, there is a background task monitoring process TaskMonitor Daemon in the computing service, which will report the actual situation of task resource usage to the reporting engine ReportEngine of the computing service in a timely manner to report the actual situation of local resources.

4. A hybrid scheduling method for privacy computing tasks according to claim 1, characterized in that: In this method, the reporting engine ReportEngine of the computing service further reports the real-time resource usage of the task to the ResourceManager of the scheduling service, and the ResourceManager updates the local ResourceTable entry of the scheduling service.

5. A hybrid scheduling method for privacy computing tasks according to claim 1, characterized in that: In this method, the Scheduler includes a StarveFIFO Scheduler, a Capacity Scheduler, and a Fair Scheduler.

6. A hybrid scheduling method for privacy computing tasks according to claim 5, characterized in that: In the design of StarveFIFO Scheduler, two queues are used to accumulate queued tasks, and a concept called hunger value is set to determine the priority of task dequeuing.

7. A hybrid scheduling method for privacy computing tasks according to claim 6, characterized in that: The detailed method for determining task priority is as follows: a new task is submitted to TaskManager, which calls StarveFIFOScheduler to make scheduling decisions.

8. A hybrid scheduling method for privacy computing tasks according to claim 7, characterized in that: In this method, the two queues inside StarveFIFO Schedule are the normal task accumulation queue and the hungry task accumulation queue. Then each task waiting to be executed will carry a hunger value. When the hunger value is greater than the minimum hunger threshold, the task will be executed first.

9. A hybrid scheduling method for privacy computing tasks according to claim 8, characterized in that: Task priority execution is divided into three situations as follows: When a new task arrives, StarveFIFO Scheduler will first check whether the hunger task accumulation queue is empty. If it is not empty, it will increase the hunger value of the new task and put it into the normal task accumulation queue, and pop out a task with the largest hunger value from the hunger task accumulation queue for execution; If the hungry task accumulation queue is empty, but the normal task accumulation queue is not empty, the new task will be increased by a hungry value and added to the normal task accumulation queue, and a head task will be popped out of the normal task accumulation queue for execution; if both queues are empty, the newly arrived task will be scheduled directly.

10. A hybrid scheduling method for privacy computing tasks according to claim 9, characterized in that: In resource allocation during task scheduling, each node can customize its own minimum usable resource unit in resource management.

Images