Task execution method and device based on privacy calculation, electronic equipment and storage medium

Abstract

The invention discloses a task execution method and device based on privacy computing, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps that a task controller obtains description information of a privacy computing task to be executed; creating a task pod according to the obtained description information of the privacy computing task; the central scheduler monitors that a task pod is created, and determines that a first computing node meeting a condition exists in the first Kubernetes cluster; the scheduler selects a task execution mode according to the task state; executing the privacy computing task, and storing and uploading an execution result; and after the calculation is finished, releasing related resources. According to the method and the device, the problem that the system is congested and even collapsed due to the fact that the participants compete for computing resources is avoided, the system stability of privacy computing is improved, the privacy computing task can be executed among the participants corresponding to different Kubernetes clusters, and the privacy computing efficiency is improved. And the utilization rate of cluster resources and the execution efficiency of privacy computing tasks can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task execution method, device, electronic device and storage medium based on privacy computing. Background Art

[0002] Kubernetes, abbreviated as k8s, is an open source, industrial-grade container orchestration platform. The emergence of k8s has promoted the popularization and implementation of popular technologies such as microservice architecture, making development, operation, maintenance and delivery easier and easier.

[0003] Privacy computing tasks often require the participation of multiple organizations. For the above-mentioned multiple participants, they may be located in different kubernetes (k8s) clusters respectively. This requires cross-cluster resource condition judgment to realize cross-cluster privacy task execution. Unlike traditional big data processing technology, privacy computing needs to be completed by multiple participants through computing interactions. This method requires multiple participants to coordinate computing resources, and the privacy computing task can only be successfully executed when each participant successfully executes their own computing logic.

[0004] Whether it is big data, machine learning or privacy computing, sufficient computing resources and robust services are the guarantee for the smooth execution of computing tasks. In the scenario where multiple participants collaborate to perform privacy computing tasks, when scheduling tasks, it is necessary to consider whether each participant has sufficient resources. Otherwise, some participants may be successfully scheduled, while others fail to be scheduled, which may lead to the failure of the entire privacy computing task.

[0005] In practical applications, multiple tasks compete for limited resources, which leads to computing system congestion or even system crash. Moreover, in the process of privacy computing by multiple participants, usually only the computing resources required by the participants to perform tasks are considered, while the computing resources required for interaction between multiple participants are not considered. As a result, the remaining resources of the participants are insufficient, and the privacy computing tasks cannot be performed normally, resulting in system congestion or even system crash. Since privacy computing tasks often require the participation of multiple institutions, how to ensure that tasks between different institutions can be efficiently scheduled has always been a hot research field in the industry. Standard k8s cannot meet the specific scenarios of multi-party privacy computing task scheduling.

[0006] Therefore, it is an urgent problem that technicians in this field need to solve to propose a task execution method, device, electronic device and storage medium based on privacy computing to solve the difficulties existing in the prior art. Summary of the invention

[0007] In view of this, the present invention provides a task execution method, device, electronic device and storage medium based on privacy computing, which avoids the problem of system congestion or even crash due to competition for computing resources among participants, improves the system stability of privacy computing, and can execute privacy computing tasks between multiple participants corresponding to different Kubernetes clusters, and can improve the utilization of cluster resources and the execution efficiency of privacy computing tasks.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A task execution method based on privacy computing involves a first participant and a second participant, which correspond to a first and a second Kubernetes cluster respectively, and is applied to a task controller and a central scheduler in the Kubernetes cluster, including:

[0010] S101, the task controller obtains description information of the privacy computing task to be executed, including: resource requirement information and task configuration information of the first participant and the second participant of the task to be executed;

[0011] S102. Create task pods for running the privacy computing task in the namespaces of the first and second participants, respectively, according to the description information of the privacy computing task obtained;

[0012] Specifically, in the first Kubernetes cluster, a first computing resource and a second computing resource for running the privacy computing task are created in the first and second namespaces corresponding to the first participant and the second participant, respectively;

[0013] The first computing resource includes: a first Pod group and a first state;

[0014] The second computing resource includes: a second Pod group and a second state;

[0015] S103. Synchronize the second computing resource to the second Kubernetes cluster through a synchronization mechanism established between the second namespace and the second participant;

[0016] S104, when the central scheduler monitors that the task pod is created and determines that there is a first computing node in the first Kubernetes cluster that satisfies the resource condition that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the value of the first state is set to the reserved state;

[0017] S105, obtaining a value of the second state from the second participant through a synchronization mechanism, and updating the first state and the second state to a schedulable state when it is determined that the values ​​of the first state and the second state are both reserved states;

[0018] S106. The central scheduler selects a task execution mode according to the task status and determines whether it is a streaming task; if so, a service is created through kubernetes to continuously receive the data stream of the streaming task; if not, a kubernetes image corresponding to the execution mode is called to schedule the first Pod group to the first computing node;

[0019] S107. Execute the privacy computing task, save the execution result, and upload the execution result to the blockchain for evidence storage through the smart contract;

[0020] S108. After the calculation is completed, relevant resources are released.

[0021] In the above method, optionally, in S102, a first task Pod scheduler corresponding to the first participant is deployed in the first Kubernetes cluster;

[0022] A first computing resource and a second computing resource are respectively created for running a privacy computing task, including:

[0023] Set the scheduler name field in the first Pod information to the first identifier of the first task Pod scheduler, so that the first Pod group is scheduled by the first task Pod scheduler;

[0024] The scheduler name in the second Pod information is set to other identifiers so that the second Pod group is not scheduled by the first task Pod scheduler.

[0025] In the above method, optionally, in S103, a first synchronization controller corresponding to the first participant is deployed in the first Kubernetes cluster, and a second synchronization controller corresponding to the second participant is deployed in the second Kubernetes cluster;

[0026] The synchronization mechanism is established through the following steps:

[0027] The second synchronization controller subscribes to the first synchronization controller for resource changes in the second namespace.

[0028] The above method optionally synchronizes the second computing resource to the second Kubernetes cluster, including: the first participant synchronization controller listens to the creation of the second computing resource, sends the second computing resource to the second participant synchronization controller, so that the second participant synchronization controller creates the received second computing resource under the namespace of the second participant in the second Kubernetes cluster.

[0029] In the above method, optionally, in S104, when it is detected that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the first computing resources are applied to the resource manager of the first participant, the first computing resources are deployed based on the first computing resources, and the second computing resources are applied to the resource manager of the second participant, so that the second participant deploys the second computing engine based on the second computing resources;

[0030] When it is detected that the remaining computing resources of the first participant are less than the first computing resources, or the remaining computing resources of the second participant are less than the second computing resources, a first message is returned to the task scheduler to prohibit the task scheduler from issuing the target task within a preset time period;

[0031] The target task is the task of the first party or the second party who performs the task.

[0032] In the above method, optionally, in S105, the task Pod status change obtained from the second participant is updated in the second namespace through a synchronization mechanism to indicate that the second Pod group is scheduled;

[0033] After the first participant and the second participant are updated to a schedulable state, the updated value of the second state is synchronized to the second Kubernetes cluster through a synchronization mechanism;

[0034] A first group of controllers corresponding to the first party is deployed in the first Kubernetes cluster;

[0035] Updating the first state and the second state to a schedulable state includes:

[0036] The first group of controllers monitors that the values ​​of the first state and the second state are set to the reserved state, and updates the values ​​of the first state and the second state to the schedulable state.

[0037] The above method optionally includes, after setting the value of the first state to the reserved state, when the first group of controllers determines that the timing reaches the first preset time length and does not hear that the value of the second state is set to the reserved state, setting the values ​​of the first state and the second state to the reservation resource failure state; and synchronizing the updated value of the second state to the second Kubernetes cluster through the synchronization mechanism;

[0038] After setting the value of the first state to the reserved state, the first group of controllers determines that the timing reaches the second preset time length, and if the values ​​of the first state and the second state are not heard to be set to a schedulable state, the values ​​of the first state and the second state are set to a failed state; and synchronizes the updated value of the second state to the second Kubernetes cluster through the synchronization mechanism.

[0039] In the above method, optionally, in S107, executing the privacy computing task and saving the execution result include:

[0040] S701, applying for a first computing resource from a resource manager of a first participant, and deploying a first computing engine based on the first computing resource, including:

[0041] Generate a task executor corresponding to the task metadata, where the task executor is used to represent an interface for abstracting the logic of privacy computing; deploy a first computing engine through the task executor, where the instance resource corresponding to the first computing engine is a first computing resource;

[0042] S702: Send the task metadata to the first computing engine, and interact with the second computing engine to perform privacy computing on the task to be executed, including:

[0043] Establishing a communication link between the first computing engine and the second computing engine, and sending the task metadata to the second computing engine through the communication link;

[0044] The first computing engine and the second computing engine perform interactive computing to obtain a privacy computing result corresponding to the task to be executed;

[0045] S703: polling the task execution status of the first computing engine performing privacy computing through the task executor, and storing the task execution status in the storage area corresponding to the first participant, and the task scheduler polling the task execution status from the storage area;

[0046] S704. Report the task execution status to the task scheduler. The task scheduler determines the target task status corresponding to the task to be executed according to the task execution status reported by the first participant and the second participant, and determines whether to issue the next task to the first participant and the second participant according to the target task status.

[0047] The above method is optional. S108, after the calculation is completed, release the relevant resources;

[0048] Specifically, the first computing engine is destroyed, and an engine destruction request is sent to the task controller of the second participant. The second computing engine is destroyed by the task controller of the second participant. A resource release instruction is sent to the resource manager of the first participant to release the first computing resource used by the first computing engine; a resource release request is sent to the resource manager of the second participant to release the second computing resource used by the second computing engine.

[0049] A task execution device based on privacy computing, executing any one of the task execution methods based on privacy computing described above, comprising:

[0050] The data acquisition module is used to obtain the task metadata corresponding to the tasks to be executed issued by the task scheduler;

[0051] A pod creation module is configured to create task pods for running privacy computing tasks in the namespaces of multiple participants.

[0052] A synchronization module, which synchronizes the second resource to a second Kubernetes cluster corresponding to the second participant through a synchronization mechanism, and synchronizes the value of the second state to the first Kubernetes cluster;

[0053] A resource determination module, which determines, based on the task metadata, a first computing resource required by a first participant to perform the task to be performed, and a second computing resource required by a second participant interacting with the first participant;

[0054] The engine deployment module, when detecting that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, applies to the resource manager of the first participant for the first computing resources, deploys the first computing engine based on the first computing resources, and applies to the resource manager of the second participant for the second computing resources, so that the second participant deploys the second computing engine based on the second computing resources;

[0055] The first processing module returns a first message to the task scheduler to prohibit the task scheduler from issuing the target task within a preset time period when detecting that the remaining computing resources of the first participant are less than the first computing resources or the remaining computing resources of the second participant are less than the second computing resources;

[0056] The task execution module sends the task metadata to the first computing engine and interacts with the second computing engine to perform privacy computing on the task to be executed;

[0057] The resource release module releases the first computing resources used by the first computing engine and the second computing resources used by the second computing engine after the computing is completed.

[0058] An electronic device comprises: a processor, a memory storing computer program instructions, and a communication interface; when the processor executes the computer program instructions, any one of the above-mentioned task execution methods based on privacy computing is implemented.

[0059] A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement any one of the above-mentioned privacy-based computing task execution methods.

[0060] It can be seen from the above technical solution that compared with the prior art, the present invention provides a task execution method, device, electronic device and storage medium based on privacy computing, which has the following beneficial effects: the present invention can realize cross-institutional pod scheduling in the scenario where multiple parties perform privacy computing tasks, and effectively improve the efficiency and resource utilization of cross-institutional scheduling; it fully considers the computing resources required in the calculation and interaction process, and privacy computing will only be performed between multiple participants when the computing resources are sufficient, avoiding the problem of system congestion or even crash of privacy computing caused by insufficient computing resources of the participants, and improving the stability of privacy computing system for privacy computing; privacy computing tasks can be executed between multiple participants corresponding to different Kubernetes clusters, and the utilization of cluster resources and the execution efficiency of privacy computing tasks can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0062] Figure 1 A flowchart of a task execution method based on privacy computing provided by the present invention;

[0063] Figure 2 A schematic diagram of the structure of a task execution device based on privacy computing provided by the present invention;

[0064] Figure 3 A schematic diagram of the structure of an electronic device for a task execution method based on privacy computing provided by the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0067] Reference Figure 1 As shown, the present invention discloses a task execution method based on privacy computing, involving a first participant and a second participant, corresponding to a first and a second Kubernetes cluster respectively, and applied to a task controller and a central scheduler in a Kubernetes cluster, including:

[0068] S101, the task controller obtains description information of the privacy computing task to be executed, including: resource requirement information and task configuration information of the first participant and the second participant of the task to be executed;

[0069] S102. Create task pods for running the privacy computing task in the namespaces of the first and second participants, respectively, according to the description information of the privacy computing task obtained;

[0070] Specifically, in the first Kubernetes cluster, a first computing resource and a second computing resource for running the privacy computing task are created in the first and second namespaces corresponding to the first participant and the second participant, respectively;

[0071] The first computing resource includes: a first Pod group and a first state;

[0072] The second computing resource includes: a second Pod group and a second state;

[0073] S103. Synchronize the second computing resource to the second Kubernetes cluster through a synchronization mechanism established between the second namespace and the second participant;

[0074] S104, when the central scheduler monitors that the task pod is created and determines that there is a first computing node in the first Kubernetes cluster that satisfies the resource condition that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the value of the first state is set to the reserved state;

[0075] S105, obtaining a value of the second state from the second participant through a synchronization mechanism, and updating the first state and the second state to a schedulable state when it is determined that the values ​​of the first state and the second state are both reserved states;

[0076] S106. The central scheduler selects a task execution mode according to the task status and determines whether it is a streaming task; if so, a service is created through kubernetes to continuously receive the data stream of the streaming task; if not, a kubernetes image corresponding to the execution mode is called to schedule the first Pod group to the first computing node;

[0077] S107. Execute the privacy computing task, save the execution result, and upload the execution result to the blockchain for evidence storage through the smart contract;

[0078] S108. After the calculation is completed, relevant resources are released.

[0079] Get the task metadata corresponding to the pending task issued by the task scheduler. The pending task is any task in the task flow. The task flow is a data flow for privacy computing described by a directed acyclic graph (DAG). Each node in the directed acyclic graph is an execution step, that is, a separate task. Each arrow in the directed acyclic graph represents a dependency relationship, pointing from the "upstream node" to the "downstream node", and the current node can only be executed when all upstream nodes of a node are executed successfully or there is no upstream node.

[0080] The above-mentioned tasks to be executed can be offline tasks or online services. Offline tasks include training tasks and prediction tasks of privacy computing models. The operators of offline tasks are executed only once, where the operators are a packaged privacy computing algorithm; online services are tasks that provide prediction services in the form of interfaces for the privacy computing models obtained through training.

[0081] In addition, the task metadata corresponding to the task to be executed is used to characterize the resource requirement information and task configuration information of multiple participants executing the task to be executed, wherein the resource requirement information may include but is not limited to shared resources or exclusive resources, queuing mode or competition mode, CPU (Central Processing Unit) requirements, memory requirements, etc.; the task configuration information includes at least the participants executing the task, operator type, input data, output data, etc.

[0082] Further, in S102, a first task Pod scheduler corresponding to the first participant is deployed in the first Kubernetes cluster;

[0083] A first computing resource and a second computing resource are respectively created for running a privacy computing task, including:

[0084] Set the scheduler name field in the first Pod information to the first identifier of the first task Pod scheduler, so that the first Pod group is scheduled by the first task Pod scheduler;

[0085] The scheduler name in the second Pod information is set to other identifiers so that the second Pod group is not scheduled by the first task Pod scheduler.

[0086] Further, in S103, a first synchronization controller corresponding to the first participant is deployed in the first Kubernetes cluster, and a second synchronization controller corresponding to the second participant is deployed in the second Kubernetes cluster;

[0087] The synchronization mechanism is established through the following steps:

[0088] The second synchronization controller subscribes to the first synchronization controller for resource changes in the second namespace.

[0089] Furthermore, synchronizing the second computing resource to the second Kubernetes cluster includes: the first participant synchronization controller listens to the creation of the second computing resource, and sends the second computing resource to the second participant synchronization controller, so that the second participant synchronization controller creates the received second computing resource under the namespace of the second participant in the second Kubernetes cluster.

[0090] The task scheduler can determine the tasks to be executed from the task flow according to the task scheduling algorithm, and send the tasks to be executed to the task controllers of each participant, so that the task controllers of each participant can perform resource scheduling, computing engine deployment and privacy computing according to the sent tasks.

[0091] Determine the first computing resource required by the first participant to perform the task to be performed and the second computing resource required by the second participant interacting with the first participant based on the task metadata; computing resources (including the first computing resource and the second computing resource) include but are not limited to resource information such as CPU, memory and system concurrency required to perform the task to be performed. Among them, computing resources can include exclusive resources and shared resources. Exclusive resources mean that the resources required for the execution of different tasks are isolated from each other, and the resources allocated to a task cannot be used by other tasks before the task is completed; while shared tasks mean that the execution of multiple tasks uses the same resources, and the resources that can be used by each task at a certain time are uncertain.

[0092] Further, in S104, when it is detected that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the first computing resources are applied to the resource manager of the first participant, the first computing resources are deployed based on the first computing resources, and the second computing resources are applied to the resource manager of the second participant, so that the second participant deploys the second computing engine based on the second computing resources;

[0093] When it is detected that the remaining computing resources of the first participant are less than the first computing resources, or the remaining computing resources of the second participant are less than the second computing resources, a first message is returned to the task scheduler to prohibit the task scheduler from issuing the target task within a preset time period;

[0094] The target task is the task of the first party or the second party who performs the task.

[0095] When it is detected that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the first computing resources are applied for from the resource manager of the first participant, a first computing engine is deployed based on the first computing resources, and the second computing resources are applied for from the resource manager of the second participant, so that the second participant deploys a second computing engine based on the second computing resources.

[0096] The computing engine (including the first computing engine and the second computing engine) is the infrastructure that supports the execution of privacy computing, which can realize calculations such as reading and writing files and databases, matrix operations, encryption and decryption of ciphertext, and network communication.

[0097] Further, in S105, the task Pod status change obtained from the second participant is updated in the second namespace through the synchronization mechanism, indicating that the second Pod group is scheduled;

[0098] After the first participant and the second participant are updated to a schedulable state, the updated value of the second state is synchronized to the second Kubernetes cluster through a synchronization mechanism;

[0099] A first group of controllers corresponding to the first party is deployed in the first Kubernetes cluster;

[0100] Updating the first state and the second state to a schedulable state includes:

[0101] The first group of controllers monitors that the values ​​of the first state and the second state are set to the reserved state, and updates the values ​​of the first state and the second state to the schedulable state.

[0102] Furthermore, after setting the value of the first state to the reserved state, the first group of controllers determines that the timing reaches the first preset time length, and does not listen to the setting of the value of the second state to the reserved state, and sets the values ​​of the first state and the second state to the reservation resource failure state; synchronizes the updated value of the second state to the second Kubernetes cluster through the synchronization mechanism;

[0103] After setting the value of the first state to the reserved state, the first group of controllers determines that the timing reaches the second preset time length, and if the values ​​of the first state and the second state are not heard to be set to a schedulable state, the values ​​of the first state and the second state are set to a failed state; and synchronizes the updated value of the second state to the second Kubernetes cluster through the synchronization mechanism.

[0104] Before performing privacy computing, the task controller of the first participant requests each participant to reserve computing resources, and the resource manager of each participant allocates resources according to the resource requirements in the task metadata. That is, the resource manager of each participant reserves computing resources for each participant to perform privacy computing.

[0105] The resource manager can provide a resource pool interface for abstract resource description and allocation methods, and provide a default resource allocation strategy. Users can also customize resource allocation strategies according to actual needs. The resource manager can also manage multiple resource pools at the same time to determine the computing resources required for the corresponding participants to implement privacy computing from multiple resource pools according to the resource allocation strategy. In the embodiment of the present application, the above-mentioned resource pool can be a hardware resource pool or a computing engine instance resource pool.

[0106] When a participant reserves computing resources, the participant will generate a resource allocation record to mark the computing resources as being "allocated". When the computing resources are released, the corresponding status of the computing resources will also be updated to "released".

[0107] Furthermore, in S107, executing the privacy computing task and saving the execution result include:

[0108] S701, applying for a first computing resource from a resource manager of a first participant, and deploying a first computing engine based on the first computing resource, including:

[0109] Generate a task executor corresponding to the task metadata, where the task executor is used to represent an interface for abstracting the logic of privacy computing; deploy a first computing engine through the task executor, where the instance resource corresponding to the first computing engine is a first computing resource;

[0110] S702: Send the task metadata to the first computing engine, and interact with the second computing engine to perform privacy computing on the task to be executed, including:

[0111] Establishing a communication link between the first computing engine and the second computing engine, and sending the task metadata to the second computing engine through the communication link;

[0112] The first computing engine and the second computing engine perform interactive computing to obtain a privacy computing result corresponding to the task to be executed;

[0113] S703: polling the task execution status of the first computing engine performing privacy computing through the task executor, and storing the task execution status in the storage area corresponding to the first participant, and the task scheduler polling the task execution status from the storage area;

[0114] S704. Report the task execution status to the task scheduler. The task scheduler determines the target task status corresponding to the task to be executed according to the task execution status reported by the first participant and the second participant, and determines whether to issue the next task to the first participant and the second participant according to the target task status.

[0115] In the process of multi-party privacy computing, not only the computing resources required for the first participant to perform the task to be performed are taken into account, but also the computing resources required for the two participants to interact. This avoids the problem of insufficient computing resources of the participants to perform privacy computing, resulting in system congestion or even crash of the privacy computing system, thereby improving the stability of the privacy computing system for privacy computing.

[0116] Further, in S108, after the calculation is completed, the relevant resources are released;

[0117] Specifically, the first computing engine is destroyed, and an engine destruction request is sent to the task controller of the second participant. The second computing engine is destroyed by the task controller of the second participant. A resource release instruction is sent to the resource manager of the first participant to release the first computing resource used by the first computing engine; a resource release request is sent to the resource manager of the second participant to release the second computing resource used by the second computing engine.

[0118] Reference Figure 2 As shown, a task execution device based on privacy computing executes any one of the task execution methods based on privacy computing described above, including:

[0119] The data acquisition module is used to obtain the task metadata corresponding to the tasks to be executed issued by the task scheduler;

[0120] A pod creation module is configured to create task pods for running privacy computing tasks in the namespaces of multiple participants.

[0121] A synchronization module, which synchronizes the second resource to a second Kubernetes cluster corresponding to the second participant through a synchronization mechanism, and synchronizes the value of the second state to the first Kubernetes cluster;

[0122] A resource determination module, which determines, based on the task metadata, a first computing resource required by a first participant to perform the task to be performed, and a second computing resource required by a second participant interacting with the first participant;

[0123] The engine deployment module, when detecting that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, applies to the resource manager of the first participant for the first computing resources, deploys the first computing engine based on the first computing resources, and applies to the resource manager of the second participant for the second computing resources, so that the second participant deploys the second computing engine based on the second computing resources;

[0124] The first processing module returns a first message to the task scheduler to prohibit the task scheduler from issuing the target task within a preset time period when detecting that the remaining computing resources of the first participant are less than the first computing resources or the remaining computing resources of the second participant are less than the second computing resources;

[0125] The task execution module sends the task metadata to the first computing engine and interacts with the second computing engine to perform privacy computing on the task to be executed;

[0126] The resource release module releases the first computing resources used by the first computing engine and the second computing resources used by the second computing engine after the computing is completed.

[0127] Reference Figure 3 As shown, an electronic device includes: a processor, a memory storing computer program instructions, and a communication interface; when the processor executes the computer program instructions, it implements any one of the above-mentioned task execution methods based on privacy computing.

[0128] A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement any one of the above-mentioned privacy-based computing task execution methods.

[0129] In a specific embodiment, Party A corresponds to Kubernetes cluster A, and Party B corresponds to Kubernetes cluster B. In Kubernetes cluster A, namespaces NamespaceA and NamespaceB are created for Party A and Party B, respectively. In Kubernetes cluster B, namespace NamespaceB is created for Party B. In addition, in Kubernetes cluster A, a task controller (TaskController), a synchronization controller (SyncController), a group controller (PartyGroupController), and a task Pod scheduler (KusciaScheduler) corresponding to Party A are also deployed. And in Kubernetes cluster B, a synchronization controller (SyncController) and a task Pod scheduler (KusciaScheduler) are also deployed.

[0130] Party A can receive the privacy computing task, and then the task controller therein can create the first and second resources for running the privacy computing task under NamespaceA and NamespaceB in Kubernetes cluster A respectively. Among them, the first resource includes Pod group A and PartyResourceA, and the second resource includes Pod group B and PartyResourceB. PartyResourceA and PartyResourceB here are customized resources CR for storing resource status. Afterwards, the second resource can be synchronized in the two clusters through the synchronization controllers in Kubernetes cluster A and Kubernetes cluster B. In addition, the resource status stored by the above-mentioned PartyResourceA and PartyResourceB can be set or updated through the group controller in Kubernetes cluster A.

[0131] In addition, when the resource status stored in PartyResourceA and PartyResourceB is in a schedulable state, the first Pod group can be scheduled by the task Pod scheduler in Kubernetes cluster A, and the second Pod group created in Kubernetes cluster B can be scheduled by the task Pod scheduler in Kubernetes cluster B, so as to realize the collaborative execution of privacy computing tasks between participants A and B.

[0132] In actual applications, in Kubernetes cluster B, a namespace NamespaceA corresponding to participant A may also be created, and task controllers, group controllers, etc. may be deployed therein, which is not limited in this specification.

[0133] Finally, the number of the above-mentioned parties is not limited to two, and can be three or more.

[0134] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0135] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A task execution method based on privacy computing, characterized in that: Involving a first participant and a second participant, corresponding to a first and a second Kubernetes cluster respectively, and applied to a task controller and a central scheduler in the Kubernetes cluster, including: S101, the task controller obtains description information of the privacy computing task to be executed, including: resource requirement information and task configuration information of the first participant and the second participant of the task to be executed; S102. Create task pods for running the privacy computing task in the namespaces of the first and second participants, respectively, according to the description information of the privacy computing task obtained; Specifically, in the first Kubernetes cluster, a first computing resource and a second computing resource for running the privacy computing task are created in the first and second namespaces corresponding to the first participant and the second participant, respectively; The first computing resource includes: a first Pod group and a first state; The second computing resource includes: a second Pod group and a second state; S103. Synchronize the second computing resource to the second Kubernetes cluster through a synchronization mechanism established between the second namespace and the second participant; S104, when the central scheduler monitors that the task pod is created and determines that there is a first computing node in the first Kubernetes cluster that satisfies the resource condition that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the value of the first state is set to the reserved state; S105, obtaining a value of the second state from the second participant through a synchronization mechanism, and updating the first state and the second state to a schedulable state when it is determined that the values ​​of the first state and the second state are both reserved states; S106. The central scheduler selects a task execution mode according to the task status and determines whether it is a streaming task; if so, a service is created through kubernetes to continuously receive the data stream of the streaming task; if not, a kubernetes image corresponding to the execution mode is called to schedule the first Pod group to the first computing node; S107. Execute the privacy computing task, save the execution result, and upload the execution result to the blockchain for evidence storage through the smart contract; S108. After the calculation is completed, relevant resources are released.

2. According to the task execution method based on privacy computing according to claim 1, it is characterized in that: In S102, a first task Pod scheduler corresponding to the first participant is deployed in the first Kubernetes cluster; A first computing resource and a second computing resource are respectively created for running a privacy computing task, including: Set the scheduler name field in the first Pod information to the first identifier of the first task Pod scheduler, so that the first Pod group is scheduled by the first task Pod scheduler; The scheduler name in the second Pod information is set to other identifiers so that the second Pod group is not scheduled by the first task Pod scheduler.

3. According to the task execution method based on privacy computing of claim 1, it is characterized in that: In S103, a first synchronization controller corresponding to the first participant is deployed in the first Kubernetes cluster, and a second synchronization controller corresponding to the second participant is deployed in the second Kubernetes cluster; The synchronization mechanism is established through the following steps: The second synchronization controller subscribes to the resource changes in the second namespace from the first synchronization controller; Synchronizing the second computing resource to the second Kubernetes cluster includes: the first participant synchronization controller listens to the creation of the second computing resource, and sends the second computing resource to the second participant synchronization controller, so that the second participant synchronization controller creates the received second computing resource under the namespace of the second participant in the second Kubernetes cluster.

4. According to the task execution method based on privacy computing of claim 1, it is characterized in that: In S104, when it is detected that the remaining computing resources of the first participant are greater than or equal to the first computing resources, and the remaining resources of the second participant are greater than or equal to the second computing resources, the first computing resources are applied to the resource manager of the first participant, the first computing resources are deployed based on the first computing resources, and the second computing resources are applied to the resource manager of the second participant, so that the second participant deploys the second computing engine based on the second computing resources; When it is detected that the remaining computing resources of the first participant are less than the first computing resources, or the remaining computing resources of the second participant are less than the second computing resources, a first message is returned to the task scheduler to prohibit the task scheduler from issuing the target task within a preset time period; The target task is the task of the first party or the second party who performs the task.

5. According to the task execution method based on privacy computing of claim 1, it is characterized in that: In S105, through the synchronization mechanism, the task Pod status change obtained from the second participant is updated in the second namespace to indicate that the second Pod group is scheduled; After the first participant and the second participant are updated to a schedulable state, the updated value of the second state is synchronized to the second Kubernetes cluster through a synchronization mechanism; A first group of controllers corresponding to the first party is deployed in the first Kubernetes cluster; Updating the first state and the second state to a schedulable state includes: The first group of controllers monitors that the values ​​of the first state and the second state are set to the reserved state, and updates the values ​​of the first state and the second state to the schedulable state; After setting the value of the first state to the reserved state, the first group of controllers determines that the timing reaches the first preset time and does not hear that the value of the second state is set to the reserved state, and sets the values ​​of the first state and the second state to the reservation resource failure state; synchronizes the updated value of the second state to the second Kubernetes cluster through the synchronization mechanism; After setting the value of the first state to the reserved state, the first group of controllers determines that the timing reaches the second preset time, and if the values ​​of the first state and the second state are not heard to be set to a schedulable state, the values ​​of the first state and the second state are set to a failed state; and synchronizes the updated value of the second state to the second Kubernetes cluster through a synchronization mechanism.

6. The task execution method based on privacy computing according to claim 1, characterized in that: In S107, the privacy computing task is executed and the execution result is saved, including: S701, applying for a first computing resource from a resource manager of a first participant, and deploying a first computing engine based on the first computing resource, including: Generate a task executor corresponding to the task metadata, where the task executor is used to represent an interface for abstracting the logic of privacy computing; deploy a first computing engine through the task executor, where the instance resource corresponding to the first computing engine is a first computing resource; S702: Send the task metadata to the first computing engine, and interact with the second computing engine to perform privacy computing on the task to be executed, including: Establishing a communication link between the first computing engine and the second computing engine, and sending the task metadata to the second computing engine through the communication link; The first computing engine and the second computing engine perform interactive computing to obtain a privacy computing result corresponding to the task to be executed; S703: polling the task execution status of the first computing engine performing privacy computing through the task executor, and storing the task execution status in the storage area corresponding to the first participant, and the task scheduler polling the task execution status from the storage area; S704. Report the task execution status to the task scheduler. The task scheduler determines the target task status corresponding to the task to be executed according to the task execution status reported by the first participant and the second participant, and determines whether to issue the next task to the first participant and the second participant according to the target task status.

7. The task execution method based on privacy computing according to claim 1, characterized in that: S108, after the calculation is completed, relevant resources are released; Specifically, a destruction operation is performed on the first computing engine, and an engine destruction request is sent to the task controller of the second participant, the task controller of the second participant destroys the second computing engine, and a resource release instruction is sent to the resource manager of the first participant to release the first computing resource used by the first computing engine; A resource release request is sent to the resource manager of the second participant to release the second computing resource used by the second computing engine.

8. A task execution device based on privacy computing, executing the task execution method based on privacy computing according to any one of claims 1 to 7, characterized in that: include: The data acquisition module is used to obtain the task metadata corresponding to the tasks to be executed issued by the task scheduler; A pod creation module is configured to create task pods for running privacy computing tasks in the namespaces of multiple participants. A synchronization module, which synchronizes the second resource to a second Kubernetes cluster corresponding to the second participant through a synchronization mechanism, and synchronizes the value of the second state to the first Kubernetes cluster; A resource determination module, which determines, based on the task metadata, a first computing resource required by a first participant to perform the task to be performed, and a second computing resource required by a second participant interacting with the first participant; The engine deployment module, when detecting that the remaining computing resources of the first participant are greater than or equal to the first computing resources and the remaining resources of the second participant are greater than or equal to the second computing resources, applies for the first computing resources from the resource manager of the first participant, deploys the first computing engine based on the first computing resources, and applies for the second computing resources from the resource manager of the second participant, so that the second participant deploys the second computing engine based on the second computing resources; The first processing module returns a first message to the task scheduler to prohibit the task scheduler from issuing the target task within a preset time period when detecting that the remaining computing resources of the first participant are less than the first computing resources or the remaining computing resources of the second participant are less than the second computing resources; The task execution module sends the task metadata to the first computing engine and interacts with the second computing engine to perform privacy computing on the task to be executed; The resource release module releases the first computing resources used by the first computing engine and the second computing resources used by the second computing engine after the computing is completed.

9. An electronic device, characterized in that: The electronic device includes: a processor, a memory storing computer program instructions, and a communication interface; when the processor executes the computer program instructions, it implements the task execution method based on privacy computing as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: Computer program instructions are stored on a computer-readable storage medium, and when the computer program instructions are executed by a processor, a task execution method based on privacy computing as described in any one of claims 1 to 7 is implemented.

Images