A cross-platform scheduling method of heterogeneous privacy computing platform algorithms

Abstract

The application discloses a cross-platform scheduling method of a heterogeneous privacy computing platform algorithm, and the method is based on Kubernetes and comprises the following steps: constructing a standardized container image for each algorithm of the heterogeneous privacy computing platform; registering an algorithm component comprising a standardized container image address and image metadata tags; receiving a job request comprising a DAG configuration, wherein the DAG configuration defines a plurality of tasks and task dependency relationships of the job request, and the task comprises an algorithm name and a task role; obtaining the algorithm component according to the algorithm name; judging whether the running role of the algorithm component contains the task role, and if not, the scheduling fails; generating a task execution sequence based on the task dependency relationship; judging whether the upstream task output data type of the task and the input data type of the task are matched, and if not, the scheduling fails; and if the scheduling does not fail, creating an isolated Pod for the task. The constraint condition of the image metadata tags ensures cross-platform compatibility, so that the algorithms of the heterogeneous privacy computing platform can be seamlessly deployed across platforms.

Description

Technical Field

[0001] This invention relates to scheduling methods for algorithms, and particularly to a cross-platform scheduling method for algorithms on heterogeneous privacy computing platforms. Background Technology

[0002] Common heterogeneous privacy-preserving computing platforms include the federated learning framework FATE (Federated AI Technology Enabler), Secure Multi-Party Computation (MPC), and the hardware-based Trusted Execution Environment (TEE). FATE, developed by WeBank's AI team, is the world's first industrial-grade open-source federated learning framework. It provides a secure computing framework based on data privacy protection, offering robust secure computing support for machine learning, deep learning, and transfer learning algorithms, and includes built-in protection for various machine learning algorithms, including linear models, tree models, and neural networks. The Trusted Execution Environment (TEE) is a hardware-based privacy protection method, referring to a secure area built on a computing platform using hardware and software methods. It ensures the confidentiality and integrity of code and data loaded within this secure area. Secure Multi-Party Computation (MPC), through the construction of a series of cryptographic tools and a decentralized deployment architecture, strictly protects data privacy from leaving the domain, enabling multi-party collaborative modeling in general scenarios.

[0003] However, privacy computing platforms such as FATE, TEE, and MPC employ closed frameworks, preventing their algorithms from being deployed across platforms. For example, the FATE algorithm's runtime environment may be based on a specific Python version, such as python:3.8-slim, a specific deep learning framework, such as PyTorch 1.13, and specific FATE libraries: fate_client, fate_flow, and numerous dependent Python packages such as numpy, pandas, and scikit-learn. In contrast, the TEE algorithm requires specific SGX drivers, SGXSDK / PSW, Gramine or Occlum LibOS, and potentially specially compiled and linked libraries and applications that utilize the SGX runtime. Data transfer between FATE tasks heavily relies on FATE-Flow's internal protocols and serialization formats, which may be Protobuf or custom binary formats. A FATE algorithm's expected input and output formats are FATE-specific; directly outputting them to a TEE algorithm may result in the TEE algorithm being unable to parse them. Applications within the TEE, such as Gramine / Occlum, may also have their own data formats and secure transmission requirements. Therefore, heterogeneous privacy computing platforms have significantly different interface protocols. When the algorithm components of these privacy computing platforms are deployed across platforms in the existing way, the large differences in dependent libraries and interface protocols result in high deployment costs, with a single algorithm migration exceeding 80 person-days. Summary of the Invention

[0004] To address the shortcomings of existing technologies, there is an urgent need for a cross-platform scheduling method for heterogeneous privacy computing platform algorithms that can solve the problems of environmental isolation and protocol compatibility.

[0005] This invention provides a cross-platform scheduling method for heterogeneous privacy computing platform algorithms. The method is based on Kubernetes and is characterized by comprising: Standardized container images are constructed for each algorithm of the heterogeneous privacy computing platform. The standardized container images include algorithm applications and algorithm runtime environments. Register an algorithm component that includes the standardized container image address and image metadata tag. The image metadata tag is used to declare the running constraints of the algorithm component. The running constraints include running role, input data type, and output data type. Receive a job request that includes a DAG configuration. The DAG configuration defines several tasks and task dependencies of the job request. The tasks include algorithm name and task role. Obtain the algorithm component based on the algorithm name; Determine whether the running role of the algorithm component includes the task role; if not, the scheduling fails. A task execution sequence is generated based on the aforementioned task dependencies; Determine whether the output data type of the upstream task in the task execution sequence matches the input data type of the task. If they do not match, the scheduling fails. If scheduling does not fail, an isolated Pod is created for the task. The isolated Pod includes: The standardized container image is pulled using the standardized container image address of the algorithm component; A container is launched based on the obtained standardized container image, and the container is used to perform the task.

[0006] Furthermore, launching a container based on the standardized container image includes: When starting a container, specified environment variables are dynamically injected into the process environment of the container runtime. The specified environment variables include parameter values ​​determined according to the task.

[0007] Furthermore, the parameter values ​​include at least one of the following: data path, participant ID, participant address information, and machine learning algorithm parameters.

[0008] Furthermore, the machine learning algorithm parameters include at least one of the following: maximum number of iterations, learning rate, batch size, regularization coefficient, and random seed.

[0009] Furthermore, the parameter values ​​are obtained based on the DAG configuration or parameters submitted by the user.

[0010] Furthermore, the algorithm's runtime environment includes the libraries, runtime, configuration, and environment variables that the algorithm depends on.

[0011] Furthermore, the standardized container image is named as follows: / vendor name / algorithm name:version.

[0012] Furthermore, it also includes: Create a task instance for each task in the task execution sequence, and initialize the task state of the task to PENDING; After starting the container, update the task status of the task to RUNNING; If the task is successfully completed, the task status will be updated to SUCCESS. If the task fails, the task status will be updated to FAILED.

[0013] Furthermore, after the container starts successfully, a dual-channel mechanism is used to monitor the task status, the dual-channel mechanism including: The task status is updated in real time based on WebSocket and by polling the task status using HTTP.

[0014] Furthermore, the method also includes: When the task status of the task is FAILED, or when the idle resources occupied by the Pod of the task exceed the time threshold, self-healing resource reclamation is performed; the self-healing resource reclamation includes: automatically destroying the container, and / or releasing the CPU and memory, and / or reclaiming the storage volume.

[0015] The technical effects of this invention are as follows: By providing key metadata information about algorithm components to the scheduling system through the image metadata tag LABEL, including role assignment and input / output management, the scheduling system can perform necessary constraint checks before and during scheduling. By using the metadata such as input / output types and roles defined in LABEL, the complex internal protocols of heterogeneous privacy computing platforms, such as the FATE-Flow protocol and SGX memory access, are abstracted into standardized interfaces that the scheduling system can understand and operate. The scheduling system only needs to perform scheduling and resource management based on these white-box metadata, without needing to understand the complex protocol details inside the black box, thus achieving cross-platform protocol compatibility. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the cross-platform scheduling method for the heterogeneous privacy computing platform algorithm provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Kubernetes (K8s) is an open-source container orchestration engine used to automate the deployment, scaling, and management of containerized applications.

[0020] The following is combined Figure 1This invention provides a flowchart illustrating a cross-platform scheduling method for a heterogeneous privacy computing platform algorithm based on Kubernetes, as illustrated in an embodiment of the present invention. The method may include: S1. Build standardized container images for each algorithm of the heterogeneous privacy computing platform. The standardized container images include algorithm applications and algorithm runtime environments.

[0021] To address the issue of algorithm components being unable to run on heterogeneous platforms, this invention employs a dual-mode container encapsulation approach. Dual-mode container encapsulation includes white-box protocol abstraction and black-box full packaging. Black-box full packaging refers to completely packaging the entire FATE, MPC, TEE, and other algorithms, along with their specific environments, into a single container image. The scheduling system treats this as an indivisible black-box unit for execution. This is the primary method for resolving environment isolation.

[0022] Taking the FATE and TEE algorithms as examples, building a standardized container image for the FATE algorithm includes: installing the specific libraries and dependencies required by FATE on the base image, such as python:3.8-slim, using the command: pip install fate_client fate_flow torch==1.13... The completed standardized container image for the FATE algorithm, fate_hetero_lr:v1.0, contains everything required for the FATE algorithm in its runtime environment. Building a standardized container image for the TEE algorithm includes: installing the algorithm code, SGXSDK, and necessary LibOS components on a base image that supports SGX, such as occlum / occlum:0.29.3-ubuntu20.04 or gramineproject / gramine, and compiling and configuring the algorithm. The completed standardized container image for the TEE algorithm, tee_hetero_lr:v1.0, contains everything required for the TEE algorithm in its runtime environment.

[0023] Meanwhile, this invention provides a unified image naming convention, standardizing image naming as: / vendor name / algorithm name:version, such as insight / hetero_lr:v1.0. This unified image naming allows the scheduling system to uniquely and clearly identify the algorithm component that needs to be run. For example, seeing that a task requires the hetero_lr algorithm, it knows to pull the / insight / hetero_lr:v1.0 image. When the FATE algorithm needs to be run on a node of a heterogeneous privacy computing platform, the node scheduler only needs to instruct the node's K8s / KubeEdge to pull and run the standardized container image fate_hetero_lr:v1.0. The node only needs to have the Docker / containerd runtime and the infrastructure required by FATE, such as the network access FATE-Flow service and storage volume mount points; it does not need to globally install all FATE dependencies beforehand. Similarly, when running the TEE algorithm, the standardized container image tee_hetero_lr:v1.0 is pulled and run. The node only needs to support the TEE hardware and drivers, but does not need to globally install the algorithm-specific SGX application environment. Standardized container images provide environmental isolation, ensuring that algorithms run in the expected, isolated environment. A Node is the unit of work in a Kubernetes cluster; it is an abstraction of computing resources such as physical machines, virtual machines, and VPS, belonging to the infrastructure layer. It is used to run Pods (container groups), provide CPU, memory, and storage resources, and receive Kubernetes control plane scheduling instructions.

[0024] S2. Register the algorithm component, including the standardized container image address and image metadata tag, on the heterogeneous privacy computing platform. The image metadata tag is used to declare the running constraints of the algorithm component, including the running role, input data type, and output data type.

[0025] The image metadata label (LABEL) must include at least the running role, input data type, and output data type: `LABEL component.role_list = "guest,host"` # Defines the running role; `LABEL component.input = "train_data"` # Defines the input data type; `LABEL component.output_types = "model"` # Defines the output data type. The `LABEL component.role_list` defines the running roles, specifically declaring the roles this algorithm component is designed to play. For example, in federated learning scenarios like FATE, the Guest typically possesses the labeled data (i.e., the initiator or client), while the Host possesses the feature data but lacks labels (i.e., the server or collaborator). Roles determine the task's responsibilities and data flow within the federation. The actual LABEL label values ​​should be based on the role definitions supported by the algorithm; for example, the coordinator (arbiter) might also be a role. Roles determine which node the task runs on and may also affect the environment variables or mounted storage volumes required by the container runtime when accessing data from different roles.

[0026] The `LABEL component.input_types` property defines the input data types, specifically declaring what type of input data this algorithm component requires at runtime. "train_data" is a type identifier representing the raw or feature data used for training, not a specific filename. Specific data instances, such as files or directories, are dynamically bound by the scheduling system based on the DAG configuration during task runtime. Other possible types include "test_data" (test data), "pre_model" (pre-trained model), and "validation_data" (validation data).

[0027] When parsing a Directed Acyclic Graph (DAG), the scheduling system checks the task inputs. For example, if task B in the DAG depends on the output of task A, and task B's algorithmic component has `LABEL component.input_types="model"`, then the scheduler optimizer will check if task A's algorithmic component has `LABEL component.output_types="model"`. This is a crucial step in input-output matching verification to ensure the compatibility of the DAG data flow. Simultaneously, when creating a Pod, the task context determines the types of data the task requires, such as `train_data` and `model` type data. Combined with this tag information, it can correctly mount the corresponding storage volumes to the specified paths within the container: ` / opt / input / train.csv`, ` / opt / model / previous.model`. The source of the mounted data is specified by the output of the upstream task in the DAG or the initial user input.

[0028] The `LABEL component.output_type` defines the output data type, specifically declaring what type of output data this algorithm component will generate after execution. "model" represents a trained model file or object; other possible types include "metrics" (evaluation metrics) and "transformed_data" (transformed data). Besides matching and validating with the `input_types` of downstream tasks, `LABELcomponent.output_types` is also used to write the algorithm container's output, such as model files, to a specified mount directory, such as ` / opt / output / model`, after computation. The scheduling or storage system is responsible for persistently storing the contents of type "model" in this directory and registering them as data available for downstream tasks. Using `LABEL component.output_types` informs the scheduling system what the algorithm will produce after execution, ensuring proper management of task output and demonstrating protocol compatibility.

[0029] Therefore, the image metadata tag (LABEL) provides the scheduling system with key metadata about the algorithm components, enabling it to perform necessary constraint checks before and during scheduling: role assignment, input / output management, etc., which is crucial for achieving automation and compatibility. The following steps will detail how to utilize the image metadata tag (LABEL) for these necessary constraint checks.

[0030] In step S2, the image metadata tag corresponds to the white-box protocol abstraction in the dual-mode container encapsulation. White-box protocol abstraction refers to using metadata such as input_types, output_types, and role_list defined in the LABEL to abstract complex internal platform protocols, such as the FATE-Flow protocol and SGX memory access, into standardized interfaces that the scheduling system can understand and operate. This includes type contracts and file mount points. The scheduling system only needs to perform scheduling and resource management based on this white-box metadata, without needing to understand the complex protocol details inside the black box. This is the core mechanism for solving cross-platform protocol compatibility and data interaction compatibility.

[0031] The mirror metadata tag (LABEL) differs from the input and output of a Directed Acyclic Graph (DAG): a DAG defines the logical data dependencies between tasks, where the output (output_A) of task A is the input (input_B) of task B. In this invention, the mirror metadata tag (LABEL) defines the physical data interface types supported by the algorithm component itself. The scheduling system utilizes the logical relationships of the DAG and the physical interface definitions of the LABEL to ensure that the logical data flow finds compatible data types for transmission and mounting, enabling the logical data flow to be physically implemented.

[0032] By setting up the LABEL of this invention, all FATE, TEE, and MPC algorithms must declare their input and output types through the LABEL. During DAG parsing, the scheduling system only checks the data type matching, that is, it only checks whether task A's output_types="model" matches task B's input_types="model". The actual data transmission protocol is handled by the underlying storage abstraction layer; the scheduling system does not concern itself with how FATE or TEE internally reads and writes data. The LABEL also provides a standardized storage interface, such as a POSIX file system or an S3-compatible object storage API. Each algorithm container only needs to be responsible for: reading data files of its declared input data type from a mounted input directory, such as / opt / input, and writing data of its declared output data type to the mounted output directory: / opt / output. The algorithm developers ensure that the algorithm internally knows how to parse the data files, and the encryption format is also handled internally by the algorithm image. The scheduling system only ensures that files are placed and retrieved according to data type and path; ensuring compatibility of data formats of the same type is sufficient for scheduling. Therefore, the data types defined by LABEL become the contract for data interaction between cross-platform algorithms. As long as platforms adhere to this contract when reading and writing specific types of data, differences in internal platform implementations can be masked, achieving white-box protocol abstraction. This invention abstracts complex platform-specific protocols into simple tags, achieving cross-platform protocol compatibility.

[0033] S3. Receive a job request including DAG configuration. The DAG configuration defines several tasks and task dependencies of the job request. The tasks include the name of the algorithm to be called and the task role. In Kubernetes, a Directed Acyclic Graph (DAG) refers to an orchestration model used to describe and execute a series of steps or tasks with explicit dependencies between them.

[0034] The initiating user submits a job request including a DAG configuration, and the scheduling system receives the DAG configuration in JSON format. Each task "component" in the DAG represents a specific privacy-preserving computation algorithm module, and its upstream and downstream dependencies are defined through inputs and outputs, thereby constructing a complete task flow.

[0035] An example of “component” is: "component":[{"name":"hetero_lr","role":"guest","input":[{"type":"dataset"}]}].

[0036] name: Represents the algorithm name, used to find the corresponding algorithm implementation; role: indicates the task's role; The meaning of “input”:[{“type”:“dataset”} is to input a custom data resource of type dataset into the container.

[0037] S4. Obtain the algorithm components based on the algorithm name of the task; The algorithm name, heteroo_lr, is obtained from the name field of components. Then, the algorithm name is mapped to the algorithm component, which includes the standardized container image address corresponding to the algorithm name, such as / insight / hetero_lr:v1.0. When the scheduling system distributes tasks to nodes, it will push the standardized container image address to the target node.

[0038] S5. Determine whether the running role of the algorithm component includes the task role. If not, the scheduling fails. When resolving the DAG configuration, the DAG specifies a task role, such as role:guest. The scheduling optimizer of the scheduling system checks whether the LABEL component.role_list of the algorithm component mapped to the task contains this role. If LABELcomponent.role_list="guest,host" contains the task role role:guest, the verification passes. If the image does not support this role, scheduling will fail. Passing the role permission verification is a crucial step, enabling constraint verification in advance to determine whether to continue scheduling.

[0039] Simultaneously, based on the task's role in the DAG configuration and the node's role label in the Kubernetes cluster: `kubectl LABELnode node -A role=host`, tasks with the role: host are assigned to nodes marked with role=host: `node_A`, and tasks with the role: guest are assigned to nodes marked with role=guest: `node_B`. Therefore, task role is also the basis for assigning tasks based on role. Taking federated computing as an example, the functional difference between Host and Guest nodes lies in the data they hold and their role in the federated computing protocol. A Guest node might be responsible for calculating loss and gradients, while a Host node is responsible for calculating local gradients and safely aggregating them.

[0040] S6. Generate task execution sequences based on task dependencies configured by DAG; When parsing the DAG configuration, a task execution sequence is generated based on the task dependencies in the DAG configuration. The task execution sequence is a sequence of tasks with upstream and downstream relationships.

[0041] S7. Determine whether the output data type of the upstream task in the task execution sequence matches the input data type of the task. If they do not match, the scheduling fails. For each task in the task execution sequence, the system checks whether the output data type of its upstream task matches its input data type. When parsing a Directed Acyclic Graph (DAG), the scheduling system examines the input of each task. For example, if task B in the DAG depends on the output of task A, and task B's algorithm component has `LABEL component.input_types="model"`, then the scheduler optimizer checks if task A's algorithm component has `LABEL component.output_types="model"`. If it does, the upstream task matches this task, and the verification passes; otherwise, the upstream task does not match this task, and scheduling fails. This input-output matching verification is a crucial step to ensure the compatibility of the DAG data stream.

[0042] S8. If scheduling does not fail, create an isolated Pod for the task.

[0043] Creating isolated Pods includes: Use the standardized container image address of the algorithm component to pull the standardized container image; Launch a container based on the obtained standardized container image; the container is then used to perform tasks.

[0044] This invention verifies the running role and validates the input and output data types by using the image metadata tag (LABEL), enabling correct task scheduling between heterogeneous privacy computing platforms. Once the verification is successful, the Kubernetes cluster can start containers and run tasks on the nodes of each heterogeneous privacy computing platform.

[0045] In Kubernetes, standardized container images are stored in an image repository. When a container is started, the node scheduler calls the Kubernetes API to send a Pod creation request to the Kubernetes cluster. After receiving the Pod creation request, the Kubernetes cluster pulls the image from the image repository using the standardized container image address, such as / insight / hetero_lr:v1.0. The image repository returns the image, and the Kubernetes cluster then starts the container on the corresponding node.

[0046] When starting a container, this invention injects different parameters into the same algorithm image through a dynamic parameter injection mechanism. Specifically, when starting a container, specified environment variables are dynamically injected into the process environment of the container runtime. These specified environment variables include parameter values ​​determined according to the task.

[0047] Dynamic parameter injection is performed by the node scheduler when creating a Pod by calling the Kubernetes API. In Kubernetes, dynamic injection typically sets environment variables through the `env` field in the Pod Spec. It can also be used in conjunction with ConfigMaps. For example, a set of basic configurations can be placed in a ConfigMap, and task-specific override parameters can be dynamically injected using `env.valueFrom`.

[0048] Here is an example of dynamic parameter injection: `docker run -e max_iter=50hetero_lr_image data_path= / input / train.csv` # Injects the number of iterations and the data path at startup.

[0049] Here, `hetero_lr_image` represents the image of the `hetero_lr` algorithm, named according to previous specifications, such as ` / insight / hetero_lr:v1.0`. In the `docker run` command, it specifies which image the container instance to be run is based on. `max_iter` represents the maximum number of iterations, a common hyperparameter for machine learning algorithms such as logistic regression (`hetero_lr`). After the algorithm program starts within the container, it reads the value of `MAX_ITER` (environment variable names are usually uppercase) (e.g., 50) from the environment variables and uses this value to control the number of iteration rounds during training. This is the runtime configuration of the algorithm and directly affects its behavior. For the same algorithm image, such as `hetero_lr:v1.0`, different `max_iter` values ​​can be injected into different DAG task instances to run the algorithm. This eliminates the need to build a new image or define multiple `ConfigMap`s for each different parameter combination, greatly improving flexibility and reusability. `max_iter` values ​​can be, for example, 50, 100, or 200. Here, `data_path` is the data path. Although the data path is often specified by mounting a volume, it can also be parameterized. Other common dynamically injected parameters include: learning rate, batch size, regularization coefficient, random seed, on / off or mode parameters for specific algorithms, and participant IDs or address information injected by the scheduler in federated scenarios.

[0050] Traditional Kubernetes (Kubernetes) uses ConfigMaps and Secrets to manage configurations. These are well-suited for storing relatively static or sensitive configuration data, such as database connection strings, API keys, and configuration file contents. Configurations typically need to be defined before Pod creation and mounted as files or set as environment variables. Updating a ConfigMap requires rebuilding the Pod or using mechanisms like sidecars to trigger application reconfiguration. This invention, however, employs dynamic parameter injection. During task execution, the scheduling system dynamically injects parameter values ​​into the container's environment variables based on the specific task instance's requirements. These requirements can originate from DAG configurations or user-submitted parameters. Since parameters are injected at container startup and read at application startup, they take effect immediately. This eliminates the need to manage complex configuration reload logic. Furthermore, the same algorithm image can run in different DAG task instances by injecting different parameter values, without needing to build a new image or define multiple ConfigMaps for each different parameter combination, significantly improving flexibility and reusability.

[0051] To address the issue of state asynchrony between heterogeneous privacy computing platforms, this invention maps the various complex states of these platforms to four core states: PENDING, RUNNING, SUCCESS, and FAILED. For example, "computing in progress" on the FATE platform is mapped to RUNNING, and "interrupted" on the TEE platform is mapped to FAILED. This allows all nodes to map their local task states to these four core states, forming a unified state language that facilitates state synchronization and monitoring management, and avoids scheduling system misjudgments caused by semantic ambiguity.

[0052] The process for switching between the four states of a task is as follows: After DAG parsing, a task instance is created for each task in the task execution sequence, and the task state is initialized to PENDING. After starting the container, update the task status to RUNNING; If the task is completed successfully, the task status will be updated to FAILED; If the task fails, the task status will be updated to FAILED.

[0053] After the container starts successfully, that is, after the container executes the task, a dual-channel mechanism is used to monitor the task status. The dual-channel mechanism includes: Task status updates are reported in real time via WebSocket and task status is polled via HTTP.

[0054] When actively reporting, the WebSocket protocol is used to actively report the task status update in real time. When the network is smooth, the delay is ≤1 second, which can realize real-time push status. At the same time, the scheduling system queries the task status every 5 seconds via the HTTP protocol, and can still accurately obtain the task status when the node crashes or the network loses packets.

[0055] The present invention also determines that if the HTTP polling task is unresponsive, or the HTTP polling timeout occurs, or the task execution timeout occurs, or the CPU utilization rate is >95% for 5 minutes, or the algorithm execution error occurs, the task status is forcibly marked as FAILED and self-healing resource reclamation is triggered.

[0056] When a task's status is FAILED, or when the task's Pod has been idle for more than a time threshold, self-healing resource reclamation is performed. Self-healing resource reclamation includes: automatically destroying containers, and / or releasing CPU and memory, and / or reclaiming storage volumes. Preferably, the resource idle time threshold is set to more than 15 minutes.

[0057] Adopting a unified task status can ensure global consistency. Using a dual-channel mechanism to monitor task status can more quickly identify task status and accelerate fault location, thereby accelerating fault recovery. Automatic recycling is achieved through self-healing resource recycling, which significantly improves resource recycling efficiency.

[0058] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-platform scheduling method for heterogeneous privacy computing platform algorithms, the method being based on Kubernetes, characterized in that, include: Standardized container images are constructed for each algorithm of the heterogeneous privacy computing platform. The standardized container images include algorithm applications and algorithm runtime environments. Register an algorithm component that includes the standardized container image address and image metadata tag. The image metadata tag is used to declare the running constraints of the algorithm component. The running constraints include running role, input data type, and output data type. Receive a job request that includes a DAG configuration. The DAG configuration defines several tasks and task dependencies of the job request. The tasks include algorithm name and task role. Obtain the algorithm component based on the algorithm name; Determine whether the running role of the algorithm component includes the task role; if not, the scheduling fails. A task execution sequence is generated based on the aforementioned task dependencies; Determine whether the output data type of the upstream task in the task execution sequence matches the input data type of the task. If they do not match, the scheduling fails. If scheduling does not fail, an isolated Pod is created for the task. The isolated Pod includes: The standardized container image is pulled using the standardized container image address of the algorithm component; A container is launched based on the obtained standardized container image, and the container is used to perform the task.

2. The method according to claim 1, characterized in that, The step of starting a container based on the standardized container image includes: when starting the container, dynamically injecting specified environment variables into the process environment of the container runtime, wherein the specified environment variables include parameter values ​​determined according to the task.

3. The method according to claim 2, characterized in that, The parameter values ​​include at least one of the following: data path, participant ID, participant address information, and machine learning algorithm parameters.

4. The method according to claim 2, characterized in that, Machine learning algorithm parameters include at least one of the following: maximum number of iterations, learning rate, batch size, regularization coefficient, and random seed.

5. The method according to claim 2, characterized in that, The parameter values ​​are obtained based on the DAG configuration or parameters submitted by the user.

6. The method according to claim 1, characterized in that, The algorithm's runtime environment includes the libraries, runtime environment, configuration environment, and environment variables that the algorithm depends on.

7. The method according to claim 1, characterized in that, The standardized container image is named as follows: / vendor name / algorithm name:version.

8. The method according to claim 1, characterized in that, Also includes: Create a task instance for each task in the task execution sequence, and initialize the task state of the task to PENDING; After starting the container, update the task status of the task to RUNNING; If the task is successfully completed, the task status will be updated to SUCCESS. If the task fails, the task status will be updated to FAILED.

9. The method according to claim 8, characterized in that, After the container starts successfully, a dual-channel mechanism is used to monitor the task status. The dual-channel mechanism includes: The task status is updated in real time based on WebSocket and by polling the task status using HTTP.

10. The method according to claim 8, characterized in that, The method further includes: When the task status of the task is FAILED, or when the idle resources occupied by the Pod of the task exceed the time threshold, self-healing resource reclamation is performed; the self-healing resource reclamation includes: automatically destroying the container, and / or releasing the CPU and memory, and / or reclaiming the storage volume.

Images