Resource scheduling method, device, equipment, medium and product

Abstract

The invention discloses a resource scheduling method, device and equipment, a medium and a product. The method comprises the steps of obtaining a total privacy calculation task load, total response time corresponding to the total privacy calculation task load, a total privacy budget corresponding to the total privacy calculation task load, accumulated privacy budget consumption, a global instantaneous privacy budget consumption rate and a physical resource index of a calculation cluster; determining a target resource scheduling strategy according to the total privacy calculation task load, the total response time corresponding to the total privacy calculation task load, the total privacy budget corresponding to the total privacy calculation task load, the accumulated privacy budget consumption, the global instantaneous privacy budget consumption rate and the physical resource index of the calculation cluster; according to the technical scheme, the privacy budget can be prevented from being exhausted too early, and it is guaranteed that the privacy calculation task is completed efficiently and reliably.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and in particular to a resource scheduling method, device, equipment, medium and product. BACKGROUND

[0002] Cloud native technology has become a key driving force for enterprises to achieve efficient innovation and business agile development. Cloud native, a technology system that combines containerization, microservices, automation, and continuous delivery, has brought unprecedented changes to the development, deployment, and management of applications. Elastic scaling, as one of the core capabilities of cloud native architecture, is a typical representative of the cloud native elastic scaling solution. The mainstream cloud native elastic scaling solution, as a general technology, can monitor the load of the application in real time based on CPU usage, memory usage, request rate, and other indicators, and automatically and dynamically adjust the number of Pods to ensure that the application is always in the best running state and efficiently cope with various complex business scenarios.

[0003] The cloud native elastic scaling solution has three fundamental technical defects:

[0004] First, it lacks privacy awareness capability, and blind expansion can sharply accelerate privacy budget consumption, causing the budget to be depleted before the task is completed, resulting in task failure due to privacy protection failure and waste of computing resources and time.

[0005] Second, the decision-making goal deviates from the demand for privacy computing, and its optimization goal is general performance and resource utilization, rather than optimal performance under the premise of meeting privacy security, and the decision-making logic is inconsistent with the fundamental goal of privacy computing.

[0006] Third, there is no budget depletion warning mechanism, the system cannot perceive the consumption trend and provide a warning, and the administrator cannot intervene in advance and can only respond passively to task failure. SUMMARY

[0007] Embodiments of the present application provide a resource scheduling method, device, equipment, medium and product to solve at least one of the above problems.

[0008] According to an aspect of the present application, a resource scheduling method is provided, comprising:

[0009] obtaining total privacy computing task quantity, total response time corresponding to total privacy computing task quantity, total privacy budget corresponding to total privacy computing task quantity, cumulative privacy budget consumption, global instantaneous privacy budget consumption rate, and physical resource indicators of a computing cluster;

[0010] The target resource scheduling strategy is determined based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0011] Resource scheduling is performed based on the target resource scheduling strategy.

[0012] According to another aspect of the present invention, a resource scheduling apparatus is provided, the resource scheduling apparatus comprising:

[0013] The acquisition module is used to acquire the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0014] The determination module is used to determine the target resource scheduling strategy based on the total amount of privacy computing tasks, the total response time corresponding to the total amount of privacy computing tasks, the total privacy budget corresponding to the total amount of privacy computing tasks, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0015] The execution module is used to perform resource scheduling based on the target resource scheduling strategy.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the resource scheduling method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the resource scheduling method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the resource scheduling method as described in any of the embodiments of the present invention.

[0022] This invention addresses the technical problem of premature privacy budget depletion caused by blind scaling in general solutions, ensuring the compliance and effectiveness of privacy computing tasks. Furthermore, this embodiment uses the privacy budget as a hard constraint and response time as the optimization target, optimizing performance while prioritizing privacy security, rather than simply pursuing performance or resource utilization. This approach better aligns with the real-world needs of privacy computing scenarios.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a resource scheduling method according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of a resource scheduling system according to an embodiment of the present invention;

[0027] Figure 3 This is a flowchart of another resource scheduling method in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a resource scheduling device according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0033] Example 1

[0034] Figure 1 This is a flowchart illustrating a resource scheduling method provided in an embodiment of the present invention. This embodiment is applicable to resource scheduling situations. The method can be executed by the resource scheduling device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0035] S110, obtain the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0036] In this embodiment, the total privacy computing task volume refers to the total number of privacy computing tasks, which can be a preset value. The total response time corresponding to the total privacy computing task volume is a preset time to complete all privacy computing tasks. For example, if the total number of privacy computing tasks is 5, then the total response time corresponding to the total privacy computing task volume is the time to complete 5 privacy computing tasks. The total privacy budget corresponding to the total privacy computing task volume is a preset total privacy budget.

[0037] In this embodiment, the cumulative privacy budget consumption is the cumulative privacy budget consumption at the current moment. For example, if the current time has passed the time length corresponding to 3 monitoring time windows, then the sum of the privacy budget consumed by all query subtasks in the privacy computing task executed in the 3 monitoring time windows is taken as the cumulative privacy budget consumption.

[0038] In this embodiment, the global instantaneous privacy budget consumption rate can be obtained as follows: obtain the monitoring time window size, the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computing task within the monitoring time window, and the total number of operator instances executing the privacy computing task; determine the instantaneous privacy budget consumption rate of each operator instance based on the monitoring time window size and the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computing task within the monitoring time window; and determine the global instantaneous privacy budget consumption rate based on the total number of operator instances executing the privacy computing task and the instantaneous privacy budget consumption rate of each operator instance.

[0039] In this embodiment, the physical resource metrics of the computing cluster include at least one of the following: average CPU utilization of CPUs occupied by each operator instance, average memory utilization of memory occupied by each operator instance, and privacy computing tasks executed.

[0040] Optionally, obtain the global instantaneous privacy budget consumption rate, including:

[0041] Obtain the size of the monitoring time window, the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computation task within the monitoring time window, and the total number of operator instances executing the privacy computation task.

[0042] In this embodiment, the monitoring time window size can be a pre-set time length. The privacy budget consumed by each operator instance in executing all query subtasks within the privacy computation task within the monitoring time window can be obtained from the Privacy Budget Monitor (PM). The total number of operator instances executing privacy computation tasks is also the total number of active operator instances.

[0043] The instantaneous privacy budget consumption rate of each operator instance is determined based on the size of the monitoring time window and the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computation task within the monitoring time window.

[0044] In this embodiment, the instantaneous privacy budget consumption rate of each operator instance can be determined based on the size of the monitoring time window and the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computing task within the monitoring time window. The ratio of the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computing task within the monitoring time window to the size of the monitoring time window can be used as the instantaneous privacy budget consumption rate of each operator instance.

[0045] The global instantaneous privacy budget consumption rate is determined based on the total number of operator instances performing privacy computation tasks and the instantaneous privacy budget consumption rate of each operator instance.

[0046] In this embodiment, the global instantaneous privacy budget consumption rate can be determined by taking the sum of the instantaneous privacy budget consumption rates of each operator instance performing the privacy computation task as the global instantaneous privacy budget consumption rate.

[0047] Optionally, the physical resource metrics of the computing cluster include: the average CPU utilization of the CPU used by each operator instance, the average memory utilization of the memory used by each operator instance, and the privacy computing tasks executed.

[0048] S120: Determine the target resource scheduling strategy based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0049] In this embodiment, the target resource scheduling strategy is determined based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster. The method is as follows: Based on the total privacy computing task volume and the executed privacy computing tasks, determine the predicted remaining time corresponding to the total privacy computing task volume; use the difference between the total response time corresponding to the total privacy computing task volume and the cumulative privacy computing task response time as the target remaining time; use the difference between the total privacy budget corresponding to the total privacy computing task volume and the cumulative privacy budget consumption as the remaining privacy budget; use the ratio of the remaining privacy budget to the global instantaneous privacy budget consumption rate as the predicted privacy budget exhaustion time; and determine the target resource scheduling strategy based on the predicted privacy budget exhaustion time, the target remaining time, and the predicted remaining time corresponding to the total privacy computing task volume.

[0050] Optionally, based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster, a target resource scheduling strategy is determined, including:

[0051] Based on the total privacy computing task volume and the physical resource indicators of the computing cluster, the predicted remaining time corresponding to the total privacy computing task volume is determined.

[0052] In this embodiment, the method for determining the predicted remaining time corresponding to the total privacy computing task volume based on the total privacy computing task volume and the physical resource indicators of the computing cluster can be as follows: determine the predicted remaining time corresponding to the total privacy computing task volume based on the total privacy computing task volume and the privacy computing tasks already executed.

[0053] In a specific example, the predicted remaining time corresponding to the total privacy computing task is calculated based on the following formula: ,in, At the current processing speed, The predicted remaining time for the total privacy computation task, i.e., at the current processing speed. Complete the remaining data volume The required time. It can be estimated based on the privacy computing tasks performed and historical data. The current processing speed can be determined based on the total number of privacy-enhancing computation tasks and the number of privacy-enhancing computation tasks already executed. It should be noted that the method for estimating the current processing speed based on executed privacy-enhancing computation tasks and historical data can be as follows: If the historical data includes the processing speed of each type of privacy-enhancing computation task, then based on the executed privacy-enhancing computation tasks and the total number of privacy-enhancing computation tasks, determine the privacy-enhancing computation tasks not yet executed; query the processing speed of each type of privacy-enhancing computation task to obtain the processing speed corresponding to the privacy-enhancing computation tasks, and use the average of the processing speeds corresponding to the privacy-enhancing computation tasks as the current processing speed. If historical data includes the total processing time for privacy-preserving computation tasks, then the ratio of the amount of data for privacy-preserving computation tasks to the total processing time for those tasks will be used as the current processing speed. .

[0054] The difference between the total response time corresponding to the total amount of privacy computing tasks and the cumulative response time of privacy computing tasks is taken as the target remaining time.

[0055] The difference between the total privacy budget corresponding to the total amount of privacy computing tasks and the cumulative privacy budget consumption is used as the remaining privacy budget.

[0056] The ratio of the remaining privacy budget to the global instantaneous privacy budget consumption rate is used as the predicted privacy budget exhaustion time.

[0057] The target resource scheduling strategy is determined based on the predicted privacy budget exhaustion time, the target remaining time, and the predicted remaining time corresponding to the total privacy computing task volume.

[0058] In this embodiment, the method for determining the target resource scheduling strategy based on the predicted privacy budget exhaustion time and the cumulative privacy computing task response time corresponding to the total privacy computing task volume can be as follows: If the predicted remaining time corresponding to the total privacy computing task volume is greater than the target remaining time, the predicted privacy budget exhaustion time is greater than the predicted remaining time corresponding to the total privacy computing task volume, the average CPU utilization of the CPU occupied by each operator instance is less than the CPU utilization threshold, the average memory utilization of the memory occupied by each operator instance is less than the memory utilization threshold, and the first condition is met, then the target resource scheduling strategy is determined to be a scaling strategy; if the predicted privacy budget exhaustion time is less than or equal to the target remaining time, then... The target resource scheduling strategy is determined to maintain the current resource state and issue an early warning. If the difference between the predicted remaining time and the target remaining time corresponding to the total privacy computing task volume is greater than the first difference threshold, the difference between the predicted privacy budget exhaustion time and the target remaining time is greater than the second difference threshold, the average CPU utilization of each operator instance at the current time is less than the average CPU utilization of each operator instance at the previous time, and the average memory utilization of each operator instance at the current time is less than the average memory utilization of each operator instance at the previous time, then the target resource scheduling strategy is determined to be a scaling-down strategy. If none of the above conditions are met, the current state is maintained.

[0059] Optionally, based on the predicted privacy budget exhaustion time, the target remaining time, and the predicted remaining time corresponding to the total privacy computing task volume, a target resource scheduling strategy is determined, including:

[0060] If the predicted remaining time corresponding to the total privacy computing task volume is greater than the target remaining time, the predicted privacy budget exhaustion time is greater than the predicted remaining time corresponding to the total privacy computing task volume, the average CPU utilization of each operator instance is less than the CPU utilization threshold, the average memory utilization of each operator instance is less than the memory utilization threshold, and the first condition is met, then the target resource scheduling strategy is determined to be the expansion strategy.

[0061] In this embodiment, the first condition includes: the sum of the remaining privacy budget and the first privacy budget is less than or equal to the product of the total privacy budget corresponding to the total privacy computing task volume and the first value, wherein the first privacy budget is determined based on the predicted privacy budget consumption rate after expansion and the target remaining time.

[0062] In this embodiment, the scaling strategy can be to increase the number of operator instances. The specific number of operator instances to be added can be a pre-set value, or it can be determined based on the adequacy of the privacy budget. The adequacy of the privacy budget can be determined based on the difference between the predicted privacy budget exhaustion time and the predicted remaining time corresponding to the total privacy computing task volume.

[0063] If the predicted time for the privacy budget to run out is less than or equal to the target remaining time, the target resource scheduling strategy is determined to keep the current resource state unchanged, and an early warning is issued.

[0064] In this embodiment, if the predicted privacy budget exhaustion time is less than or equal to the target remaining time, it indicates that the quantification boundary of privacy protection has been breached before the task is completed. The target remaining time is the difference between the total response time corresponding to the total amount of privacy computing tasks and the cumulative response time of the privacy computing tasks, which is the estimated time required to complete the current privacy computing task (such as the remaining training rounds of federated learning or the number of data queries).

[0065] In this embodiment, if the predicted privacy budget exhaustion time is less than or equal to the target remaining time, the target resource scheduling strategy is determined to keep the current resource state unchanged, an early warning is issued, and any expansion operation is prohibited to prioritize security. The system can then switch to a degraded mode.

[0066] This embodiment provides a proactive early warning mechanism. It can predict and alert administrators in advance the risk of privacy budget exhaustion, giving them valuable buffer time for manual intervention, avoiding losses caused by sudden failures of privacy computing tasks, and improving the reliability and maintainability of the system.

[0067] If the difference between the target remaining time and the predicted remaining time corresponding to the total privacy computing task volume is greater than the first difference threshold, the difference between the predicted privacy budget exhaustion time and the target remaining time is greater than the second difference threshold, the average CPU utilization of each operator instance at the current time is less than the average CPU utilization of each operator instance at the previous time, and the average memory utilization of each operator instance at the current time is less than the average memory utilization of each operator instance at the previous time, then the target resource scheduling strategy is determined to be a scaling-down strategy.

[0068] In this embodiment, the first difference threshold can be a preset value, and the second difference threshold can be a preset value.

[0069] It should be noted that if the average CPU utilization of each operator instance at the current moment is less than the average CPU utilization of each operator instance at the previous moment, and the average memory utilization of each operator instance at the current moment is less than the average memory utilization of each operator instance at the previous moment, then it indicates that the resource utilization is low.

[0070] In this embodiment, if the predicted remaining time corresponding to the total privacy computing task is much less than the target remaining time, the predicted privacy budget exhaustion time is much greater than the target remaining time, the average CPU utilization of each operator instance at the current time is less than the average CPU utilization of each operator instance at the previous time, and the average memory utilization of each operator instance at the current time is less than the average memory utilization of each operator instance at the previous time, then the target resource scheduling strategy is determined to be a scaling-down strategy.

[0071] In this embodiment, the scaling-down strategy can be to reduce the number of operator instances. The specific number of operator instances to be reduced can be a pre-set value, or it can be determined based on performance and budget. Performance can be determined based on the difference between the target remaining time and the predicted remaining time corresponding to the total privacy computing task volume, and budget can be determined based on the difference between the predicted privacy budget exhaustion time and the target remaining time.

[0072] Optionally, the first condition includes:

[0073] The sum of the remaining privacy budget and the first privacy budget is less than or equal to the product of the total privacy budget corresponding to the total privacy computing task volume and the first value, wherein the first privacy budget is determined based on the predicted privacy budget consumption rate after expansion and the target remaining time.

[0074] In this embodiment, the first value can be a pre-set safety factor.

[0075] It should be noted that the first condition ensures that the increased privacy consumption remains within the safe range of the total budget.

[0076] S130, Perform resource scheduling based on the target resource scheduling strategy.

[0077] The technical solution provided in this embodiment can overcome the shortcomings of existing general scaling solutions, and provide a resource scheduling method that can sense and make decisions based on the privacy budget consumption rate, take privacy and security as rigid constraints, and intelligently balance performance and security, thereby avoiding the premature exhaustion of the privacy budget and ensuring that privacy computing tasks are completed efficiently and reliably.

[0078] In a specific example, a resource scheduling system is provided that overcomes the fundamental shortcomings of general-purpose scalability solutions in privacy-preserving computing scenarios by introducing a dynamic feedback loop with the Privacy Budget Consumption Rate (PBCR) as the core control variable. The core of the resource scheduling system lies in its decision-making process, which is a real-time optimization problem seeking the optimal solution within a three-dimensional constraint space of (privacy security × performance × resources). Figure 2As shown, the Task Manager (TM) is the system's entry point, responsible for receiving and parsing user-submitted privacy computing tasks. Its core function is to extract the user-defined dual constraint objective: the total response time corresponding to the total amount of privacy computing tasks. Total privacy computing tasks and the corresponding total privacy budget , forming a constrained binary tuple Subsequently, it generates a globally unique identifier (job ID) for the task and sends the task context (job ID, Φ) to the dynamic decision engine, thus providing a clear optimization objective and security boundary for the closed-loop control of the entire system. The Privacy Budget Monitor (PM) is the cornerstone of achieving "privacy-aware" capabilities and is deeply integrated into each operator instance. Its core function is to perform real-time, quantitative measurement of privacy consumption. For a fixed monitoring time window... PM will accurately calculate the privacy budget consumed by all query subtasks in the privacy computation task executed by sub-instance i within the monitoring time window. And calculate the instantaneous privacy budget consumption rate of operator instance i. , Subsequently, all clusters in the aggregation cluster The instantaneous privacy budget consumption rate of each operator instance performing privacy computation tasks is used to obtain the global instantaneous privacy budget consumption rate. , and tuple in, This accumulates privacy budget consumption. It is continuously reported to the dynamic decision engine, providing security-related data for decision-making. The Resource Monitor (RM) is responsible for monitoring the status of traditional cloud computing resources and acts as a performance sensor. Its function is to continuously collect physical resource utilization and performance metrics of the computing cluster and organize them into a vector of physical resource metrics for the computing cluster. This mainly includes the average CPU utilization of the CPU used by operator instance k. Average memory utilization of operator instance k and privacy computing tasks already performed These metrics are reported to the Dynamic Decision Engine (DDE) to assess the system's current performance load and determine if sufficient resources are available for scaling operations. The DDE is the system's intelligent hub, responsible for executing core multi-constraint optimization algorithms. It receives inputs from the TM, PM, and RM. , , , ), and determine the cumulative privacy computing task response time. First, calculate the key predicted values, including the remaining time to the target. Remaining privacy budget Predict when privacy budget will run out And the predicted remaining time corresponding to the predicted total privacy computing task volume. Subsequently, it is based on rules (e.g., only when...). , And physical resources are abundant Only when the first condition is met. Under the premise of triggering expansion, joint reasoning is performed, and finally the optimal decision is output, which is the scaling decision instruction D(k), which directs the system behavior.

[0079] The Lifecycle Manager (LCM) and Data Allocator (DAE) are the system's execution mechanisms. The LCM receives instructions D(k) from the DDE and, by calling the underlying cloud infrastructure's management interface, executes the creation or destruction of operator instances, thereby changing the instance count from N(k) to N(k+1). The DAE, upon changing the instance count, immediately performs dynamic repartitioning and load balancing of the data to be processed, ensuring that data flows are automatically and efficiently distributed to all active operator instances (including newly created ones). This ensures that scaling up or down operations are truly effective and avoids creating new performance bottlenecks.

[0080] like Figure 3 As shown, the system's workflow is a dynamic control closed loop operating at a fixed period Δt, and its data flow and decision logic are... Figure 2 The architecture shown is tightly coupled, and the specific process is as follows: Step 1: State awareness: At the beginning of each decision cycle k, such as Figure 3 As shown, the dynamic decision engine receives data streams from various monitoring sources: it obtains the user-defined raw constraints from the TaskManager (TM). , This represents the total response time corresponding to the total number of privacy-preserving computation tasks. The total privacy budget corresponds to the total amount of privacy-preserving computation tasks; obtain the physical resource indicator vector of the computing cluster from the resource monitor. , The average CPU utilization of operator instance k. The average memory utilization rate of the memory occupied by operator instance k. The privacy computation task has been performed; obtain tuples representing the privacy and security status from the Privacy Budget Monitor (PM). , For the global instantaneous privacy budget consumption rate, To accumulate privacy budget consumption, , , Let i be the instantaneous privacy budget consumption rate of operator instance i. The number of operator instances required to perform privacy-preserving computation tasks. The privacy budget consumed by operator instance i during the execution of all query subtasks in the privacy computation task within the monitoring time window. This is to monitor the size of the time window. Simultaneously, the Dynamic Decision Engine (DDE) internally calculates the time elapsed for privacy-preserving computation tasks, which is the cumulative response time of the privacy-preserving computation tasks. This step completes a comprehensive sampling of the system's current state in the "performance-resources-security" three-dimensional state space.

[0081] Step 2: Prediction and Evaluation: After aggregating all state data, DDE then performs prediction functions, the core of which is calculating two key time indicators related to the final decision: remaining time to the target. and remaining privacy budget ,in, Remaining time for the target This represents the total response time corresponding to the total number of privacy-preserving computation tasks. To accumulate the response time of privacy computing tasks, For the remaining privacy budget, The total privacy budget corresponds to the total amount of privacy-preserving computation tasks. This calculates the cumulative privacy budget consumption. Then, based on the current state, a linear extrapolation is performed to predict when the privacy budget will be exhausted. , This represents the global instantaneous privacy budget consumption rate. It also indicates the point in time when the privacy budget will be exhausted if the consumption rate remains constant; and the predicted remaining time corresponding to the total privacy computation workload. This means that at the current processing speed (can be based on) (and historical throughput estimation) Complete the remaining data volume Time required. (Based on...) The direct comparison constitutes the fundamental basis for subsequent decision-making logic.

[0082] Step 3: Multi-Constraint Decision Making: This step is the core algorithm of DDE (Distributed Decision Analysis). It solves for the optimal decision D(k) under multiple constraints based on a series of conditional rules. For example...Figure 3 As shown, after completing the calculation, DDE generates instructions and outputs them to the execution component. Its decision rules are as follows:

[0083] Rule 1 (Secure Expansion): When performance is not up to standard ( ), sufficient privacy budget ( And physical resources are abundant. When ), the expansion decision D(k) = +ΔN is triggered. To ensure safety, the number of operator instances after expansion is... The first condition must be met ,in, This is the estimated rate of new resource consumption after capacity expansion. For safety, this formula ensures that expansion will not cause the budget to run out before the task is completed.

[0084] Rule 2 (Budget Warning): Once a threat to privacy and security is anticipated ( If the system fails to perform any expansion operations, a warning D(k)=WARNING will be triggered immediately, and any expansion operations will be prohibited to prioritize security. The system can then switch to a downgrade mode.

[0085] Rule 3 (Safe Scaling): When performance is excessive ( The budget is far from exhausted. Furthermore, when resource utilization is low, a shrinkage decision D(k) = -ΔN is triggered to optimize resource efficiency.

[0086] Rule 4 (Maintain the status quo): If none of the above conditions are met, output D(k)=0, and the system maintains its current state.

[0087] Step 4: Decision Execution: The decision instruction D(k) is issued from the DDE and flows to the Lifecycle Manager (LCM) and Data Allocator (DAE). After receiving the instruction, the LCM calls the underlying cluster API to actually perform the creation or destruction of operator instances, changing the cluster size N(k) to N(k+1). Almost simultaneously, the DAE is triggered, which dynamically repartitions and reroutes the unprocessed data stream to ensure that the workload is evenly distributed across the new set of instances. After execution, the system enters the next monitoring cycle k+1, and the process repeats, thus forming a continuously optimized, self-adjusting closed-loop feedback system.

[0088] In summary, the elastic scaling problem of privacy-preserving computation tasks is formalized as a constrained online optimization problem. A leap in state awareness is achieved by introducing a privacy monitoring plane. Its core algorithm, through rigorous mathematical definitions and logical rules, ensures that any decision is made within rigid privacy-preserving boundaries, seeking the optimal solution for performance.

[0089] The technical solution of this embodiment obtains the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster. Based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster, a target resource scheduling strategy is determined. Resource scheduling is executed based on the target resource scheduling strategy, which solves the technical problem of premature exhaustion of the privacy budget due to blind scaling in general solutions, and ensures the compliance and effectiveness of privacy computing tasks. In addition, this embodiment uses the privacy budget as a hard constraint and response time as the optimization target for decision-making, optimizing performance under the premise of meeting privacy security, rather than simply pursuing performance or resource utilization, which is more in line with the real needs of privacy computing scenarios.

[0090] Example 2

[0091] Figure 4 This is a schematic diagram of a resource scheduling device provided in an embodiment of the present invention. This embodiment is applicable to resource scheduling scenarios. The device can be implemented using software and / or hardware, and can be integrated into any device that provides resource scheduling functionality, such as… Figure 4 As shown, the resource scheduling device specifically includes: an acquisition module 410, a determination module 420, and an execution module 430.

[0092] The acquisition module is used to acquire the total amount of privacy computing tasks, the total response time corresponding to the total amount of privacy computing tasks, the total privacy budget corresponding to the total amount of privacy computing tasks, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0093] The determination module is used to determine the target resource scheduling strategy based on the total amount of privacy computing tasks, the total response time corresponding to the total amount of privacy computing tasks, the total privacy budget corresponding to the total amount of privacy computing tasks, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster.

[0094] The execution module is used to perform resource scheduling based on the target resource scheduling strategy.

[0095] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0096] Example 3

[0097] Figure 5A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0098] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0099] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as resource scheduling methods.

[0101] In some embodiments, the resource scheduling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the resource scheduling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the resource scheduling method by any other suitable means (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0107] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0108] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0109] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the resource scheduling method according to any embodiment of the invention.

[0110] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A resource scheduling method, characterized in that, include: Obtain the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster; The target resource scheduling strategy is determined based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster. Resource scheduling is performed based on the target resource scheduling strategy.

2. The method according to claim 1, characterized in that, Obtain the global instantaneous privacy budget consumption rate, including: Get the size of the monitoring time window, the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computation task within the monitoring time window, and the total number of operator instances executing the privacy computation task; Based on the size of the monitoring time window and the privacy budget consumed by each operator instance in executing all query subtasks in the privacy computation task within the monitoring time window, determine the instantaneous privacy budget consumption rate of each operator instance. The global instantaneous privacy budget consumption rate is determined based on the total number of operator instances performing privacy computation tasks and the instantaneous privacy budget consumption rate of each operator instance.

3. The method according to claim 1, characterized in that, The physical resource metrics of the computing cluster include: the average CPU utilization of each operator instance, the average memory utilization of each operator instance, and the number of privacy computing tasks executed.

4. The method according to claim 3, characterized in that, Based on the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster, the target resource scheduling strategy is determined, including: Based on the total amount of privacy computing tasks and the physical resource indicators of the computing cluster, determine the predicted remaining time corresponding to the total amount of privacy computing tasks. The difference between the total response time and the cumulative response time of the total number of privacy computing tasks is taken as the target remaining time. The difference between the total privacy budget corresponding to the total amount of privacy computing tasks and the cumulative privacy budget consumption is taken as the remaining privacy budget; The ratio of the remaining privacy budget to the global instantaneous privacy budget consumption rate is used as the predicted privacy budget exhaustion time. The target resource scheduling strategy is determined based on the predicted privacy budget exhaustion time, the target remaining time, and the predicted remaining time corresponding to the total privacy computing task volume.

5. The method according to claim 4, characterized in that, Based on the predicted privacy budget exhaustion time, the target remaining time, and the predicted remaining time corresponding to the total privacy computing task volume, a target resource scheduling strategy is determined, including: If the predicted remaining time corresponding to the total privacy computing task is greater than the target remaining time, the predicted privacy budget exhaustion time is greater than the predicted remaining time corresponding to the total privacy computing task, the average CPU utilization of each operator instance is less than the CPU utilization threshold, the average memory utilization of each operator instance is less than the memory utilization threshold, and the first condition is met, then the target resource scheduling strategy is determined to be the expansion strategy. If the predicted time for the privacy budget to run out is less than or equal to the target remaining time, the target resource scheduling strategy is determined to maintain the current resource state and an early warning is issued. If the difference between the target remaining time and the predicted remaining time corresponding to the total privacy computing task volume is greater than the first difference threshold, the difference between the predicted privacy budget exhaustion time and the target remaining time is greater than the second difference threshold, the average CPU utilization of each operator instance at the current time is less than the average CPU utilization of each operator instance at the previous time, and the average memory utilization of each operator instance at the current time is less than the average memory utilization of each operator instance at the previous time, then the target resource scheduling strategy is determined to be a scaling-down strategy.

6. The method according to claim 5, characterized in that, The first condition includes: The sum of the remaining privacy budget and the first privacy budget is less than or equal to the product of the total privacy budget corresponding to the total privacy computing task volume and the first value, wherein the first privacy budget is determined based on the predicted privacy budget consumption rate after expansion and the target remaining time.

7. A resource scheduling device, characterized in that, include: The acquisition module is used to acquire the total privacy computing task volume, the total response time corresponding to the total privacy computing task volume, the total privacy budget corresponding to the total privacy computing task volume, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster. The determination module is used to determine the target resource scheduling strategy based on the total amount of privacy computing tasks, the total response time corresponding to the total amount of privacy computing tasks, the total privacy budget corresponding to the total amount of privacy computing tasks, the cumulative privacy budget consumption, the global instantaneous privacy budget consumption rate, and the physical resource indicators of the computing cluster. The execution module is used to perform resource scheduling based on the target resource scheduling strategy.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the resource scheduling method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the resource scheduling method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the resource scheduling method according to any one of claims 1-6.

Images