A serverless adaptive scheduling method and system based on consistency perception

Abstract

The application discloses a Serverless adaptive scheduling method and system based on consistency perception, the method monitors key shared states through a perception layer, collects multi-source data, calculates risk features and pre-processes, outputs feature vectors and system state snapshots; a decision layer calculates risk scores based on the feature vectors provided by the perception layer, and performs elastic scheduling of consistency perception according to dynamic threshold values and real-time conflict rate feedback; the execution layer routes tasks to different execution environments according to the risk scores, and feeds back execution results to the perception layer to form a closed-loop optimization. In the field of information scheduling, high-risk tasks are isolated for execution (consistency is preserved), low-risk tasks are executed on a FaaS platform (elasticity is preserved), and then the two are dynamically balanced through closed-loop optimization, thereby ingeniously solving the problem that elasticity and consistency cannot be achieved simultaneously in the existing scheduling method.

Description

Technical Field

[0001] This invention belongs to the field of information scheduling, and specifically relates to a serverless adaptive scheduling method and system based on consistency awareness. Background Technology

[0002] In modern information technology, many data-intensive applications with stringent data consistency requirements, such as clinical multimodal data analysis, real-time financial transaction processing, and IoT sensor information aggregation, have become core components supporting critical business decisions. These applications integrate multi-source heterogeneous information and execute complex processing flows including data cleaning, correlation analysis, and model calculation to uncover the inherent relationships and business value between data, thereby improving the understanding of complex problems and the accuracy of decision-making. Because their results often directly impact the reliability of major decisions and the stability of business systems, extremely high requirements are placed on the integrity, consistency, and reproducibility of the entire data processing flow. Any data deviation, calculation anomaly, or inconsistency in any stage can be amplified step by step in the analysis chain, ultimately weakening the validity of the output conclusions. This not only concerns the business quality evaluation system but may also lead to serious economic or social impacts. With the rapid development of IoT, high-throughput sequencing, and real-time sensing technologies, the data processed by these applications is showing an explosive growth and a high degree of heterogeneity. Against this backdrop, how to efficiently and reliably schedule massive concurrent tasks while ensuring processing quality has become a key bottleneck restricting the intelligent transformation of various industries.

[0003] To cope with the continuous growth in data volume and concurrency demands, complex workflows are typically broken down into numerous parallel tasks with data dependencies and shared state. Existing mainstream solutions, including batch scheduling systems based on Apache Airflow or persistent microservice clusters based on Kubernetes, still have inherent limitations in handling such high-concurrency, state-dependent tasks. The fundamental reason is that most systems rely on static rules or lagging system-level monitoring metrics (such as CPU and memory utilization) for scheduling decisions, failing to perceive application-layer semantics (such as data conflict risks and task priorities) in real time, thus employing static or semi-static resource allocation mechanisms. When faced with sudden computing demands, system scaling is slow, often requiring several minutes or even longer to complete resource adjustments, leading to a mismatch between resource supply and demand: on the one hand, this causes decreased resource utilization and task backlog; on the other hand, it results in resource idleness and waste. Especially in scenarios with stringent requirements for low latency and high real-time performance, such as real-time financial transaction processing, clinical multimodal data analysis, and IoT sensor information aggregation, this sluggish scheduling based on underlying metrics directly weakens the system's responsiveness and business timeliness.

[0004] Furthermore, in these specific scenarios, complex state dependencies and data sharing often exist between tasks. For example, in real-time financial transactions, multiple risk control tasks may concurrently update a user's credit limit; in clinical data analysis, different processing flows (such as image recognition and genomic analysis) need to read and write the same patient mediation results; in IoT aggregation scenarios, a large number of sensor data tasks will compete to update the global state counter. Traditional schedulers cannot perceive these application-layer data competition risks, which can easily lead to state inconsistencies and cause serious problems such as erroneous risk control decisions, distorted medical analysis conclusions, or inaccurate aggregation counts.

[0005] Therefore, there is an urgent need for a new scheduling architecture that can detect application-layer consistency risks, has second-level elastic scaling and fine-grained resource scheduling capabilities, so as to simultaneously meet the multiple requirements of real-time performance, resource efficiency and consistency assurance.

[0006] Serverless architecture, with its extreme elasticity and fine-grained scheduling, is considered an ideal solution for handling sudden loads of massive, short-term tasks, and theoretically can fundamentally solve the efficiency bottlenecks in traditional architectures. However, its stateless and short-lifecycle characteristics fundamentally conflict with the need for stateful maintenance and complex dependency management in complex workflows. Directly applying existing serverless platforms often leads to data inconsistencies and computational errors due to blind scaling mechanisms that disrupt intermediate states and task dependencies.

[0007] Therefore, despite Serverless's exceptional elasticity and fine-grained scheduling potential, several core challenges remain to be addressed before it can be truly applied to complex workflows with stateful tasks. Specifically, these challenges mainly focus on the following two aspects:

[0008] (1) Challenges of state and consistency management

[0009] In high-concurrency, elastic environments, the stateless nature of Serverless systems clashes significantly with the stateful requirements of workflows. Some existing solutions attempt to maintain state through external storage or middleware, but this often introduces additional communication overhead and consistency risks. When numerous function instances simultaneously initiate read / write operations and share data, data races and dependency disorder are highly likely to occur. Furthermore, state consistency is difficult to guarantee during dynamic scaling; delays or loss of updates to some instances can lead to global state inconsistencies, thus propagating and amplifying computational errors. Meanwhile, over-reliance on centralized state storage or coordination components can introduce single-point performance bottlenecks and reliability vulnerabilities. In the event of a failure, the correctness and availability of the entire system can be severely impacted. Therefore, achieving fine-grained state maintenance and consistency control without compromising the elasticity advantages of Serverless systems has become a key challenge for supporting the reliable operation of such analytical tasks.

[0010] (2) Challenges in resource scheduling efficiency

[0011] In high-concurrency computing scenarios, task size and arrival rates are often highly uncertain, making traditional scheduling strategies based on static rules or simple CPU / memory metrics insufficient to meet sudden surges in demand. When request volume increases dramatically, the system may experience resource scaling lag, leading to task backlog and amplified latency; conversely, when the load decreases, over-allocated resources result in significant waste. This "one-size-fits-all" static scaling mechanism directly undermines the cost-effectiveness and responsiveness advantages of Serverless architecture. Furthermore, complex tasks involve multiple layers of dependencies and data transfer. Relying solely on underlying resource utilization for scheduling decisions often ignores application-layer signals such as data dependencies and sudden risks, easily leading to uneven resource allocation, priority mismatch, and even partial blocking. Frequent scheduling and migration also incur additional overhead, impacting overall throughput and service quality. Therefore, achieving intelligent elastic scheduling with context awareness and application semantics, while ensuring real-time performance and improving resource utilization and system robustness, has become another core challenge in driving the deployment of Serverless in complex scenarios. Summary of the Invention

[0012] To address the consistency risks and scheduling inefficiencies inherent in Serverless architectures when handling high-concurrency stateful tasks, this invention proposes a high-concurrency Serverless closed-loop scheduling method based on consistency risk quantification.

[0013] (1) Based on the multi-dimensional characteristics of task operation type, concurrency intensity, write range and historical anomaly, a consistency risk quantification model is designed to score the risk of tasks and use it as the input signal for subsequent scheduling, thereby avoiding data competition and dependency disorder during large-scale concurrency.

[0014] (2) A consistency-aware elastic scheduling mechanism is proposed, which takes risk scoring as the core driver, dynamically adjusts the concurrency limit, queue strategy and expansion speed, and constructs a three-stage process of "initial judgment - feedback correction - stability constraint" to achieve a dynamic balance between state consistency and resource elasticity.

[0015] This method aims to improve overall resource utilization and throughput performance while ensuring system consistency and reliability, thereby building an efficient and scalable serverless computing platform.

[0016] The technical solution provided by this invention is as follows:

[0017] Firstly, a consistency-aware Serverless adaptive scheduling method includes:

[0018] Step 1: The perception layer collects and preprocesses data based on the tasks submitted by the user, generates core feature vectors and system status, and sends them to the decision layer;

[0019] The core feature vector includes the risk cardinality of operation type, write range, concurrency intensity, and historical anomalies;

[0020] Step 2: The decision layer calculates the task risk score S based on the data sent by the perception layer, and generates corresponding scheduling instructions by combining the dynamic threshold and sending them to the execution layer;

[0021] S = clip(B + aO + bW + cE, 0, 1)

[0022] Where B, O, W, and E are the task's operation type risk base, write range, concurrency intensity, and historical anomalies, respectively; clip represents the pruning function; and a, b, and c represent the weighting coefficients of O, W, and E's contribution to the overall risk, respectively. The initial values ​​are set based on business experience and are dynamically optimized using a feedback mechanism during scheduling and operation.

[0023] Step 3: The execution layer schedules tasks according to the scheduling instructions and feeds back the data of completed task execution to the perception layer;

[0024] Low-risk tasks are processed via FaaS, medium-risk tasks are placed in the container's standard queue, and high-risk tasks are executed in isolation and sequentially.

[0025] FaaS, or Function as a Service, is a cloud computing service that allows developers to build, compute, run, and manage application packages as functions without maintaining their own infrastructure.

[0026] Furthermore, the specific process for obtaining the core feature vector is as follows:

[0027] (1) Risk base of operation type;

[0028] Operation type risk base is used to characterize the inherent risk level of a task operation type. Its value is derived from the risk value corresponding to the operation type declared in the task metadata. The operation type-risk value mapping table is set according to the principle that write operation risk is higher than read operation risk, concurrent write risk is higher than exclusive write risk, and the value range is [0.0, 1.0].

[0029] Read operations are assigned a value close to zero as a low-risk baseline, while various write operations are assigned increasing risks based on their concurrency.

[0030] (2) Data writing range;

[0031] The data writing range is used to objectively characterize the data range affected by the task operation. Based on the specific resource identifier accessed by the task, the data granularity level is automatically determined by querying the integrated metadata catalog or resource classification rules. Then, according to the general principle that "the finer the granularity of the data object affected by the operation, the lower the risk value; the coarser the granularity, the higher the risk value," a configurable "data granularity-risk value" mapping table is queried. The data granularity-risk value mapping table clearly defines an increasing risk value for each granularity level in the range of [0.0, 1.0] to accurately reflect the risk differences of different impact ranges.

[0032] (3) Concurrency intensity;

[0033] Concurrency intensity is used to characterize the real-time contention pressure on the target resource, and its value is a real number between 0 and 1:

[0034]

[0035] in, The number of currently active tasks is indicated by a dedicated counting probe deployed on the resource side, provided in real time. It is a configurable normalization threshold.

[0036] (4) Historical anomalies;

[0037] Historical anomalies are used to quantify the frequency of consistency anomalies occurring in the target resource within a configurable sliding time window:

[0038] ;

[0039] in, This indicates the number of times a consistency anomaly event occurred within the window. This indicates the preset threshold for the number of anomalies, which is set based on the business's tolerance for data consistency, the historical anomaly baseline of the resource, or a principle proportional to the length of the sliding time window.

[0040] The consistency exception event refers to an event that causes task execution to fail or roll back due to data competition, version conflict or state inconsistency, and can be captured by the system.

[0041] Furthermore, the optimization process for the weighting coefficients of O, W, and E's contribution to overall risk is as follows:

[0042] The risk prediction score S is periodically compared with the result of whether the task actually conflicts, and the weight coefficients a, b, c are iteratively updated by minimizing the prediction error.

[0043] Furthermore, the gradient descent method is used to optimize the formula... Defined loss function L:

[0044]

[0045] in, This represents the actual conflict result of the i-th task, with 1 indicating occurrence and 0 indicating non-occurrence; N represents the number of samples. Let represent the risk coefficient of the i-th task; the weight coefficients are updated as follows: η is the learning rate, used to control the update step size. Its value needs to be determined by balancing convergence speed and stability. X represents the weight coefficients to be updated, where a, b, and c represent the weights. This represents the updated weight parameters. This represents the weight parameters before the update.

[0046] Furthermore, the dynamic security threshold in step 2 includes a lower limit. and upper limit The general setting principles are as follows: the initial value is set according to the distribution of historical task risk scores, or configured according to the business's initial expectations for the proportion of low, medium and high risk tasks.

[0047] Furthermore, based on dynamic security thresholds, the task scheduling strategy is as follows:

[0048] S < Tasks are scheduled to the FaaS platform; ≤ S < Tasks are scheduled to the standard queue of the container cluster; S ≥ The task is scheduled to the container cluster's isolated queue and serialized.

[0049] Among them, the concurrency limit of FaaS is limited by the platform service quota; the concurrency limit of container queues can be initially estimated and dynamically optimized based on the node resource specifications and the average resource requirements of tasks.

[0050] Furthermore, the optimization process for the dynamic security threshold is as follows: The decision-making layer calculates the actual conflict occurrence rate during task scheduling based on statistics from the perception layer. Conflict rate with preset target The deviation is periodically adjusted to adjust the upper limit. Actual conflict incidence rate The target conflict rate refers to the proportion of tasks that experienced consistency conflicts within the most recent statistical period out of the total number of tasks. It is the upper limit of the expected conflict rate set by the administrator based on the business's tolerance for data consistency;

[0051] The adjustment logic is as follows: If > If there are too many conflicts, adjust the step size by the preset step size δ. To tighten controls and identify more tasks as high-risk; conversely, if < Then adjust upwards according to the phase length δ. To relax standards and release more tasks to a highly elastic resource pool;

[0052] Then and Maintain a fixed interval Δ for linkage adjustment, that is, always satisfy... - = Δ, to maintain the relative stability of the low, medium and high risk range structure. The step size δ and the interval Δ are both configurable parameters, which are set according to the needs of task scheduling to adjust sensitivity and stability.

[0053] Furthermore, (1) when the consistency risk quantification fails, the decision layer does not score the risk, but directly executes static routing based on the operation type declared in the task metadata: routing read operation (READ) tasks to the FaaS platform, and routing write operation (including EXCLUSIVE_WRITE, CONCURRENT_WRITE) tasks to the container cluster;

[0054] (2) When the monitoring data stream of the perception layer is interrupted, causing the real-time features to fail to be updated, the adaptive learning of the weight coefficients a, b, c and the dynamic safety threshold is paused, and the previous effective feature values ​​and parameters are used to continue to calculate the task risk score until the data link is restored.

[0055] The aforementioned consistency risk quantification failure refers to the occurrence of at least one of the following situations:

[0056] ① When the task risk score S exceeds the [0,1] range or an invalid value is output;

[0057] ②The perception layer does not provide complete data on operation type risk base B, write range W, concurrency intensity O, and historical anomalies E.

[0058] Secondly, a system employing the aforementioned consistency-aware Serverless adaptive scheduling method includes:

[0059] Perception layer: Based on the tasks submitted by the user, it collects and preprocesses data, generates core feature vectors and system status, and sends them to the decision layer;

[0060] The core feature vector includes the risk cardinality of operation type, write range, concurrency intensity, and historical anomalies;

[0061] Decision layer: Calculates task risk score based on data sent by perception layer, and generates corresponding scheduling instructions based on dynamic thresholds and sends them to execution layer;

[0062] S = clip(B + aO + bW + cE, 0, 1)

[0063] Where B, O, W, and E are the task's operation type risk base, write range, concurrency intensity, and historical anomalies, respectively; clip represents the pruning function; and a, b, and c represent the weighting coefficients of O, W, and E's contribution to the overall risk, respectively. The initial values ​​are set based on business experience and are dynamically optimized using a feedback mechanism during scheduling and operation.

[0064] Execution layer: Schedules tasks according to scheduling instructions and feeds back the data of completed task execution to the perception layer;

[0065] Low-risk tasks are processed via FaaS, medium-risk tasks are placed in the container's standard queue, and high-risk tasks are executed in isolation and sequentially.

[0066] A third aspect is an electronic device, comprising:

[0067] One or more processors;

[0068] Storage device for storing one or more programs.

[0069] When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0070] The beneficial effects of the technical solution of this invention:

[0071] The core lies in introducing a "closed-loop scheduling" mechanism. Through the collaborative linkage and information feedback of the perception layer, decision-making layer, and execution layer, a dynamic and adaptive intelligent scheduling system is constructed, aiming to achieve an intelligent balance between consistency assurance and elastic scaling in high-concurrency scenarios. It relies on a quantitative model built based on four dimensions of features: operation type, write range, concurrency intensity, and historical anomalies. Compared to traditional scheduling methods that only rely on underlying resource indicators, it can more comprehensively and accurately capture application-layer risks such as data competition and state inconsistency, significantly improving the accuracy of high-risk task identification. The closed-loop feedback mechanism enables dynamic optimization of weight coefficients and security thresholds, flexibly adapting to business load fluctuations and data pattern changes without manual intervention, continuously maintaining the optimal balance between "consistency and elasticity." Through a layered scheduling strategy, low-risk tasks fully leverage the elasticity advantages of the FaaS platform, while medium- and high-risk tasks are isolated through container queues and serialized processing to ensure consistency. This avoids resource idleness and waste, reduces conflict losses, effectively improves resource utilization, and lowers the conflict rate and average latency of high-risk tasks. A robust degradation and fault tolerance mechanism can ensure business continuity in abnormal scenarios and can flexibly adapt to customized needs in key areas such as financial transactions, clinical data analysis, and IoT information aggregation, promoting the large-scale deployment of Serverless architecture in stateful and complex workflows. Attached Figure Description

[0072] Figure 1 This diagram illustrates the scheduling system framework in the technical solution of this invention.

[0073] Figure 2 This is a flowchart illustrating the technical solution of the present invention. Detailed Implementation

[0074] The present invention will now be further described in conjunction with the accompanying drawings and embodiments.

[0075] like Figure 1 and Figure 2 As shown, a consistency-aware Serverless adaptive scheduling method includes:

[0076] Step 1: The perception layer collects and preprocesses data based on the tasks submitted by the user, generates core feature vectors and system status, and sends them to the decision layer;

[0077] The collected data includes data from public cluster monitoring systems, FaaS platforms and container runtime logs, execution layer reported data, and two types of dedicated probes: public cluster monitoring systems provide system load and queue status; FaaS and container logs provide application process and error information; execution layer reported task final status codes, actual time consumption, and runtime error codes to form a closed-loop feedback; non-general counting probes designed specifically for counting concurrency provide real-time resource access counts by querying distributed locks or resource interfaces; and non-general detection probes designed specifically for proactively detecting consistency conflicts provide data version conflicts and transaction rollback events by listening to the message bus or parsing transaction logs.

[0078] Using the above-mentioned system load, queue status, application process error information, task final status code, actual time consumption, runtime error code, real-time resource access count, data version conflict and transaction rollback events as input, the feature collection is completed through the collaborative operation type mapping module, concurrency intensity collection module and abnormal event collection module.

[0079] The core feature vector includes the risk cardinality of operation type, write range, concurrency intensity, and historical anomalies;

[0080] The specific process for obtaining the core feature vector is as follows:

[0081] (1) Risk base of operation type;

[0082] Operation type risk base is used to characterize the inherent risk level of a task operation type. Its value is derived from the risk value corresponding to the operation type declared in the task metadata. The operation type-risk value mapping table is set according to the principle that write operation risk is higher than read operation risk, concurrent write risk is higher than exclusive write risk, and the value range is [0.0, 1.0].

[0083] Read operations are assigned a value close to zero as a low-risk baseline, while various write operations are assigned increasing risks based on their concurrency.

[0084] In a preferred embodiment, the mapping table is configured such that READ (read operation) is mapped to 0.0, EXCLUSIVE_WRITE (exclusive write operation) is mapped to 0.3, and CONCURRENT_WRITE (concurrent write operation) is mapped to 0.6. This numerical sequence intuitively reflects the core quantitative relationship that the risk of concurrent write is twice that of exclusive write. For example, in clinical data analysis, "querying a gene sequence" is declared as a read-only operation, so it is mapped to 0.0; while "concurrently updating a medical examination report" is declared as a concurrent write operation, so it is mapped to 0.6. This feature value provides a standardized behavioral pattern risk input for subsequent risk fusion calculations.

[0085] First, the `operation_type` field in the task metadata is parsed. If the field value is `READ`, a fixed value within the range of [0.00, 0.10] is assigned; if it is `EXCLUSIVE_WRITE`, a fixed value within the range of [0.20, 0.40] is assigned; and if it is `CONCURRENT_WRITE`, a fixed value within the range of [0.50, 0.80] is assigned. For other write types, based on their concurrency risk level, linear or piecewise interpolation is performed within the above range according to the monotonically non-decreasing principle. If the interpolation result is higher than 0.80, it is capped at 0.80, and under no circumstances can B be greater than 1.0. The data write range W is automatically determined by the system based on the schema-table-column containment relationship inherent in relational database standards, and is mapped in a closed manner to FIELD 0.10, COLUMN 0.20, ROW 0.40, TABLE 0.70, SCHEMA 0.90, GLOBAL 0.50, and GLOBAL 0.60. Version 1.00: The mapping table is hot-updated using a local configuration file. After the update, it remains monotonically non-decreasing and the interval [0.1, 1.0] remains unchanged.

[0086] (2) Data writing range;

[0087] The data writing range is used to objectively characterize the data range affected by the task operation. Based on the specific resource identifier accessed by the task, the data granularity level is automatically determined by querying the integrated metadata catalog or resource classification rules. Then, according to the general principle that "the finer the granularity of the data object affected by the operation, the lower the risk value; the coarser the granularity, the higher the risk value," a configurable "data granularity-risk value" mapping table is queried. The data granularity-risk value mapping table clearly defines an increasing risk value for each granularity level in the range of [0.0, 1.0] to accurately reflect the risk differences of different impact ranges.

[0088] In a preferred embodiment, the mapping table is configured as follows: FIELD (field level) is mapped to 0.1, ROW (row level) to 0.3, TABLE (table level) or FILE (file level) to 0.7, DATASET (dataset level) to 0.9, and GLOBAL (global level) to 1.0. For example, a task targeting a "patient information table" would be classified as TABLE-level and mapped to a risk value of 0.7. This feature provides a standardized breadth of data impact input for subsequent risk calculations.

[0089] (3) Concurrency intensity;

[0090] Concurrency intensity is used to characterize the real-time contention pressure on the target resource, and its value is a real number between 0 and 1:

[0091]

[0092] in, The number of currently active tasks is indicated by a dedicated counting probe deployed on the resource side, provided in real time. It is a configurable normalization threshold;

[0093] In this example, In the initial calibration phase, the maximum concurrent connection count for database resources is set to 60% to 80%, or the maximum concurrent connection count for cache resources is set to 70% to 75% of the maximum timeout limit. Then, during the 48- to 96-hour stress test period, the load is increased in increments of 50% to 150%. The maximum concurrent connection count with a conflict rate of ≤0.1% and the latency meets the standard is selected as the calibration value. The value is then dynamically optimized within a 24-hour sliding window. If the actual concurrent connection count is lower than 50% of the threshold for three consecutive windows, the value is increased by 5% to 20%. If the conflict rate is higher than 0.1% for two consecutive windows, the value is decreased by 5% to 20%. After each adjustment, the value is retested for 6 hours for verification.

[0094] Dedicated counting probes deployed on the resource side exist as a system service, atomically updating their internal counters by listening to resource access lifecycle events (such as task start / end access). The perception layer directly obtains the precise counter at that moment by calling the standardized query interface provided by this probe (such as a REST endpoint or internal API). Values; for example, for a file that supports a maximum of 10 concurrent writes, you can set... =10. When When this threshold is reached or exceeded, the O value will be limited to 1.0, indicating that competition has reached saturation. The closer this value is to 1, the more intense the resource competition. For example, when multiple processing flows such as image recognition and genomic analysis simultaneously access the "same patient mediation outcome file," the resource's... As the value increases, the O value will increase significantly.

[0095] (4) Historical anomalies;

[0096] Historical anomalies are used to quantify the frequency of consistency anomalies occurring in the target resource within a configurable sliding time window:

[0097] ;

[0098] in, This indicates the number of times a consistency anomaly event occurred within the window. This indicates the preset threshold for the number of anomalies, which is set based on the business's tolerance for data consistency, the historical anomaly baseline of the resource, or a principle proportional to the length of the sliding time window.

[0099] The consistency exception event refers to an event that causes task execution to fail or roll back due to data competition, version conflict or state inconsistency, and can be captured by the system.

[0100] By monitoring dedicated consistency detection probes, FaaS platform and container runtime logs, database driver and transaction manager logs, and receiving specific error codes reported by the task execution layer, consistency anomaly signals are captured and aggregated, and the number of times they occur within a window is counted. In the formula This is a preset threshold for the number of anomalies, used to normalize statistical values ​​to the [0, 1] range. This threshold is set based on the business's tolerance for data consistency, the historical anomaly baseline of the resource, or a principle proportional to the length of the sliding window. For example, if the business requires an alert for a resource exceeding 5 anomalies within a 5-minute window, then this threshold is set... = 5.

[0101] Step 2: The decision layer calculates the task risk score S based on the data sent by the perception layer, and generates corresponding scheduling instructions by combining the dynamic threshold and sending them to the execution layer;

[0102] S = clip(B + aO + bW + cE, 0, 1)

[0103] Where B, O, W, and E are the task's operation type risk base, write range, concurrency intensity, and historical anomalies, respectively; clip represents the pruning function; and a, b, and c represent the weighting coefficients of O, W, and E's contribution to the overall risk, respectively. The initial values ​​are set based on business experience and are dynamically optimized using a feedback mechanism during scheduling and operation.

[0104] The optimization process for the weighting coefficients of O, W, and E's contribution to overall risk is as follows:

[0105] The risk prediction score S is periodically compared with the result of whether the task actually conflicts, and the weight coefficients a, b, c are iteratively updated by minimizing the prediction error.

[0106] Optimize using gradient descent method from the formula Defined loss function L:

[0107]

[0108] in, This represents the actual conflict result of the i-th task, with 1 indicating occurrence and 0 indicating non-occurrence; N represents the number of samples. Let represent the risk coefficient of the i-th task; the weight coefficients are updated as follows: η is the learning rate, used to control the update step size. Its value needs to be determined by balancing convergence speed and stability. X represents the weight coefficients to be updated, where a, b, and c represent the weights. This represents the updated weight parameters. This represents the weight parameters before the update.

[0109] The dynamic security threshold in step 2 includes a lower limit. and upper limit The general setting principles are as follows: the initial value is set according to the distribution of historical task risk scores, or configured according to the business's initial expectations for the proportion of low, medium and high risk tasks.

[0110] Based on dynamic safety thresholds, the task scheduling strategy is as follows:

[0111] S < Tasks are scheduled to the FaaS platform; ≤ S < Tasks are scheduled to the standard queue of the container cluster; S ≥ The task is scheduled to the container cluster's isolated queue and serialized.

[0112] Among them, the concurrency limit of FaaS is limited by the platform service quota; the concurrency limit of container queues can be initially estimated and dynamically optimized based on the node resource specifications and the average resource requirements of tasks.

[0113] In one embodiment of clinical data analysis, the initial configuration is... =0.3, =0.7.

[0114] The optimization process for dynamic security thresholds is as follows: The decision-making layer calculates the actual conflict occurrence rate during task scheduling based on statistics from the perception layer. Conflict rate with preset target The deviation is periodically adjusted to adjust the upper limit. Actual conflict incidence rate The target conflict rate refers to the proportion of tasks that experienced consistency conflicts within the most recent statistical period out of the total number of tasks. It is the upper limit of the expected conflict rate set by the administrator based on the business's tolerance for data consistency;

[0115] The adjustment logic is as follows: If > If there are too many conflicts, adjust the step size by the preset step size δ. To tighten controls and identify more tasks as high-risk; conversely, if < Then adjust upwards according to the phase length δ. To relax standards and release more tasks to a highly elastic resource pool;

[0116] Then and Maintain a fixed interval Δ for linkage adjustment, that is, always satisfy... - = Δ, to maintain the relative stability of the low, medium and high risk range structure. The step size δ and the interval Δ are both configurable parameters, which are set according to the needs of task scheduling to adjust sensitivity and stability.

[0117] The decision-making level periodically uses feedback information to drive two optimization cycles: first, by comparing the predicted risk score S with the actual conflict results, the risk model weights (a, b, c) are updated using gradient descent; second, based on the actual conflict occurrence rate of the system... Conflict rate with target To adjust the dynamic safety threshold based on the deviation. and linkage .

[0118] Step 3: The execution layer schedules tasks according to the scheduling instructions and feeds back the data of completed task execution to the perception layer;

[0119] Low-risk tasks are processed via FaaS, medium-risk tasks are placed in the container's standard queue, and high-risk tasks are executed in isolation and sequentially.

[0120] To ensure system reliability, a tiered degradation scheme is designed for core components. If the consistency risk quantification module fails, the decision layer will bypass risk scoring and directly execute static routing based on the operation type declared in the task metadata to maintain basic task distribution capabilities. That is, tasks declared as write operations will be directly routed to the container cluster, and read operations will be routed to the FaaS cluster. If the monitoring data stream of the perception layer is interrupted, causing real-time features to fail to update, the system will pause the adaptive learning module for weight coefficients and security thresholds. In this case, the risk quantification module will continue to run using the last valid feature values ​​and parameters obtained before the failure until the data link is restored.

[0121] Consistency risk quantification failure judgment: 1) The risk score output by the module exceeds the reasonable range of [0,1], or the output value is invalid; 2) The perception layer does not provide complete B / O / W / E feature data.

[0122] Example 2

[0123] A system employing the above-mentioned consistency-aware Serverless adaptive scheduling method includes:

[0124] Perception layer: Based on the tasks submitted by the user, it collects and preprocesses data, generates core feature vectors and system status, and sends them to the decision layer;

[0125] The core feature vector includes the risk cardinality of operation type, write range, concurrency intensity, and historical anomalies;

[0126] Decision layer: Calculates task risk score based on data sent by perception layer, and generates corresponding scheduling instructions based on dynamic thresholds and sends them to execution layer;

[0127] S = clip(B + aO + bW + cE, 0, 1)

[0128] Where B, O, W, and E are the task's operation type risk base, write range, concurrency intensity, and historical anomalies, respectively; clip represents the pruning function; and a, b, and c represent the weighting coefficients of O, W, and E's contribution to the overall risk, respectively. The initial values ​​are set based on business experience and are dynamically optimized using a feedback mechanism during scheduling and operation.

[0129] Execution layer: Schedules tasks according to scheduling instructions and feeds back the data of completed task execution to the perception layer;

[0130] Low-risk tasks are processed via FaaS, medium-risk tasks are placed in the container's standard queue, and high-risk tasks are executed in isolation and sequentially.

[0131] A heterogeneous resource scheduling strategy is adopted: This strategy does not simply rely on whether a task is stateful, but rather uses consistency risk quantification results as an intelligent guide to dynamically match tasks to the most suitable resource pool. Low-risk tasks are scheduled to the FaaS platform to fully utilize its millisecond-level elasticity and high concurrency advantages; medium- and high-risk tasks are scheduled to the container cluster, and differentiated isolation (standard queues or isolated queues) is implemented according to the risk level. The container orchestration capability ensures the persistence of task state and the consistency of the computation process. This architecture design is particularly suitable for scenarios with stringent data consistency requirements, such as clinical multimodal data analysis. In this scenario, low-risk tasks such as "querying specific gene sequences" are intelligently scheduled to FaaS for efficient processing; while high-consistency-risk tasks such as "merging and updating patient diagnostic reports" are dynamically routed to the isolated environment of the container cluster to ensure strong consistency. The perception, decision-making, and execution layers form a closed optimization loop through continuous data flow, control flow, and feedback flow, thereby intelligently resolving the inherent contradiction between elasticity and consistency in the Serverless architecture.

[0132] The perception layer comprehensively collects data and outputs decision-making basis; the decision-making layer performs calculations and scheduling; the execution layer strictly executes instructions and reports local results, which are then collected again by the perception layer to drive the next round of optimization. Through this "perception-decision-execution-feedback" closed loop, the system achieves an adaptive balance between resource elasticity and state consistency in a high-concurrency serverless environment.

[0133] The specific implementation process of each layer is described in the above method. This invention will not repeat the details here. The division of the functional modules is only an example. In some embodiments, some functional modules can be merged and some functional modules can be split. Each functional module can be implemented in software, hardware, or a combination of software and hardware. The software and hardware devices include, but are not limited to, general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0134] Example 3

[0135] An electronic device includes: one or more processors; and a storage device for storing one or more programs.

[0136] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described consistency-aware Serverless adaptive scheduling method.

[0137] For the specific implementation process of each step, please refer to the above method content; the present invention will not be described in detail here.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A Serverless adaptive scheduling method based on consistency awareness, characterized in that, Comprise: Step 1: The perception layer collects and preprocesses data according to the user-submitted task, generates core feature vectors and system states, and sends them to the decision layer; The core feature vector includes operation type risk base, write range, concurrency strength, and historical anomaly; Step 2: The decision layer calculates the task risk score S according to the data sent by the perception layer, and generates corresponding scheduling instructions combined with the dynamic safety threshold and sends them to the execution layer; S = clip(B + aO + bW + cE, 0, 1) Where B, O, W, E are the operation type risk base, write range, concurrency strength, and historical anomaly of the task in turn, clip represents the clipping function, a, b, c are the weight coefficients of the contribution degree of O, W, E to the overall risk, the initial value is set according to business experience, and the feedback mechanism is used for dynamic optimization during scheduling and running; Step 3: The execution layer schedules tasks according to the scheduling instructions, and feeds back the data after task execution to the perception layer; Low-risk tasks go to FaaS, medium-risk tasks enter the container standard queue, and high-risk tasks are executed in isolation and series; The specific acquisition process of the core feature vector is as follows: (1) Operation type risk base; The operation type risk base is used to represent the inherent risk level of the operation type of the task, and the value is derived from the risk value corresponding to the operation type declared in the task metadata. The operation type-risk value mapping table is set according to the principle that write operation risk is higher than read operation, and concurrent write risk is higher than exclusive write. The value range is [0.0, 1.0]; Read operation is assigned a value close to zero as a low-risk baseline, and various write operations are incremented according to their concurrency risk; (2) Data write range; Data write range is used to objectively represent the data range affected by task operation. According to the specific resource identifier accessed by the task, the integrated metadata directory or resource classification rules are queried to automatically determine the data granularity level it belongs to, and then a configurable "data granularity-risk value" mapping table is queried according to the general rule "the finer the data object granularity affected by operation, the lower the risk value; the coarser the granularity, the higher the risk value" to obtain the risk value; The data granularity-risk value mapping table defines an increasing risk value for each granularity level in the [0.0, 1.0] interval to accurately reflect the risk difference of different impact ranges; (3) Concurrency strength; Concurrency strength is used to represent the real-time competition pressure of the target resource, and its value is a real number between 0 and 1: ； wherein, represents the current number of active tasks, provided in real-time by a dedicated counting probe deployed at the resource side; is a configurable normalization threshold; (4) Historical anomaly; Historical anomaly is used to quantitatively represent the frequency of consistency anomaly events occurring within a configurable sliding time window: ； wherein, represents the number of occurrences of the consistency abnormal event within the window, represents a preset threshold of the number of abnormalities, which is set according to the tolerance of the business to the data consistency, the historical abnormal baseline of the resource, or the principle of being proportional to the length of the sliding time window. The consistency anomaly event refers to an event that causes task execution failure or rollback due to data competition, version conflict, or state inconsistency, and can be captured by the system; The dynamic security threshold in step 2 includes a lower limit and an upper limit The general setting principle is as follows: the initial value is set according to the distribution of historical task risk scores, or configured according to the initial expectation of the business for the proportion of low, medium and high risk tasks; Based on the dynamic safety threshold, the task scheduling strategy is as follows: S < scheduling tasks to the FaaS platform; ≤ S < scheduling tasks to the container cluster standard queue; S≥ scheduling tasks to the container cluster isolated queue and performing serialization processing; Where the concurrency upper limit of FaaS is limited by the platform service quota; the concurrency upper limit of the container queue can be initially estimated and dynamically optimized according to the node resource specification and the average resource demand of the task; The optimization process for dynamic security thresholds is as follows: The decision-making layer calculates the actual conflict occurrence rate during task scheduling based on statistics from the perception layer. Conflict rate with preset target The deviation is periodically adjusted to adjust the upper limit. Actual conflict incidence rate The target conflict rate refers to the proportion of tasks that experienced consistency conflicts within the most recent statistical period out of the total number of tasks. It is the upper limit of the expected conflict rate set by the administrator based on the business's tolerance for data consistency; The adjustment logic is as follows: if , it indicates that the current conflict is too much, and the threshold is adjusted downward by a preset step size δ to tighten the control and identify more tasks as high-risk; otherwise, if , the threshold is adjusted upward by the same step size δ to relax the standard and release more tasks to the high-flexibility resource pool;​​ then linkage adjustment with fixed interval Δ, that is, always satisfy - = Δ, to maintain the relative stability of the low, medium and high risk interval structure, both step δ and interval Δ are configurable parameters, which are set according to the requirements of adjustment sensitivity and stability of task scheduling.

2. The method of claim 1, wherein, The optimization process of the weight coefficients of the contribution degree of O, W, E to the overall risk is as follows: Periodically compare the risk prediction score S with the result of whether the task actually conflicts, and iteratively update the weight coefficients a, b, c by minimizing the prediction error.

3. The method of claim 2, wherein, The loss function L defined by the formula is optimized using the gradient descent method. ； wherein, represents the actual conflict result of the i-th task, occurrence is 1, and non-occurrence is 0; N represents the sample number, represents the risk coefficient of the i-th task; the updating mode of the weight coefficient is: η is the learning rate, used to control the updating step, and its value needs to balance the convergence speed and stability, X represents the weight coefficient to be updated, and represents a, b, and c; represents the updated weight parameter, represents the weight parameter before updating.

4. The method of claim 1, wherein, (1) When the consistency risk quantification fails, the decision layer does not risk scoring, and directly performs static routing according to the operation type declared in the task metadata: routing read operation tasks to the FaaS platform, and routing write operation tasks to the container cluster; (2) When the perception layer monitoring data flow is interrupted to cause real-time features to be unable to update, suspend the adaptive learning of the weight coefficients a, b, c and the dynamic security threshold, continue to calculate the task risk score with the previous valid feature values and parameters until the data link is restored; The consistency risk quantification failure refers to at least one of the following situations: ① When the task risk score S exceeds the interval [0, 1] or outputs an invalid value; ② The perception layer does not provide complete operation type risk base B, write range W, concurrency strength O, and historical anomaly E data.

5. A system employing the consistency-aware Serverless adaptive scheduling method according to any one of claims 1-4, characterized in that, The perception layer includes: The perception layer: according to the task submitted by the user, performs data collection and preprocessing, generates a core feature vector and a system state, and sends them to the decision layer; The core feature vector includes the operation type risk base, the write range, the concurrency strength, and the historical anomaly; The decision layer: calculates the task risk score based on the data sent by the perception layer, and generates corresponding scheduling instructions in combination with the dynamic threshold and sends them to the execution layer; S = clip(B + aO + bW + cE, 0, 1) Where B, O, W, E are the operation type risk base, write range, concurrency strength, and historical anomaly of the task, respectively, clip represents the clipping function, a, b, c are weight coefficients representing the contribution of O, W, E to the overall risk, and the initial values are set according to business experience and dynamically optimized using a feedback mechanism during scheduling and running; The execution layer: schedules tasks according to scheduling instructions, and feeds back the data of task execution to the perception layer; Low-risk tasks go to FaaS, medium-risk tasks enter the container standard queue, and high-risk tasks are executed in isolation and in series.

6. An electronic device, comprising: The one or more processors; The storage device is used to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4. ​

Images