Dynamic computing power distribution method and system based on reinforcement learning

Abstract

The invention belongs to the technical field of computing power distribution, and particularly relates to a dynamic computing power distribution method and system based on reinforcement learning, and the method comprises the following specific steps: S1, covering cloud, edge and end full-node scenes, and collecting computing power resource states, task demand features and cross-domain network condition data in real time; s2, on the basis of standardized data output by a cross-domain computing power sensing module, by constructing a state space fusing computing power, tasks and a network, defining an action space of computing power scheduling direction and proportion, and designing a multi-target reward function for balancing the resource utilization rate, the task satisfaction rate and long-term conflict avoidance; and realizing self-learning and self-iteration scheduling strategy generation based on a reinforcement learning algorithm. According to the invention, the reinforcement learning agent autonomously learns the computing power demand of the emergency scene and the new type of task, the rule does not need to be manually preset and modified, and the method has the advantage of realizing dynamic adaptation of computing power distribution to complex and changeable scenes.

Description

Technical Field

[0001] This invention relates to the field of computing power allocation technology, specifically to a dynamic computing power allocation method and system based on reinforcement learning. Background Technology

[0002] With the acceleration of digital transformation, the computing power demand of applications such as AI, big data, and IoT (e.g., autonomous driving, telemedicine, and industrial internet) exhibits dynamic fluctuations: for example, AI training tasks have low computing power demand during the data loading stage, but the computing power demand increases sharply during the model iteration stage; smart parks require a large amount of computing power for security monitoring video analysis during peak hours (e.g., 10:00-12:00 on weekdays), but the computing power demand decreases sharply during off-peak hours.

[0003] Current mainstream computing power allocation methods have significant limitations:

[0004] Static allocation: Pre-allocating fixed computing power to tasks or nodes; for example, allocating 20% ​​of GPU resources to a certain AI training task cannot meet dynamic needs, resulting in insufficient computing power during peak hours and waste of resources during off-peak hours.

[0005] Rule-based dynamic allocation: Adjustments are made based on manually preset rules (such as "if the node load exceeds 80%, transfer 10% of the tasks"). While this offers some flexibility, the rules rely on manual design, cannot cover complex scenarios, and are mostly limited to a single domain (such as only the cloud or only the edge), lacking cross-domain collaboration capabilities.

[0006] Based on the above, the existing technology has the following specific problems:

[0007] (i) Strong dependence on rules and insufficient dynamic adaptability

[0008] The core logic of existing rule-based dynamic allocation methods relies on manual pre-setting, which cannot cope with unforeseen scenarios or new types of tasks. For example, a cloud computing center designed a rule for AI training tasks that "for every 10% increase in data volume, computing power increases by 5%". When a training task suddenly receives a 50% increase in high-resolution image data (far exceeding the rule adjustment range), the system cannot adapt in time, resulting in a training delay of more than 2 hours. Moreover, when a new type of quantum computing simulation task is introduced, because there is no corresponding rule, the configuration needs to be manually modified, resulting in a response delay of more than 4 hours.

[0009] (ii) Lack of cross-domain collaboration and low resource utilization.

[0010] Existing solutions are mostly limited to a single domain (such as cloud or edge), failing to establish interconnected computing power channels between the cloud, edge, and endpoints, resulting in the inability to reuse idle resources. For example, in a smart park, the edge (responsible for security video analysis) reaches 95% load (computing power saturation, video stuttering) between 10:00 and 12:00, while the cloud (responsible for big data statistics) only has a 30% load during the same period (a large amount of idle power), and the terminal (park's smart server) has 30% idle computing power. Due to the lack of a cross-domain scheduling mechanism, the overall resource utilization rate is only 40%, resulting in both waste and stuttering.

[0011] (iii) Lack of long-term optimization leads to subsequent resource conflicts

[0012] Existing solutions only focus on whether current computing power is sufficient, without predicting subsequent task demands, leading to optimal short-term decisions but insufficient long-term resources. For example, a hospital system allocates 80% of its computing power to non-urgent medical record statistics tasks (lasting 2 hours) at 9:00 AM. At 10:30 AM, a high-priority CT image diagnosis task (requiring 50% computing power) suddenly occurs. Due to the lack of reserved resources, the diagnosis is delayed by 1 hour, posing a medical risk. Summary of the Invention

[0013] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0014] A dynamic computing power allocation method based on reinforcement learning includes the following specific steps:

[0015] S1: Covers all node scenarios in cloud, edge, and terminal, and collects real-time data on computing resource status, task requirements, and cross-domain network conditions.

[0016] S2: First, based on the standardized data output by the cross-domain computing power perception module, the action space for computing power scheduling direction and proportion is defined by constructing a state space that integrates computing power, tasks, and networks. Then, a multi-objective reward function is designed to generate a scheduling strategy that can achieve self-learning and self-iteration by relying on reinforcement learning algorithms. Finally, a cross-domain computing power scheduling instruction adapted to the current scenario is output.

[0017] S3: It receives the computing power scheduling action instructions output by the reinforcement learning decision module, and conducts real-time verification from three dimensions: hardware adaptability, network support, and task compatibility. It only transmits instructions that can be implemented and feeds back instructions that cannot be implemented to the reinforcement learning decision module for adjustment and optimization. At the same time, it synchronously verifies the results and updates the status of the cross-domain computing power perception module.

[0018] S4: Receives compliant instructions that have been verified for feasibility, executes cross-domain computing power migration and precise task-computing power matching between cloud, edge, and device; at the same time, it collects data after the instructions are executed in real time and feeds it back to the cross-domain computing power perception module.

[0019] As a preferred embodiment of the dynamic computing power allocation method based on reinforcement learning described in this invention, the specific steps of step S1 are as follows:

[0020] S11: Real-time acquisition of computing power data;

[0021] S12: Identify task type, priority, and computing power requirement curve;

[0022] S13: Collect cross-domain network bandwidth, latency, and packet loss rate to ensure data transmission feasibility during computing power scheduling.

[0023] As a preferred embodiment of the dynamic computing power allocation method based on reinforcement learning described in this invention, the specific steps of step S2 are as follows:

[0024] S21: Fuse the data from the cross-domain computing power perception module into a state vector S=[cloud node load, remaining computing power of edge nodes, number of high-priority tasks, cross-domain network latency];

[0025] S22: Define computing power allocation action A;

[0026] S23: Design a multi-objective reward function ;in, This represents the current resource utilization rate. For task requirement fulfillment rate; This is a predicted value for resource conflicts in the next hour; , , Configurable weighting coefficients;

[0027] S24: The PPO algorithm is adopted. It is first trained offline using historical data, and then fine-tuned in real time during actual operation to continuously optimize decision-making capabilities.

[0028] As a preferred embodiment of the dynamic computing power allocation method based on reinforcement learning described in this invention, the specific steps of step S3 are as follows:

[0029] S31: Retrieve the hardware parameter threshold of the target node of the cross-domain computing power execution module and verify whether the scheduling instruction exceeds the hardware carrying capacity;

[0030] S32: Combining the network data transmitted in real time by the cross-domain computing power perception module, the computing power scheduling-network consumption mapping model is used to predict whether the network can continuously support data transmission during the scheduling process;

[0031] S33: Associate the task requirement data of the cross-domain computing power perception module and verify whether the scheduling instructions conflict with the task characteristics;

[0032] S34: When the verification is successful, a feasibility verification report is generated, and the scheduling instructions and the report are synchronously transmitted to the cross-domain computing power execution module to trigger the execution process; when the verification fails, adjustment suggestions are generated and fed back to the reinforcement learning decision module. At the same time, the reason for the verification failure is transmitted to the cross-domain computing power perception module to update the status data and assist the reinforcement learning decision module in regenerating instructions.

[0033] As a preferred embodiment of the dynamic computing power allocation method based on reinforcement learning described in this invention, the specific steps of step S4 are as follows:

[0034] S41: Adopts the distributed computing power resource description specification, abstracts cloud, edge, and terminal computing power resources into a unified computing power unit and binds it with a unique identifier, and realizes target node addressing through distributed directory service;

[0035] S42: Data encryption transmission is implemented based on the TLS 1.3 protocol. The legality and integrity of migration are ensured through identity certificate verification and hash verification. Incremental migration technology is used to improve efficiency.

[0036] S43: Employs a two-phase commit protocol to synchronize the operations of the source node and the target node, maintaining the consistency of the computing power status table of all nodes;

[0037] S44: Execute cross-domain computing power migration based on action A output by the decision module;

[0038] S45: Match the scheduled computing power with the tasks;

[0039] S46: Feedback the node status and task running status after execution to the cross-domain computing power perception module to form a closed-loop control.

[0040] A dynamic computing power allocation system based on reinforcement learning, comprising:

[0041] The cross-domain computing power perception module is used to cover all nodes in the cloud, edge, and terminal scenarios, and collect computing power resource status, task requirement characteristics, and cross-domain network condition data in real time.

[0042] The reinforcement learning decision module is used to construct a state space that integrates computing power, tasks, and networks, define the action space for computing power scheduling direction and proportion, and design a multi-objective reward function that balances resource utilization, task satisfaction rate, and long-term conflict avoidance. It relies on reinforcement learning algorithms to generate self-learning and self-iterative scheduling strategies and finally outputs cross-domain computing power scheduling instructions adapted to the current scenario.

[0043] The cross-domain scheduling feasibility verification module is used to receive the computing power scheduling action instructions output by the reinforcement learning decision module. It conducts real-time verification from three dimensions: hardware adaptability, network support, and task compatibility. It only transmits instructions that can be implemented and feeds back instructions that cannot be implemented to the reinforcement learning decision module for adjustment and optimization. At the same time, it synchronizes the verification results to the cross-domain computing power perception module to update the status.

[0044] The cross-domain computing power execution module is used to receive compliant instructions that have been verified for feasibility, execute cross-domain computing power migration from cloud to edge to device and precise matching of tasks and computing power; at the same time, it collects data after the instructions are executed in real time and feeds it back to the cross-domain computing power perception module.

[0045] As a preferred embodiment of the dynamic computing power allocation system based on reinforcement learning described in this invention, the cross-domain computing power sensing module includes:

[0046] The computing power data acquisition module is used to be deployed on cloud nodes, edge nodes, and terminal nodes to collect computing power data in real time.

[0047] The task requirements analysis module is used to identify task type, priority, and computing power requirement curve;

[0048] The network status monitoring module is used to collect cross-domain network bandwidth, latency, and packet loss rate to ensure the feasibility of data transmission during computing power scheduling.

[0049] As a preferred embodiment of the dynamic computing power allocation system based on reinforcement learning described in this invention, the reinforcement learning decision module includes:

[0050] The state space construction module is used to fuse the data from the cross-domain computing power perception module into a state vector S=[cloud node load, remaining computing power of edge nodes, number of high-priority tasks, cross-domain network latency];

[0051] The action space design module is used to define the computing power allocation action A;

[0052] The reward function optimization module is used to design multi-objective reward functions. ;in, This represents the current resource utilization rate. For task requirement fulfillment rate; This is a predicted value for resource conflicts in the next hour; , , Configurable weighting coefficients;

[0053] The agent training module uses the PPO algorithm to first train offline using historical data, and then fine-tunes it in real time during actual operation to continuously optimize decision-making capabilities.

[0054] As a preferred embodiment of the dynamic computing power allocation system based on reinforcement learning described in this invention, the cross-domain scheduling feasibility verification module includes:

[0055] The hardware compatibility verification module is used to retrieve the hardware parameter thresholds of the target node of the cross-domain computing power execution module and verify whether the scheduling instructions exceed the hardware carrying capacity.

[0056] The network support prediction module is used to combine the network data transmitted in real time by the cross-domain computing power perception module and predict whether the network can continuously support data transmission during the scheduling process through the computing power scheduling-network consumption mapping model.

[0057] The task compatibility verification module is used to associate the task requirement data of the cross-domain computing power perception module and verify whether the scheduling instructions conflict with the task characteristics.

[0058] The verification result processing mechanism module is used to generate a feasibility verification report when the verification is successful, and transmit the scheduling instructions and the report synchronously to the cross-domain computing power execution module to trigger the execution process; when the verification fails, it generates adjustment suggestions, feeds them back to the reinforcement learning decision module, and transmits the reason for the verification failure to the cross-domain computing power perception module to update the status data and assist the reinforcement learning decision module in regenerating instructions.

[0059] As a preferred embodiment of the dynamic computing power allocation system based on reinforcement learning described in this invention, the cross-domain computing power execution module includes:

[0060] The cross-domain resource abstraction and addressing module is used to abstract cloud, edge, and terminal computing resources into a unified computing unit and bind it with a unique identifier using the distributed computing power resource description specification, and realize target node addressing through distributed directory service.

[0061] The trusted migration module is used to implement encrypted data transmission based on the TLS 1.3 protocol. It ensures the legitimacy and integrity of the migration through identity certificate verification and hash verification, and improves efficiency by using incremental migration technology.

[0062] The consistency control module is used to synchronize the operations of the source node and the target node using a two-phase commit protocol, and maintain the consistency of the computing power status table of all nodes.

[0063] The computing power scheduling module is used to perform cross-domain computing power migration based on action A output by the decision module.

[0064] The task adaptation module is used to match the scheduled computing power with the task;

[0065] The status feedback module is used to feed back the node status and task running status after execution to the cross-domain computing power perception module to form a closed-loop control.

[0066] Compared with existing technologies:

[0067] 1. By using reinforcement learning agents to autonomously learn the computing power requirements of sudden scenarios and new types of tasks, without the need for manual pre-setting and modification of rules, it has the advantage of dynamically adapting computing power allocation to complex and changing scenarios.

[0068] 2. By collecting full-dimensional computing power data from cloud nodes, edge nodes, and terminal nodes through the cross-domain computing power perception module, and combining the cross-domain computing power migration and task adaptation capabilities of the cross-domain computing power execution module, the cloud, edge, and terminal computing power channels are opened up, which has the advantages of realizing cross-domain reuse of idle computing power and reducing resource waste.

[0069] 3. By incorporating future resource conflict predictions into the reward function through the reinforcement learning decision-making module, the agent is guided to consider both current computing power needs and subsequent task planning when making decisions. This has the advantages of avoiding subsequent computing power resource conflicts in advance and ensuring the supply of computing power for high-priority tasks. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0071] Figure 2 This is a schematic diagram of the cross-domain computing power perception module framework of the present invention;

[0072] Figure 3 This is a schematic diagram of the reinforcement learning decision module framework of the present invention;

[0073] Figure 4 This is a schematic diagram of the cross-domain scheduling feasibility verification module framework of the present invention;

[0074] Figure 5 This is a schematic diagram of the cross-domain computing power execution module framework of the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0076] This invention provides a dynamic computing power allocation method based on reinforcement learning. Please refer to [link / reference]. Figures 1-5 The specific steps are as follows:

[0077] S1: Covers all node scenarios in cloud, edge, and terminal, and collects real-time data on computing resource status, task requirements, and cross-domain network conditions.

[0078] The specific steps of S1 are as follows:

[0079] S11: Real-time collection of computing power data such as CPU / GPU utilization, memory usage, remaining computing power, and energy consumption;

[0080] S12: Identify task type (AI training, real-time interaction, data processing), priority (high / medium / low, such as medical diagnosis as high priority), and computing power demand curve (computing power demand that changes over time).

[0081] S13: Collect cross-domain network bandwidth, latency, and packet loss rate to ensure data transmission feasibility during computing power scheduling.

[0082] S2: First, based on the standardized data output by the cross-domain computing power perception module, the action space for computing power scheduling direction and proportion is defined by constructing a state space that integrates computing power, tasks, and networks. Then, a multi-objective reward function is designed to generate a scheduling strategy that can achieve self-learning and self-iteration by relying on reinforcement learning algorithms. Finally, a cross-domain computing power scheduling instruction adapted to the current scenario is output.

[0083] The specific steps of S2 are as follows:

[0084] S21: Fuse the data from the cross-domain computing power perception module into a state vector S=[cloud node load, remaining computing power of edge nodes, number of high-priority tasks, cross-domain network latency];

[0085] S22: Define computing power allocation action A, such as "30% computing power is allocated from the cloud to the edge", "idle computing power on the terminal is allocated to high-priority tasks", and "low-priority tasks are reduced by 5% of computing power".

[0086] The action of "allocating idle computing power of the terminal to high-priority tasks" must meet the terminal hardware characteristics (low power consumption, weak computing power) and operational constraints (power consumption, network stability), and its technical feasibility is achieved through the following mechanisms:

[0087] 1. Terminal resource proactive reporting and status assessment mechanism:

[0088] The terminal deploys a lightweight resource broker (such as a micro client based on MQTT, consuming less than 5% of CPU and less than 10MB of memory), which periodically (every 30 seconds by default, which can be dynamically adjusted) reports the following information to the edge nodes:

[0089] Basic hardware information: CPU architecture (e.g., ARMv8), number of available cores, remaining memory, computing power type (general computing / AI acceleration, such as whether NPU is supported);

[0090] Real-time status parameters: current battery level (percentage), network type (Wi-Fi / cellular), network connection status (online / weak / offline), and user workload (e.g., whether applications are running in the foreground).

[0091] Idle computing power declaration: The upper limit of schedulable computing power calculated based on its own load (e.g., "Current load is 20%, providing 1 CPU core + 512MB memory idle resources, estimated duration 10 minutes"). Edge nodes aggregate data reported by terminals to build a "terminal resource whitelist," only including terminals that meet basic conditions (battery level ≥ 30%, network latency ≤ 100ms, computing power matching task requirements) into the schedulable pool.

[0092] 2. Lightweight security isolation and task adaptation:

[0093] To address the limited resources of the terminal, a combination of "Function Compute + Resource Sandbox" is used to achieve task isolation.

[0094] Task preprocessing: High-priority tasks (such as lightweight image recognition for security events) are broken down into terminal-adapted micro-functions (such as WebAssembly-based compilation packages with a size of <5MB) at edge nodes, limiting single-task CPU usage to ≤30% and memory usage to ≤200MB to avoid affecting the terminal's native functions.

[0095] Sandbox isolation: Terminals run microfunctions through lightweight containers (such as DockerLite) or system-level sandboxes (such as LinuxNamespace), prohibiting access to local terminal storage (such as mobile phone photo albums, raw data from IoT device sensors), and only allowing reading of task input data and output calculation results;

[0096] Access control: End users can preset computing power scheduling permissions (such as "scheduling is allowed only when charging" and "maximum scheduling time of 30 minutes per day"). The agent module verifies permissions in real time and rejects unauthorized tasks.

[0097] 3. Network adaptation and network outage fault tolerance mechanism:

[0098] Solving terminal NAT traversal and network instability issues:

[0099] NAT traversal: Edge nodes act as relay proxies, and terminals establish communication with cloud / task nodes through UDP hole punching or edge forwarding (e.g., the terminal registers a temporary port with the edge node, and the edge node maintains a "terminal ID-temporary port" mapping table to realize the transmission of computing power scheduling instructions across NAT).

[0100] Weak network / network outage handling: When the terminal detects that the network quality is lower than the threshold (e.g., packet loss rate > 20%), it automatically pauses the task and saves the intermediate results (local temporary cache, occupying space < 100MB), and sends a "pause notification" to the edge node. If the network outage lasts for more than 30 seconds, the edge node triggers the task rollback mechanism to migrate the unfinished task to other terminals or edge nodes to avoid task interruption.

[0101] 4. Dynamic power and load balancing mechanism:

[0102] To prevent terminals from consuming excessive power or becoming overloaded due to excessive computing power scheduling:

[0103] Power threshold control: Set three power thresholds (high ≥50%, medium 30%-50%, low <30%). High-priority tasks are only scheduled when the power is ≥30%. When the power drops below 30%, the terminal agent actively sends an "unsubscribe notification" to the edge node, and the edge node stops allocating new tasks and gradually reclaims the scheduled computing power.

[0104] Load feedback adjustment: The terminal monitors its own load in real time (e.g., CPU utilization > 70%). If the load exceeds the limit due to scheduled tasks, it requests "download" from the edge node. The edge node dynamically reduces the task computing power allocation (e.g., from 1 core to 0.5 cores) or migrates some tasks.

[0105] S23: Design a multi-objective reward function ;in, The current resource utilization rate (the higher the rate, the higher the reward); Task requirement fulfillment rate (the weight of fulfillment rate for high-priority tasks is doubled); The predicted resource conflict value for the next hour (the lower the value, the higher the reward). , , Configurable weighting coefficients;

[0106] First of all The data source consists of 90 days of historical full data collected by the cross-domain computing power perception module (including task arrival time, execution duration, interruption recovery records, and resource consumption time-series curves for each node in the cloud-edge-device architecture) and real-time task queue data (including task types, priorities, and estimated power requirements). The historical data sampling frequency is 1 minute / time, and the real-time data update frequency is 10 seconds / time. High-concurrency reading is ensured through distributed time-series database storage.

[0107] also, This is achieved through a three-tier architecture: "task arrival modeling + load prediction + online correction".

[0108] 1) Task arrival modeling: For different types of tasks such as AI training (periodic), video analysis (Poisson arrival), and terminal interaction (bursting), arrival rate prediction sub-models are constructed using autoregressive model (ARIMA) and Poisson process model respectively, outputting the number of tasks arriving every 5 minutes in the next hour;

[0109] 2) Load prediction: Based on the LSTM model, the "task arrival prediction result + historical resource usage curve + real-time load" are fused to output the resource load prediction value every 5 minutes;

[0110] 3) Online Correction: Real-time data is compared with previous predictions every 10 minutes, and model parameters are corrected using Kalman filtering; a load threshold of 80% is set as the conflict line. =1 - (Number of conflict periods / 12), the fewer the number of conflict periods... The higher.

[0111] S24: The PPO (Proximity Policy Optimization) algorithm is adopted. It is first trained offline using historical data, and then fine-tuned in real time during actual operation to continuously optimize decision-making capabilities.

[0112] Specifically, to address the on-policy nature of the PPO algorithm (which relies on sampling data from the current policy), a two-level training system of "offline pre-training - online lightweight fine-tuning" is designed. This system resolves the conflict between real-time performance and stability through sampling optimization, training acceleration, and security mechanisms, as detailed below:

[0113] 1. Offline pre-training phase (data accumulation and model initialization):

[0114] 1.1 Historical Data Preprocessing: Extract scenario data (including cloud / edge / end resource status, task characteristics, network conditions and historical scheduling results) from the historical database of the cross-domain computing power perception module for the past 90 days. Divide the dataset according to scenario type (such as weekday peak / valley, sudden task scenario), remove invalid samples (such as abnormal scheduling caused by network interruption), and construct a training set containing more than 1 million samples.

[0115] 1.2 Parallel Sampling Environment Construction: 100 parallel simulation environments (simulating a cloud-edge-device heterogeneous architecture) were built using Docker containers. Each environment loaded historical data from different scenarios and underwent initial training using the policy network (actor) and value network (critic) of the PPO algorithm.

[0116] Policy network: Input state vector S, output action probability distribution (such as the direction and proportion of scheduling computing power);

[0117] Value network: Evaluates the value of the current state to assist in policy updates;

[0118] Training parameters: 2048 trajectories sampled per round, mini-batch size 512, learning rate 3e-4, ClippedSurrogate objective function (clip range [0.8, 1.2]), trained for 1000 epochs until reward function R converges (fluctuation ≤ 5%).

[0119] 1.3 Pre-trained model output: Save the converged policy network parameters (such as PyTorch model files) as the initial model for online fine-tuning, ensuring that the model has the basic scene scheduling capability.

[0120] 2. Online real-time fine-tuning stage (lightweight updates to adapt to dynamic scenarios):

[0121] To meet the real-time requirements of the production environment, an architecture of "edge-side sampling + center-side incremental training" is designed, with an end-to-end control latency of ≤5 seconds.

[0122] 2.1 Lightweight Parallel Sampling Mechanism:

[0123] Sampling Nodes: Deploy lightweight sampling agents (CPU usage ≤10%) on edge nodes. Each edge node is responsible for collecting real-time status and action feedback (such as load changes after the execution of scheduling instructions) of local and associated terminals, avoiding the network overhead of centralized sampling at the central node.

[0124] Sampling window: A 5-second sampling window is set. Within each window, 10 edge nodes collect a total of 128 trajectories in parallel (to meet the data requirements of the on-policy algorithm for the current policy). After the sampled data is cleaned by the cross-domain data fusion preprocessing module, it is compressed and transmitted to the central training node (data size ≤ 1MB).

[0125] 2.2 Incremental Training Strategy:

[0126] Freeze the low-level feature extraction network of the pre-trained model (such as the first 3 fully connected layers), update only the top-level policy head (the last 2 layers), and reduce the size of the training parameters (from 1 million parameters to 100,000).

[0127] Training iteration: Each round uses 128 new trajectories for 1-2 iterations (instead of multiple rounds of offline training), and low-precision computation (FP16) is used to accelerate backpropagation, with a single round of training taking ≤2 seconds;

[0128] Policy Constraints: A new "Policy Smoothing Factor" is added to limit the KL divergence between the updated policy and the original policy to ≤0.01, thus preventing policy abrupt changes caused by fine-tuning.

[0129] 2.3 Latency and Resource Consumption Control:

[0130] Sampling latency: Local sampling at edge nodes + compressed transmission ≤ 2 seconds;

[0131] Training latency: Incremental training + parameter update ≤ 2 seconds;

[0132] Resource consumption: The central training node uses a single GPU (such as NVIDIA T4), and the GPU utilization rate during training is ≤60%, which does not affect the core business of the system.

[0133] 3. Model deployment and security rollback mechanism:

[0134] 3.1 Dual-model hot deployment: The central node simultaneously deploys the "currently effective model" and the "model to be updated". After fine-tuning, the new model is first tested in a gray-scale test on 10% of the edge nodes (lasting 1 minute) to verify its scheduling effect (such as resource utilization improvement ≥5% and no task delay).

[0135] 3.2 Performance Monitoring: Monitor in real time whether the action commands output by the model cause abnormalities (such as node overload after scheduling, increased task delay), and set three alarm thresholds (minor abnormality: 1 time / minute; moderate abnormality: 3 times / minute; severe abnormality: 5 times / minute).

[0136] 3.3 Safety Rollback: If a moderate or higher level of anomaly is triggered, the deployment of the new model will be stopped immediately, and the system will automatically switch back to the "currently effective model". At the same time, the abnormal data will be recorded (as negative samples for the next round of fine-tuning) to avoid system failure due to model updates.

[0137] S3: It receives the computing power scheduling action instructions output by the reinforcement learning decision module, and conducts real-time verification from three dimensions: hardware adaptability, network support, and task compatibility. It only transmits instructions that can be implemented and feeds back instructions that cannot be implemented to the reinforcement learning decision module for adjustment and optimization. At the same time, it synchronously verifies the results and updates the status of the cross-domain computing power perception module.

[0138] The specific steps of S3 are as follows:

[0139] S31: Retrieve the hardware parameter thresholds (preset in the module database, including the maximum CPU / GPU load limit, memory capacity limit, and computing power access interface bandwidth) of the target node (such as the edge node or terminal node receiving computing power) of the cross-domain computing power execution module, and verify whether the scheduling instruction exceeds the hardware carrying capacity.

[0140] Example: The decision module outputs the instruction "the cloud allocates 40% of computing power to edge node C". The module queries the hardware threshold of edge node C (GPU maximum load 90%, currently at 60% load) and calculates that "the maximum computing power that can be received is 30%". It determines that the original instruction exceeds the hardware limit and triggers the adjustment mechanism.

[0141] S32: Combining the real-time network data (bandwidth, latency, packet loss rate) transmitted by the cross-domain computing power perception module, and through the computing power scheduling-network consumption mapping model (preset model, such as scheduling 10% of computing power requires 5Mbps bandwidth and latency ≤50ms), predict whether the network can continuously support data transmission during the scheduling process;

[0142] Example: The decision module commands "Terminal node D to allocate 20% of computing power to edge node E". The module calculates that it needs 10Mbps bandwidth. It monitors in real time that the current bandwidth of the link is 8Mbps and there is no trend of improvement in the next minute. It determines that the network support is insufficient and feeds back to the decision module to reduce the allocation ratio.

[0143] S33: Associate the task requirement data (task type, latency sensitivity threshold, data privacy level) of the cross-domain computing power awareness module, and verify whether the scheduling instructions conflict with the task characteristics;

[0144] Example: The decision module command "schedule the computing power of diagnostic tasks related to medical privacy data from the local terminal to the public cloud". The module verifies the privacy level of the task (high privacy, needs to be processed locally), determines that the command violates privacy requirements, triggers the "privacy priority" adjustment, and changes to scheduling the computing power of local edge nodes;

[0145] S34: When verification passes, a feasibility verification report (including hardware margin, network redundancy, and compatibility conclusions) is generated, and the scheduling instructions and report are synchronously transmitted to the cross-domain computing power execution module to trigger the execution process; when verification fails, adjustment suggestions are generated (such as "Edge node C only supports 30% computing power access, it is recommended to reduce the scheduling ratio to 30%" or "Insufficient link bandwidth, it is recommended to prioritize scheduling the computing power of terminals in the same area"), and fed back to the reinforcement learning decision module. At the same time, the reason for the verification failure is transmitted to the cross-domain computing power perception module to update the status data (such as marking "Edge node C hardware load limit 30%)" to assist the reinforcement learning decision module in regenerating instructions.

[0146] S4: Receives compliant instructions that have been verified for feasibility, executes cross-domain computing power migration and precise task-computing power matching between cloud, edge, and device; at the same time, it collects data such as node load changes, task operation effects, and resource utilization after instruction execution in real time, and feeds them back to the cross-domain computing power perception module.

[0147] The specific steps of S4 are as follows:

[0148] S41: Adopts a distributed computing power resource description specification to abstract computing power resources such as CPU, GPU, and memory of cloud, edge, and terminal nodes into a unified format of "computing power unit". Each unit is bound to a unique identifier (including node type, geographical location, and resource specifications). Through the distributed computing power node directory service, the mapping relationship between full node identifiers and network addresses is maintained in real time to achieve fast addressing of target nodes.

[0149] S42: End-to-end encryption of computing power migration data is performed based on the TLS1.3 protocol, and the legitimacy of both parties in the migration is ensured through the node identity certificate verification mechanism; incremental migration technology is adopted, only the incremental data associated with the computing power to be migrated is transmitted, and data integrity is guaranteed through hash verification.

[0150] S43: Perform cross-domain consistency guarantee: A two-phase commit protocol is adopted. In the first phase, pre-migration instructions are sent to the source node and the target node to confirm that the resources of both parties are ready. In the second phase, the computing power stripping and access operations are performed synchronously, and the computing power status table of the full node is updated synchronously to ensure that computing power is not duplicated or omitted during the migration process.

[0151] S44: Based on the output action A, perform cross-domain computing power migration (such as transferring computing power resources from the cloud to the edge).

[0152] S45: Match the scheduled computing power with tasks (e.g., assign low-latency tasks to the edge and high-computing-power tasks to the cloud).

[0153] S46: Feedback the node status and task running status after execution to the cross-domain computing power perception module to form a closed-loop control.

[0154] A dynamic computing power allocation system based on reinforcement learning, comprising:

[0155] The cross-domain computing power perception module is used to cover all nodes in the cloud, edge, and terminal scenarios, and collect computing power resource status, task requirement characteristics, and cross-domain network condition data in real time.

[0156] The reinforcement learning decision module is used to construct a state space that integrates computing power, tasks, and networks, define the action space for computing power scheduling direction and proportion, and design a multi-objective reward function that balances resource utilization, task satisfaction rate, and long-term conflict avoidance. It relies on reinforcement learning algorithms to generate self-learning and self-iterative scheduling strategies and finally outputs cross-domain computing power scheduling instructions adapted to the current scenario.

[0157] The cross-domain scheduling feasibility verification module is used to receive the computing power scheduling action instructions output by the reinforcement learning decision module. It conducts real-time verification from three dimensions: hardware adaptability, network support, and task compatibility. It only transmits instructions that can be implemented and feeds back instructions that cannot be implemented to the reinforcement learning decision module for adjustment and optimization. At the same time, it synchronizes the verification results to the cross-domain computing power perception module to update the status.

[0158] The cross-domain computing power execution module is used to receive compliant instructions that have been verified for feasibility, execute cross-domain computing power migration from cloud to edge to device and precise matching of tasks and computing power; at the same time, it collects data such as node load changes, task operation effects and resource utilization after instruction execution in real time and feeds them back to the cross-domain computing power perception module.

[0159] The cross-domain computing power sensing module includes:

[0160] The computing power data acquisition module is used to deploy on cloud nodes, edge nodes, and terminal nodes to collect computing power data such as CPU / GPU utilization, memory usage, remaining computing power, and energy consumption in real time.

[0161] The task requirements analysis module is used to identify task type (AI training, real-time interaction, data processing), priority (high / medium / low, such as medical diagnosis as high priority), and computing power requirement curve (computing power requirement that changes over time).

[0162] The network status monitoring module is used to collect cross-domain network bandwidth, latency, and packet loss rate to ensure the feasibility of data transmission during computing power scheduling.

[0163] The reinforcement learning decision module includes:

[0164] The state space construction module is used to fuse the data from the cross-domain computing power perception module into a state vector S=[cloud node load, remaining computing power of edge nodes, number of high-priority tasks, cross-domain network latency];

[0165] The action space design module is used to define computing power allocation action A, such as "dispatch 30% of computing power from the cloud to the edge", "allocate idle computing power of the terminal to high-priority tasks", and "reduce the computing power of low-priority tasks by 5%".

[0166] The reward function optimization module is used to design multi-objective reward functions. ;in, The current resource utilization rate (the higher the rate, the higher the reward); Task requirement fulfillment rate (the weight of fulfillment rate for high-priority tasks is doubled); The predicted resource conflict value for the next hour (the lower the value, the higher the reward). , , Configurable weighting coefficients;

[0167] The agent training module uses the PPO (Proximal Policy Optimization) algorithm to first train offline using historical data, and then fine-tune in real time during actual operation to continuously optimize decision-making capabilities.

[0168] The cross-domain scheduling feasibility verification module includes:

[0169] The hardware compatibility verification module is used to retrieve the hardware parameter thresholds (preset in the module database, including the maximum CPU / GPU load limit, memory capacity limit, and computing power access interface bandwidth) of the target node of the cross-domain computing power execution module (such as the edge node or terminal node receiving computing power) and verify whether the scheduling instructions exceed the hardware carrying capacity.

[0170] Example: The decision module outputs the instruction "the cloud allocates 40% of computing power to edge node C". The module queries the hardware threshold of edge node C (GPU maximum load 90%, currently at 60% load) and calculates that "the maximum computing power that can be received is 30%". It determines that the original instruction exceeds the hardware limit and triggers the adjustment mechanism.

[0171] The network support prediction module is used to combine the network data (bandwidth, latency, packet loss rate) transmitted in real time by the cross-domain computing power perception module, and predict whether the network can continuously support data transmission during the scheduling process through the computing power scheduling-network consumption mapping model (preset model, such as scheduling 10% of computing power requires 5Mbps bandwidth and latency ≤50ms).

[0172] Example: The decision module commands "Terminal node D to allocate 20% of computing power to edge node E". The module calculates that it needs 10Mbps bandwidth. If it detects in real time that the current bandwidth of the link is 8Mbps, it will determine that the network support is insufficient and will feed back to the decision module to reduce the allocation ratio.

[0173] The task compatibility verification module is used to associate the task requirement data (task type, latency sensitivity threshold, data privacy level) of the cross-domain computing power awareness module and verify whether the scheduling instructions conflict with the task characteristics.

[0174] Example: The decision module command "schedule the computing power of diagnostic tasks related to medical privacy data from the local terminal to the public cloud". The module verifies the privacy level of the task (high privacy, needs to be processed locally), determines that the command violates privacy requirements, triggers the "privacy priority" adjustment, and changes to scheduling the computing power of local edge nodes;

[0175] The verification result processing module generates a feasibility verification report (including hardware redundancy, network redundancy, and compatibility conclusions) when verification is successful. It then transmits the scheduling instructions and the report synchronously to the cross-domain computing power execution module, triggering the execution process. When verification fails, it generates adjustment suggestions (such as "Edge node C only supports 30% computing power access, it is recommended to reduce the scheduling ratio to 30%" or "Insufficient link bandwidth, it is recommended to prioritize scheduling computing power from terminals in the same region"), feeds them back to the reinforcement learning decision module, and transmits the reason for the verification failure to the cross-domain computing power perception module to update the status data (such as marking "Edge node C hardware load limit 30%)" to assist the reinforcement learning decision module in regenerating instructions.

[0176] The cross-domain computing power execution module includes:

[0177] The cross-domain resource abstraction and addressing module is used to abstract computing resources such as CPU, GPU, and memory of cloud, edge, and terminal nodes into a unified format of "computing power units" using the distributed computing power resource description specification. Each unit is bound to a unique identifier (including node type, geographical location, and resource specifications). Through the distributed computing power node directory service, the mapping relationship between full node identifiers and network addresses is maintained in real time to achieve fast addressing of target nodes.

[0178] The trusted migration module is used to perform end-to-end encryption of computing power migration data based on the TLS 1.3 protocol, and ensure the legitimacy of both parties during the migration through a node identity certificate verification mechanism; it adopts incremental migration technology, only transmits incremental data associated with the computing power to be migrated, and ensures data integrity through hash verification.

[0179] The consistency control module is used to employ a two-phase commit protocol. In the first phase, pre-migration instructions are sent to the source and target nodes to confirm that the resources of both parties are ready. In the second phase, the computing power stripping and access operations are executed synchronously, and the computing power status table of all nodes is updated synchronously to ensure that computing power is not duplicated or omitted during the migration process.

[0180] The computing power scheduling module is used to perform cross-domain computing power migration (such as transferring computing power resources from the cloud to the edge) based on action A output by the decision module.

[0181] The task adaptation module is used to match the scheduled computing power with the task (e.g., assigning low-latency tasks to the edge and high-computing-power tasks to the cloud).

[0182] The status feedback module is used to feed back the node status and task running status after execution to the cross-domain computing power perception module to form a closed-loop control.

[0183] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A dynamic computing power allocation method based on reinforcement learning, characterized in that, The specific steps are as follows: S1: Covers all node scenarios in cloud, edge, and terminal, and collects real-time data on computing resource status, task requirements, and cross-domain network conditions. S2: First, based on the standardized data output by the cross-domain computing power perception module, the action space for computing power scheduling direction and proportion is defined by constructing a state space that integrates computing power, tasks, and networks. Then, a multi-objective reward function is designed to generate a scheduling strategy that can achieve self-learning and self-iteration by relying on reinforcement learning algorithms. Finally, a cross-domain computing power scheduling instruction adapted to the current scenario is output. S3: It receives the computing power scheduling action instructions output by the reinforcement learning decision module, and conducts real-time verification from three dimensions: hardware adaptability, network support, and task compatibility. It only transmits instructions that can be implemented and feeds back instructions that cannot be implemented to the reinforcement learning decision module for adjustment and optimization. At the same time, it synchronously verifies the results and updates the status of the cross-domain computing power perception module. S4: Receives compliant instructions that have been verified for feasibility, executes cross-domain computing power migration and precise task-computing power matching between cloud, edge, and device; at the same time, it collects data after the instructions are executed in real time and feeds it back to the cross-domain computing power perception module.

2. The dynamic computing power allocation method based on reinforcement learning according to claim 1, characterized in that, The specific steps of S1 are as follows: S11: Real-time acquisition of computing power data; S12: Identify task type, priority, and computing power requirement curve; S13: Collect cross-domain network bandwidth, latency, and packet loss rate to ensure data transmission feasibility during computing power scheduling.

3. The dynamic computing power allocation method based on reinforcement learning according to claim 1, characterized in that, The specific steps of S2 are as follows: S21: Fuse the data from the cross-domain computing power perception module into a state vector S=[cloud node load, remaining computing power of edge nodes, number of high-priority tasks, cross-domain network latency]; S22: Define computing power allocation action A; S23: Design a multi-objective reward function ;in, This represents the current resource utilization rate. For task requirement fulfillment rate; This is a predicted value for resource conflicts in the next hour; , , Configurable weighting coefficients; S24: The PPO algorithm is adopted. It is first trained offline using historical data, and then fine-tuned in real time during actual operation to continuously optimize decision-making capabilities.

4. The dynamic computing power allocation method based on reinforcement learning according to claim 1, characterized in that, The specific steps of S3 are as follows: S31: Retrieve the hardware parameter threshold of the target node of the cross-domain computing power execution module and verify whether the scheduling instruction exceeds the hardware carrying capacity; S32: Combining the network data transmitted in real time by the cross-domain computing power perception module, the computing power scheduling-network consumption mapping model is used to predict whether the network can continuously support data transmission during the scheduling process; S33: Associate the task requirement data of the cross-domain computing power perception module and verify whether the scheduling instructions conflict with the task characteristics; S34: When the verification is successful, a feasibility verification report is generated, and the scheduling instructions and the report are synchronously transmitted to the cross-domain computing power execution module to trigger the execution process; when the verification fails, adjustment suggestions are generated and fed back to the reinforcement learning decision module. At the same time, the reason for the verification failure is transmitted to the cross-domain computing power perception module to update the status data and assist the reinforcement learning decision module in regenerating instructions.

5. The dynamic computing power allocation method based on reinforcement learning according to claim 1, characterized in that, The specific steps of S4 are as follows: S41: Adopts the distributed computing power resource description specification, abstracts cloud, edge, and terminal computing power resources into a unified computing power unit and binds it with a unique identifier, and realizes target node addressing through distributed directory service; S42: Data encryption transmission is implemented based on the TLS 1.3 protocol. The legality and integrity of migration are ensured through identity certificate verification and hash verification. Incremental migration technology is used to improve efficiency. S43: Employs a two-phase commit protocol to synchronize the operations of the source node and the target node, maintaining the consistency of the computing power status table of all nodes; S44: Execute cross-domain computing power migration based on action A output by the decision module; S45: Match the scheduled computing power with the tasks; S46: Feedback the node status and task running status after execution to the cross-domain computing power perception module to form a closed-loop control.

6. A dynamic computing power allocation system based on reinforcement learning, characterized in that, include: The cross-domain computing power perception module is used to cover all nodes in the cloud, edge, and terminal scenarios, and collect computing power resource status, task requirement characteristics, and cross-domain network condition data in real time. The reinforcement learning decision module is used to construct a state space that integrates computing power, tasks, and networks, define the action space for computing power scheduling direction and proportion, and design a multi-objective reward function that balances resource utilization, task satisfaction rate, and long-term conflict avoidance. It relies on reinforcement learning algorithms to generate self-learning and self-iterative scheduling strategies and finally outputs cross-domain computing power scheduling instructions adapted to the current scenario. The cross-domain scheduling feasibility verification module is used to receive the computing power scheduling action instructions output by the reinforcement learning decision module. It conducts real-time verification from three dimensions: hardware adaptability, network support, and task compatibility. It only transmits instructions that can be implemented and feeds back instructions that cannot be implemented to the reinforcement learning decision module for adjustment and optimization. At the same time, it synchronizes the verification results to the cross-domain computing power perception module to update the status. The cross-domain computing power execution module is used to receive compliant instructions that have been verified for feasibility, execute cross-domain computing power migration from cloud to edge to device and precise matching of tasks and computing power; at the same time, it collects data after the instructions are executed in real time and feeds it back to the cross-domain computing power perception module.

7. A dynamic computing power allocation system based on reinforcement learning according to claim 6, characterized in that, The cross-domain computing power sensing module includes: The computing power data acquisition module is used to be deployed on cloud nodes, edge nodes, and terminal nodes to collect computing power data in real time. The task requirements analysis module is used to identify task type, priority, and computing power requirement curve; The network status monitoring module is used to collect cross-domain network bandwidth, latency, and packet loss rate to ensure the feasibility of data transmission during computing power scheduling.

8. A dynamic computing power allocation system based on reinforcement learning according to claim 6, characterized in that, The reinforcement learning decision module includes: The state space construction module is used to fuse the data from the cross-domain computing power perception module into a state vector S=[cloud node load, remaining computing power of edge nodes, number of high-priority tasks, cross-domain network latency]; The action space design module is used to define the computing power allocation action A; The reward function optimization module is used to design multi-objective reward functions. ;in, This represents the current resource utilization rate. For task requirement fulfillment rate; This is a predicted value for resource conflicts in the next hour; , , Configurable weighting coefficients; The agent training module uses the PPO algorithm to first train offline using historical data, and then fine-tunes it in real time during actual operation to continuously optimize decision-making capabilities.

9. A dynamic computing power allocation system based on reinforcement learning according to claim 6, characterized in that, The cross-domain scheduling feasibility verification module includes: The hardware compatibility verification module is used to retrieve the hardware parameter thresholds of the target node of the cross-domain computing power execution module and verify whether the scheduling instructions exceed the hardware carrying capacity. The network support prediction module is used to combine the network data transmitted in real time by the cross-domain computing power perception module and predict whether the network can continuously support data transmission during the scheduling process through the computing power scheduling-network consumption mapping model. The task compatibility verification module is used to associate the task requirement data of the cross-domain computing power perception module and verify whether the scheduling instructions conflict with the task characteristics. The verification result processing mechanism module is used to generate a feasibility verification report when the verification is successful, and transmit the scheduling instructions and the report synchronously to the cross-domain computing power execution module to trigger the execution process; when the verification fails, it generates adjustment suggestions, feeds them back to the reinforcement learning decision module, and transmits the reason for the verification failure to the cross-domain computing power perception module to update the status data and assist the reinforcement learning decision module in regenerating instructions.

10. A dynamic computing power allocation system based on reinforcement learning according to claim 6, characterized in that, The cross-domain computing power execution module includes: The cross-domain resource abstraction and addressing module is used to abstract cloud, edge, and terminal computing resources into a unified computing unit and bind it with a unique identifier using the distributed computing power resource description specification, and realize target node addressing through distributed directory service. The trusted migration module is used to implement encrypted data transmission based on the TLS 1.3 protocol. It ensures the legitimacy and integrity of the migration through identity certificate verification and hash verification, and improves efficiency by using incremental migration technology. The consistency control module is used to synchronize the operations of the source node and the target node using a two-phase commit protocol, and maintain the consistency of the computing power status table of all nodes. The computing power scheduling module is used to perform cross-domain computing power migration based on action A output by the decision module. The task adaptation module is used to match the scheduled computing power with the task; The status feedback module is used to feed back the node status and task running status after execution to the cross-domain computing power perception module to form a closed-loop control.

Images