Heterogeneous perception task scheduling method and system based on personalized federal reinforcement learning

Abstract

The invention discloses a heterogeneous perception task scheduling method and system based on personalized federal reinforcement learning, and the method comprises the steps: constructing a task scheduling environment with a scene boundary based on load data, and constructing a local scheduling model based on the task scheduling environment through employing a dual-reviewer near-end strategy optimization algorithm; according to the method, public reviewer network model parameters in a constructed local scheduling model are aggregated, a personalized model is generated based on multi-head attention weights, and value estimation of local reviews and public reviews is fused by introducing Dual-Critic PPO, so that the problem of performance reduction of a global model in a heterogeneous environment is effectively relieved; and meanwhile, the stability of advantage estimation and the reliability of strategy updating are improved. According to the method, a server side carries out personalized aggregation on public reviewer parameters through a multi-head attention mechanism, so that different clients can obtain public models matched with environment characteristics of the clients instead of depending on a general average model, and thus faster convergence and stronger personalized adaptation are realized.

Description

Technical Field

[0001] This invention belongs to the technical field of federated learning and reinforcement learning, specifically relating to a heterogeneous perception task scheduling method and system based on personalized federated reinforcement learning. Background Technology

[0002] The increasing complexity of workload data generated in large-scale cloud environments makes collaborative training a viable direction for obtaining more efficient task schedulers; under the constraints of privacy, security, and data migration costs, federated reinforcement learning (FRL) is considered a promising solution. However, multiple studies have shown that environmental heterogeneity leads to a decline in the performance of FRL, making this problem even more challenging. Furthermore, simply aggregating workload data from different cloud providers is neither practical nor secure; although FRL can collaboratively train task scheduling models without exposing privacy data, it still has limitations. Further, different clouds are highly heterogeneous due to diverse service infrastructures and business participants. In this context, proposed methods include federated averaging (FPA), momentum-assisted federated policy optimization (MFPO), and personalized federated aggregation (PFA). FedPA boasts advantages such as simplicity, efficiency, and low communication overhead, but it is prone to slow convergence and suboptimal solutions in non-independent and identically distributed scenarios. MFPO, by introducing momentum and importance sampling to suppress gradient shift, exhibits better stability and sample utilization, but it is sensitive to hyperparameters and communication frequency, and increases directional storage and computational costs. In contrast, PFA has attracted attention for its ability to adaptively generate personalized models based on client differences, but it requires learning additional aggregation weights and increases model management complexity. However, in cloud environments, the data distribution and system capabilities of multiple clients are typically highly heterogeneous, making it difficult for traditional methods to achieve an effective balance between model convergence speed and optimal decision strategy acquisition. Summary of the Invention

[0003] The purpose of this invention is to provide a heterogeneous perception task scheduling method and system based on personalized federated reinforcement learning, so as to overcome the problems of slow global model convergence, degraded decision performance and lack of personalized adaptation in existing federated reinforcement learning in heterogeneous environments.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A heterogeneous perception task scheduling method based on personalized federated reinforcement learning includes the following steps: S1, Construct a task scheduling environment with scene boundaries based on load data, wherein the scene boundaries include state space, action space, and reward function; S2, a local scheduling model is constructed based on the task scheduling environment using a dual-critic near-end strategy optimization algorithm; S3 aggregates the parameters of the public critic network model in the constructed local scheduling model and generates a personalized model based on multi-head attention weights. S4. Based on the personalized model, update the parameters of the public critic network and the local critic network in the local scheduling model until the local scheduling model converges. Then, use the converged local scheduling model to generate a scheduling strategy based on the current environment and task information.

[0005] Preferably, the state space of the cloud computing task scheduling environment includes the machine's remaining capacity, CPU running status, and waiting queue task information.

[0006] Preferably, the action space is defined as a set {1,…,L,-1}, where 1 to L represent assigning the current task to the VM with the corresponding number, and -1 represents the current decision step performing a waiting operation. When any resource dimension of the selected VM cannot meet the task requirements, a constraint penalty mechanism based on resource utilization is triggered.

[0007] Preferably, the reward function uses a weighted combination of response time and load balancing terms:

[0008] in It increases as the task waiting time and execution time decrease. The penalty increases with the improvement of cluster resource load balancing; for invalid actions or resource over-provisioning, additional penalties based on resource utilization are applied.

[0009] Preferably, the value function of the dual-critic proximal policy optimization algorithm employs an adaptive weighted fusion of the two critic networks:

[0010] Among them, the weighting coefficient The weighting is dynamically adjusted based on the evaluation error of the two networks on the current trajectory batch, with the network with the smaller error receiving a higher weighting.

[0011] Preferably, the policy network update employs a truncated objective function for near-end policy optimization: the importance sampling ratio and the advantage estimate are compared. Construct a constrained optimization objective:

[0012] The advantage estimation is then jointly optimized by combining the critic mean squared error loss and the policy entropy regularization term with a pre-defined weighted total loss function. Calculated based on the generalized dominance estimation method.

[0013] Preferably, the personalized model is set on the server side, and the task scheduling environment is set on the client side. When the preset communication conditions or performance threshold are reached, the client only uploads the public critic parameters to the server. The server collects the public critic parameters from several clients, calculates the correlation weight between the parameters based on the multi-head attention mechanism, and uses the weight vector to perform weighted combination of the parameter set to generate personalized public critic parameters adapted to each client. At the same time, it maintains the global public critic parameters for clients that do not participate in this round to initialize.

[0014] A heterogeneous perception task scheduling system based on personalized federated reinforcement learning includes a parameter uploading unit, a local training unit, a personalized aggregation unit, and a joint update unit. The parameter upload unit constructs a task scheduling environment with scene boundaries based on load data. The scene boundaries include a state space, an action space, and a reward function. The local training unit constructs a local scheduling model based on a dual-critic proximal strategy optimization algorithm within the task scheduling environment. The personalized aggregation unit aggregates the parameters of the public critic network model in the constructed local scheduling model and generates a personalized model based on multi-head attention weights. The joint update unit updates the parameters of the public critic network and the local critic network in the local scheduling model based on the personalized model until the local scheduling model converges. Then, it uses the converged local scheduling model to generate a scheduling strategy based on the current environment and task information.

[0015] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned heterogeneous perception task scheduling method and system based on personalized federated reinforcement learning.

[0016] A computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements the aforementioned heterogeneous perception task scheduling method and system based on personalized federated reinforcement learning.

[0017] Compared with the prior art, the present invention has the following beneficial technical effects: This invention presents a heterogeneous perceptual task scheduling method based on personalized federated reinforcement learning. It constructs a task scheduling environment with scene boundaries based on load data. A local scheduling model is built using a dual-critic proximal policy optimization algorithm within this environment. The parameters of the common critic network model in the constructed local scheduling model are aggregated, and a personalized model is generated based on multi-head attention weights. The parameters of both the common and local critic networks in the local scheduling model are updated based on the personalized model until the local scheduling model converges. The converged local scheduling model generates a scheduling policy based on the current environment and task information. By introducing Dual-Critic PPO to fuse the value estimates of local and common critics, the performance degradation of the global model in heterogeneous environments is effectively mitigated, while simultaneously improving the stability of advantage estimation and the reliability of policy updates. On the server side, a multi-head attention mechanism is used to personalize the aggregation of common critic parameters, enabling different clients to obtain a common model that matches their environmental characteristics, rather than relying on a general average model, thus achieving faster convergence and stronger personalized adaptation.

[0018] Preferably, the invention only uploads parameters from public critics, which reduces communication burden and lowers the risk of privacy leaks. Simultaneously, personalized aggregation and local joint updates ensure the organic integration of global knowledge with the local environment. This method significantly improves key indicators such as task response time, completion time, resource utilization, and load balancing in multi-cloud heterogeneous and mixed load scenarios, demonstrating stronger robustness and long-term performance advantages. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the heterogeneous perception task scheduling method based on personalized federated reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] like Figure 1 As shown, this invention provides a heterogeneous perceptual task scheduling method based on personalized federated reinforcement learning, to achieve rapid convergence and long-term performance improvement of personalized scheduling strategies in multi-cloud heterogeneous and hybrid load environments. Specifically, it includes the following steps: S1, Construct a task scheduling environment with scene boundaries based on load data, formalized as a Markov decision process; the scene boundaries include state space, action space, and reward function; In a specific embodiment of this application, a task scheduling environment based on load data is constructed on the client and formalized as a Markov decision process.

[0023] Specifically, the client defines a state space and an action space locally. The state space includes the machine's remaining capacity, CPU running status, and waiting queue task information. The action space includes assigning a task to a VM or choosing to wait. The reward function is composed of a weighted average of response time and load balancing, and a constraint penalty is imposed when resources are insufficient or scheduling fails.

[0024] In a specific embodiment of the present invention, the action space is defined as a set {1,…,L,-1}, where 1 to L represent assigning the current task to the VM with the corresponding number, and -1 represents the current decision step performing a waiting operation. When any resource dimension of the selected VM cannot meet the task requirements, a constraint penalty mechanism based on resource utilization is triggered.

[0025] The reward function uses a weighted combination of response time and load balancing terms:

[0026] in It increases as the task waiting time and execution time decrease. The penalty increases with the improvement of cluster resource load balancing; for invalid actions or resource over-provisioning, additional penalties based on resource utilization are applied.

[0027] S2, based on the task scheduling environment, uses the Dual-Critic Proximal Policy Optimization (DPO) algorithm to build a local scheduling model. Specifically, it obtains the local scheduling model by interacting with the task scheduling environment and iteratively updating it based on Dual-Critic PPO, so as to maximize the cumulative reward obtained. The client performs local training based on the Dual-Critic PPO. Specifically, each client maintains both local and public critics, fusing their outputs through adaptive weights to obtain a stable state value estimate. Advantage values ​​are calculated using GAE, and the policy network is updated by truncating the PPO objective function. The critic network is optimized using mean squared error regression, improving the convergence speed and robustness of local training.

[0028] The dual-critic proximal policy optimization algorithm employs an adaptive weighted fusion of two critic networks:

[0029] Among them, the weighting coefficient The weighting is dynamically adjusted based on the evaluation error of the two networks on the current trajectory batch, with the network with the smaller error receiving a higher weighting.

[0030] The policy network is updated using a truncated objective function of near-end policy optimization: the importance sampling ratio and the advantage estimate are compared. Construct a constrained optimization objective:

[0031] The advantage estimation is then jointly optimized by combining the critic mean squared error loss and the policy entropy regularization term with a pre-defined weighted total loss function. Calculated based on the Generalized Advantage Estimation (GAE) method.

[0032] S3 aggregates the parameters of the public critic network model in the constructed local scheduling model and generates a personalized model based on multi-head attention weights. Specifically, the client uploads public critic parameters, and the server performs personalized aggregation: when the preset communication conditions or performance thresholds are reached, the client only uploads public critic parameters to the server. The server collects public critic parameters from several clients, calculates the correlation weights between parameters based on a multi-head attention mechanism, and uses weight vectors to weight and combine the parameter sets to generate personalized public critic parameters adapted to each client. At the same time, it maintains global public critic parameters for clients that did not participate in this round of initialization.

[0033] S4. Based on the personalized model, update the parameters of the public critic network and the local critic network in the local scheduling model until the local scheduling model converges. Then, use the converged local scheduling model to generate a scheduling strategy based on the current environment and task information.

[0034] In a specific embodiment of the invention, the server distributes a personalized model, and the client performs joint updates. Specifically, the server distributes personalized public critics to the corresponding clients. Upon receiving the data, the clients reset or align the public critics and local critics, and continue Dual-Critic PPO training to achieve joint optimization of personalized and global knowledge. This process is repeated until the task scheduling model converges. Specifically, training terminates when the reward function value stabilizes, the policy update magnitude decreases, or the maximum number of communication rounds is reached. The personalized task scheduling model is then output and used for online inference and task allocation.

[0035] The personalized aggregation on the server side is achieved through a multi-head attention mechanism. Specifically, the public critic parameters uploaded by each client are used as input to calculate the correlation matrix of the parameter space, generate a normalized attention weight vector, and perform a weighted summation of the parameter set based on this weight vector to generate personalized public critic parameters adapted to each client. At the same time, the global public critic parameters are protected for initialization by clients that did not participate in this round of communication.

[0036] The parameters of the public critic network and the local critic network in the local scheduling model are updated based on the personalized model until the local scheduling model converges. Specifically, a federated communication strategy is adopted, which follows these rules: parameter uploading is only initiated when a preset communication frequency or performance trigger threshold is reached; K clients are selected to participate in each round of aggregation; the uploaded content is limited to public critic network parameters, while the policy network parameters are kept updated locally to reduce communication overhead and privacy risks.

[0037] Newly connected clients are initialized using the global public critic model distributed by the server and participate in the model aggregation process in subsequent rounds.

[0038] In this embodiment of the invention, the security model adopts the honest but curious server assumption; without changing the core process, a security aggregation protocol, differential privacy mechanism or trusted execution environment can be integrated to enhance the privacy protection capabilities of client updates.

[0039] In one embodiment of the present invention, the client constructs a cloud computing task scheduling environment and formalizes the problem as a Markov decision process, defining the state space, action space, and reward function of the Markov decision process, so as to solve it using reinforcement learning methods. The specific process is as follows: Define the state space of a Markov decision process as:

[0040] The state consists of three parts: : Indicates the remaining capacity of the VM, specifically ,in Indicates the first i The VM in the first d The remaining amount of resources. : Indicates the running status of the virtual CPU, described by a percentage of completion, i.e., the percentage of completion. i The utilization of the k-th virtual CPU in a VM during task execution is denoted as . ,in . : Represents the task waiting queue, specifically in the form of ,in For the first i The waiting task is for the first... d The demand for similar resources.

[0041] The system can simultaneously characterize resource supply (VM capacity), resource usage (CPU utilization), and task demands (waiting queue), thereby ensuring that the scheduling agent has a comprehensive grasp of the environmental situation when making decisions. In cases of insufficient VMs, invalid values ​​are used to fill in the gaps, ensuring consistency in the state dimensions.

[0042] Define the action space as follows:

[0043] Where 1 to L indicate that the current task will be assigned to the VM with the corresponding number, and -1 indicates that the task will be waited for and will not be assigned immediately.

[0044] This design enables agents not only to learn how to select the most suitable VM, but also to learn when not to schedule and to delay decisions, thereby avoiding resource conflicts caused by blind allocation.

[0045] The reward function is designed as follows:

[0046] in, This is a weighting factor used to balance task response time and load balancing.

[0047] Task response time is defined as:

[0048] in, For task execution time, The waiting time for the task. The shorter the response time, the higher the service quality.

[0049] The corresponding reward for the response is defined as follows:

[0050] in, This is the normalization term. The formula ensures that the shorter the response time, the higher the reward value, thereby guiding the agent to learn and improve scheduling efficiency.

[0051] System load balancing is defined as:

[0052] in, This represents the load rate of the VM at time t for the i-th type of resource, where AvgLoad(t,i) is the average load rate. The more balanced the load, the more stable the system operation.

[0053] The reward for load balancing is defined as follows:

[0054] in, This represents the change in load balancing before and after task scheduling. The reward is 1 when task scheduling improves load balancing; if it exacerbates the load imbalance, the reward decreases accordingly.

[0055] To prevent agents from engaging in unreasonable scheduling behavior, a penalty term is introduced:

[0056] in, Indicates the VM at time [time]. No. Resource utilization. If a scheduling action results in exceeding the available resources of the VM or unreasonable waiting, a negative reward will be applied to prevent the agent from falling into a bad strategy of long waiting.

[0057] In one embodiment of the present invention, in order to simultaneously utilize the generalization ability of the global model and the environment adaptability of the local model, the client employs the Dual-Critic PPO method for training. By introducing a collaborative mechanism between the local critic network and the public critic network, the suboptimal policy problem caused by environmental heterogeneity is effectively alleviated. The specific process is as follows: Assume the first n The policy network of each client is denoted as Its parameters are At the k-th update, the optimization objective is defined as:

[0058] in, For the old policy, J is the core objective function of Proximal Policy Optimization (PPO):

[0059]

[0060] The above formula ensures that the policy update range is limited, avoiding excessive parameter fluctuations during training, thereby improving the stability of learning.

[0061] The advantage function is defined as:

[0062] in, For estimating the action value function, This is an estimate of the state value function.

[0063] The state-value function is represented by a weighted fusion of two critic networks:

[0064] In one embodiment of the invention, the weight coefficient α is adaptively adjusted based on the evaluation error of the two-way critic network:

[0065] in, and These represent the mean squared errors of local critics and public critics on their empirical trajectories, respectively. Critics with smaller evaluation errors will receive higher weights.

[0066] Through this mechanism, the agent can achieve a dynamic balance between local and global knowledge. Specifically, if the local environment differs significantly, the weight of local critics is increased; if the global model is more stable, the weight of public critics is increased.

[0067] The parameters of both critic networks are updated via gradient descent, ensuring that they converge synchronously during training and continuously provide high-quality value estimates for the policy network. The specific formula is as follows: Local critic network parameters updated:

[0068] Public critics network parameters updated:

[0069] in, Let τ be the client's experience buffer, and τ be the trajectory. For instant rewards.

[0070] In one embodiment of the present invention, to overcome the performance degradation problem of traditional FedAvg in heterogeneous environments, the server-side employs a multi-head attention mechanism to personalize the aggregation of public critic parameters uploaded by clients. Compared with existing aggregation methods, this method can automatically learn the relevance weights between clients, generating personalized models that match the environmental characteristics of different clients, thereby improving global convergence speed and local adaptability. The specific process is as follows: The server calculates the relevance weights between clients, making parameters with higher weights more closely resemble the environmental characteristics of the target client. The relevance weights are calculated using the following formula:

[0071] Where Q is the query matrix, representing the characteristics of the current target client; K is the key matrix, representing the characteristics of parameters uploaded by other clients; and V is the value matrix, corresponding to the set of parameters to be weighted. This is a scaling factor used to prevent gradient instability caused by excessively large values.

[0072] To enhance the model's expressive power, the server-side further employs a multi-head attention mechanism, allowing different attention heads to capture the similarities between clients from multiple perspectives, thereby more comprehensively modeling environmental differences. The calculation method is as follows:

[0073] The definition of each head is as follows:

[0074] , , The first Projected weight matrix of the head; These are the output weights after concatenation; The number of attention heads.

[0075] In one embodiment of the present invention, the above technical solution is described. Assume a federated cloud environment comprised of multiple cloud service providers (client A, client B, and client C), each with different hardware configurations, workload patterns, and user needs, resulting in a highly heterogeneous environment. These providers wish to collaboratively optimize their task scheduling strategies to improve resource utilization and task response speed while protecting their respective sensitive load data. In the local environments of clients A, B, and C, a task scheduling environment is constructed based on their historical load data. For example, client A's load data might show that its virtual machines (VMs) typically handle compute-intensive tasks, while client B handles more I / O-intensive tasks. Each client defines its state space (e.g., including the CPU, memory, and remaining disk capacity of each VM, as well as the type, size, and priority of tasks in the waiting queue), action space (e.g., assigning the current task to a VM or choosing to wait), and reward function (e.g., comprehensively considering task completion time, resource utilization, and load balancing).

[0076] Each client builds and trains its local scheduling model based on its locally constructed task scheduling environment using the Dual-Critic Proximity Policy Optimization (DPPO) algorithm. For example, client A's local scheduling model would include a policy network and a dual-critic network. The policy network is responsible for outputting the probability of task assignment based on the current state, while the dual-critic network evaluates the value of these assignments. During local training, client A continuously interacts with its environment, collects empirical data, and updates the parameters of its policy network and dual-critic network using the Dual-Critic DPPO algorithm.

[0077] When training reaches preset communication conditions (every 1000 training time steps or a certain performance improvement), clients A, B, and C upload the public critic network parameters from their local scheduling models to the central server. Upon receiving these parameters, the central server aggregates them using a multi-head attention mechanism and generates a personalized model for each client. Specifically, the server may analyze the similarity and differences between the parameters uploaded by client A and those uploaded by clients B and C, and calculate attention weights based on these correlations. For example, if the load patterns of clients A and B are similar in some aspects, their attention weights may be higher. The server uses these weights to weightedly combine the aggregated public critic parameters, thereby generating a personalized public critic model for client A that is more suitable for its specific environment.

[0078] The server distributes the generated personalized public critic model to the corresponding client. Upon receiving its personalized model, client A combines it with the parameters of its locally trained local critic network and continues local training. For example, client A might use the personalized public critic model as an update to its public critic network and continue training its local critic network and policy network using local data. This process continues until client A's local scheduling model converges, meaning its performance on local task scheduling no longer shows a significant improvement. Once converged, client A's local scheduling model can be put into practical use, generating optimal scheduling policies based on arriving tasks and the current state of its VMs, such as assigning a newly arrived computationally intensive task to a VM with low CPU utilization and sufficient memory.

[0079] Through the above methods, each client's local scheduling model not only learns from local data but also benefits from the experience of other clients through federated aggregation and personalization mechanisms. This improves task scheduling performance in heterogeneous environments while maintaining data privacy. The heterogeneous perceptual task scheduling method based on personalized federated reinforcement learning proposed in this invention demonstrates a significant technical contribution to addressing the challenges faced by federated reinforcement learning in heterogeneous cloud environments.

[0080] The present invention also provides a system for scheduling heterogeneous perception tasks based on the above-mentioned personalized federated reinforcement learning method, including a parameter uploading unit, a local training unit, a personalized aggregation unit and a joint update unit. The parameter upload unit constructs a task scheduling environment with scene boundaries based on load data. The scene boundaries include a state space, an action space, and a reward function. The local training unit constructs a local scheduling model based on a dual-critic proximal strategy optimization algorithm within the task scheduling environment. The personalized aggregation unit aggregates the parameters of the public critic network model in the constructed local scheduling model and generates a personalized model based on multi-head attention weights. The joint update unit updates the parameters of the public critic network and the local critic network in the local scheduling model based on the personalized model until the local scheduling model converges. Then, it uses the converged local scheduling model to generate a scheduling strategy based on the current environment and task information.

[0081] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a heterogeneous perception task scheduling method based on personalized federated reinforcement learning.

[0082] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the heterogeneous perception task scheduling method based on personalized federated reinforcement learning in the above embodiments.

[0083] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a specific manner, such that the instructions stored in the computer-readable storage medium produce a manufacturer including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A heterogeneous perception task scheduling method based on personalized federated reinforcement learning, characterized in that, Includes the following steps: S1, Construct a task scheduling environment with scene boundaries based on load data, wherein the scene boundaries include state space, action space, and reward function; S2, a local scheduling model is constructed based on the task scheduling environment using a dual-critic near-end strategy optimization algorithm; S3 aggregates the parameters of the public critic network model in the constructed local scheduling model and generates a personalized model based on multi-head attention weights. S4. Based on the personalized model, update the parameters of the public critic network and the local critic network in the local scheduling model until the local scheduling model converges. Then, use the converged local scheduling model to generate a scheduling strategy based on the current environment and task information.

2. The heterogeneous perception task scheduling method based on personalized federated reinforcement learning according to claim 1, characterized in that, The state space of the cloud computing task scheduling environment includes the machine's remaining capacity, CPU running status, and waiting queue task information.

3. The heterogeneous perception task scheduling method based on personalized federated reinforcement learning according to claim 1, characterized in that, The action space is defined as a set {1,…,L,-1}, where 1 to L represent assigning the current task to the VM with the corresponding number, and -1 represents the current decision step performing a waiting operation. When any resource dimension of the selected VM cannot meet the task requirements, a constraint penalty mechanism based on resource utilization is triggered.

4. The heterogeneous perception task scheduling method based on personalized federated reinforcement learning according to claim 1, characterized in that, The reward function uses a weighted combination of response time and load balancing terms: in It increases as the task waiting time and execution time decrease. The penalty increases with the improvement of cluster resource load balancing; for invalid actions or resource over-provisioning, additional penalties based on resource utilization are applied.

5. The heterogeneous perception task scheduling method based on personalized federated reinforcement learning according to claim 1, characterized in that, The value function of the dual-critic proximal policy optimization algorithm employs an adaptive weighted fusion of two critic networks: Among them, the weighting coefficient The weighting is dynamically adjusted based on the evaluation error of the two networks on the current trajectory batch, with the network with the smaller error receiving a higher weighting.

6. The heterogeneous perception task scheduling method based on personalized federated reinforcement learning according to claim 5, characterized in that, The policy network is updated using a truncated objective function of near-end policy optimization: the importance sampling ratio and the advantage estimate are compared. Construct a constrained optimization objective: The advantage estimation is then jointly optimized by combining the critic mean squared error loss and the policy entropy regularization term with a pre-defined weighted total loss function. Calculated based on the generalized dominance estimation method.

7. The heterogeneous perception task scheduling method based on personalized federated reinforcement learning according to claim 1, characterized in that, The personalized model is set on the server side, and the task scheduling environment is set on the client side. When the preset communication conditions or performance threshold are reached, the client only uploads the public critic parameters to the server. The server collects the public critic parameters from several clients, calculates the correlation weight between the parameters based on the multi-head attention mechanism, and uses the weight vector to weight and combine the parameter set to generate personalized public critic parameters adapted to each client. At the same time, it maintains the global public critic parameters for clients that do not participate in this round to initialize.

8. A heterogeneous perceptual task scheduling system based on personalized federated reinforcement learning, characterized in that, It includes a parameter upload unit, a local training unit, a personalized aggregation unit, and a joint update unit; The parameter upload unit constructs a task scheduling environment with scene boundaries based on load data. The scene boundaries include a state space, an action space, and a reward function. The local training unit constructs a local scheduling model based on a dual-critic proximal strategy optimization algorithm within the task scheduling environment. The personalized aggregation unit aggregates the parameters of the public critic network model in the constructed local scheduling model and generates a personalized model based on multi-head attention weights. The joint update unit updates the parameters of the public critic network and the local critic network in the local scheduling model based on the personalized model until the local scheduling model converges. Then, it uses the converged local scheduling model to generate a scheduling strategy based on the current environment and task information.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the heterogeneous perception task scheduling method and system based on personalized federated reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the heterogeneous perception task scheduling method and system based on personalized federated reinforcement learning as described in any one of claims 1-7.

Images