Heterogeneous equipment maintenance task distribution system and method based on A2C reinforcement learning

Abstract

The invention discloses a heterogeneous equipment maintenance task distribution system and method based on A2C reinforcement learning, and belongs to the technical field of reinforcement learning. The implementation method comprises the steps that 1, a server side establishes an A2C-GA hybrid model used for distribution of maintenance tasks of heterogeneous equipment in a workbench-task state, and hyper-parameters are configured for the model; the client side carries out parameter initialization on the working table-task state of the server side; 2, the server side updates the Actor network and the Critic network by utilizing action state interaction evaluation of the task allocation set, and performs parameter updating on the A2C model by utilizing the gradient loss of the Actor network and the Critic network; 3, executing the step 1 and the step 2 in an iterative loop mode until the fitness is improved to be less than 1%; 4, the server side visually outputs the task allocation set and the total time of task completion through the client side; compared with the prior art, the technical problem of improving the equipment maintenance task allocation accuracy in the existing heterogeneous equipment maintenance task allocation is solved.

Description

Technical Field

[0001] This invention relates to a heterogeneous equipment maintenance task allocation system and method based on A2C reinforcement learning, belonging to the field of reinforcement learning technology, and applied to the allocation environment between heterogeneous equipment maintenance tasks and workbenches. Background Technology

[0002] Enterprises are placing increasingly stringent demands on equipment reliability, operational efficiency, and cost control. Modern industrial equipment is also transforming towards higher complexity, higher integration, and higher automation. In today's rapidly evolving industrial manufacturing landscape, equipment maintenance processes must consider not only the time constraints of task allocation but also multi-dimensional optimization goals such as the actual working efficiency of workbenches and spare parts availability. Traditional precise methods, such as branch and bound, are no longer sufficient to meet such complex needs. However, with the advancement of AI technology providing computational support for real-time decision-making, the industry has begun to actively explore the application of reinforcement learning in equipment maintenance.

[0003] The core of equipment maintenance problems lies in how to formulate the optimal maintenance strategy to maximize equipment reliability or minimize maintenance costs under limited time, resources, and cost constraints. Currently, two main approaches are used to address equipment maintenance problems: First, exact methods are employed to guarantee finding the optimal solution. However, these methods are typically unsuitable for large-scale problems due to their extremely high computational complexity. For example, while dynamic programming can store solutions to subproblems and avoid redundant calculations in the 0-1 knapsack problem, it suffers from the curse of dimensionality when the knapsack capacity is too large. Second, heuristic methods are used to find feasible solutions within a reasonable timeframe through approximation strategies. However, these methods are overly sensitive to parameter selection, which significantly impacts computational power. In short, exact and heuristic methods can solve problems faster but produce suboptimal solutions with poor scalability and flexibility.

[0004] Therefore, in the existing allocation of maintenance tasks for heterogeneous equipment, how to improve the accuracy of equipment maintenance task allocation has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the technical problem of improving the accuracy of equipment maintenance task allocation in existing heterogeneous equipment maintenance task allocation systems, and to propose a heterogeneous equipment maintenance task allocation system and method based on A2C reinforcement learning.

[0006] The workflow of this invention is as follows: The A2C-GA hybrid model of this invention adopts an Actor-Critic architecture and a genetic algorithm for collaborative optimization. First, the server initializes an A2C agent population containing multiple hyperparameter variants, extracts and concatenates the feature vectors of the workbench data and task data into a state vector, providing a basis for network action selection. Second, each A2C agent selects an action and updates its environment state based on the current task-workbench state and its advantage function, while simultaneously calculating the immediate reward value. During training, the Critic network evaluates the merits of the Actor network's decisions and provides policy updates, while the genetic algorithm module periodically intervenes in the agent parameter optimization stage: it selects the optimal A2C agent through intergenerational fitness evaluation, performs hyperparameter cross-mutation on non-optimal A2C agents, specifically including learning rate interpolation adjustment, exploration rate perturbation, and discount factor optimization, and diffuses the superior parameter combinations to all A2C agents. Through the dual optimization mechanism of reinforcement learning and evolutionary algorithms, the state interaction-policy evaluation-parameter evolution cycle is repeated until the model converges to the optimal allocation method.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] This invention discloses a method for allocating maintenance tasks for heterogeneous devices based on A2C reinforcement learning, applicable to nodes containing multiple servers and multiple clients, comprising the following steps:

[0009] Step 1: On the server side, an A2C-GA hybrid model for heterogeneous equipment maintenance task allocation based on workbench-task status is established, and hyperparameters are configured for the model; on the client side, parameters for the workbench-task status on the server side are initialized.

[0010] Step 1.1: On the server side, establish an A2C model for the allocation of heterogeneous equipment maintenance tasks, consisting of workbench status and task status.

[0011] Step 1.2: The server automatically configures and initializes the hyperparameters for the A2C model, and initializes the genetic algorithm parameters at the same time;

[0012] Step 1.2.1: Set the default core parameters for reinforcement learning on the server side, including the discount factor, the learning rate of the Actor network, and the learning rate of the Critic network;

[0013] Step 1.2.2: Set the default training process parameters on the server side, including the number of training rounds, hidden layer dimension, batch size, memory capacity, and convergence threshold;

[0014] Step 1.2.3: Set the genetic method parameters on the server side, including population size, generation interval, crossover probability, and mutation probability; construct a single A2C agent using the A2C model; combine the single A2C agents according to a threshold number to form an A2C agent set; initialize the A2C agents on the server side.

[0015] Step 1.3: The client initializes the parameters for the server-side workbench-task status; specifically, this involves the workbench parameter θ. g and task parameters θ r Configure the number of worktables respectively and number of tasks

[0016] Step 1.4: Using the table parameter θ g and task parameters θ r Building the server-side workbench - task status parameters {θ g ,θ r};

[0017] Step 1.5: The server uses the workbench-task status parameters to build the workbench-task environment;

[0018] Step 1.5.1: The server initializes the environment parameters consisting of the initialization workbench state, initialization task state, and initialization scheduling record;

[0019] Step 1.5.2: The server-side configures the environment reset method, which consists of resetting all workbench states, resetting all task states, and clearing air conditioning temperature records;

[0020] Step 1.5.3: Extract state and concatenate the beginning and end of the state space using the task state and workbench state to form the current workbench-task state space ρ. x ;

[0021] Step 1.5.3.1: Construct the task state space ρ r and the workbench state space ρ g Set the task status to not started, in progress, or finished; set the workbench status to idle or working.

[0022] Step 1.5.3.2: Use the task state and the workbench state to concatenate the beginning and end of the state space to form the current workbench-task state space ρ. x ;

[0023] Step 1.6: The server uses the ε-greedy method to allocate tasks and workbenches, updates the workbench-task status, and then obtains immediate rewards.

[0024] Step 1.6.1: In the current workbench-task state space ρ x Next, Set the number of task assignment sets; select workbench-task pairs using the task-workbench task assignment probability distribution π(a|s);

[0025] Step 1.6.2: Task allocation is performed using the ε-greedy method. At each time step, random exploration is performed with a probability of ε, i.e., a random action is selected; the action with the highest probability output of the current task allocation set is selected with a probability of (1-ε), i.e., the deterministic output of the Actor network.

[0026] Step 1.6.3: Advance the simulation time using the actual task processing time, and update the workbench-task status using the workbench-task pair;

[0027] Step 1.6.4: Utilize the updated workbench - task status to obtain instant rewards;

[0028] Step 2: The server uses the action-state interaction evaluation of the task allocation set to update the Actor network and Critic network, and uses the gradient loss of the Actor network and Critic network to update the parameters of the A2C model.

[0029] Step 2.1: Construct an A2C model consisting of an Actor network and a Critic network; the server generates a task allocation set through the Actor network model and feeds back the allocated workbench-task status to the Critic network;

[0030] Step 2.1.1: Construct a system consisting of state dimensions Action dimension An Actor network consisting of an input layer and an output layer;

[0031] Step 2.1.2: The server uses the forward propagation of the input layer to extract features, obtains the probability distribution of the task allocation set as shown in Equation (1), and normalizes the probability distribution;

[0032] h=ReLU(W2·ReLU(W1s+b1)+b2),

[0033] π(a|s)=Softmax(W3h+b3) (1)

[0034] Where s is the workbench-task state space ρ x W1 and W2 are weight matrices, b1 and b2 are bias terms; π(a|s) is the task assignment set function, representing the probability of choosing action a in state s; W3 and b3 are the weights and biases of the output layer.

[0035] Step 2.1.3: Use π(a|s) to select tasks and obtain the workbench-task assignment status;

[0036] Step 2.2: The server evaluates the value of the workbench-task allocation state of the Actor network using the Critic network model and feeds it back to the Actor network.

[0037] Step 2.2.1: Construct a system consisting of state dimensions Action dimension The Critic network consists of an input layer and an output layer;

[0038] Step 2.2.2: The Critic network uses equation (3) to evaluate the workbench-task allocation status;

[0039] A(s,a)=R+γV(s ′ )-V(s) (3)

[0040] Where R is the immediate reward, i.e., the reward obtained after performing action a; γ is the discount factor, usually 0.9 to 0.99, used to reduce the influence of future rewards; and V(s) is the Critic network's estimate of the value of the current state. ′ ) represents the Critic network's estimate of the value of the next state;

[0041] Step 2.3: The server-side uses the gradient loss of the Actor network and Critic network as shown in Equation (4) to optimize the parameters of the A2C model;

[0042]

[0043] Wherein, V(s) i R represents the Critic network's estimate of the value of the current state. i For instant rewards, γ is the discount factor, and S i ‘ This is the Critic network's estimate of the value of the next state.

[0044] Step 2.4: The server samples the buffer data and uses the Adam optimizer to backpropagate and update the A2C model parameters.

[0045] Step 2.5: The server stores the interaction data, consisting of state, action, reward, next state, and termination flag, into the experience replay buffer using the experience storage method;

[0046] Step 2.6: All A2C agents on the server side share all information in the experience replay buffer and filter the parameters of the applicable A2C agents;

[0047] Step 2.6.1: The A2C agents on the server side share the interaction data in the experience replay buffer and perform A2C agent parameter interaction;

[0048] Step 2.6.2: Set the number of rounds for the generational interval, and use the average reward per round to obtain the fitness of the A2C agent;

[0049] Step 2.6.3: Set a fitness threshold and sort the fitness of A2C agents in descending order; save the parameters of A2C agents with fitness greater than the fitness threshold to the next round;

[0050] Step 2.6.4: Set the exploration rate ε to balance the exploration and obtain the action that tends to be applicable. Adjust the parameters of A2C agents with fitness less than the fitness threshold to the parameters of A2C agents with fitness greater than the fitness threshold in a linear manner until the exploration rate ε converges.

[0051] Step 2.6.5: Store A2C agents with fitness greater than the fitness threshold as snapshots for fault-tolerant performance rollback; use the snapshots for linear adjustment of A2C agent parameters in iterative rounds;

[0052] Step 3: Execute Step 1 and Step 2 iteratively until the fitness improvement is less than 1%;

[0053] Step 4: The server outputs the task allocation set and the total task completion time to the client for visualization.

[0054] This invention discloses a heterogeneous equipment maintenance task allocation system based on A2C reinforcement learning, used to implement the above-mentioned method. The heterogeneous equipment maintenance task allocation system based on A2C reinforcement learning disclosed in this invention includes a task management module, a scheduling optimization module, a resource management module, and a decision support module.

[0055] The task management module is used to monitor and adjust the lifecycle status of maintenance tasks in real time. Through an intelligent state machine mechanism, it dynamically marks the task status based on real-time equipment operating data, fault severity, and resource availability; this will serve as the input to the scheduling optimization module.

[0056] The scheduling optimization module is used to make intelligent allocation decisions for maintenance tasks based on the A2C-GA hybrid reinforcement learning algorithm. It dynamically generates a task allocation set by analyzing the workbench status, task priority, and resource constraints in real time, which will serve as the input to the resource management module.

[0057] The resource management module is used to obtain the types and quantities of spare parts required for maintenance tasks and automatically deduct virtual inventory; it will serve as input for the decision support module.

[0058] The decision support module is used to visualize the task allocation set, the instant reward trend chart, and the total time to complete the task.

[0059] Compared with existing technologies, it has the following beneficial effects:

[0060] 1. Multi-dimensional state modeling: The workbench state and maintenance task state are dynamically integrated, and the state vector is composed of the workbench state vector and the maintenance task state vector.

[0061] 2. Policy gradient optimization: In this experiment, the output layer of the Actor network implicitly implements a mask to avoid ineffective exploration of inefficient service stations; when calculating the advantage function, the Critic network implicitly learns the value assessment of the efficiency difference of service stations, guiding the Actor to prioritize the allocation of efficient service stations.

[0062] 3. Hybrid reward mechanism: The reward function in this experiment combines task completion rewards and time penalties, which is more in line with actual production conditions and balances the situation of excessively high and low maintenance efficiency; at the same time, a global time penalty is set that is negatively correlated with the service desk utilization rate, so that the more efficient service desks are allocated, the smaller the total time penalty will be. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the process of the present invention;

[0064] Figure 2 This is a schematic diagram of the fine-grained classification layer adjustment of the present invention;

[0065] Figure 3 This is a task allocation diagram for the present invention;

[0066] Figure 4 This is a reward trend chart for the present invention. Detailed Implementation

[0067] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0068] Example

[0069] like Figure 1 As shown in the figure, the specific implementation steps of the heterogeneous device maintenance task allocation method based on A2C reinforcement learning in this embodiment are as follows:

[0070] Step 1: On the server side, an A2C-GA hybrid model for heterogeneous equipment maintenance task allocation based on workbench-task status is established, and hyperparameters are configured for the model; on the client side, parameters for the workbench-task status on the server side are initialized.

[0071] Step 1.1: On the server side, establish an A2C model for the allocation of heterogeneous equipment maintenance tasks, consisting of workbench status and task status.

[0072] Step 1.2: The server automatically configures and initializes the hyperparameters for the A2C model, and initializes the genetic algorithm parameters at the same time;

[0073] Step 1.2.1: Set the default core parameters for reinforcement learning on the server side, including the discount factor, the learning rate of the Actor network, and the learning rate of the Critic network;

[0074] Step 1.2.2: Set the default training process parameters on the server side, including the number of training rounds, hidden layer dimension, batch size, memory capacity, and convergence threshold;

[0075] Step 1.2.3: Set the genetic method parameters on the server side, including population size, generation interval, crossover probability, and mutation probability; construct a single A2C agent using the A2C model; combine the single A2C agents according to a threshold number to form an A2C agent set; initialize the A2C agents on the server side.

[0076] Step 1.3: The client initializes the parameters for the server-side workbench-task status; specifically, this involves the workbench parameter θ. g and task parameters θ r Configure the number of worktables respectively and number of tasks

[0077] Step 1.4: Using the table parameter θ g and task parameters θ r Building the server-side workbench - task status parameters {θ g ,θ r};

[0078] Step 1.5: The server uses the workbench-task status parameters to build the workbench-task environment;

[0079] Step 1.5.1: The server initializes the environment parameters consisting of the initialization workbench state, initialization task state, and initialization scheduling record;

[0080] Step 1.5.2: The server-side configures the environment reset method, which consists of resetting all workbench states, resetting all task states, and clearing air conditioning temperature records;

[0081] Step 1.5.3: Extract state and concatenate the beginning and end of the state space using the task state and workbench state to form the current workbench-task state space ρ. x ;

[0082] Step 1.5.3.1: Construct the task state space ρ r and the workbench state space ρ g Set the task status to not started, in progress, or finished; set the workbench status to idle or working.

[0083] Step 1.5.3.2: Use the task state and the workbench state to concatenate the beginning and end of the state space to form the current workbench-task state space ρ. x ;

[0084] Step 1.6: The server uses the ε-greedy method to allocate tasks and workbenches, updates the workbench-task status, and then obtains immediate rewards.

[0085] Step 1.6.1: In the current workbench-task state space ρ x Next, Set the number of task assignment sets; select workbench-task pairs using the task-workbench task assignment probability distribution π(a|s);

[0086] Step 1.6.2: Task allocation is performed using the ε-greedy method. At each time step, random exploration is performed with a probability of ε, i.e., a random action is selected; the action with the highest probability output of the current task allocation set is selected with a probability of (1-ε), i.e., the deterministic output of the Actor network.

[0087] Step 1.6.3: Advance the simulation time using the actual task processing time, and update the workbench-task status using the workbench-task pair;

[0088] Step 1.6.4: Utilize the updated workbench - task status to obtain instant rewards;

[0089] In this embodiment, the server first configures the core parameters for reinforcement learning: setting the Actor network learning rate to 0.001, the Critic network learning rate to 0.002, and the discount factor to 0.95. Simultaneously, the genetic algorithm population size is initialized to 8 agents, with a generation interval of 30 training rounds. The client inputs initial configuration data for 10 workbenches and 15 maintenance tasks via an API interface, including feature parameters such as workbench type and task quantity. The server concatenates the workbench states and task states into a 25-dimensional state vector to construct the state space. The initial task allocation probability distribution π(a|s) is initialized with each element having equal value. Tasks are allocated according to the ε-greedy method. During the task allocation phase, a dynamic exploration rate ε is set with an initial value of 0.2, decaying by 5% per generation. The simulation time is advanced after each round of task allocation. In calculating the immediate reward, a validity check is first performed, i.e., if a task has been allocated or the machine is busy, a penalty reward τ is applied. Positive reward: completing task ε, with an additional 10ε for completing all tasks; Negative reward: total time penalty -0.1 * total time. The reward for the first round is 24.0, the reward for the second round is 86.1, the reward after 50 episodes is 24.0, the reward after 100 episodes is 120.3, the reward after 200 episodes is 142.5, and the reward converges to 168.9 at the 568th episode. After updating the workbench-task state in each round, the state transition data is stored through an experience replay mechanism with priority sampling.

[0090] Step 2: The server uses the action-state interaction evaluation of the task allocation set to update the Actor network and Critic network, and uses the gradient loss of the Actor network and Critic network to update the parameters of the A2C model.

[0091] Step 2.1: Construct an A2C model consisting of an Actor network and a Critic network; the server generates a task allocation set through the Actor network model and feeds back the allocated workbench-task status to the Critic network;

[0092] Step 2.1.1: Construct a system consisting of state dimensions Action dimension An Actor network consisting of an input layer and an output layer;

[0093] Step 2.1.2: The server uses the forward propagation of the input layer to extract features, obtains the probability distribution of the task allocation set as shown in Equation (1), and normalizes the probability distribution;

[0094] h=ReLU(W2·ReLU(W1s+b1)+b2),

[0095] π(a|s)=Softmax(W3h+b3) (1)

[0096] Where s is the workbench-task state space ρ x W1 and W2 are weight matrices, b1 and b2 are bias terms; π(a|s) is the task assignment set function, representing the probability of choosing action a in state s; W3 and b3 are the weights and biases of the output layer.

[0097] Step 2.1.3: Use π(a|s) to select tasks and obtain the workbench-task assignment status;

[0098] Step 2.2: The server evaluates the value of the workbench-task allocation state of the Actor network using the Critic network model and feeds it back to the Actor network.

[0099] Step 2.2.1: Construct a system consisting of state dimensions Action dimension The Critic network consists of an input layer and an output layer;

[0100] Step 2.2.2: The Critic network uses equation (3) to evaluate the workbench-task allocation status;

[0101] A(s,a)=R+γV(s ′ )-V(s) (3)

[0102] Where R is the immediate reward, i.e., the reward obtained after performing action a; γ is the discount factor, usually 0.9 to 0.99, used to reduce the influence of future rewards; and V(s) is the Critic network's estimate of the value of the current state. ′ ) represents the Critic network's estimate of the value of the next state;

[0103] Step 2.3: The server-side uses the gradient loss of the Actor network and Critic network as shown in Equation (4) to optimize the parameters of the A2C model;

[0104]

[0105] Wherein, V(s) i R represents the Critic network's estimate of the value of the current state. i For instant rewards, γ is the discount factor, and S i ‘ This is the Critic network's estimate of the value of the next state.

[0106] Step 2.4: The server samples the buffer data and uses the Adam optimizer to backpropagate and update the A2C model parameters.

[0107] Step 2.5: The server stores the interaction data, consisting of state, action, reward, next state, and termination flag, into the experience replay buffer using the experience storage method;

[0108] Step 2.6: All A2C agents on the server side share all information in the experience replay buffer and filter the parameters of the applicable A2C agents;

[0109] Step 2.6.1: The A2C agents on the server side share the interaction data in the experience replay buffer and perform A2C agent parameter interaction;

[0110] Step 2.6.2: Set the number of rounds for the generational interval, and use the average reward per round to obtain the fitness of the A2C agent;

[0111] Step 2.6.3: Set a fitness threshold and sort the fitness of A2C agents in descending order; save the parameters of A2C agents with fitness greater than the fitness threshold to the next round;

[0112] Step 2.6.4: Set the exploration rate ε to balance the exploration and obtain the action that tends to be applicable. Adjust the parameters of A2C agents with fitness less than the fitness threshold to the parameters of A2C agents with fitness greater than the fitness threshold in a linear manner until the exploration rate ε converges.

[0113] Step 2.6.5: Store A2C agents with fitness greater than the fitness threshold as snapshots for fault-tolerant performance rollback; use the snapshots for linear adjustment of A2C agent parameters in iterative rounds;

[0114] In this embodiment, the server extracts 256 batches of data from the experience replay buffer, containing a 25-dimensional state vector, a 10×15-dimensional action record history allocation matrix, and a normalized reward value calculated based on task completion time. First, the Critic network calculates the state value and generates an estimate of the advantage function (step 2.2.2). Then, the Actor network updates its parameters based on the current policy gradient, while using an Adam optimizer with a learning rate of 0.001 to perform backpropagation, and performs norm clipping of the gradient with a threshold of 0.5 to ensure training stability. After every 30 episodes, the system automatically calculates the performance metrics of each A2C agent, i.e., the average instantaneous reward, and selects the top three A2C agents. Agent C retains its parameters, while the remaining agents mutate the network weights and hyperparameters using Gaussian perturbations (μ = 0, σ = 0.1). The mutated parameter combinations are then loaded into the next generation of agents for further training. After 30 episodes, the average real-time rewards of the top three A2C agents were 16.8, 15.2, and 15.1, respectively, with a maximum discount factor of 0.99 and a maximum exploration rate ε of 0.19. After 60 episodes, the average real-time rewards of the top three A2C agents were 30.6, 29.6, and 28.4, respectively, with a maximum discount factor of 0.98 and a maximum exploration rate ε of 0.12. All interaction data were timestamped and stored in logs for subsequent analysis.

[0115] Step 3: Execute Step 1 and Step 2 iteratively until the fitness improvement is less than 1%;

[0116] In this embodiment, the parameters of the eight A2C agents are updated every 30 episodes. The optimal parameters for each generation, such as the learning rate and discount factor, are selected to make the parameters of the remaining A2C agents converge to the optimal parameters. When the optimal fitness improvement of the three consecutive generations is less than 1% is detected in the 600th episode, the algorithm terminates and the server records the task allocation set and the total task completion time.

[0117] Step 4: The server outputs the task allocation set and the total task completion time to the client for visualization.

[0118] In an embodiment, such as Figure 3 As shown, the horizontal axis represents time, the vertical axis represents the workstations, the working rate of each workstation is included, the rectangles of different colors represent each task, the entire Gantt chart shows the allocation of tasks and workstations, and the Mackespan in the header is the total working time of 2.77.

[0119] This embodiment provides a heterogeneous equipment maintenance task allocation system based on A2C reinforcement learning, used to implement the above-described method. This heterogeneous equipment maintenance task allocation system based on A2C reinforcement learning includes a task management module, a scheduling optimization module, a resource management module, and a decision support module.

[0120] The task management module is used to monitor and adjust the lifecycle status of maintenance tasks in real time. Through an intelligent state machine mechanism, it dynamically marks the task status based on real-time equipment operating data, fault severity, and resource availability; this will serve as the input to the scheduling optimization module.

[0121] The scheduling optimization module is used to make intelligent allocation decisions for maintenance tasks based on the A2C-GA hybrid reinforcement learning algorithm. It dynamically generates a task allocation set by analyzing the workbench status, task priority, and resource constraints in real time, which will serve as the input to the resource management module.

[0122] The resource management module is used to obtain the types and quantities of spare parts required for maintenance tasks and automatically deduct virtual inventory; it will serve as input for the decision support module.

[0123] The decision support module is used to visualize the task allocation set, the instant reward trend chart, and the total time to complete the task.

[0124] To further illustrate the advantages of this invention, the explanation is provided in conjunction with experimental data. For example... Figure 4 As shown, the reward curve of the A2C-GA hybrid model of this invention exhibits three-stage characteristics:

[0125] 1. Rounds 0-100: Rapid growth phase (average reward increases from -0.5 to 1.2), indicating that the genetic algorithm effectively accelerates the initial exploration;

[0126] 2. 100-300 rounds: Fluctuation optimization period (standard deviation decreases from 0.8 to 0.3), showing the effect of hyperparameter adaptive adjustment;

[0127] 3. After 300 rounds: Stable convergence period (100-round moving average reward reaches 1.85±0.05), which is faster than the traditional A2C model.

[0128] The results show that the present invention realizes a heterogeneous equipment maintenance task allocation system and method based on A2C reinforcement learning. By using the A2C-GA hybrid model, it overcomes the problem that heuristic algorithms produce suboptimal solutions with poor flexibility and insufficient generalization, and improves the accuracy of equipment maintenance task allocation.

[0129] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning, characterized in that: Includes the following steps, Step 1: On the server side, an A2C-GA hybrid model for heterogeneous equipment maintenance task allocation based on workbench-task status is established, and hyperparameters are configured for the model; on the client side, parameters for the workbench-task status on the server side are initialized. Step 2: The server uses the action-state interaction evaluation of the task allocation set to update the Actor network and Critic network, and uses the gradient loss of the Actor network and Critic network to update the parameters of the A2C model. Step 2.1: Construct an A2C model consisting of an Actor network and a Critic network; the server generates a task allocation set through the Actor network model and feeds back the allocated workbench-task status to the Critic network; Step 2.2: The server evaluates the value of the workbench-task allocation state of the Actor network using the Critic network model and feeds it back to the Actor network. Step 2.3: The server-side uses the gradient loss of the Actor network and Critic network as shown in Equation (4) to optimize the parameters of the A2C model; Wherein, V(s) i R represents the Critic network's estimate of the value of the current state. i For instant rewards, γ is the discount factor, and S i 'Estimation of the value of the next state for the Critic network;' Step 2.4: The server samples the buffer data and uses the Adam optimizer to backpropagate and update the A2C model parameters; Step 2.5: The server stores the interaction data, consisting of state, action, reward, next state, and termination flag, into the experience replay buffer using the experience storage method; Step 2.6: All A2C agents on the server side share all information in the experience replay buffer and filter the parameters of the applicable A2C agents; Step 3: Execute Step 1 and Step 2 iteratively until the fitness improvement is less than 1%; Step 4: The server outputs the task allocation set and the total task completion time to the client for visualization.

2. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: On the server side, establish an A2C model for the allocation of heterogeneous equipment maintenance tasks, consisting of workbench status and task status. Step 1.2: The server automatically configures and initializes the hyperparameters for the A2C model, and initializes the genetic algorithm parameters at the same time; Step 1.3: The client initializes the parameters for the server-side workbench-task status; specifically, this involves the workbench parameter θ. g and task parameters θ r Configure the number of worktables respectively and number of tasks Step 1.4: Using the table parameter θ g and task parameters θ r Building the server-side workbench - task status parameters {θ g ,θ r }; Step 1.5: The server uses the workbench-task status parameters to build the workbench-task environment; Step 1.6: The server uses the ε-greedy method to allocate tasks and workbenches, update the workbench-task status, and then obtain instant rewards.

3. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 2, characterized in that: Step 1.2 is implemented as follows: Step 1.2.1: Set the default core parameters for reinforcement learning on the server side, including the discount factor, the learning rate of the Actor network, and the learning rate of the Critic network; Step 1.2.2: Set the default training process parameters on the server side, including the number of training rounds, hidden layer dimension, batch size, memory capacity, and convergence threshold; Step 1.2.3: Set the genetic method parameters on the server side, including population size, generation interval, crossover probability and mutation probability; construct a single A2C agent using the A2C model; combine the single A2C agents according to the threshold number to form an A2C agent set; initialize the A2C agents on the server side.

4. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 2, characterized in that: Step 1.5 is implemented as follows: Step 1.5.1: The server initializes the environment parameters consisting of the initialization workbench state, initialization task state, and initialization scheduling record; Step 1.5.2: The server-side configures the environment reset method, which consists of resetting all workbench states, resetting all task states, and clearing air conditioning temperature records; Step 1.5.3: Extract state and concatenate the beginning and end of the state space using the task state and workbench state to form the current workbench-task state space ρ. x .

5. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 4, characterized in that: The implementation method for step 1.5.3 is as follows: Step 1.5.3.1: Construct the task state space ρ r and the workbench state space ρ g Set the task status to not started, in progress, or finished; set the workbench status to idle or working. Step 1.5.3.2: Use the task state and the workbench state to concatenate the beginning and end of the state space to form the current workbench-task state space ρ. x .

6. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 2, characterized in that: Step 1.6 is implemented as follows: Step 1.6.1: In the current workbench-task state space ρ x Next, Set the number of task assignment sets; select workbench-task pairs using the task-workbench task assignment probability distribution π(a|s); Step 1.6.2: Task allocation is performed using the ε-greedy method. At each time step, random exploration is performed with a probability of ε, i.e., a random action is selected; the action with the highest probability output of the current task allocation set is selected with a probability of (1-ε), i.e., the deterministic output of the Actor network. Step 1.6.3: Advance the simulation time using the actual task processing time, and update the workbench-task status using the workbench-task pair; Step 1.6.4: Utilize the updated workbench - task status to obtain instant rewards.

7. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 1, characterized in that: Step 2.1 is implemented as follows: Step 2.1.1: Construct a system consisting of state dimensions Action dimension An Actor network consisting of an input layer and an output layer; Step 2.1.2: The server uses the forward propagation of the input layer to extract features, obtains the probability distribution of the task allocation set as shown in Equation (1), and normalizes the probability distribution; h=ReLU(W2·ReLU(W1s+b1)+b2), π(a|s)=Softmax(W3h+b3) (1) Where s is the workbench-task state space ρ x W1 and W2 are weight matrices, b1 and b2 are bias terms; π(a|s) is the task assignment set function, representing the probability of choosing action a in state s; W3 and b3 are the weights and biases of the output layer. Step 2.1.3: Use π(a|s) to select task assignments and obtain the workbench-task assignment status.

8. The method for assigning maintenance tasks for heterogeneous equipment based on A2C reinforcement learning as described in claim 1, characterized in that: Step 2.2 is implemented as follows: Step 2.2.1: Construct a system consisting of state dimensions Action dimension The Critic network consists of an input layer and an output layer; Step 2.2.2: The Critic network uses equation (3) to evaluate the workbench-task allocation status; A(s,a)=R+γV(s ′ )-V(s) (3) Where R is the immediate reward, i.e., the reward obtained after performing action a; γ is the discount factor, usually 0.9 to 0.99, used to reduce the influence of future rewards; and V(s) is the Critic network's estimate of the value of the current state. ′ ) represents the Critic network's estimate of the value of the next state.

9. The method for assigning maintenance tasks to heterogeneous devices based on A2C reinforcement learning as described in claim 1, characterized in that: Step 2.6 is implemented as follows: Step 2.6.1: The A2C agents on the server side share the interaction data in the experience replay buffer and perform A2C agent parameter interaction; Step 2.6.2: Set the number of rounds for the generational interval, and use the average reward per round to obtain the fitness of the A2C agent; Step 2.6.3: Set a fitness threshold and sort the fitness of A2C agents in descending order; save the parameters of A2C agents with fitness greater than the fitness threshold to the next round; Step 2.6.4: Set the exploration rate ε to balance the exploration and obtain the action that tends to be applicable. Adjust the parameters of A2C agents with fitness less than the fitness threshold to the parameters of A2C agents with fitness greater than the fitness threshold in a linear manner until the exploration rate ε converges. Step 2.6.5: Store A2C agents with fitness greater than the fitness threshold as snapshots for fault-tolerant performance rollback; use the snapshots for linear adjustment of A2C agent parameters in iterative rounds.

10. A heterogeneous equipment maintenance task allocation system based on A2C reinforcement learning to implement the method as described in claim 1, characterized in that: It includes a task management module, a scheduling optimization module, a resource management module, and a decision support module; The task management module is used to monitor and adjust the life cycle status of maintenance tasks in real time. Through an intelligent state machine mechanism, the task status is dynamically marked according to the real-time operating data of the equipment, the severity of the fault, and the availability of resources. This will be used as input to the scheduling optimization module; The scheduling optimization module is used to make intelligent allocation decisions for maintenance tasks based on the A2C-GA hybrid reinforcement learning algorithm. It dynamically generates a task allocation set by analyzing the workbench status, task priority, and resource constraints in real time, which will serve as the input to the resource management module. The resource management module is used to obtain the types and quantities of spare parts required for maintenance tasks and automatically deduct virtual inventory. This will be used as input to the decision support module; The decision support module is used to visualize the task allocation set, the instant reward trend chart, and the total time to complete the task.

Images