Dynamic personnel scheduling optimization method based on DDQN

Abstract

The invention relates to the technical field of intelligent manufacturing and dynamic personnel scheduling optimization, and discloses a dynamic personnel scheduling optimization method based on DDQN. According to the method, a DDQN intelligent decision framework oriented to the industrial scheduling problem is constructed, and the problem of Q value over-estimation in a traditional DQN algorithm is effectively solved by designing a multi-dimensional state space containing a process state, a team and a parallel mark and directly mapping an action space of an actual scheduling decision; and process states and worker behaviors are modeled through a deep reinforcement learning framework, and a task allocation and resource scheduling scheme is dynamically adjusted in combination with a real-time worker behavior prediction module, so that the total completion time is minimized and the resource utilization rate is maximized in a multi-process and multi-constraint scene, and the working efficiency is improved. The method is suitable for complex industrial environments such as chemical workshops and flexible manufacturing systems, scheduling challenges caused by process dependence, worker heterogeneity and dynamic task requirements can be effectively handled, and the production efficiency and the response capability of the intelligent manufacturing system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent manufacturing and dynamic personnel scheduling optimization, in particular to a dynamic personnel scheduling optimization method based on DDQN. BACKGROUND

[0002] In intelligent manufacturing systems, the workforce scheduling problem (WSP) is one of the key factors affecting production efficiency. Traditional scheduling methods, such as priority rules, linear programming, and heuristic algorithms, are effective in specific scenarios, but when faced with complex industrial environments such as multi-process dependence, worker skill differences, and dynamic task requirements, they often have low computational efficiency and weak generalization ability. In addition, traditional methods usually assume constant worker ability, ignoring dynamic behavior factors such as fatigue and task adaptability, resulting in deviations between scheduling schemes and actual execution results.

[0003] In recent years, deep reinforcement learning (DRL) has shown significant advantages in dynamic scheduling due to its strong adaptive learning ability and high-dimensional state space processing capability. However, existing research mostly focuses on static task allocation or fixed worker ability models, making it difficult to adapt to the dynamic changes in worker state in real production environments. In addition, existing methods still have room for improvement in parallel task scheduling and critical path optimization, especially in scenarios with strict process constraints and intense resource competition.

[0004] Therefore, there is an urgent need for an intelligent optimization method that combines dynamic prediction of worker behavior, adaptive task allocation, and efficient parallel scheduling to address the challenges of dynamic personnel scheduling in complex industrial environments. SUMMARY

[0005] The purpose of the present application is to provide a dynamic personnel scheduling optimization method based on DDQN to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides the following technical solution: a dynamic personnel scheduling optimization method based on DDQN, comprising the following steps: Step 1: Consider a typical chemical plant, with the processing process divided into multiple processes and the processes are grouped according to stages where P is the total number of stages, the current stage index; Step 2: The processing time of the process is affected by the number of processing workers assigned to the process, and the relationship between the number of processing workers and the processing time of the process is: ; where Tj D represents the team collaboration processing time for process j. j This indicates the originally specified baseline processing time for the process. S represents the number of workers assigned to process j for team k. k Let k be the total number of people in team k. This indicates the factor influencing the impact of personnel allocation on processing time. This represents the acceleration factor for teamwork, therefore the completion time C j : C j =s j +T j； s j Indicates the start time of process j; Step 3: Since personnel need to be assigned to process each stage of the process, therefore: ; Processing is carried out sequentially between stages; therefore, the processing of a step in a stage can only begin after all steps in the previous stage have been completed. ; Indicates the preceding stage intermediate process Completion time; Step 4: If the processes in a stage are processed sequentially, the processing time for a process can only begin after the previous process has been completed. ; When parallel processing exists, each team member can only handle one process at any given time. That is, the same person cannot participate in the processing of multiple processes simultaneously, and must wait for one process to be completed before starting the next process. ; ; in Representation phase Completion time, Representation phase Start time; Step 5: The goal of this scheme is to minimize the maximum processing time, where C J Indicates the last process: ; ; Step 6: In the scenario of this invention, the processing in the chemical workshop includes multiple stages, each stage has some procedures, and teams of different numbers of people are responsible for operating these procedures. The scheduling system, as an intelligent agent, uses a dual-deep Q-network algorithm to autonomously make decisions on the scheduling of relevant personnel. Step 7: Model the above chemical workshop personnel scheduling problem as a Markov decision process (MDP). In this framework, the scheduling system interacts with the production environment as an intelligent agent. The simulation process includes MaxIter training rounds, each round consisting of N time steps, and each step is defined as a decision moment (Episode). Step 8: After modeling the dynamic personnel scheduling optimization method as a Markov process, the scheduling center agent is trained using the DDQN algorithm; Step 9: After completing the training of the DDQN model, the next step is the implementation phase. In the implementation phase, the scheduling center agent uses its trained DDQN model to perform autonomous operations. Based on its trained DDQN model, the scheduling center agent observes the environment, obtains the current observation state, selects an optimal discrete action from the discrete action space, and determines the specific action based on the current state and the output of the policy network. The scheduling center agent executes the selected discrete action within the current episode.

[0007] Preferably, in step 1, the processing is divided into multiple steps, and each step can be handled by different teams of workers. Completed, each team The total number of people is S k Processes at the same stage can be processed in parallel or sequentially, using binary variables. express, stage The intermediate processes are processed in sequence. Representation phase The processes in the process can be carried out in parallel.

[0008] Preferably, step 5 also includes personnel constraints: when the process is carried out sequentially, team members need to cooperate to complete the process, and the number of participants must not exceed the total number of team members. When there is parallel processing at a certain stage, team members can be reasonably allocated so that they can participate in the processing of each parallel process. Regardless of sequential or parallel processing, at any stage, the total number of people participating in the processing must not exceed the total number of team members.

[0009] Preferably, the dual deep Q-network algorithm in step 6 works collaboratively through two deep neural networks: the online network is responsible for evaluating the Q-value of each action in real time and outputting the value estimate of all possible actions in the current state; the target network periodically synchronizes the parameters of the online network to provide a stable benchmark reference for training. The agent adopts an ε-greedy strategy for action selection, achieving a balance between exploration (randomly trying new actions) and utilization (selecting the current best action). During training, the system stores transfer samples through an experience replay mechanism and uses a dual Q-learning update method to avoid overestimation of value.

[0010] Preferably, in step 7, at the beginning of each episode, the scheduling agent first observes the environmental state and obtains information such as the current process progress, worker status, and resource availability. Based on the observed state and the current strategy, the agent calculates the Q-value of each discrete action (such as team allocation) through the DDQN policy network and selects actions based on the ε-greedy policy. After the action is executed, the environmental state is updated according to the process logic and resource constraints. At the same time, the agent obtains an immediate reward value (usually a negative value, reflecting time consumption). The policy network outputs the action probability distribution to guide the agent to learn the optimal scheduling decision in the dynamic environment. Finally, by minimizing the cumulative negative reward (i.e., the total completion time), the agent gradually optimizes the scheduling strategy to achieve efficient resource allocation and process collaboration.

[0011] Preferably, the specific process of the training phase in step 8 is as follows: Step 8.1: Initialize DDQN algorithm parameters (policy network θ, target network θ) - (1. Experience replay pool D, discount factor γ, exploration rate ε); The training is set to MaxIter loops, with each loop containing N episodes; Step 8.2: For the nth episode, take the nth episode as the current episode. Before the current episode starts, the agent observes the environment and obtains the current observation state s(n). This observation state includes various dynamic information in the current environment, such as process progress (not started / in progress / completed), worker team status, etc., where the value of n is n = 1, 2, ..., N; Step 8.3: At the start of the current episode, the scheduling agent calculates the Q-value of each discrete action (team assignment) based on the observed environmental state s(n) through the DDQN policy network, and selects actions using an ε-greedy policy: randomly selects a team assignment action with probability ε to achieve exploration; otherwise, it selects the action with the largest Q-value in the current state. Then the scheduling center agent executes hybrid actions within the current episode. After the action is executed, the environment is updated to the new state s(n+1) according to the process constraints and team collaboration time model. At the same time, the agent receives an immediate reward r(n), completing the state transition and reward feedback process of the current episode. Step 8.4: The scheduling center agent stores its own memories (including state, action, reward, and next state) in the experience replay pool within the current episode; Step 8.5: Let n = n + 1, start from the nth episode and move to the (n+1)th episode, then return to step 8.3 and continue execution until the N episodes of one loop are completed, then execute step 8.6; Step 8.6: If n%L = 0, it means that the condition n is divisible by L is met at time step n, i.e., an operation is performed every L time steps. The agent begins the parameter update phase: First, B transfer samples are randomly sampled from the experience replay pool to form a mini-batch of data; then, the target Q value is calculated using a dual-network architecture, where the target network θ - The parameters are used for stability evaluation; then, the policy network parameters θ are updated by minimizing the loss function; after every L parameter updates, the target network parameters are synchronized. Simultaneously, the exploration rate is adjusted according to a linear decay strategy. This ensures that the training process gradually transitions from full exploration to strategy utilization; Step 8.7: Set n=1, then return to step 8.2 and continue execution until the MaxIter loops end, completing the training phase of the scheduling center agent and obtaining the trained DDQN model.

[0012] Preferably, in step 7, the state space is obtained by the scheduling center agent observing the environment before the start of the nth episode. Defined as , where X j T represents the current status of the j-th process (not started / in progress / completed). j s represents the cooperative processing time of process j. j This represents the start time of the j-th process. This means that processes in the same stage can be processed in parallel or sequentially. The scheduling center, as an intelligent agent, can fully observe changes in the environment, collect all the above information, and update this information in each time slot. By accurately collecting state information, the intelligent agent can learn and apply the optimal strategy.

[0013] Preferably, in the action space of step 7, the online network and target network of the DDQN algorithm are initialized. Before the start of each episode, the scheduling center agent observes the environment and sets the current observation state. As input, it is fed into the online network, and each time slot n is the action executed by the scheduling center. , This represents the number of people assigned to team k in process j. The determination of the action at this point indicates that the allocation of the team by the scheduling center agent in the nth time slot has been determined.

[0014] Preferably, the reward function in step 7 is calculated at the end of each time step. Since the objective is to minimize the completion time, we define the immediate reward function as follows: ,in This represents the sum of the baseline times for all processes. To control the reward range within [-1, 0], avoid gradient explosion due to absolute time differences.

[0015] Preferably, in step 9, after the discrete action is completed, the environmental state changes, and the scheduling center agent observes the environment again to obtain the new observation state.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a DDQN intelligent decision-making framework for industrial scheduling problems. By designing a multi-dimensional state space that includes process status, team and parallel flags, and an action space that directly maps to actual scheduling decisions, the system can autonomously adapt to complex changes in the production environment. Furthermore, by adopting a separate online network and target network architecture, it effectively solves the problem of Q-value overestimation in the traditional DQN algorithm.

[0017] 2. In view of the problem of policy inaccuracy caused by limited training samples, this invention introduces the priority experience playback technology into the field of industrial scheduling, and dynamically adjusts the sample priority based on the time difference error.

[0018] 3. This invention enhances cross-domain adaptability, making it applicable not only to manufacturing scenarios such as chemical workshops but also extending to areas like emergency resource scheduling and multi-robot collaboration. By modeling process states and worker behavior using a deep reinforcement learning framework, combined with a real-time worker behavior prediction module, it dynamically adjusts task allocation and resource scheduling schemes to minimize total completion time and maximize resource utilization in multi-process, multi-constraint scenarios. This invention is suitable for complex industrial environments such as chemical workshops and flexible manufacturing systems, effectively addressing scheduling challenges arising from process dependencies, worker heterogeneity, and dynamic task requirements, thereby improving the production efficiency and responsiveness of intelligent manufacturing systems. Attached Figure Description

[0019] Figure 1 The network structure diagram of the DDQN algorithm provided by this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 As shown, a dynamic personnel scheduling optimization method based on DDQN includes the following steps: Step 1: Consider a typical chemical plant where the processing is divided into multiple steps. Furthermore, the processes are grouped according to stages. Where P is the total number of stages. Index for the current stage; Step 2: The processing time of a process is affected by the number of workers assigned to that process. The relationship between the number of workers and the processing time is as follows: ; Where T j D represents the team collaboration processing time for process j. j This indicates the originally specified baseline processing time for the process. S represents the number of workers assigned to process j for team k. k Let k be the total number of people in team k. This indicates the factor influencing the impact of personnel allocation on processing time. This represents the acceleration factor for teamwork, therefore the completion time C j : C j =s j +T j； s j Indicates the start time of process j; Step 3: Since personnel need to be assigned to process each stage of the process, therefore: ; Processing is carried out sequentially between stages; therefore, the processing of a step in a stage can only begin after all steps in the previous stage have been completed. ; Indicates the preceding stage intermediate process Completion time; Step 4: If the processes in a stage are processed sequentially, the processing time for a process can only begin after the previous process has been completed. ; When parallel processing exists, each team member can only handle one process at any given time. That is, the same person cannot participate in the processing of multiple processes simultaneously, and must wait for one process to be completed before starting the next process. ; ; in Representation phase Completion time, Representation phase Start time; Step 5: The goal of this scheme is to minimize the maximum processing time, where C J Indicates the last process: ; ; Step 6: In the scenario of this invention, the processing in the chemical workshop includes multiple stages, each stage has some procedures, and teams of different numbers of people are responsible for operating these procedures. The scheduling system, as an intelligent agent, uses a dual-deep Q-network algorithm to autonomously make decisions on the scheduling of relevant personnel. Step 7: Model the above chemical workshop personnel scheduling problem as a Markov decision process (MDP). In this framework, the scheduling system interacts with the production environment as an intelligent agent. The simulation process includes MaxIter training rounds, each round consisting of N time steps, and each step is defined as a decision moment (Episode). Step 8: After modeling the dynamic personnel scheduling optimization method as a Markov process, the scheduling center agent is trained using the DDQN algorithm; Step 9: After completing the training of the DDQN model, the next step is the implementation phase. In the implementation phase, the scheduling center agent uses its trained DDQN model to perform autonomous operations. Based on its trained DDQN model, the scheduling center agent observes the environment, obtains the current observation state, selects an optimal discrete action from the discrete action space, and determines the specific action based on the current state and the output of the policy network. The scheduling center agent executes the selected discrete action within the current episode.

[0022] Step 1 divides the processing into multiple steps, each of which can be handled by different teams of workers. Completed, each team The total number of people is S k Processes at the same stage can be processed in parallel or sequentially, using binary variables. express, stage The intermediate processes are processed in sequence. Representation phase The processes in the process can be carried out in parallel.

[0023] Step 5 also includes personnel constraints: when the process is carried out sequentially, team members must cooperate to complete the process, and the number of participants must not exceed the total number of team members. When there is parallel processing at a certain stage, team members can be reasonably allocated to participate in the processing of each parallel process. Regardless of sequential or parallel processing, at any stage, the total number of people participating in the processing must not exceed the total number of team members.

[0024] The dual-deep Q-network algorithm in step 6 works collaboratively through two deep neural networks: the online network is responsible for evaluating the Q-value of each action in real time and outputting the value estimate of all possible actions in the current state; the target network periodically synchronizes the parameters of the online network to provide a stable benchmark reference for training. The agent adopts an ε-greedy strategy for action selection, achieving a balance between exploration (randomly trying new actions) and utilization (selecting the current best action). During training, the system stores transfer samples through an experience replay mechanism and uses a dual-Q learning update method to avoid overestimating value.

[0025] In step 7, at the beginning of each episode, the scheduling agent first observes the environmental state, obtaining information such as the current process progress, worker status, and resource availability. Based on the observed state and the current policy, the agent calculates the Q-value of each discrete action (such as team assignment) through the DDQN policy network, and selects actions based on the ε-greedy policy. After the action is executed, the environmental state is updated according to the process logic and resource constraints. At the same time, the agent obtains an immediate reward value (usually a negative value, reflecting time consumption). The policy network outputs the action probability distribution to guide the agent to learn the optimal scheduling decision in the dynamic environment. Finally, by minimizing the cumulative negative reward (i.e., the total completion time), the agent gradually optimizes the scheduling policy to achieve efficient resource allocation and process collaboration.

[0026] The specific process of the training phase in step 8 is as follows: Step 8.1: Initialize DDQN algorithm parameters (policy network θ, target network θ) - (1. Experience replay pool D, discount factor γ, exploration rate ε); The training is set to MaxIter loops, with each loop containing N episodes; Step 8.2: For the nth episode, take the nth episode as the current episode. Before the current episode starts, the agent observes the environment and obtains the current observation state s(n). This observation state includes various dynamic information in the current environment, such as process progress (not started / in progress / completed), worker team status, etc., where the value of n is n = 1, 2, ..., N; Step 8.3: At the start of the current episode, the scheduling agent calculates the Q-value of each discrete action (team assignment) based on the observed environmental state s(n) through the DDQN policy network, and selects actions using an ε-greedy policy: randomly selects a team assignment action with probability ε to achieve exploration; otherwise, it selects the action with the largest Q-value in the current state. Then the scheduling center agent executes hybrid actions within the current episode. After the action is executed, the environment is updated to the new state s(n+1) according to the process constraints and team collaboration time model. At the same time, the agent receives an immediate reward r(n), completing the state transition and reward feedback process of the current episode. Step 8.4: The scheduling center agent stores its own memories (including state, action, reward, and next state) in the experience replay pool within the current episode; Step 8.5: Let n = n + 1, start from the nth episode and move to the (n+1)th episode, then return to step 8.3 and continue execution until the N episodes of one loop are completed, then execute step 8.6; Step 8.6: If n%L = 0, it means that the condition n is divisible by L is met at time step n, i.e., an operation is performed every L time steps. The agent begins the parameter update phase: First, B transfer samples are randomly sampled from the experience replay pool to form a mini-batch of data; then, the target Q value is calculated using a dual-network architecture, where the target network θ - The parameters are used for stability evaluation; then, the policy network parameters θ are updated by minimizing the loss function; after every L parameter updates, the target network parameters are synchronized. Simultaneously, the exploration rate is adjusted according to a linear decay strategy. This ensures that the training process gradually transitions from full exploration to strategy utilization; Step 8.7: Set n=1, then return to step 8.2 and continue execution until the MaxIter loops end, completing the training phase of the scheduling center agent and obtaining the trained DDQN model.

[0027] In step 7, the scheduling center agent observes the environment and obtains the state space before the start of the nth episode. Defined as , where X j T represents the current status of the j-th process (not started / in progress / completed). j s represents the cooperative processing time of process j. j This represents the start time of the j-th process. This means that processes in the same stage can be processed in parallel or sequentially. The scheduling center, as an intelligent agent, can fully observe changes in the environment, collect all the above information, and update this information in each time slot. By accurately collecting state information, the intelligent agent can learn and apply the optimal strategy.

[0028] In step 7, the action space is initialized, including the online network and target network of the DDQN algorithm. Before each episode begins, the scheduling center agent observes the environment and sets the current observation state. As input, it is fed into the online network, and each time slot n is the action executed by the scheduling center. , This represents the number of people assigned to team k in process j. The determination of the action at this point indicates that the allocation of the team by the scheduling center agent in the nth time slot has been determined.

[0029] The reward function in step 7 is calculated at the end of each time step. Since the objective is to minimize the completion time, we define the instantaneous reward function as follows: ,in This represents the sum of the baseline times for all processes. To control the reward range within [-1, 0], avoid gradient explosion due to absolute time differences.

[0030] In step 9, after the discrete action is completed, the environmental state changes, and the scheduling center agent observes the environment again to obtain the new observation state.

[0031] In the scenario of this invention, there is a scheduling center agent, multiple teams, and processes. The scheduling center agent operates using the standard DoubleDeepQ-Network (DDQN) algorithm. Communication between the scheduling center and team members is achieved through dedicated positioning devices to ensure the reliability and stability of communication. DDQN is a value-based method that uses two deep Q-networks with identical structures but different parameters. The first network, called the Online Network (parameter θ), is responsible for evaluating the value of discrete actions in real time, taking the state as input and outputting the value corresponding to each legal action. The second network is called the target network (parameter θ). - ), periodically synchronizing parameters from the online network to stably calculate the target Q value, thereby mitigating the overestimation of value problem. For example Figure 1As shown, both the online network and the target network contain an input layer, hidden layers, and an output layer. The hidden layer consists of two fully connected layers, and the number of neurons in the input layer and the two fully connected layers are 256, 128, and 64, respectively. The output layer has the same dimension as the discrete action space and directly provides the Q-value estimate for each action. During training, the system randomly samples mini-batch transfer samples through an experience replay mechanism, updates the gradients only in the online network, and completely copies the parameters of the online network to the target network every C steps to ensure stable convergence of training.

[0032] The dynamic personnel scheduling optimization problem described above is modeled as a Markov Decision Process (MDP). In this process, the scheduling center agent interacts with the chemical workshop production environment. The simulation consists of MaxIter training rounds, each round comprising N decision moments (Episodes). Before each Episode begins, the agent first observes the environment to obtain the current observation state. This state is composed of the process progress (not started / in progress / completed), the duration and start time of the j-th process, whether processes in the same stage can be processed in parallel or serially, and the current time step n. Based on this observation state, the agent selects an optimal discrete action from the discrete action space according to the Q-value distribution output by the DDQN policy network. After executing the action, the environment deterministically transitions to the next state based on process constraints, worker behavior models, and resource limitations. Simultaneously, the agent receives an immediate reward. The action space is dynamically generated according to process dependencies and personnel constraints. During training, experience playback is used to store data. It periodically synchronizes the target network until the MaxIter round ends.

[0033] In this specific example, in step 2, the observation state obtained by the scheduling center agent before the start of the nth episode is represented as: , Where n = 1, 2, ..., N, and N represents the total number of steps taken in a preset cycle. In the experiment, N = 1000, X j T represents the current status of the j-th process (not started / in progress / completed). j Let s represent the duration of the j-th process. j This represents the start time of the j-th process. This indicates that processes at the same stage can be processed in parallel or sequentially. All the aforementioned information is collected and updated in each time slot n. By accurately collecting state information, the agent can learn and apply the optimal policy. To address the policy bias caused by the scarcity or redundancy of samples in traditional uniform sampling experience replay, this system introduces Prioritized Experience Replay (PER). PER assigns priority based on the temporal difference error of each experience, enabling the agent to reuse transition samples that contribute significantly to policy improvement more frequently during training. This significantly improves the convergence speed and policy robustness of DDQN under limited sample conditions.

[0034] In a specific implementation case, the action space in step 2 initializes the online network and target network for the DDQN algorithm. Before each episode begins, the scheduling center agent checks the environment and sets the current state. As input, it is fed into the online network. The action selected by the scheduling center agent from the action space in the nth time slot. The determination process is as follows: Step 2.1: Discrete actions of the scheduling center agent , : The number of people allocated by the scheduling center in time slot n. If... Then the number of people assigned to team k in process j is The action network branch structure goes from the input layer to the hidden layer (two fully connected layers), and then to the output layer. At this point, the determination of the action indicates that the dispatch center agent has determined the team's allocation in the nth time slot.

[0035] In this specific embodiment, in step 2, the reward function is calculated at the end of each time step. Since the objective is to minimize the completion time, we define the immediate reward function as: ,in This represents the sum of the baseline times for all processes. To limit the reward range to [-1, 0], avoid gradient explosion due to absolute time differences. In this example, there are 6 teams to choose from, with 5, 10, 10, 5, 15, and 5 members respectively. They are responsible for processing 15 steps, with corresponding baseline processing times of 10, 5, 8, 6, 7, 9, 6, 7, 6, 7, 7, 7, 4, 7, and 5.

[0036] Here, Markov decision processes (MDFs) are an existing technique. They are discrete-time stochastic processes represented by a quadruple (S, A, H, R). Here, S represents the set of all possible system states, A represents the set of all possible actions the agent can take, H represents the state transition probability function, and R represents the reward function given to the agent, reflecting the value of each action in a specific environment. After modeling the scenario of minimizing the completion time of a chemical plant as a Markov decision process, deep reinforcement learning algorithms can be applied. Through continuous interaction between the agent and the environment, by calculating the value function of each state or the value function of a state-action pair and performing iterative optimization, the optimal policy is gradually approximated, thus forming an agent model capable of autonomous observation and decision-making in the scenario.

[0037] Step 3: After modeling the scenario of minimizing the completion time of the chemical workshop as a Markov decision process, the scheduling center agent is trained using the DDQN algorithm. The specific process of the training phase is as follows: Step 3.1: Initialize the algorithm parameters of the DDQN algorithm used by the scheduling center agent; initialize the experience replay pool of the agent; set the training phase to a total of MaxIter loops, with N episodes in each loop. In this example, MaxIter = 100 and N = 1000.

[0038] Step 3.2: For the nth episode, the nth episode is taken as the current episode. Before the current episode starts, the agent observes the environment and obtains the current observation state s(n). This observation state includes the current process state (not started / in progress / completed), the duration of the process, the start time of the process, and whether processes in the same stage can be processed in parallel or serially.

[0039] Step 3.3: Based on the observation state s(n) obtained by the scheduling center agent before the start of the current episode, including the process status and personnel team allocation, the agent inputs s(n) into the online Q-network using the DDQN algorithm. This yields Q-value estimates for all legal discrete actions. Then, an ε-greedy strategy is employed to select the action with the highest Q-value with a probability of 1-ε. By randomly exploring with probability ε, the action selected by the scheduling center agent within the current episode is obtained. Then the scheduling center agent executes actions within the current episode. After an action is completed, meaning team members are assigned to different processes, each process enters a different state, the environment is updated, and the scheduling center agent receives a reward value r(n) within the current episode. Then, at the end of the current episode (before the next episode begins), the agent observes the environment and obtains a new observation state s(n+1).

[0040] Step 3.4: The scheduling center agent stores its own memories (including state, action, reward, and next state) in the experience replay pool within the current episode.

[0041] Step 3.5: Let n = n+1, start from the nth episode and move to the (n+1)th episode, then return to step 3.3 and continue execution until the N episodes of a loop are completed, then execute step 3.6.

[0042] Step 3.6: If n%L = 0, it means that the condition n is divisible by L is met at time step n, i.e., an operation is performed every L time steps. DDQN will perform several time-based iterations on the collected data. In each iteration, the algorithm randomly collects a small batch of samples from the experience memory. Each batch of samples contains B pieces of experience data, which are generated by the online network. Generate. Utilize the collected quadruplets. With the target network Calculate the target value, input the batch status into the online network to obtain the current Q value, then minimize the mean square error and update the online network parameters θ; after completing a fixed number of steps, perform a hard copy. This is to maintain the stability of the target network. Then execute 3.7.

[0043] Step 3.7: Set n=1, then return to step 3.2 and continue execution until the MaxIter loops end, completing the training phase of the scheduling center agent and obtaining the trained DDQN model.

[0044] The dynamic personnel scheduling method based on dual-depth Q-network (DDQN) proposed in this invention adopts an "online-target" dual-network structure, including an online Q-network (parameter θ) and a target Q-network (parameter θ). - Both networks have hidden layers consisting of two fully connected layers, with 256, 128, and 64 neurons in the input layer and the two hidden layers, respectively; they directly output the Q-value of each legal action 'a'. The strategy and action value functions are unified by... This indicates that during the environment interaction phase, the scheduling center agent sends the current state s(n) to the online network. The formula for calculating the target Q-value of DDQN is: By using a policy network to select actions and a target network to evaluate the value of those actions, the overestimation problem is effectively mitigated. (Online network) Action selection based on the ε-greedy strategy: Execute discrete actions (That is, "the number of people assigned to team k in process j is") After that, the environment transitions to the next state based on process constraints and worker behavior models. And return the immediate reward r. The environment provides feedback on the immediate reward r and the next state. Generate quadruplets Store in experience replay pool D, and sort by TD - absolute value of error. Update the priority of this sample: ,in As the discount factor, this example takes Priority is defined as This ensures that samples that contribute more to strategy improvement are replayed more frequently. Once the number of samples in the replay pool reaches a preset threshold, the algorithm enters the learning phase. Every L=5 time steps, a batch of B experience samples is sampled from D according to priority F. Minimize the weighted mean square error update online network: The Adam optimizer (learning rate α = 0.01) is used to update the gradient of the online network parameters θ. After each update, the new TD error is calculated. Write back the corresponding sample and update its priority. After every 5 gradient steps, perform a hard copy. This ensures that the target network remains consistent with the online network.

[0045] Step 4: After completing the training of the DDQN model, the next step is the implementation phase. In the implementation phase, the scheduling center agent uses its trained DDQN model to perform autonomous operations. The scheduling center agent observes the environment based on its trained DDQN model, obtains the current observation state, selects an optimal discrete action from the discrete action space, and determines the specific action based on the current state and the output of the online network. The scheduling center agent executes the selected discrete action within the current episode. After the discrete action is completed, the environmental state changes, and the scheduling center agent observes the environment again to obtain the new observation state.

[0046] The feasibility and effectiveness of the DDQN dynamic personnel scheduling method of the present invention were further verified through simulation.

[0047] Experimental environment: Based on a typical chemical workshop, the following methods were adopted. Construct a discrete event simulator. The workshop comprises 5 processing stages and 15 processes (process number AO), with corresponding baseline processing times of 10, 5, 8, 6, 7, 9, 6, 7, 6, 7, 7, 7, 4, 7, 5. Six teams are responsible for completing these processes, with team sizes of 5, 10, 10, 5, 15, and 5 members respectively. Serial / parallel constraints exist between processes, perfectly corresponding to actual chemical processes.

[0048] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dynamic personnel scheduling optimization method based on DDQN, characterized in that, Includes the following steps: Step 1: Consider a typical chemical plant where the processing is divided into multiple steps. Furthermore, the processes are grouped according to stages. Where P is the total number of stages. Index for the current stage; Step 2: The processing time of a process is affected by the number of workers assigned to that process. The relationship between the number of workers and the processing time is as follows: ； Where T j D represents the team collaboration processing time for process j. j This indicates the originally specified baseline processing time for the process. S represents the number of workers assigned to process j for team k. k Let k be the total number of people in team k. This indicates the factor influencing the impact of personnel allocation on processing time. This represents the acceleration factor for teamwork, therefore the completion time C j : C j =s j +T j ; s j Indicates the start time of process j; Step 3: Since personnel need to be assigned to process each stage of the process, therefore: ； Processing is carried out sequentially between stages; therefore, the processing of a step in a stage can only begin after all steps in the previous stage have been completed. ； Indicates the preceding stage intermediate process Completion time; Step 4: If the processes in a stage are processed sequentially, the processing time for a process can only begin after the previous process has been completed. ; When parallel processing exists, each team member can only handle one process at any given time. That is, the same person cannot participate in the processing of multiple processes simultaneously, and must wait for one process to be completed before starting the next process. ； ； in Representation phase Completion time, Representation phase The start time; Step 5: The goal of this scheme is to minimize the maximum processing time, where C J Indicates the last process: ； ； Step 6: In the scenario of this invention, the processing in the chemical workshop includes multiple stages, each stage has some procedures, and teams of different numbers of people are responsible for operating these procedures. The scheduling system, as an intelligent agent, uses a dual-deep Q-network algorithm to autonomously make decisions on the scheduling of relevant personnel. Step 7: Model the above chemical workshop personnel scheduling problem as a Markov decision process (MDP). In this framework, the scheduling system interacts with the production environment as an intelligent agent. The simulation process includes MaxIter training rounds, each round consisting of N time steps, and each step is defined as a decision moment (Episode). Step 8: After modeling the dynamic personnel scheduling optimization method as a Markov process, the scheduling center agent is trained using the DDQN algorithm; Step 9: After completing the training of the DDQN model, the next step is the implementation phase. In the implementation phase, the scheduling center agent uses its trained DDQN model to perform autonomous operations. Based on its trained DDQN model, the scheduling center agent observes the environment, obtains the current observation state, selects an optimal discrete action from the discrete action space, and determines the specific action based on the current state and the output of the policy network. The scheduling center agent executes the selected discrete action within the current episode.

2. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: In step 1, the processing is divided into multiple steps, each of which can be handled by different teams of workers. Completed, each team The total number of people is S k Processes at the same stage can be processed in parallel or sequentially, using binary variables. express, stage The intermediate processes are processed in sequence. Representation phase The processes in the process can be carried out in parallel.

3. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: Step 5 also includes personnel constraints: when the process is carried out sequentially, team members need to cooperate to complete the process, and the number of participants must not exceed the total number of team members. When there is parallel processing at a certain stage, team members can be reasonably allocated to participate in the processing of each parallel process. Regardless of sequential or parallel processing, at any stage, the total number of people participating in the processing must not exceed the total number of team members.

4. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: The dual-deep Q-network algorithm in step 6 works collaboratively through two deep neural networks: the online network is responsible for evaluating the Q-value of each action in real time and outputting the value estimate of all possible actions in the current state; the target network periodically synchronizes the parameters of the online network to provide a stable benchmark reference for training. The agent adopts an ε-greedy strategy for action selection, achieving a balance between exploration (randomly trying new actions) and utilization (selecting the current best action). During training, the system stores transfer samples through an experience replay mechanism and uses a dual-Q learning update method to avoid overestimation of value.

5. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: In step 7, at the beginning of each episode, the scheduling agent first observes the environmental state and obtains information such as the current process progress, worker status, and resource availability. Based on the observed state and the current strategy, the agent calculates the Q-value of each discrete action (such as team assignment) through the DDQN policy network and selects actions based on the ε-greedy policy. After the action is executed, the environmental state is updated according to the process logic and resource constraints. At the same time, the agent obtains an immediate reward value (usually a negative value, reflecting time consumption). The policy network outputs the action probability distribution to guide the agent to learn the optimal scheduling decision in the dynamic environment. Finally, by minimizing the cumulative negative reward (i.e., the total completion time), the agent gradually optimizes the scheduling strategy to achieve efficient resource allocation and process collaboration.

6. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: The specific process of the training phase in step 8 is as follows: Step 8.1: Initialize DDQN algorithm parameters (policy network θ, target network θ) - (1. Experience replay pool D, discount factor γ, exploration rate ε); Set the training to a total of MaxIter loops, with each loop containing N episodes; Step 8.2: For the nth episode, take the nth episode as the current episode. Before the current episode starts, the agent observes the environment and obtains the current observation state s(n). This observation state includes various dynamic information in the current environment, such as process progress (not started / in progress / completed), worker team status, etc., where the value of n is n = 1, 2, ..., N; Step 8.3: At the start of the current episode, the scheduling agent calculates the Q-value of each discrete action (team assignment) based on the observed environmental state s(n) through the DDQN policy network, and selects actions using an ε-greedy policy: randomly selects a team assignment action with probability ε to achieve exploration; otherwise, it selects the action with the largest Q-value in the current state. ; Then the dispatch center agent performs hybrid actions within the current episode. After the action is executed, the environment is updated to the new state s(n+1) according to the process constraints and team collaboration time model. At the same time, the agent receives an immediate reward r(n), completing the state transition and reward feedback process of the current episode. Step 8.4: The scheduling center agent stores its own memories (including state, action, reward, and next state) in the experience replay pool within the current episode; Step 8.5: Let n = n + 1, start from the nth episode and move to the (n+1)th episode, then return to step 8.3 and continue execution until the N episodes of one loop are completed, then execute step 8.6; Step 8.6: If n%L = 0, it means that the condition n is divisible by L is met at time step n, i.e., an operation is performed every L time steps. The agent begins the parameter update phase: First, B transfer samples are randomly sampled from the experience replay pool to form a mini-batch of data; then, the target Q value is calculated using a dual-network architecture, where the target network θ - The parameters are used for stability evaluation; then, the policy network parameters θ are updated by minimizing the loss function; after every L parameter updates, the target network parameters are synchronized. Simultaneously, the exploration rate is adjusted according to a linear decay strategy. This ensures that the training process gradually transitions from full exploration to strategy utilization; Step 8.7: Set n=1, then return to step 8.2 and continue execution until the MaxIter loops end, completing the training phase of the scheduling center agent and obtaining the trained DDQN model.

7. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: In step 7, the scheduling center agent observes the environment before the start of the nth episode, and the obtained state space is defined as: , where X j T represents the current status of the j-th process (not started / in progress / completed). j s represents the cooperative processing time of process j. j This represents the start time of the j-th process. This means that processes in the same stage can be processed in parallel or sequentially. The scheduling center, as an intelligent agent, can fully observe changes in the environment, collect all the above information, and update this information in each time slot. By accurately collecting state information, the intelligent agent can learn and apply the optimal strategy.

8. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: In step 7, the action space is initialized, including the online network and the target network of the DDQN algorithm. Before each episode begins, the scheduling center agent observes the environment, inputting the current observation state (State) into the online network. Each time slot n is then executed by the scheduling center. , This represents the number of people assigned to team k in process j. The determination of the action at this point indicates that the allocation of the team by the scheduling center agent in the nth time slot has been determined.

9. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: The reward function in step 7 is calculated at the end of each time step. Since the objective is to minimize the completion time, we define the instantaneous reward function as follows: ,in This represents the sum of the baseline times for all processes. To control the reward range within [-1, 0], avoid gradient explosion due to absolute time differences.

10. The dynamic personnel scheduling optimization method based on DDQN according to claim 1, characterized in that: In step 9, after the discrete action is completed, the environmental state changes, and the scheduling center agent observes the environment again to obtain the new observation state.

Images