Flexible assembly job shop green dynamic multi-target scheduling method under personnel learning effect

Abstract

The invention discloses a flexible assembly job shop green dynamic multi-target scheduling method under a personnel learning effect, and relates to the technical field of shop scheduling. According to the method, a multi-objective double-agent planning model is established according to green dynamic multi-objective scheduling requirements of a flexible assembly job shop under a personnel learning effect, and the dynamic multi-objective scheduling problem of the model is solved based on a double-layer deep reinforcement learning framework method in combination with machine and personnel double-resource constraints; a state space and an action space are designed according to a workshop simulation environment, and reward functions of instant reward and round reward are combined; the intelligent agent interacts with a scheduling environment, and obtains a better scheduling rule at a scheduling point; through the above method, the decision-making efficiency of a manufacturing enterprise is improved, a better solution can be adaptively and rapidly generated, the loss caused by delay time is effectively reduced, the energy consumption is reduced, and the method has certain generalization and stability.

Description

Technical Field

[0001] The present invention relates to the field of workshop scheduling, and in particular to a green dynamic multi-objective scheduling method for a flexible assembly workshop under a personnel learning effect based on deep reinforcement learning. Background Art

[0002] In recent years, with the development of globalization and computer technology, many manufacturing companies have gradually shifted from the traditional job shop model to the assembly job shop model, which can reduce material costs and improve factory production efficiency. Compared with the classic flexible job shop scheduling problem, the products scheduled by the low-carbon flexible assembly job shop are composed of multiple modular parts. After each part is processed in the flexible job shop, it is transported to the assembly shop for assembly to complete the finished product. Therefore, the assembly job production model is more suitable for the actual production environment and can better meet the user's personalized customization needs. The field of deep reinforcement learning scheduling based on assembly workshops is still in its early and middle stages, which makes it challenging to design scheduling solutions. In addition, for multi-objective scheduling algorithms, the shortcomings of many algorithm frameworks are that at each rescheduling point, the algorithm only chooses to optimize a certain goal, and cannot better coordinate the conflicts between multiple goals.

[0003] With the implementation and promotion of industrial manufacturing strategies, manufacturing systems are developing towards a more customized trend, and human-machine collaboration has become the mainstream development trend of intelligent manufacturing. In the manufacturing process of large-scale equipment, complex welding or assembly processes make it difficult to rely on robot automation, and high-quality workers have become a key production resource. The personnel learning effect is an important factor affecting the scheduling problem of dual-resource flexible job shops. Therefore, workshop managers need to decide on the efficiency allocation of machines and workers. In addition, workshop job scheduling is one of the important means for manufacturing companies to reduce carbon emissions. Researchers have made certain contributions to green scheduling, but these contributions are mainly theoretical and cannot meet the requirements of actual scheduling. For example, they do not consider energy saving during actual machine operation, use green scheduling indicators to build a model that is more in line with actual production scheduling, and develop more effective algorithms to solve green scheduling.

[0004] In summary, flexible job shop scheduling is an NP-Hard problem, and the extended topic will be more complicated with the addition of human factors and environmental factors. The low-carbon dynamic multi-objective scheduling problem of flexible assembly job shops under dual resource constraints is a novel and meaningful topic, representing a complex production environment including job shops and assembly shops, with the characteristics of large scale, dual constraints, complex environment, and strong dynamics. And because the scheduling process is dynamic, emergencies such as emergency orders may occur during the production process. These incidental events may affect the efficiency of the previously generated scheduling plan, and the scheduling goal will change from the optimality of the plan to rapid rationality. Summary of the invention

[0005] The present application provides a green dynamic multi-objective scheduling method for a flexible assembly workshop under the personnel learning effect, which takes minimizing the maximum completion time and the total energy consumption of the processing process as the optimization goal, and solves the dynamic low-carbon multi-objective scheduling problem of a flexible assembly workshop under dual resource constraints by using a dual-agent-based solution method; According to the first aspect disclosed in the present application, the present application provides a green dynamic multi-objective scheduling method for a flexible assembly workshop under the personnel learning effect, comprising the following steps: S1: According to the green dynamic multi-objective scheduling requirements of flexible assembly workshops under the personnel learning effect, a green dynamic multi-objective scheduling planning model for flexible assembly workshops under the personnel learning effect is established; S2: Use a dual-agent structure to solve the green dynamic multi-objective scheduling planning model of a flexible assembly workshop under the human learning effect, and build a two-layer deep reinforcement learning framework to coordinate and solve multi-objective conflicts; S3: Construct a state space and action space that matches the problem model and algorithm framework, and propose an immediate reward function and a delayed reward function.

[0006] In a feasible implementation manner, the step S1, according to the green dynamic multi-objective scheduling requirements of the flexible assembly workshop under the personnel learning effect, considers the maximum completion time and total energy consumption of multiple workpieces in the processing and assembly process, and establishes a green dynamic multi-objective scheduling planning model for the flexible assembly workshop under the personnel learning effect. The model parameters are as follows: (1) The objective function includes the function for calculating the total processing time and the total energy consumption function : The maximum completion time is the maximum end time of the last process of all workpieces. The function is composed of all workpieces Maximum completion time calculate, is the total number of workpieces, Used to return the largest value from a set of values. Used to return the smallest value from a set of values; ; The total energy consumption of the processing process includes processing energy consumption and idle energy consumption; Indicates processing energy consumption: ; in, Indicates workers Operating Equipment Execution process The initial processing time, Representation of workpiece No. process, Indicates the device Unit processing energy consumption, Indicates the judgment process Whether by workers On the device 0-1 decision variables performed on Indicates workers By machine Processing procedures The actual learning rate at Indicates the total number of devices. is the total number of personnel, For workpiece The total number of processes; Indicates the idle energy consumption of computing equipment: ; in, Indicates the idle energy consumption of computing equipment. Indicates the device Unit idle energy consumption, Indicates the process By workers On the device The start time on Indicates the process By workers On the device The end time on Indicates the judgment process On the device Is the post-process on 0-1 decision variables, Indicates the judgment process Whether by workers On the device 0-1 decision variables performed on For workpiece The total number of processes, For workpiece The total number of processes; (2) A series of constraints include: Limit a process to be operated by one person on one piece of equipment; ; The actual processing time of a process is equal to the end time minus the start time. By the workers On the device Upper operation process The initial processing time is the actual processing time, which is obtained from the initial processing time and learning rate. By the workers On the device Upper operation process The end time of the operation, By the workers On the device Upper operation process The start time of the operation, Indicates workers By machine Processing procedures The actual learning rate at ; ; Process Processing completion time =Equal to the start processing time of the process plus the process processing time, equal to the worker Operating Equipment Processing procedures End time: ; The process of each workpiece must follow the priority order from front to back, that is, the process By workers In the machine Start time on Not less than the previous process of the same workpiece By workers In the machine End time of the run ; ; If you want to process different workpieces on one device, they must be done in sequence. 1, indicating a process On the device The post-process on , 1, indicating a process On the device The post-process on , only one of the two situations is true, process By workers In the machine The start time of the above operation is , end time is , process By workers In the machine The start time of the above operation is , end time is ;

[0007] ; Any machine can only operate one process at a time, that is, the worker In the machine The last process after processing Start time No less than workers On the same machine Previous process End time , is a positive number; ; Any person can only operate one process at a time. Indicates workers Operate the machine first Re-operate the machine , Indicates workers Can operate the machine The start time of ; Worker Operating the machine Processing procedures Start time Not less than the machine The end time of the previous process is equal to the worker In the machine The start time of the operation ; ; Worker Operate the machine again The interval before It is equivalent to the last time the machine was operated. End time To operate the machine again Start time The interval time between ; Worker Processing to The actual learning rate of the worker at the time Number of times worked on related, Dynamically adjust the learning rate, A worker The learning effect coefficient, It is the incompressible coefficient of the learning effect. The more times, the lower the learning rate, and vice versa. ; Assembling workpieces The first process Start time is no earlier than The end time of all predecessor jobs, Indicated by workers Operating the machine Processing procedures The start time of Representation of workpiece The set of pre-requisites, Representation of workpiece The last step, Indicated by workers operate Processing procedures End time of .

[0008] In a feasible implementation, the step S2 proposes a dual-agent framework to solve the green dynamic multi-objective scheduling planning model of a flexible assembly workshop under the learning effect of personnel, and constructs a two-layer deep reinforcement learning framework to coordinate and solve multi-objective conflicts; Dual agents are used to achieve multiple goals, and a hierarchical multi-action space is designed. Each space is managed by an agent based on DQN (DeepQ-Network). When the framework is running, the state information will be first transmitted to the upper-layer DQN agent. After the output result is obtained, the result and the environment state characteristics will be transmitted to the lower-layer DQN agent to obtain the final scheduling result of this time step. The high-level DQN agent is a controller. The output result is used to determine the preference value of the optimization target of the current time step, guide the selection of scheduling actions by the lower-level DQN agent, and thus affect the scheduling of the entire environment state; using the target preference value will better coordinate the conflict between multiple targets, and will not make the lower-level action selection only optimize target 1 or target 2, but will be biased on the basis of balance; the low-level DQN agent receives the optimization preference value of the high-level DQN agent, uses it as the target parameter of the reward function, and then selects the current scheduling point according to the rules of the action space scheduling algorithm through the input environment state information. The action is right; The dual-agent real-time control workshop production process is as follows: (1) High-level DQN The high-level DQN agent acts as a controller. At each rescheduling point, it takes the state feature as input and the Q value of each target as output. It passes the output target probability value as a weight to the low-level DQN agent. Target value in DQN algorithm The calculation method is as follows: ; in, is in state Next action After receiving the instant reward, is the discount factor, are the target network parameters, The target network For the next state Maximum Q value estimation of ; The error calculation formula between the estimated value and the target value in the current state, that is, the loss function, is as follows: ; The agent is in state Next action The immediate reward obtained from the environment reflects the direct effect of the current action, is the discount factor, is the next state of the target network The maximum Q value estimate is, Is the main network's current state and actions Q value estimation of ; (2) Low-level PPO The low-level PPO (Proximal Policy Optimization) agent acts as an executor, taking the state features and the output targets of the high-level DQN agent as input, and the Q value of the workpiece-machine sequence pair as output. The workpiece-machine with the highest Q value is selected as the action selected by the current scheduling point. The following is the objective function of the PPO algorithm, by finding the parameters θ Maximize this function: ; It is The objective function of PPO at the iteration is: yes t The state of the moment, yes tActions taken at all times, In the current state Take Action strategy, is the importance sampling ratio, which represents the probability ratio of the new strategy to the old strategy and is used to measure the change of the new strategy compared to the old strategy. Indicates that the importance sampling ratio is limited to between, is the parameter that limits the range; It is The strategy parameters at the iteration, is the advantage function, It is The advantage function at the iteration is To evaluate the status Take action The advantages and disadvantages of It means obtaining the minimum value among a set of target values, which can effectively prevent excessive deviation from the old strategy when the strategy is updated; The decision-making part of the algorithm adopts the Actor-Critic algorithm, which is divided into the Critic evaluation network and the Actor strategy network. The former inputs the environment state and action to evaluate the value of taking the action in the current state, while the latter inputs the current state and outputs the probability distribution of the action. The Critic network is then used to evaluate the quality of the action and adjust the strategy to improve the expected return of future actions. The Critic network uses the mean square error loss function, that is, ; is the total number of samples, Corresponding to samples, is the mean square error loss function value obtained by the Critic network, and Samples to calculate the cumulative reward value Compare the status with the Critic network and actions Q-value estimation The mean square error between them promotes the Critic network to quickly correct the error, and finally The sum of the errors of the samples is averaged to obtain ; The optimization goals of the Actor network are as follows: ; is the policy function, Is the current state Take action The logarithmic probability of the policy function is used to measure the current policy in the state Next select action possibility; is the estimated value of the advantage function advantage, indicating that in this state Take action Compared with the average strategy, if it is greater than 0, it means it is better, so the probability of selecting the action is increased; if it is less than 0, it means it is worse, so the probability of selecting the action is reduced. The error of each sample is summed and averaged to get the loss function value of the Actor network. .

[0009] In a feasible implementation manner, the step S3 constructs a state space and an action space that match the problem model and the algorithm framework, and proposes an immediate reward function and a delayed reward function; (1) State space The production system of low-carbon flexible assembly workshop scheduling consists of three elements: workpiece, processing machine, and personnel. The global state characteristics include three elements: workpiece characteristics, processing machine characteristics, and personnel characteristics; The workpiece characteristic indicators include actual completion time, remaining processing time, expected completion time and average completion time, and also include the predecessor-successor relationship between workpieces to indicate assembly operations; the processing machine characteristic indicators include machine utilization rate and machine completion time; the personnel characteristic indicators include workers' initial learning rate, learning rate, and forgetting curve; Rescheduling point or decision point Average utilization of all devices on the , is the total number of devices: ; in, Indicates that at a rescheduling point or decision point On the device Equipment utilization rate; ; Indicates the device The completion time of the last process; Rescheduling point or decision point The standard deviation of equipment utilization on : ; All artifacts at rescheduling points or decision points The average completion rate : ; Indicates that at a rescheduling point or decision point Loading workpiece The number of completed operations, Indicates Number of processes per workpiece; All processes are at the rescheduling point or decision point The average completion rate , is the total number of artifacts: ; in, Indicates that at a rescheduling point or decision point Loading workpiece completion rate; ; At a rescheduling point or decision point Estimated maximum completion time for loading workpiece : ; Indicates that at a rescheduling point or decision point Loading workpiece Completed process set, Indicates the process The running time, Indicates the process On the device 0-1 decision variable for the above operation; Workpiece The actual running time of the completed operation Determined by the personnel learning rate: ; Personnel learning rate Is the number of times personnel learn And the forgetting curve yields: ; At a rescheduling point or decision point Energy consumption index of the completed process : ; in, Indicates that at a rescheduling point or decision point The actual energy consumption of the processing step is Indicates that at a rescheduling point or decision point The minimum energy consumption required to complete the process, Indicates that at a rescheduling point or decision point The median energy consumption required to complete the process; ; ; ; in, Indicates the processing steps The operating energy consumption generated when Indicates the process The start time to the same device Previous process Idle energy consumption at the end time, Indicates the process used to execute Processing equipment set, Indicates that at a rescheduling point or decision point The maximum energy consumption required to complete the process, Indicates the process used to execute Processing personnel set, Indicates the process By workers On the device The end time of ; ; (2) Action Space The high-level DQN agent derives the target emphasis probability ratio based on the current state, and the output result indicates the preference for the two targets in the current state; The low-level PPO agent generates the optimal process-machine action pair by combining the high-level target proportion and the current environment state, constructs a heterogeneous graph neural network HGNN, uses a multi-layer perceptron MLP to embed operation nodes and machine nodes, and builds a graph attention network GAT model to learn using the attention mechanism; then screens out operation-machine node pairs that do not meet the scheduling conditions; finally, inputs the Actor network to obtain the probability vector of the action, and adopts ε-Greedy when selecting action a to increase randomness for exploration: ; is the strategy function of the Actor network, is a deterministic strategy, which means choosing The biggest move, Is a random strategy, calculated based on the Actor network The probability distribution randomly selects actions; (3) Reward Function The reward function includes immediate rewards and delayed rewards; Instant Rewards : Instant rewards include economic indicators and energy consumption indicators; Calculating economic indicators , based on the average equipment utilization , Standard deviation of equipment utilization , the estimated maximum completion time during training To calculate, Indicates the current time point. Indicates the next rescheduling point:

[0010]

[0011]

[0012] Calculate energy consumption index , according to the energy consumption index , the minimum total energy consumption during training and current total energy consumption To calculate:

[0013]

[0014] The weighted sum of economic indicators and energy consumption indicators is used as the immediate reward, and the parameters Used to balance economic indicators and energy consumption indicators, where the parameters and It is the target optimization probability ratio passed from the high-level DQN to the lower level, but the positive and negative values ​​of the reward need to be specified by the state without being affected by the weight parameters: ; Delayed Rewards : The delay reward is a negative value; the total completion time is generated after the lower-level agent selects the action pair and completes the scheduling process in the workshop environment and total energy consumption ,parameter and They are the sum of weighted probabilities of the two objectives in a complete scheduling process; ; The larger the total completion time and total energy consumption, the greater the penalty the environment gives back to the agent. The experimental scheduling results are presented using a Gantt chart; Compared with the prior art, this application has the following beneficial effects: The present invention uses a dual-agent structure to solve a green dynamic multi-objective scheduling problem of a flexible assembly workshop under the effect of personnel learning, and uses two-layer deep reinforcement learning and a reward function that combines immediate rewards and delayed rewards to coordinately solve the problem of multi-objective conflicts; the method of the present invention consists of two-layer deep reinforcement learning, the high-level combines the learning ability of deep convolutional neural networks with the decision-making ability of reinforcement learning, and the low-level adopts a strategy optimization algorithm according to the high-level results to obtain a better scheduling strategy to solve the scheduling problem, using the real-time production environment as the state space, using machine and personnel dual resource constraints, and obtaining a better real-time scheduling strategy based on real-time production environment information, achieving a balance between solution quality and algorithm dynamics; in addition, a reward function that combines immediate rewards and cumulative rewards is designed in the framework, and the agent continuously interacts with the environment to obtain the optimal scheduling action for each rescheduling point or decision point. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a low-carbon flexible assembly workshop scheduling framework diagram; Figure 2 It is the flow chart of the algorithm for generating the optimal process-machine action pair; Figure 3 It is the personnel learning effect curve; Figure 4 This is a Gantt chart of the experimental scheduling results. DETAILED DESCRIPTION

[0016] A dual-agent structure is used to solve a green dynamic multi-objective scheduling problem in a flexible assembly workshop under the effect of personnel learning. A double-layer deep reinforcement learning is used to coordinate and solve multi-objective conflicts. A reward function combining immediate rewards and delayed rewards is adopted. The low-carbon flexible assembly workshop architecture is as follows: Figure 1 As shown, the data information such as the environmental status is input into the high-level intelligent agent to obtain the probability weight, that is, the target preference value, and then the weight is passed to the low-level intelligent agent, and the optimal process-machine action pair is obtained by combining the data information. After selecting the personnel, the action scheduling is completed, and the above operations are repeated until the scheduling of all workpieces in the algorithm is completed; 1. Mathematical model Considering the maximum completion time and maximum energy consumption of multiple workpieces during processing and assembly operations, a multi-objective mathematical programming model was established. The model parameters are shown in the following table:

[0017] (1) The objective function includes the function for calculating the total processing time and the total energy consumption function : The maximum completion time is the maximum end time of the last process of all workpieces. The function is composed of all workpieces Maximum completion time calculate, is the total number of workpieces, It represents the maximum completion time of all workpieces, that is, the total processing time of workpiece scheduling. Used to return the largest value from a set of values. Used to return the smallest value from a set of values; ; The total energy consumption of the processing process includes processing energy consumption and idle energy consumption; ; Indicates processing energy consumption: ; in, Indicates workers Operating Equipment Execution process The initial processing time, Representation of workpiece No. process, Indicates the device Unit processing energy consumption, Indicates the judgment process Whether by workers On the device 0-1 decision variables performed on Indicates workers By machine Processing procedures The actual learning rate at Indicates the total number of devices. is the total number of personnel, For workpiece The total number of processes; Indicates the idle energy consumption of computing equipment: ; in, Indicates the idle energy consumption of computing equipment. Indicates the device Unit idle energy consumption, Indicates the process By workers On the device The start time on Indicates the process By workers On the device The end time on Indicates the judgment process On the device Is the post-process on 0-1 decision variables, Indicates the judgment process Whether by workers On the device 0-1 decision variables performed on For workpiece The total number of processes, For workpiece The total number of processes; (2) A series of constraints include: Limit a process to be operated by one person on one piece of equipment; ; The actual processing time of a process is equal to the end time minus the start time. By the workers On the device Upper operation process The initial processing time is the actual processing time, which is obtained from the initial processing time and learning rate. By the workers On the device Upper operation process The end time of the operation, By the workers On the device Upper operation process The start time of the operation, Indicates workers By machine Processing procedures The actual learning rate at ; ; Process Processing completion time =Equal to the start processing time of the process plus the process processing time, equal to the worker Operating Equipment Processing procedures End time: ; The process of each workpiece must follow the priority order from front to back, that is, the process By workers In the machine Start time on Not less than the previous process of the same workpiece By workers In the machine End time of the run ; ; If you want to process different workpieces on one device, they must be done in sequence. 1, indicating a process On the device The post-process on , 1, indicating a process On the device The post-process on , only one of the two situations is true, process By workers In the machine The start time of the above operation is , end time is , process By workers In the machine The start time of the above operation is , end time is ;

[0018] ; Any machine can only operate one process at a time, that is, the worker In the machine The last process after processing Start time No less than workers On the same machine Previous process End time , is a positive number; ; Any person can only operate one process at a time. Indicates workers Operate the machine first Re-operate the machine , Indicates workers Can operate the machine The start time of ; Worker Operating the machine Processing procedures Start time Not less than the machine The end time of the previous process is equal to the worker In the machine The start time of the operation : ; Worker Operate the machine again The interval before It is equivalent to the last time the machine was operated. End time To operate the machine again Start time The interval time between ; Worker Processing to The actual learning rate of the worker at the time Number of times worked on related, Dynamically adjust the learning rate, A worker The learning effect coefficient, It is the incompressible coefficient of the learning effect. The more times, the lower the learning rate, and vice versa. ; Assembling workpieces The first process Start time is no earlier than The end time of all predecessor jobs, Indicated by workers Operating the machine Processing procedures The start time of Representation of workpiece The set of pre-requisites, Representation of workpiece The last step, Indicated by workers operate Processing procedures End time of .

[0019] 2. Dual Agents Dual agents are used to achieve multiple goals, and a hierarchical multi-action space is designed. Each space is managed by an agent based on DQN (DeepQ-Network). When the framework is running, the state information will be first transmitted to the upper-layer DQN agent. After the output result is obtained, the result and the environment state characteristics will be transmitted to the lower-layer DQN agent to obtain the final scheduling result of this time step. The high-level DQN agent is a controller. The output result is used to determine the preference value of the optimization target of the current time step, guide the selection of scheduling actions by the lower-level DQN agent, and thus affect the scheduling of the entire environment state; using the target preference value will better coordinate the conflict between multiple targets, and will not make the lower-level action selection only optimize target 1 or target 2, but will be biased on the basis of balance; the low-level DQN agent receives the optimization preference value of the high-level DQN agent, uses it as the target parameter of the reward function, and then selects the current scheduling point according to the rules of the action space scheduling algorithm through the input environment state information. The action is right; The dual-agent real-time control workshop production process is as follows: (1) High-level DQN The high-level DQN agent acts as a controller. At each rescheduling point, it takes the state feature as input and the Q value of each target as output. It passes the output target probability value as a weight to the low-level DQN agent. Target value in DQN algorithm The calculation method is as follows: ; in, is in state Next action After receiving the instant reward, is the discount factor, are the target network parameters, The target network For the next state Maximum Q value estimation of ; The error calculation formula between the estimated value and the target value in the current state, that is, the loss function, is as follows: ; The agent is in state Next action The immediate reward obtained from the environment reflects the direct effect of the current action, is the discount factor, is the next state of the target network The maximum Q value estimate is, Is the main network's current state and actions Q value estimation of ; (2) Low-level PPO The low-level PPO (Proximal Policy Optimization) agent acts as an executor, taking the state features and the output targets of the high-level DQN agent as input, and the Q value of the workpiece-machine sequence pair as output. The workpiece-machine with the highest Q value is selected as the action selected by the current scheduling point. The following is the objective function of the PPO algorithm, by finding the parameters θ Maximize this function: ; It is The objective function of PPO at the iteration is: yes t The state of the moment, yes t Actions taken at all times, In the current state Take Action strategy, is the importance sampling ratio, which represents the probability ratio of the new strategy to the old strategy and is used to measure the change of the new strategy compared to the old strategy. Indicates that the importance sampling ratio is limited to between, is the parameter that limits the range; It is The strategy parameters at the iteration, is the advantage function, It is The advantage function at the iteration is To evaluate the status Take action The advantages and disadvantages of It means obtaining the minimum value among a set of target values, which can effectively prevent excessive deviation from the old strategy when the strategy is updated; The decision-making part of the algorithm adopts the Actor-Critic algorithm, which is divided into the Critic evaluation network and the Actor strategy network. The former inputs the environment state and action to evaluate the value of taking the action in the current state, while the latter inputs the current state and outputs the probability distribution of the action. The Critic network is then used to evaluate the quality of the action and adjust the strategy to improve the expected return of future actions. The Critic network uses the mean square error loss function, that is, ; is the total number of samples, Corresponding to samples, is the mean square error loss function value obtained by the Critic network, and Samples to calculate the cumulative reward value Compare the status with the Critic network and actions Q-value estimation The mean square error between them promotes the Critic network to quickly correct the error, and finally The sum of the errors of the samples is averaged to obtain ; The optimization goals of the Actor network are as follows: ; is the policy function, Is the current state Take action The logarithmic probability of the policy function is used to measure the current policy in the state Next select action possibility; is the estimated value of the advantage function advantage, indicating that in this state Take action Compared with the average strategy, if it is greater than 0, it means it is better, so the probability of selecting the action is increased; if it is less than 0, it means it is worse, so the probability of selecting the action is reduced. The error of each sample is summed and averaged to get the loss function value of the Actor network. .

[0020] The low-level PPO algorithm flow is as follows: Preparation: Critic network learning rate and network parameters , Actor network learning rate and network parameters ; 1) Standardize current status information; 2) Heterogeneous neural networks In the second iteration, the action pairs that do not meet the conditions are filtered out and input into the Actor network to obtain the action probability; Select actions based on rules , calculate the reward value ; Will Store in cache; 3) Round strategy optimization: Generated from the interaction between the agent and the environment Select from collection Group; Get action evaluation through the Critic network; 4) The old strategy gets the new strategy: Calculate the network loss value: ,

[0021] Update network parameters: , .

[0022] 3. State Space The production system of low-carbon flexible assembly workshop scheduling consists of three elements: workpiece, processing machine, and personnel. The global state characteristics include three elements: workpiece characteristics, processing machine characteristics, and personnel characteristics; The workpiece characteristic indicators include actual completion time, remaining processing time, expected completion time and average completion time, and also include the predecessor-successor relationship between workpieces to indicate assembly operations; the processing machine characteristic indicators include machine utilization rate and machine completion time; the personnel characteristic indicators include workers' initial learning rate, learning rate, and forgetting curve; Rescheduling point or decision point Average utilization of all devices on the , is the total number of devices: ; in, Indicates that at a rescheduling point or decision point On the device Equipment utilization rate; ; Indicates the device The completion time of the last process; At a rescheduling point or decision point The standard deviation of equipment utilization on : ; All artifacts at rescheduling points or decision points The average completion rate : ; Indicates that at a rescheduling point or decision point Loading workpiece The number of completed operations, Indicates Number of processes per workpiece; All processes are at the rescheduling point or decision point The average completion rate , is the total number of artifacts: ; in, Indicates that at a rescheduling point or decision point Loading workpiece completion rate; ; At a rescheduling point or decision point Estimated maximum completion time for loading workpiece : ; Indicates that at a rescheduling point or decision point Loading workpiece Completed process set, Indicates the process The running time, Indicates the process On the device 0-1 decision variable for the above operation; Workpiece The actual running time of the completed operation Determined by the personnel learning rate, as follows Figure 3 As shown, the blue lines represent workers The learning curve of Operating Equipment The vertical axis represents the actual running time of the process. Because the running time of each process on different equipment is inconsistent, the vertical axis scale cannot be expressed in actual values. Point 1 is the worker On the device Upper operation process The initial running time of workers during continuous learning On the device Upper operation process The actual running time will decrease along the learning curve, passing through points 2 and 3. At point 3, the worker Left the device That is, forgetting occurs and regresses to the previous n The learning rate at scheduling point 2, continuing to learn will reduce the actual running time along the learning curve of the blue line. The red line represents the worker There is a learning lower limit, that is, the actual running time will not be 0: ; Personnel learning rate Is the number of times personnel learn And the forgetting curve: ; Rescheduling point or decision point Energy consumption indicators for completing the process : ; in, Indicates that at a rescheduling point or decision point The actual energy consumption of the processing step is Indicates that at a rescheduling point or decision point The minimum energy consumption required to complete the process, Indicates that at a rescheduling point or decision point The median energy consumption required to complete the process; ; ; ; in, Indicates the processing steps The operating energy consumption generated when Indicates the process The start time to the same device Previous process Idle energy consumption at the end time, Indicates the process used to execute Processing equipment set, Indicates that at a rescheduling point or decision point The maximum energy consumption required to complete the process, Indicates the process used to execute Processing personnel set, Indicates the process By workers On the device The end time of ; ; 4. Action Space The high-level DQN agent outputs the target focus probability ratio according to the current state, and the output result indicates the preference for the two targets in the current state; The low-level PPO agent generates the optimal process-machine action pair by combining the high-level target proportion and the current environment state, constructs a heterogeneous graph neural network HGNN, uses a multi-layer perceptron MLP to embed operation nodes and machine nodes, and builds a graph attention network GAT model to learn using the attention mechanism; then screens out operation-machine node pairs that do not meet the scheduling conditions; finally, inputs the Actor network to obtain the probability vector of the action, and adopts ε-Greedy when selecting action a to increase randomness for exploration: ; is the strategy function of the Actor network, is a deterministic strategy, which means choosing The biggest move, Is a random strategy, calculated based on the Actor network The probability distribution randomly selects actions; The low-level algorithm action selection process is as follows Figure 2 As shown in the figure, after the current state information is standardized, the operation nodes and machine nodes are loaded into the heterogeneous graph neural network using a multi-layer perceptron and graph attention network model. The data after L rounds of iterations are pooled and filtered and then passed into the Actor network to obtain the action probabilities of all process-machine action pairs. 5. Reward Function The reward function includes immediate rewards and delayed rewards; Instant Rewards : Instant rewards include economic indicators and energy consumption indicators; Calculating economic indicators , based on the average equipment utilization , Standard deviation of equipment utilization , the estimated maximum completion time during training To calculate, Indicates the current time point. Indicates the next rescheduling point:

[0023]

[0024]

[0025] Calculate energy consumption index , according to the energy consumption index , the minimum total energy consumption during training and current total energy consumption To calculate:

[0026]

[0027] The weighted sum of economic indicators and energy consumption indicators is used as the immediate reward, and the parameters Used to balance economic indicators and energy consumption indicators, where the parameters and It is the target optimization probability ratio passed from the high-level DQN to the lower level, but the positive and negative values ​​of the reward need to be specified by the state without being affected by the weight parameters: ; Delayed Rewards : The delay reward is a negative value; the total completion time is generated after the lower-level agent selects the action pair and completes the scheduling process in the workshop environment and total energy consumption ,parameter and They are the sum of weighted probabilities of the two objectives in a complete scheduling process; ; The larger the total completion time and total energy consumption, the greater the penalty the environment gives back to the agent. The experimental scheduling results are presented using a Gantt chart.

[0028] The Gantt chart results of the successful experimental assembly job scheduling based on the dual-agent deep reinforcement learning structure are as follows: Figure 4 As shown in the figure, the scheduling sequence obtained by the algorithm completes the scheduling operation of the assembly workpiece under the dual resource constraints of machines and workers. Indicates different workpieces, each workpiece has a different color. Indicates different processes of the same workpiece, displayed in the same color. to The operations of five machines to complete all workpieces in the scheduling sequence are displayed on the timeline, achieving multi-objective optimization of minimizing the maximum completion time and the total energy consumption of the machining process.

[0029] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A green dynamic multi-objective scheduling method for a flexible assembly workshop under the personnel learning effect, characterized in that: The following steps are involved: S1: According to the green dynamic multi-objective scheduling requirements of flexible assembly workshops under the personnel learning effect, a green dynamic multi-objective scheduling planning model for flexible assembly workshops under the personnel learning effect is established; S2: Use a dual-agent structure to solve the green dynamic multi-objective scheduling planning model of a flexible assembly workshop under the human learning effect, and build a two-layer deep reinforcement learning framework to coordinate and solve multi-objective conflicts; S3: Construct a state space and action space that matches the problem model and algorithm framework, and propose an immediate reward function and a delayed reward function.

2. According to the green dynamic multi-objective scheduling method for a flexible assembly workshop under the personnel learning effect of claim 1, it is characterized in that: In step S1, according to the green dynamic multi-objective scheduling requirements of the flexible assembly workshop under the personnel learning effect, the maximum completion time and total energy consumption of multiple workpieces in the processing and assembly process are considered, and a green dynamic multi-objective scheduling planning model for the flexible assembly workshop under the personnel learning effect is established. The model parameters are as follows; (1) The objective function includes the function for calculating the total processing time and the total energy consumption function : The maximum completion time is the maximum end time of the last process of all workpieces. The function is composed of all workpieces Maximum completion time calculate, is the total number of workpieces, Used to return the largest value from a set of values. Used to return the smallest value from a set of values; ； The total energy consumption of the processing process includes processing energy consumption and idle energy consumption; ； Indicates processing energy consumption: ； in, Indicates workers Operating equipment Execution process The initial processing time, Representation of workpiece No. process, Indicates the device Unit processing energy consumption, Indicates the judgment process Whether by workers On the device 0-1 decision variables performed on Indicates workers By machine Processing procedures The actual learning rate at Indicates the total number of devices. is the total number of personnel, For workpiece The total number of processes; Indicates the idle energy consumption of computing equipment: ； in, Indicates the idle energy consumption of computing equipment. Indicates the device Unit idle energy consumption, Indicates the process By workers On the device The start time on Indicates the process By workers On the device The end time on Indicates the judgment process On the device Is the post-process on 0-1 decision variables, Indicates the judgment process Whether by workers On the device 0-1 decision variables performed on For workpiece The total number of processes, For workpiece The total number of processes; (2) A series of constraints include: Limit a process to be operated by one person on one piece of equipment; ； The actual processing time of a process is equal to the end time minus the start time. By the workers On the device Upper operation process The initial processing time is the actual processing time, which is obtained from the initial processing time and learning rate. By the workers On the device Upper operation process The end time of the operation, By the workers On the device Upper operation process The start time of the operation, Indicates workers By machine Processing procedures The actual learning rate at ; ； Process Processing completion time =Equal to the start processing time of the process plus the process processing time, equal to the worker Operating equipment Processing procedures End time: ； The process of each workpiece must follow the priority order from front to back, that is, the process By workers In the machine Start time on Not less than the previous process of the same workpiece By workers In the machine End time of the run ; ； If you want to process different workpieces on one device, they must be done in sequence. 1, indicating a process On the device The post-process on , 1, indicating a process On the device The post-process on , only one of the two situations is true, process By workers In the machine The start time of the above operation is , end time is , process By workers In the machine The start time of the above operation is , end time is ; ； Any machine can only operate one process at a time, that is, the worker In the machine The last process after processing Start time No less than workers On the same machine Previous process End time , is a positive number; ； Any person can only operate one process at a time. Indicates workers Operate the machine first Re-operate the machine , Indicates workers Can operate the machine The start time of ； Worker Operating the machine Processing procedures Start time Not less than the machine The end time of the previous process is equal to the worker In the machine The start time of the operation ; ； Worker Operate the machine again The interval before It is equivalent to the last time the machine was operated. End time To operate the machine again Start time The interval time between ； Worker Processing to The actual learning rate of the worker at the time Number of times worked on related, Dynamically adjust the learning rate, A worker The learning effect coefficient, It is the incompressible coefficient of the learning effect. The more times, the lower the learning rate, and vice versa. ； Assembling workpieces The first process Start time is no earlier than The end time of all predecessor jobs, Indicated by workers Operating the machine Processing procedures The start time of Representation of workpiece The set of pre-requisites, Representation of workpiece The last step, Indicated by workers operate Processing procedures End time of 。 3. The green dynamic multi-objective scheduling method for a flexible assembly workshop under the personnel learning effect according to claim 1 is characterized in that: The step S2 proposes a dual-agent framework to solve the green dynamic multi-objective scheduling planning model of a flexible assembly workshop under the learning effect of personnel, and constructs a two-layer deep reinforcement learning framework to coordinate and solve multi-objective conflicts; Dual agents are used to achieve multiple goals, and a hierarchical multi-action space is designed. Each space is managed by an agent based on DQN (Deep Q-Network). When the framework is running, the state information will be first transmitted to the upper-layer DQN agent. After the output result is obtained, the result and the environment state characteristics will be transmitted to the lower-layer DQN agent to obtain the final scheduling result of this time step. The high-level DQN agent is a controller. The output result is used to determine the preference value of the optimization target of the current time step, guide the selection of scheduling actions by the lower-level DQN agent, and thus affect the scheduling of the entire environment state; using the target preference value will better coordinate the conflict between multiple targets, and will not make the lower-level action selection only optimize target 1 or target 2, but will be biased on the basis of balance; the low-level DQN agent receives the optimization preference value of the high-level DQN agent, uses it as the target parameter of the reward function, and then selects the current scheduling point according to the rules of the action space scheduling algorithm through the input environment state information. The action is right; The dual-agent real-time control workshop production process is as follows: (1) High-level DQN The high-level DQN agent acts as a controller. At each rescheduling point, it takes the state feature as input and the Q value of each target as output. It passes the output target probability value as a weight to the low-level DQN agent. Target value in DQN algorithm The calculation method is as follows: ； in, is in state Next action After receiving the instant reward, is the discount factor, are the target network parameters, The target network For the next state Maximum Q value estimation of ; The error calculation formula between the estimated value and the target value in the current state, that is, the loss function, is as follows: ； The agent is in state Next action The immediate reward obtained from the environment reflects the direct effect of the current action, is the discount factor, is the next state of the target network The maximum Q value estimate is, Is the main network's current state and actions Q value estimation of ; (2) Low-level PPO The low-level PPO (Proximal Policy Optimization) agent acts as an executor, taking the state features and the output targets of the high-level DQN agent as input, and the Q value of the workpiece-machine sequence pair as output. The workpiece-machine with the highest Q value is selected as the action selected by the current scheduling point. The following is the objective function of the PPO algorithm, by finding the parameters θ Maximize this function: ； It is The objective function of PPO at the iteration is: yes t The state of the moment, yes t Actions taken at all times, In the current state Take Action strategy, is the importance sampling ratio, which represents the probability ratio of the new strategy to the old strategy and is used to measure the change of the new strategy compared to the old strategy. Restricts the importance sampling ratio to between, is the parameter that limits the range; It is The strategy parameters at the iteration, is the advantage function, It is The advantage function at the iteration is To evaluate the status Take action The pros and cons of It means to obtain the minimum value among a set of target values; The decision-making part of the algorithm adopts the Actor-Critic algorithm, which is divided into the Critic evaluation network and the Actor strategy network. The former inputs the environment state and action to evaluate the value of taking the action under the current state, while the latter inputs the current state and outputs the probability distribution of the action. The Critic network is then used to evaluate the quality of the action and adjust the strategy. The Critic network uses the mean square error loss function, that is, ; is the total number of samples, Corresponding to samples, is the mean square error loss function value obtained by the Critic network, and Samples to calculate the cumulative reward value Compare the status with the Critic network and actions Q-value estimation The mean square error between them promotes the Critic network to quickly correct the error, and finally The sum of the errors of the samples is averaged to obtain ; The optimization goals of the Actor network are as follows: ; is the policy function, Is the current state Take action The logarithmic probability of the policy function is used to measure the current policy in the state Next select action possibility; is the estimated value of the advantage function advantage, indicating that in this state Take action Compared with the average strategy, if it is greater than 0, it means it is better, so the probability of selecting the action is increased; if it is less than 0, it means it is worse, so the probability of selecting the action is reduced. The error of each sample is summed and averaged to get the loss function value of the Actor network. .

4. The green dynamic multi-objective scheduling method for a flexible assembly workshop under the personnel learning effect according to claim 2 is characterized in that: The step S3 constructs a state space and an action space that match the problem model and the algorithm framework, and proposes an immediate reward function and a delayed reward function; (1) State space The production system of low-carbon flexible assembly workshop scheduling consists of three elements: workpiece, processing machine, and personnel. The global state characteristics include three elements: workpiece characteristics, processing machine characteristics, and personnel characteristics; The workpiece characteristic indicators include actual completion time, remaining processing time, expected completion time and average completion time, and also include the predecessor-successor relationship between workpieces to indicate assembly operations; the processing machine characteristic indicators include machine utilization and machine completion time; The personnel characteristic indicators include workers’ initial learning rate, learning rate, and forgetting curve; Rescheduling point or decision point Average utilization of all devices on the , is the total number of devices: ； in, Indicates that at a rescheduling point or decision point On the device Equipment utilization rate; ； Indicates the device The completion time of the last process; At a rescheduling point or decision point The standard deviation of equipment utilization on : ； All artifacts at rescheduling points or decision points The average completion rate : ； Indicates that at a rescheduling point or decision point Loading workpiece The number of completed operations, Indicates Number of processes per workpiece; All processes are at the rescheduling point or decision point The average completion rate , is the total number of artifacts: ； in, Indicates that at a rescheduling point or decision point Loading workpiece completion rate; ； At a rescheduling point or decision point Estimated maximum completion time for loading workpiece : ； Indicates that at a rescheduling point or decision point Loading workpiece Completed process set, Indicates the process The running time, Indicates the process On the device 0-1 decision variable for the above operation; Workpiece The actual running time of the completed operation By personnel learning rate Decide: ； Personnel learning rate Is the number of times personnel learn And the forgetting curve: ； Rescheduling point or decision point Energy consumption indicators for completing the process : ； in, Indicates that at a rescheduling point or decision point The actual energy consumption of the processing step is Indicates that at a rescheduling point or decision point The minimum energy consumption required to complete the process, Indicates that at a rescheduling point or decision point The median energy consumption required to complete the process; ； ； ； in, Indicates the processing steps The operating energy consumption generated when Indicates the process The start time to the same device Previous process Idle energy consumption at the end time, Indicates the process used to execute Processing equipment set, Indicates that at a rescheduling point or decision point The maximum energy consumption required to complete the process, Indicates the process used to execute Processing personnel set, Indicates the process By workers On the device The end time of ； ； (2) Action Space The high-level DQN agent outputs the target focus probability ratio according to the current state, and the output result indicates the preference for the two targets in the current state; The low-level PPO agent generates the optimal process-machine action pair by combining the high-level target proportion and the current environment state, constructs a heterogeneous graph neural network HGNN, uses a multi-layer perceptron MLP to embed operation nodes and machine nodes, and builds a graph attention network GAT model to learn using the attention mechanism; then screens out operation-machine node pairs that do not meet the scheduling conditions; finally, inputs the Actor network to obtain the probability vector of the action, and adopts ε-Greedy when selecting action a to increase randomness for exploration: ； is the strategy function of the Actor network, is a deterministic strategy, which means choosing The biggest move, Is a random strategy, calculated based on the Actor network The probability distribution randomly selects actions; (3) Reward Function The reward function includes immediate rewards and delayed rewards; Instant Rewards : Instant rewards include economic indicators and energy consumption indicators; Calculating economic indicators , based on the average equipment utilization , Standard deviation of equipment utilization , the estimated maximum completion time during training To calculate, Indicates the current time point. Indicates the next rescheduling point: Calculate energy consumption index , according to the energy consumption index , the minimum total energy consumption during training and current total energy consumption To calculate: The weighted sum of economic indicators and energy consumption indicators is used as the immediate reward, and the parameters Used to balance economic indicators and energy consumption indicators, where the parameters and It is the target optimization probability ratio passed from the high-level DQN to the lower level, but the positive and negative values ​​of the reward need to be specified by the state without being affected by the weight parameters: ； Delayed Rewards : The delay reward is a negative value; the total completion time is generated after the lower-level agent selects the action pair and completes the scheduling process in the workshop environment and total energy consumption ,parameter and They are the sum of weighted probabilities of the two objectives in a complete scheduling process; 。

Images