A disassembly line balancing method based on deep reinforcement learning

Abstract

This invention belongs to the field of artificial intelligence technology and discloses a disassembly line balancing method based on deep reinforcement learning, comprising the following steps: S1: preprocessing disassembly product information; S2: defining factors affecting human fatigue; S3: defining symbols, decision variables, and constraints to establish a disassembly objective function; establishing a relationship between fatigue and the objective function, and building a mathematical model for the disassembly line balancing problem affected by human fatigue; S4: converting the mathematical model established in S3 into a Markov decision process (MDP); and S5: designing a DDQN network model structure based on a Q-learning algorithm to obtain the optimal objective function value. This invention embeds fatigue accumulation and recovery formulas into disassembly line optimization, establishes a mathematical model with minimizing disassembly time as the objective function, and uses the DDQN reinforcement learning algorithm combined with a transformer to solve the objective function, achieving disassembly line balancing optimization.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to deep learning network model design and reinforcement learning optimization methods. Specifically, it relates to a disassembly line balance optimization method that takes into account the influence of human fatigue value factors, and especially to a disassembly line balance method based on deep reinforcement learning. Background Art

[0002] The disassembly of used electrical appliances is an important way to recycle resources. Currently, most home appliance disassembly processes still rely on manual disassembly. Workers suffer from muscle fatigue due to long-term continuous work, which affects the disassembly efficiency. Therefore, it is very meaningful to consider human fatigue.

[0003] The Demolition Line Balancing Problem (DLBP) involves finding a reasonable task scheduling sequence based on the type of demolition line and the objective function. Common demolition line types include linear, U-shaped, bilateral, parallel, and mixed. Compared to U-shaped and mixed lines, linear demolition lines are the most common and the most concise and intuitive. To address this problem, Guo et al. proposed a multi-objective evolutionary algorithm to efficiently solve the problem, considering the impact of human fatigue on work efficiency.

[0004] Reinforcement learning (RL) involves learning through interactions between an agent and its environment to maximize a cumulative reward. Deep reinforcement learning (DRL) combines deep neural networks with traditional reinforcement learning methods to effectively model and make decisions in complex environments and high-dimensional state spaces. In recent years, some researchers have begun using DRL to solve the DLBP problem. Wang J et al. proposed a soft actor-critic (SAC) algorithm to solve the mixed disassembly line balancing problem, aiming to maximize disassembly profits. Liu et al. considered using an improved Q-learning algorithm based on reinforcement learning to solve this two-sided disassembly line balancing problem in the context of human-computer interaction. When dealing with complex environments and large-scale state spaces, Q-learning algorithms suffer from slow convergence and are unable to find an efficient global optimal solution. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a disassembly line balancing method based on deep reinforcement learning to make up for the defects and shortcomings of traditional methods, achieve disassembly line balance optimization, and improve the efficiency of home appliance disassembly.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A disassembly line balancing method based on deep reinforcement learning includes the following steps:

[0008] S1: Preprocess the disassembled product information and use the constraint matrix to represent the constraint relationship between product tasks;

[0009] S2: Define factors affecting human fatigue;

[0010] S3: Define symbols, decision variables, and constraints to establish a disassembly objective function; establish the relationship between fatigue and the objective function, transform the practical problem into a mathematical problem, and then establish a mathematical model for the disassembly line balance problem affected by human fatigue;

[0011] S4: Convert the mathematical model established in S3 into a Markov decision process (MDP) and design the state space and action space.

[0012] S5: Design the DDQN network model structure based on the Q-learning algorithm, set the reward function calculation method according to the objective function in S3, and obtain the optimal objective function value.

[0013] Furthermore, in S1, the priority matrix To express the constraint relationship between two tasks, the element It is expressed as follows:

[0014] ;

[0015] Where p is the product ID, i and j are task indexes.

[0016] Furthermore, in S2, the human fatigue factor refers to human muscle fatigue, which is affected by working hours. That is, the accumulation of working hours will lead to muscle fatigue. When the accumulated fatigue reaches the maximum accumulated fatigue, the worker cannot work and is forced to rest. The specific accumulated fatigue is calculated by the following formula:

[0017] (1);

[0018] (2);

[0019] (3);

[0020] in represents the disassembly time of task j of product p at workstation w, represents the fatigue accumulation rate of workstation w, represents the waiting time of task j of product p at workstation w, represents the fatigue recovery rate of workstation w, , , that is, the fatigue accumulation at any moment is less than the maximum fatigue accumulation ; represents the accumulated fatigue after workstation w starts to execute task j of product p, Indicates that workstation w waits before executing task j of product p After the fatigue accumulates, rest Fatigue accumulates over time; Indicates continuing after the previous break Fatigue accumulates after hours of work.

[0021] Furthermore, in S3, the details are as follows:

[0022] S3-1: Define symbols as follows:

[0023] Product Collection , P represents the total number of products, and the task set of any product p , , Indicates the total number of tasks for product p, task index ;

[0024] Workstation collection = {1,2,......,W}, W represents the total number of workstations, ;

[0025] S3-2: Define the decision variables as follows:

[0026] ;

[0027] Indicates whether task j of product p is executed on workstation w, 1 if it is executed, 0 otherwise;

[0028] ;

[0029] Indicates whether the fatigue value of workstation w has reached the maximum fatigue accumulation , if reached, it is 1, otherwise it is 0;

[0030] ;

[0031] Indicates whether task j of product p is the immediate predecessor of task k, that is, if task j must be completed before task k, then it is 1, otherwise it is 0;

[0032] ;

[0033] Indicates whether workstation w is enabled, if enabled, it is 1, otherwise it is 0;

[0034] S3-3: Define the objective function, which is expressed as follows:

[0035] (4);

[0036] The objective function consists of two parts: the first part is the task disassembly time, and the second part is the task waiting time; the objective function satisfies the constraint relationship;

[0037] S3-4: The constraint relationship is expressed as follows:

[0038] (5);

[0039] Formula (5) ensures that each task is executed at most once;

[0040] (6);

[0041] Formula (6) ensures that the number of enabled workstations cannot exceed the total number of workstations;

[0042] (7);

[0043] Formula (7) ensures that tasks can only be assigned to enabled workstations;

[0044] (8);

[0045] Formula (8) Task j for product p is executed on workstation w, For product p, task k is executed on workstation w. If task j needs to be executed before task k, then task k must satisfy the allocation order and be executed on the same workstation as task j or on a workstation after task j. If task j is a predecessor of task k, assuming task j is assigned to workstation 2 and task k is assigned to workstation 1, then the requirements of this formula are not met.

[0046] (9);

[0047] (10);

[0048] In formula (9), M is a positive number. Formulas (9) and (10) guarantee that Meet or exceed hour, is 1; otherwise, is 0.

[0049] Furthermore, in S4, the Markov decision process MDP includes a state space S, an action space A, a state transition probability P, and a reward function R; wherein the state space S is composed of two parts: a workstation task assignment sequence and a legal operation identifier. The first part is used to represent the task assignment sequence of the current workstation, which contains the allocation information of the assigned tasks on the workstation. The second part represents the legal operations that can be performed. These legal operations define the range of actions that are feasible in the current state, and they are subject to the constraints of the disassembly priority relationship; the action space A represents the assignment of any task to any workstation; the reward function R represents the disassembly time calculated according to the objective function after executing each action; the state transition probability P represents the probability of one state transferring to another state.

[0050] Furthermore, in S5, the DDQN network model based on the Q-learning algorithm is defined as a DDTQN network model, including the Q network and target network integrated with the Transformer Encoder, as well as the experience replay pool and environment. The processing process is as follows:

[0051] First, the Q network evaluates the current disassembly line state space and selects an action. The Q network receives the current disassembly line state as input and outputs estimated Q values for each possible action. The agent uses a greedy strategy to select actions and executes them in the disassembly environment. The environment then provides feedback on the next state and corresponding reward, which is stored in the experience replay pool.

[0052] Next, a batch of experience data is randomly sampled from the experience replay pool. The Q network outputs an estimated Q value based on the current state obtained by sampling. The target network outputs the Q value of the corresponding action based on the next state obtained by sampling and calculates the target Q value based on the reward information.

[0053] Finally, the loss function is constructed by comparing the estimated Q value of the Q network with the target Q value, and the back propagation algorithm is used to update the Q network parameters.

[0054] Furthermore, the inputs of the Q network and the target network are position codes and action labels. The entire network structure of the Q network and the target network gradually extracts and integrates key information from the input position codes and action labels through multi-layer feature extraction and processing. Specifically:

[0055] First, the position code and action label are input into the network and processed by a linear layer to extract preliminary features. Next, the data enters the pre-feedback layer to capture the long-term and short-term dependencies in the disassembly line data. The data is then normalized and passed to the multi-head attention layer. In the disassembly line balancing problem, the multi-head attention layer performs attention calculations on the data from multiple perspectives in parallel, comprehensively capturing the correlations between different positions and different features during the disassembly process, thereby making action decisions to achieve disassembly line balance.

[0056] After passing through the multi-head attention layer, the data is again processed by layer normalization. Finally, the multi-layer processed data passes through another linear layer to generate the action predicted by the network for the current disassembly line state.

[0057] Compared with the prior art, the present invention has the following advantages:

[0058] (1) Establishing a fatigue dynamic model: In view of the fact that most companies still rely on manual labor for disassembly, this paper proposes a linear disassembly balance problem that takes into account the influence of human fatigue. For the first time, the fatigue accumulation and recovery formula is embedded in the disassembly line optimization, and a mathematical model is established with minimizing the disassembly time as the objective function.

[0059] (2) Multi-objective optimization and MDP design: The objective function simultaneously optimizes the disassembly time and fatigue penalty. Task allocation, fatigue status, and timing dependencies are integrated into the state space through a Markov decision process (MDP). The action space supports dynamic task reallocation, and the state transition probability covers task priority constraints and fatigue thresholds, achieving global optimization.

[0060] (3) Design of the DDQN network model structure based on the Q-learning algorithm: Using the double deep Q network reinforcement learning algorithm (DDQN) combined with the transformer to solve the objective function, solve the problem of overestimation of Q value, and improve stability. Through experiments on different scale cases, the results are better than DDQN, DQN, and A2C algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a technical flow chart of the present invention;

[0062] Figure 2 This is a schematic diagram of work-rest fatigue accumulation;

[0063] Figure 3 This is the structure diagram of the DDTQN model;

[0064] Figure 4 It is a network structure diagram of the Q network and the target network of the present invention. DETAILED DESCRIPTION

[0065] The present invention is further explained and illustrated below through specific embodiments in conjunction with the accompanying drawings.

[0066] Example 1

[0067] In this embodiment, a disassembly line balance optimization method considering the influence of human fatigue value is designed, especially a disassembly line balance method based on deep reinforcement learning. The basic technical flow chart of this method is as follows: Figure 1 As shown, the design idea is: product information preprocessing - fatigue factor definition - mathematical model establishment - algorithm design coding (implementation of DDTQN network model).

[0068] Specifically, the following steps are included:

[0069] S1: Preprocess the disassembled product information and use the constraint matrix to represent the constraint relationship between product tasks.

[0070] As a preferred embodiment, in S1, the priority matrix is used To express the constraint relationship between two tasks, the element It is expressed as follows:

[0071] ;

[0072] Where p is the product ID, i and j are task indexes.

[0073] S2: Define factors that affect human fatigue.

[0074] The factor affecting human fatigue refers to muscle fatigue, which is affected by working hours. That is, the accumulation of working hours will lead to muscle fatigue. When the fatigue accumulation reaches the maximum fatigue accumulation, the worker cannot work and is forced to rest. The specific fatigue accumulation is calculated by the following formula:

[0075] (1);

[0076] (2);

[0077] (3);

[0078] in represents the disassembly time of task j of product p at workstation w, represents the fatigue accumulation rate of workstation w, represents the waiting time of task j of product p at workstation w, represents the fatigue recovery rate of workstation w, , , that is, the fatigue accumulation at any moment is less than the maximum fatigue accumulation ,The maximum fatigue accumulation refers to the maximum allowable value of human fatigue accumulation. If this threshold is exceeded, you must be forced to rest; represents the accumulated fatigue after workstation w starts to execute task j of product p, Indicates that workstation w waits before executing task j of product p After the fatigue accumulates, rest Fatigue accumulates over time; Indicates continuing after the previous break The relationship between human fatigue accumulation and time is as follows: Figure 2 shown.

[0079] S3: Define symbols, decision variables and constraint relationships, and establish the disassembly objective function; establish the relationship between fatigue and the objective function, transform the practical problem into a mathematical problem, and then establish a mathematical model for the disassembly line balance problem affected by human fatigue.

[0080] As a preferred embodiment, in S3, the details are as follows:

[0081] S3-1: Define symbols as follows:

[0082] Product Collection , P represents the total number of products, and the task set of any product p , , Indicates the total number of tasks for product p, task index ;

[0083] Workstation collection = {1,2,......,W}, W represents the total number of workstations, ;

[0084] S3-2: Define the decision variables as follows:

[0085] ;

[0086] Indicates whether task j of product p is executed on workstation w, 1 if it is executed, 0 otherwise;

[0087] ;

[0088] Indicates whether the fatigue value of workstation w has reached the maximum fatigue accumulation , if reached, it is 1, otherwise it is 0;

[0089] ;

[0090] Indicates whether task j of product p is the immediate predecessor of task k, that is, if task j must be completed before task k, then it is 1, otherwise it is 0;

[0091] ;

[0092] Indicates whether workstation w is enabled, if enabled, it is 1, otherwise it is 0;

[0093] S3-3: Define the objective function, which is expressed as follows:

[0094] (4);

[0095] The objective function consists of two parts: the first part is the task disassembly time, and the second part is the task waiting time; the objective function satisfies the constraint relationship;

[0096] S3-4: The constraint relationship is expressed as follows:

[0097] (5);

[0098] Formula (5) ensures that each task is executed at most once;

[0099] (6);

[0100] Formula (6) ensures that the number of enabled workstations cannot exceed the total number of workstations;

[0101] (7);

[0102] Formula (7) ensures that tasks can only be assigned to enabled workstations;

[0103] (8);

[0104] Formula (8) Task j for product p is executed on workstation w, For product p, task k is executed on workstation w. If task j needs to be executed before task k, then task k must satisfy the allocation order and be executed on the same workstation as task j or on a workstation after task j. If task j is a predecessor of task k, assuming task j is assigned to workstation 2 and task k is assigned to workstation 1, then the requirements of this formula are not met.

[0105] (9);

[0106] (10);

[0107] In formula (9), M is a positive number. Formulas (9) and (10) guarantee that Meet or exceed hour, is 1; otherwise, is 0.

[0108] S4: Convert the mathematical model established in S3 into a Markov decision process (MDP) and design the state space and action space.

[0109] In S4, the Markov decision process (MDP) includes the state space S, the action space A, the state transition probability P, and the reward function R.

[0110] The state space S consists of two parts: the workstation task assignment sequence and the legal operation identifier. The first part represents the current workstation's task assignment sequence, which contains information about the distribution of assigned tasks on the workstation. The second part represents the legal operations that can be performed. These legal operations define the range of actions that are feasible in the current state and are constrained by the disassembly precedence relation. The action space A represents the assignment of any task to any workstation. The reward function R represents the disassembly time calculated according to the objective function after executing each action. The state transition probability P represents the probability of transitioning from one state to another.

[0111] Markov Decision Process (MDP), a Markov decision process can be described by the following four main parts:

[0112] S4-1: State Space (S)

[0113] State Space (S): represents the set of all possible states of the system. Describes the specific situation of the system at a certain moment.

[0114] The state space consists of two parts. The first part represents the current task allocation sequence of the workstation. This part contains the distribution information of the assigned tasks on the workstation. The second part represents the legal operations that can be performed. These legal operations define the range of actions that are feasible in the current state and are constrained by the disassembly precedence relation.

[0115] For example, in the example of mobile phone disassembly at three workstations, each state can be represented as ,in Indicates the task sequence that workstation i has been assigned to, specifically expressed as , ,when =1 indicates that task j has been assigned to workstation i. =0 means that task j is not assigned to workstation i. a represents a legal operation that can be performed, specifically , ,when =1 means that task j can perform disassembly operation. =0, disassembly cannot be performed.

[0116] S4-2: Action Space (A)

[0117] Action Space (A): represents the set of all actions that the agent can choose in each state. Each action Describes the actions that the agent can take in a specific state.

[0118] The action space allows any task to be assigned to any workstation. This means that in each state, any task can be assigned to any available workstation. This flexibility allows the agent to make the best task assignment decision based on the current state and goal. For example, in the experimental case, the action space size is 12*3. Specifically expressed as in Indicates the value of action i, which means Assignment of tasks to Workstation No.

[0119] S4-3: State Transition Probability (P)

[0120] State Transition Probability (P) describes the probability of transitioning from one state to another. Indicates that after executing action a in state s, it transitions to state probability.

[0121] S4-4: Reward Function (R)

[0122] The reward function (R) defines the immediate reward obtained after performing an action in a certain state. It represents the reward obtained after executing action a in state s.

[0123] S5: Design the DDQN network model structure (DDTQN network model) based on the Q-learning algorithm, set the reward function calculation method according to the objective function in S3, and obtain the optimal objective function value.

[0124] Double DQN (DDQN) is a value-based deep reinforcement learning algorithm for solving discrete action spaces. This algorithm is recoded, with a new Q network and target network designed. This algorithm's update method mitigates the overestimation problem of the original algorithm.

[0125] The pseudo code of the algorithm is shown in Table 1 below:

[0126] Table 1 Pseudocode

[0127]

[0128] The DDQN network model based on the Q-learning algorithm proposed in this paper is defined as the DDTQN network model. DDTQN is an improvement over the traditional DDQN, focusing primarily on the design of the Q network and target network to address the problem of Q-value overestimation, thereby improving the stability of Q-learning in deep reinforcement learning. Compared with the traditional DDQN, DDTQN introduces the Transformer Encoder to enhance the encoding capability of the input data. In the DDTQN model architecture, the Transformer Encoder is integrated into the Q network and target network to more effectively capture the semantic and structural information of the input data. The design of the Q network and target network addresses the problem of Q-value overestimation, thereby improving the stability of Q-learning in deep reinforcement learning. This design enables the Q network and target network to better understand the local structure and global relationships in the state space, thereby improving the model's understanding of the environment and learning effectiveness. To increase the independence between samples, four-tuples (s, a, r, s1) are randomly drawn from the experience pool for training. Training process: The experience replay pool stores interaction data. The dual network update alleviates the problem of overestimation of Q value and captures the dependency between states through the self-attention mechanism.

[0129] Figure 3 The DDTQN model structure of the present invention, which integrates the Transformer Encoder, is demonstrated. The DDTQN network model includes a Q network integrated with the Transformer Encoder and a target network, as well as an experience replay pool and an environment. The processing process is as follows:

[0130] First, the agent evaluates the current disassembly line state space based on the Q network and selects an action. The Q network receives the current disassembly line state as input and outputs estimated Q values for each possible action. The agent selects an action using methods such as greedy strategies and executes it in the disassembly environment. The environment then provides feedback on the next state and corresponding reward value, which is stored in the experience replay pool.

[0131] Next, a batch of experience data is randomly sampled from the experience replay pool. The Q network outputs an estimated Q value based on the current state obtained by sampling. The target network outputs the Q value of the corresponding action based on the next state obtained by sampling and calculates the target Q value based on the reward information.

[0132] Finally, the loss function is constructed by comparing the estimated Q value of the Q network with the target Q value, and the back propagation algorithm is used to update the Q network parameters.

[0133] By introducing the target network, Q network and combining it with the experience replay pool, the problem of overestimation of Q value in traditional deep Q network is effectively alleviated, the learning efficiency and decision accuracy of the agent in the disassembly line balancing problem are improved, and the agent is prompted to converge to a better strategy faster, thereby improving the balance performance of the disassembly line.

[0134] In the problem of disassembly line balancing, the designed Q network and target network have important application value. Figure 4 The Q-network and target-network architectures of the present invention are presented. The inputs to the Q-network and target-network are position codes and action labels, which reflect the status of different locations on the disassembly line and the possible actions to be taken. The entire Q-network and target-network architecture gradually extract and integrate key information from the input position codes and action labels through multi-layer feature extraction and processing.

[0135] like Figure 4 As shown, specifically:

[0136] First, the position code and action label enter the network and undergo preliminary processing through a linear layer, which extracts preliminary features through linear transformation, laying the foundation for the subsequent analysis of the disassembly line status and action.

[0137] Next, the data enters the forward feedback layer, which can capture the long-term and short-term dependencies in the disassembly line data, such as the temporal correlation or contextual relationship between different disassembly steps, helping the network better understand the dynamic changes in the disassembly process, thereby providing richer features for subsequent decision-making.

[0138] Then, the data is normalized by layers, which helps to accelerate the convergence of the network in solving the line balance problem, reduce the gradient disappearance or explosion problem during training, and make the network training more stable and efficient.

[0139] The data then enters the multi-head attention layer, a core component of the network. For the disassembly line balancing problem, the multi-head attention layer concurrently calculates attention on the data from multiple perspectives, comprehensively capturing the correlations between different positions and features during the disassembly process. This allows for more accurate action decisions to achieve line balancing, such as determining the next disassembly location or selecting the appropriate disassembly tool.

[0140] After the multi-head attention layer, the data is again normalized to further ensure that the data is within the appropriate range and maintain the stability of the network when dealing with the problem of disassembly line balance.

[0141] Finally, the data processed by multiple layers passes through another linear layer and outputs the final action. This linear layer integrates and maps the features obtained from the previous processing to generate the action predicted by the network for the current state of the disassembly line, so as to push the disassembly line towards a balanced state.

[0142] The entire network structure gradually extracts and integrates key information from the input position codes and action labels through multi-layer feature extraction and processing, providing strong decision support for achieving disassembly line balance.

[0143] In this DDTQN architecture, the overall structure of the Q network and target network is similar to that of the traditional DDQN, but the introduction of the Transformer Encoder gives the model stronger expressiveness and learning capabilities. The advantage of this design is that the Transformer Encoder can dynamically learn the correlations between input data, thereby more effectively encoding information in the state space. Therefore, while retaining the advantages of the traditional DDQN, the integration of the Transformer Encoder further improves the model's performance and stability, enabling it to perform even better in deep reinforcement learning tasks.

[0144] The Transformer Encoder effectively encodes the input sequence through a self-attention mechanism. Its core idea is to establish correlations between different positions, enabling the model to better capture the semantic and structural information of the input data. The self-attention mechanism allows the model to dynamically adjust its focus on different positions when encoding the input sequence, thereby better adapting to changes in local structure while preserving global information. This capability is particularly important for reinforcement learning models, as reinforcement learning tasks often involve understanding and making decisions about the global state of the environment. The Transformer Encoder can help the model more accurately capture the relationships and interactions between different states in the state space. Therefore, using the Transformer Encoder as a feature extractor can significantly improve the reinforcement learning model's understanding and learning ability of the state space, allowing it to more effectively solve complex reinforcement learning tasks.

[0145] To improve the Q network's fit to the action-value function, in this example, the 48-dimensional state tensor is mapped to a 1024-dimensional one-dimensional tensor through a fully connected layer. This one-dimensional tensor is then split equally into 8-by-128 tensors, and the action token is concatenated to form a 9-by-128 tensor. After adding a positional encoding to each tensor, it passes through multiple encoder layers, extracting the action token output from the final encoder layer and passing it through a fully connected layer as the final Q network output, representing the value of each action.

[0146] Example 2

[0147] This example uses the deep reinforcement learning-based disassembly line balancing method provided in Example 1 as a foundation, and conducts experiments on three different home appliances: a mobile phone, a television, and a refrigerator. Specific example information is shown in Table 2 below.

[0148] Table 2 Instance information

[0149]

[0150] This experiment was run on a server running the Ubuntu 18.04.5 operating system, equipped with an Intel(R) Xeon(R) Silver 4210 CPU @2.20GHz and an NVIDIA GeForce RTX 2080 Ti graphics card. This server, equipped with 62GB of memory, is suitable for high-performance computing, deep learning, and large-scale data processing.

[0151] The Deep Q-Network (DQN) is a reinforcement learning algorithm that combines Q-learning and deep neural networks to solve problems with high-dimensional state spaces. DQN was proposed by the DeepMind team in 2013 and has achieved breakthrough results on several Atari games. DDQN is an improvement on DQN, designed to address the overestimation of Q values in DQN. DDQN reduces this overestimation bias by using two networks to select and evaluate actions separately. Advantage Actor-Critic (A2C) is a policy gradient method that combines a policy network (actor) and a value network (critic). A2C simultaneously optimizes policy and estimates value, offering excellent sample efficiency and stability. These three algorithms are all classic deep reinforcement learning algorithms. In this experiment, we will use DDTQN, DDQN, DQN, and A2C to solve the shortest disassembly time solution for the different disassembly examples described above. Each algorithm was iteratively trained 3000 times. The comparison of training time for different algorithms to solve the optimal disassembly solution is shown in Table 3. The comparison of optimal disassembly time for different algorithms is shown in Table 4. The specific disassembly solution solved by DDTQN is shown in Table 5.

[0152] Table 3 Comparison of training time for different algorithms to solve the optimal disassembly plan

[0153]

[0154] Table 4 Comparison of optimal disassembly time obtained by different algorithms

[0155]

[0156] Table 5 Specific disassembly solutions solved by DDTQN

[0157]

[0158] A comparison of the training time and optimal disassembly solutions of different algorithms shows that, after the same training period, the DDTQN algorithm significantly outperforms the traditional DDQN and DQN algorithms in solving the disassembly line problem. The optimal disassembly sequence duration solved by DDTQN is, on average, 21%, 25%, and 15% shorter than those of DDQN, DQN, and A2C, respectively. While the DDTQN neural network incorporates the Transform's self-attention mechanism, the model structure is somewhat larger, but the solution time remains within an acceptable range. Therefore, DDTQN has a clear advantage in solving the problem of finding the shortest disassembly sequence duration.

[0159] Experiments have shown that the present invention has the following beneficial effects:

[0160] Higher optimization efficiency: Table 4 shows that the disassembly time of DDTQN is 21%, 25%, and 15% shorter than that of DDQN, DQN, and A2C, respectively. This advantage is particularly evident in complex tasks such as refrigerator disassembly.

[0161] More adaptable: The fatigue model dynamically adjusts task allocation to avoid worker overwork and reduced efficiency (e.g. Figure 2 ), suitable for high-load disassembly scenarios.

[0162] Improve resource recovery rate: By optimizing the disassembly sequence (Table 5), reducing task waiting time (the second part of Formula 4), the disassembly throughput of used home appliances can be increased.

[0163] Protect workers' health: Fatigue restraint and forced rest mechanism reduce the risk of muscle injury and meet ergonomic requirements.

[0164] In addition, it should be noted that the present invention has good scalability, and the model can be migrated to other labor-intensive disassembly scenarios (such as power battery recycling) by simply adjusting the fatigue parameters.

[0165] In summary, the present invention: (1) embeds a dynamic fatigue model. For the first time, the accumulation and recovery mechanism of human fatigue (Formulas 1-3) is quantitatively embedded in the mathematical model of the disassembly line balance problem (DLBP), and fatigue constraints are proposed to dynamically reflect the impact of worker fatigue on disassembly efficiency. Traditional methods mostly rely on static rules (such as fixed rotation), while the present invention achieves dynamic adjustment through fatigue formulas (such as exponential decay models), which is more in line with actual production scenarios. (2) Multi-objective optimization and MDP design. The objective function simultaneously optimizes disassembly time and fatigue penalty (Formula 4), and integrates task allocation, fatigue state and timing dependency into the state space through the Markov decision process (MDP). The action space supports dynamic task reallocation, and the state transition probability covers task priority constraints (such as Formula 8) and fatigue thresholds (Formulas 9-10), achieving global optimization. (3) An improved deep reinforcement learning algorithm (DDTQN) is proposed. Based on the traditional DDQN, the Transformer Encoder is introduced to process the state sequence, and the long-range dependency between tasks (such as Figure 3-4 The dual network structure (Q network + target network) combined with the experience replay pool solves the problem of overestimation of Q values and improves convergence stability.

[0166] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the scope of protection of the present invention.

Claims

1. A disassembly line balancing method based on deep reinforcement learning, characterized in that: The method comprises the following steps: S1: Preprocess the disassembled product information and use the constraint matrix to represent the constraint relationship between product tasks; S2: Define factors affecting human fatigue; S3: Define symbols, decision variables, and constraints to establish a disassembly objective function; establish the relationship between fatigue and the objective function, transform the practical problem into a mathematical problem, and then establish a mathematical model for the disassembly line balance problem affected by human fatigue; S4: Convert the mathematical model established in S3 into a Markov decision process (MDP) and design the state space and action space. In S4, the Markov decision process MDP includes a state space S, an action space A, a state transition probability P, and a reward function R. The state space S consists of two parts: a workstation task assignment sequence and a legal operation identifier. The first part is used to represent the task assignment sequence of the current workstation, which contains the allocation information of the assigned tasks on the workstation. The second part represents the executable legal operations. These legal operations define the range of actions that are feasible in the current state. These legal operations are subject to the constraints of the disassembly priority relationship. The action space A represents the assignment of any task to any workstation. The reward function R represents the disassembly time calculated according to the objective function after executing each action. The state transition probability P represents the probability of one state transitioning to another state. S5: Design the DDQN network model structure based on the Q-learning algorithm, set the reward function calculation method according to the objective function in S3, and obtain the optimal objective function value.

2. A disassembly line balancing method based on deep reinforcement learning according to claim 1, characterized in that: In S1, using the priority matrix To express the constraint relationship between two tasks, the element It is expressed as follows: Where p is the product ID, i and j are task indexes.

3. A disassembly line balancing method based on deep reinforcement learning according to claim 1, characterized in that: In S2, the human fatigue factor refers to human muscle fatigue, which is affected by working hours. That is, the accumulation of working hours will lead to muscle fatigue. When the accumulated fatigue reaches the maximum accumulated fatigue, the worker cannot work and is forced to rest. The specific accumulated fatigue is calculated by the following formula: in represents the disassembly time of task j of product p at workstation w, λ w represents the fatigue accumulation rate of workstation w, represents the waiting time of task j of product p at workstation w, μ w represents the fatigue recovery rate of workstation w, That is, the fatigue accumulation at any moment is less than the maximum fatigue accumulation F w,max ; represents the accumulated fatigue after workstation w starts to execute task j of product p, Indicates that workstation w waits before executing task j of product p After the fatigue accumulates, rest Fatigue accumulates over time; Indicates continuing after the previous break Fatigue accumulates after hours of work.

4. A disassembly line balancing method based on deep reinforcement learning according to claim 3, characterized in that: The details of S3 are as follows: S3-1: Define symbols as follows: Product Collection P represents the total number of products, and the task set of any product p I p Indicates the total number of tasks for product p, task index Workstation collection W represents the total number of workstations. S3-2: Define the decision variables as follows: Indicates whether task j of product p is executed on workstation w, 1 if it is executed, 0 otherwise; y w Indicates whether the fatigue value of workstation w has reached the maximum fatigue accumulation F w，max , if reached, it is 1, otherwise it is 0; Indicates whether task j of product p is the immediate predecessor of task k, that is, if task j must be completed before task k, then it is 1, otherwise it is 0; u w Indicates whether workstation w is enabled, if enabled, it is 1, otherwise it is 0; S3-3: Define the objective function, which is expressed as follows: The objective function consists of two parts: the first part is the task disassembly time, and the second part is the task waiting time; the objective function satisfies the constraint relationship; S3-4: The constraint relationship is expressed as follows: Formula (5) ensures that each task is executed at most once; Formula (6) ensures that the number of enabled workstations cannot exceed the total number of workstations; (7) Ensure that tasks can only be assigned to enabled workstations; Formula (8) Task j for product p is executed on workstation w, For product p, task k is executed on workstation w. If task j needs to be executed before task k, then task k must satisfy the allocation order and be executed on the same workstation as task j or on a workstation after task j. If task j is a predecessor of task k, assuming task j is assigned to workstation 2 and task k is assigned to workstation 1, then the requirements of this formula are not met. F w -F w，max ≤M·(1-y w ) (9); F w -F w，max ≥-M·y w (10); In formula (9), M is a positive number. Formulas (9) and (10) guarantee that F w Reach or exceed F w，max When y w is 1; otherwise, y w is 0.

5. The disassembly line balancing method based on deep reinforcement learning according to claim 1, characterized in that: In S5, the DDQN network model based on the Q-learning algorithm is defined as the DDTQN network model, including the Q network and target network fused with the TransformerEncoder, as well as the experience replay pool and environment. The processing process is as follows: First, the current disassembly line state space is evaluated and an action is selected based on the Q network. The Q network receives the current disassembly line state as input and outputs an estimated Q value corresponding to each possible action. The agent selects an action using a greedy strategy and executes it in the disassembly environment. The environment then feeds back the next state and the corresponding reward value, which is stored in the experience replay pool. Next, a batch of experience data is randomly sampled from the experience replay pool. The Q network outputs an estimated Q value based on the current state obtained by the sample. The target network outputs the Q value of the corresponding action based on the next state obtained by the sample and calculates the target Q value based on the reward information. Finally, the loss function is constructed by comparing the estimated Q value of the Q network with the target Q value, and the back propagation algorithm is used to update the Q network parameters.

6. A disassembly line balancing method based on deep reinforcement learning according to claim 5, characterized in that: The input of the Q network and the target network is the position code and the action label. The entire network structure of the Q network and the target network gradually extracts and integrates key information from the input position code and action label through multi-layer feature extraction and processing. Specifically: First, the position code and action label are input into the network and processed by a linear layer to extract preliminary features. Then, the data enters the pre-feedback layer to capture the long-term and short-term dependencies in the disassembly line data. The data then undergoes layer normalization before entering the multi-head attention layer. In the disassembly line balancing problem, the multi-head attention layer concurrently calculates attention on the data from multiple perspectives, comprehensively capturing the correlation between different positions and features during the disassembly process, thereby making action decisions to achieve disassembly line balance. After passing through the multi-head attention layer, the data is again processed by layer normalization. Finally, the multi-layer processed data passes through another linear layer to generate the action predicted by the network for the current disassembly line state.

Images