Disassembly line balancing optimization method and system based on multi-objective reinforcement learning, and medium

Abstract

The application relates to the field of data processing technology prediction, and discloses a disassembly line balancing optimization method and system based on multi-target reinforcement learning and a medium. The method comprises the following steps: S1, input data is acquired and preprocessed; S2, a state space, an action space and a target function are constructed, and a training environment is deployed; S3, a multi-target reinforcement learning model is trained: a preference vector is introduced through an MO-DQN algorithm to dynamically adjust target weights, a target function is calculated according to disassembly data acquired in S1, multiple targets in S2 are optimized, and an action strategy is selected in combination with an epsilon-greedy strategy and an NSGA-II algorithm; and S4, an optimal strategy generated by the trained model is applied to task allocation in a disassembly line. By introducing the preference vector, the target weights are dynamically adjusted according to specific requirements of tasks, conflicts between targets can be effectively handled, an optimization strategy can be automatically adjusted according to changes in different situations, and the efficiency of task scheduling and resource allocation in a disassembly task is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a disassembly line balance optimization method, system and medium based on multi-objective reinforcement learning. Background Art

[0002] Traditional heuristic algorithms are widely used to solve the problem of disassembly line balancing, relying primarily on empirical rules and regulations for task scheduling and resource allocation. While these methods can effectively optimize system performance in static environments, their limitations gradually become apparent when faced with the uncertainty and dynamic changes in the disassembly process. Specifically, traditional heuristic algorithms are unable to respond in real time to fluctuations in task times, temporary changes in resources, or interference from other environmental factors. As a result, when the disassembly task changes, the original balancing scheme may become ineffective, and effective dynamic rebalancing cannot be performed. Therefore, although heuristic algorithms can show certain advantages in certain stable environments, their adaptability and robustness are limited in complex and dynamic task environments.

[0003] Compared with traditional heuristic algorithms, reinforcement learning (RL) methods can self-adjust strategies in more complex and dynamic environments through interactive learning between intelligent agents and the environment, demonstrating greater flexibility and adaptability. However, traditional reinforcement learning algorithms still have significant flaws when dealing with multi-objective optimization problems. Typically, traditional RL methods optimize by weighting multiple objectives and fixing the weights, but this approach cannot flexibly adjust the relative importance of each objective according to actual needs. In disassembly tasks, there are often conflicts and competition between different objectives. Because traditional RL methods fix the weights of the weighted objectives and cannot dynamically balance between objectives, when faced with complex changes in the environment, this method finds it difficult to effectively adapt to real-time changes in tasks and actual needs.

[0004] Therefore, the application of traditional heuristic algorithms and reinforcement learning methods in disassembly line balancing faces many challenges, especially in terms of multi-objective optimization and adaptability to dynamic environments, and lacks sufficient flexibility and adjustment capabilities. Summary of the Invention

[0005] The technical problem to be solved by this application is to overcome the shortcomings of the existing technology. This application provides a disassembly line balance optimization method, system and medium based on multi-objective reinforcement learning. By introducing preference vectors, the target weights can be dynamically adjusted according to the specific requirements of the task, thereby achieving efficient multi-objective optimization. It can not only effectively handle conflicts between targets, but also automatically adjust the optimization strategy according to changes in different situations, significantly improving the task scheduling and resource allocation efficiency in disassembly tasks.

[0006] To achieve the above objectives, the present application provides, in a first aspect, a disassembly line balance optimization method based on multi-objective reinforcement learning, comprising the following steps:

[0007] S1. Get input data and preprocess it:

[0008] Obtain various data during the disassembly task execution and perform preprocessing, record the disassembly priority relationship constraints, construct a disassembly priority relationship constraint matrix, obtain the hazard attributes and demand attributes of the disassembly task, and obtain the disassembly time, disassembly cost, and disassembly energy consumption of different disassembly tasks at manual workstations and robot workstations respectively;

[0009] S2. Construct the state space, action space, and objective function, and deploy the training environment:

[0010] The state space includes a plurality of state features, and the state features include at least one of the following states: workstation load state, robot load state, task allocation state, predecessor task completion state, successor task inspiration state, task-station fitting state, and cycle offset state; the action space includes a plurality of action strategies, and the action strategies include at least one of the following strategies: longest execution time priority strategy, shortest execution time priority strategy, least predecessor task number priority strategy, most successor task number priority strategy, cycle time matching optimal strategy, and random strategy; the objective function is designed based on the three objectives of workstation number, disassembly cost, and disassembly energy consumption, and a multi-objective reinforcement learning model is constructed based on the MO-DQN algorithm;

[0011] S3. Training multi-objective reinforcement learning model:

[0012] The multi-objective reinforcement learning model introduces a preference vector through the MO-DQN algorithm to dynamically adjust the objective weights. Based on the disassembly data obtained in S1, the objective function is calculated, and multiple objectives in S2 are optimized. By combining the ε-greedy strategy and the NSGA-II algorithm to select the action strategy, it finally converges to an optimal model that can balance multiple objectives.

[0013] S4. Apply the optimal strategy generated by the trained multi-objective reinforcement learning model to task allocation in the actual disassembly line.

[0014] Optionally, in S2, a state space is constructed based on the disassembly line balance optimization problem, including: the workstation load state represents the ratio of the total time consumed by the currently assigned tasks to each workstation to the theoretical cycle time, reflecting the relative load situation between the workstations; the robot load state represents the total time consumed by the tasks currently undertaken by each robot, which is used to measure the global distribution balance of multi-robot resources; the task allocation state uses a Boolean value to mark whether the task has been assigned, which is used to dynamically identify the current feasible task set and is an important prerequisite for the subsequent action selection strategy; the predecessor task completion state is used to dynamically record whether the predecessor task of each task has been completed to ensure the correctness of the task execution order; the successor task inspiration state is used to indicate the number of subsequent tasks activated after the current task is completed, reflecting the contribution of the task to the overall scheduling advancement; the task-workstation fitting state is used to measure whether a specific task will exceed the theoretical cycle time of a workstation after being assigned to the workstation; the cycle offset state is used to record the deviation between the load of the heaviest loaded workstation in the current system and the theoretical cycle time, as a global performance indicator of the scheduling effect.

[0015] Optionally, S2 constructs an action space based on the disassembly line balance optimization problem, including: the longest execution time priority strategy indicates that the task with the longest execution time among the currently executable tasks is prioritized; the shortest execution time priority strategy indicates that the task with the shortest execution time among the currently executable tasks is prioritized; the least number of predecessor tasks priority strategy indicates that the task with the least predecessor tasks among the currently executable tasks is prioritized; the largest number of successor tasks priority strategy indicates that the task with the greatest potential impact is selected, its upstream nodes are processed first, and the scheduling depth is reduced; the cycle time matching optimal strategy indicates that the task allocation scheme is prioritized to make the target workstation load close to the theoretical cycle time; the random strategy indicates that tasks are randomly selected for allocation, which is used to explore new strategies or as a benchmark for control experiments.

[0016] Optionally, S1 obtains input data and preprocesses it, including:

[0017] Sensors are deployed on each workstation and robot in the disassembly line to monitor various data involved in the disassembly task execution in real time, including task execution time, workstation load, robot load, and task priority. These data are collected and pre-processed.

[0018] Optionally, the objective function in S2 is expressed as:

[0019] ;

[0020] in, 、 、 is the weight coefficient of each goal, reflecting the relative importance of each goal in decision-making and achieving collaborative optimization among different goals. Indicates the number of workstations, represents the dismantling cost, Indicates disassembly energy consumption.

[0021] Optionally, the multi-objective reinforcement learning model includes an intelligent agent and a disassembly line balancing environment, which continuously interact with each other, wherein the intelligent agent adopts a deep Q network, the intelligent agent receives state characteristics of the disassembly line balancing environment, selects actions according to the action strategy and objective function, and the disassembly line balancing environment returns rewards and updates network parameters;

[0022] The multi-objective reinforcement learning model is trained in S3. During the training process, the MO-DQN algorithm combines multiple target values ​​into a vector Q-value function, and uses different preference weights as conditional inputs during training to learn a generalized mapping from preference to Q-value. In each round of environmental interaction, the preference vector is sampled. The deep Q network takes the concatenation of the state vector s and the preference vector ω as input, and outputs the multi-objective Q value vector of each action according to the disassembly priority relation constraint matrix. The final action selection is selected by combining the ε-greedy strategy with the NSGA-II algorithm to select the action strategy. The NSGA-II algorithm performs non-dominated sorting on the multi-objective Q value vectors corresponding to the action and calculates the crowding distance to determine the optimal action.

[0023] Optionally, the architecture of the deep Q network includes an input layer, a hidden layer and an output layer. The input layer concatenates the state vector s and the preference vector ω to form a mixed input with a dimension of state dimension plus preference vector dimension, that is, state dimension plus target dimension; the hidden layer is a three-layer fully connected neural network, adopts ReLU activation function, and adds Dropout to prevent overfitting; the output layer outputs the multi-target Q vector corresponding to each action for subsequent action selection.

[0024] To achieve the above objectives, the second aspect of the present application provides a disassembly line balance optimization system based on multi-objective reinforcement learning, the optimization system comprising:

[0025] The acquisition unit is used to obtain input data and preprocess it: it obtains various types of data during the execution of the disassembly task and preprocesses them, records the disassembly priority relationship constraints, constructs the disassembly priority relationship constraint matrix, obtains the hazard attributes and demand attributes of the disassembly task, and obtains the disassembly time, disassembly cost, and disassembly energy consumption of different disassembly tasks on manual workstations and robot workstations respectively;

[0026] A construction unit is used to construct a state space, an action space and a reward function, and deploy a training environment: the state space includes a plurality of state features, and the state features include at least one of the following states: workstation load state, robot load state, task allocation state, predecessor task completion state, successor task inspiration state, task-station fitting state and cycle offset state; the action space includes a plurality of action strategies, and the action strategies include at least one of the following strategies: longest execution time priority strategy, shortest execution time priority strategy, least predecessor task number priority strategy, most successor task number priority strategy, cycle time matching optimal strategy and random strategy; the objective function is designed based on the three objectives of the number of workstations, disassembly cost and disassembly energy consumption, and a multi-objective reinforcement learning model is constructed based on the MO-DQN algorithm;

[0027] A training unit is used to train a multi-objective reinforcement learning model: the multi-objective reinforcement learning model dynamically adjusts the objective weights by introducing a preference vector through the MO-DQN algorithm, calculates the objective function based on the disassembly data obtained in S1, optimizes multiple objectives in S2, and selects action strategies by combining the ε-greedy strategy and the NSGA-II algorithm, ultimately converging to an optimal model that can balance multiple objectives;

[0028] The optimization unit is used for task scheduling and resource allocation optimization: the optimal strategy generated by the trained multi-objective reinforcement learning model is applied to task allocation in the actual disassembly line.

[0029] To achieve the above-mentioned purpose, the third aspect of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the method described above.

[0030] After adopting the above technical solution, this application has the following beneficial effects compared with the prior art:

[0031] This application introduces the MO-DQN algorithm to dynamically optimize task scheduling and resource allocation for multiple objectives in disassembly tasks, such as minimizing the number of workstations, minimizing disassembly costs, and minimizing disassembly energy consumption. This effectively addresses the uncertainty and multi-objective conflicts in human-machine collaborative disassembly lines. By introducing preference vectors, the weights of different objectives can be flexibly adjusted, balancing conflicts between multiple objectives in real time, improving the intelligence and flexibility of the disassembly line system and effectively optimizing disassembly efficiency.

[0032] The specific implementation methods of the present application are further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are part of this application and are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application but do not constitute an undue limitation of this application. Obviously, the drawings described below are only some embodiments. For those of ordinary skill in the art, other drawings can be derived from these drawings without inventive effort.

[0034] In the attached figure:

[0035] Figure 1 This is a flow chart of a disassembly line balance optimization method based on multi-objective reinforcement learning in this specific implementation manner;

[0036] Figure 2 Schematic diagram of the reinforcement learning framework of the disassembly line balance optimization method based on multi-objective reinforcement learning in this specific embodiment;

[0037] Figure 3 This is a schematic diagram of the framework of the MO-DQN algorithm of the disassembly line balance optimization method based on multi-objective reinforcement learning in this specific implementation;

[0038] Figure 4 This is a reward diagram of the training process of the three algorithms MO-DQN, MO-A2C, and MO-DDPG in the multi-objective reinforcement learning task in this specific implementation;

[0039] Figure 5 Schematic diagram of the structure of the disassembly line balance optimization system based on multi-objective reinforcement learning in this specific implementation. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not used to limit the scope of the present application.

[0041] This application aims to solve the problem of dynamic balance optimization of disassembly lines driven by multi-objective reinforcement learning, especially the uncertainty and multi-objective conflict problems faced in human-machine collaborative disassembly lines. At present, traditional heuristic algorithms mainly rely on empirical rules when dealing with disassembly line balance. They are difficult to cope with dynamic changes such as task time fluctuations and uneven resource allocation, and lack the ability to adjust uncertainties in the disassembly process in real time. At the same time, traditional reinforcement learning methods usually rely on a simple weighting method with fixed weights. This method shows low flexibility and adaptability when facing changes and conflicts between multiple objectives in the task. Therefore, traditional methods are often unable to effectively optimize in dynamic environments.

[0042] To overcome this limitation, this application introduces a multi-objective deep reinforcement learning algorithm, namely the MO-DQN algorithm, and combines it with the multi-objective requirements of the disassembly task, such as minimizing the number of workstations, minimizing the disassembly cost, minimizing the disassembly energy consumption, etc., to optimize task scheduling and resource allocation using the MO-DQN algorithm. By introducing a preference vector, MO-DQN can dynamically adjust the weights of each objective to flexibly respond to conflicts between objectives and adapt to changes during task execution. This method provides a more intelligent and flexible dynamic balance solution for the multi-objective optimization problem in the disassembly line, significantly improving the system's adaptability and optimization capabilities in complex and dynamic environments.

[0043] Based on this, see Figure 1-Figure 4 , the present application provides a disassembly line balance optimization method based on multi-objective reinforcement learning, comprising the following steps:

[0044] S1. Get input data and preprocess it:

[0045] Obtain various data during the disassembly task execution and perform preprocessing, record the disassembly priority relationship constraints, construct a disassembly priority relationship constraint matrix, obtain the hazard attributes and demand attributes of the disassembly task, and obtain the disassembly time, disassembly cost, and disassembly energy consumption of different disassembly tasks at manual workstations and robot workstations respectively;

[0046] S2. Construct the state space, action space, and objective function, and deploy the training environment:

[0047] The state space includes several state features, including at least one of the following states: workstation load state, robot load state, task allocation state, predecessor task completion state, successor task inspiration state, task-station fitting state, and cycle offset state; the action space includes several action strategies, including at least one of the following strategies: longest execution time priority strategy, shortest execution time priority strategy, least predecessor task number priority strategy, most successor task number priority strategy, cycle time matching optimal strategy, and random strategy; the objective function is designed based on the three objectives of workstation number, disassembly cost, and disassembly energy consumption, and a multi-objective reinforcement learning model is constructed based on the MO-DQN algorithm;

[0048] S3. Training multi-objective reinforcement learning model:

[0049] The multi-objective reinforcement learning model uses the MO-DQN algorithm to introduce a preference vector to dynamically adjust the objective weights. Based on the disassembly data obtained in S1, the objective function is calculated and multiple objectives in S2 are optimized. By combining the ε-greedy strategy and the NSGA-II algorithm to select the action strategy, it eventually converges to an optimal model that can balance multiple objectives.

[0050] S4. Task scheduling and resource allocation optimization:

[0051] The optimal strategy generated by the trained multi-objective reinforcement learning model is applied to task allocation in the actual disassembly line.

[0052] It should be noted that the execution subject of the disassembly line balance optimization method based on multi-objective reinforcement learning in this embodiment is an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, etc., and the non-mobile electronic device can be a server or a personal computer, etc., which is not specifically limited in this application. The following describes the disassembly line balance optimization method based on multi-objective reinforcement learning in this embodiment, taking the execution subject as a server as an example.

[0053] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly specified.

[0054] This embodiment improves upon the disassembly scenario by introducing a manual candidate mechanism to ensure the continuity of the disassembly process and construct a fine-grained state space and action space that characterizes the current scenario. The multi-objective reinforcement learning algorithm employed in this embodiment, the MO-DQN algorithm, differs from the previous weighted summation approach for multi-objective optimization. Instead, it uses the target values ​​of multiple objective functions as a reward vector, training a single policy for all preference vectors. Furthermore, the NSGA-II algorithm is introduced for action selection to help select the optimal solution under multiple objectives.

[0055] This embodiment introduces a multi-objective deep reinforcement learning algorithm, namely MO-DQN, which can not only optimize the trade-offs between multiple objectives, but also dynamically adjust the task allocation strategy to cope with the uncertainty and interference generated during the disassembly process. MO-DQN provides a solution to embed the target preference weights into the policy structure. By constructing the preference condition Q function Q(s,a,ω), this method can not only handle multi-objective tasks, but also has excellent preference generalization and strategy reuse capabilities. Combined with the "divergent" characteristics of the disassembly task, this method can quickly adapt to the user's needs for trade-offs between different objectives, such as flexibly switching between the "lowest disassembly energy consumption" or "lowest disassembly cost" strategies, thereby improving the intelligence and real-time performance of the disassembly decision-making system.

[0056] In practical applications, S1. Obtaining and preprocessing input data: Sensors are deployed at each workstation and robot in the disassembly line to monitor various data during the disassembly task execution in real time, including task execution time, workstation load, robot load, and task priority. Data from different disassembly tasks is collected and preprocessed to ensure data quality and consistency. Data preprocessing includes operations such as denoising and normalization to provide high-quality input for the subsequent multi-objective reinforcement learning model.

[0057] As a specific implementation, S2, constructing the state space, action space, and objective function, and deploying the training environment: In this stage, a multi-dimensional state space is constructed, covering key elements in the task scheduling process, such as workstation load status, robot load status, and task allocation status. Based on the collected real-time data, a simulated disassembly environment is generated for training the reinforcement learning model.

[0058] To enable the reinforcement learning agent to globally perceive and accurately make decisions about complex disassembly systems, a high-dimensional state space vector structure is constructed, encompassing multiple dimensions of critical information. This structure comprehensively captures key elements of the task scheduling process, including structural characteristics, resource load, logical dependencies, and scheduling potential, aiming to provide sufficient environmental representation and learning support for the policy network.

[0059] As a specific implementation method, S2 constructs a state space based on the disassembly line balance optimization problem, including: the workstation load state represents the ratio of the total time of the tasks currently assigned to each workstation to the theoretical cycle time, reflecting the relative load situation between each workstation; the robot load state represents the total time of the tasks currently undertaken by each robot, which is used to measure the distribution balance of multi-robot resources in the global scope; the task allocation state uses a Boolean value to mark whether the task has been assigned, which is used to dynamically identify the current feasible task set and is an important prerequisite for the subsequent action selection strategy; the predecessor task completion state is used to dynamically record whether the predecessor task of each task has been completed to ensure the correctness of the task execution order; the successor task inspiration state is used to characterize the number of subsequent tasks that can be activated after the current task is completed, reflecting the contribution of the task to the overall scheduling advancement; the task-workstation fitting state is used to measure whether a specific task will exceed the theoretical cycle time of a workstation after being assigned to the workstation; the cycle offset state is used to record the deviation between the load of the heaviest loaded workstation in the current system and the theoretical cycle time, as a global performance indicator of the scheduling effect.

[0060] To improve the training stability and convergence efficiency of the neural network, all state features are normalized before input, uniformly mapping their value range to the interval [0, 1]. This avoids gradient instability caused by inconsistent dependent variable dimensions. Furthermore, considering that the number of tasks may vary across different task instances, a zero-padding mechanism is used to expand and fill the state vector to ensure that the input dimension of the neural network structure is fixed. This ensures that the model maintains structural consistency and training compatibility across different problem sizes.

[0061] It's important to note that the purpose of task-station fit and cycle offset is to ensure that the disassembly time at a station does not exceed the cycle time, and remains as close to the cycle time as possible, thus achieving load balancing. Ensuring that the disassembly time at a station does not exceed the cycle time is the most fundamental metric for disassembly line balancing. Load balancing is also a common metric in this area.

[0062] Considering the huge number of disassembly tasks and the complex dependencies between tasks, if the action space is directly modeled as a discrete combination of "task-station-robot" triplets, its dimension will increase exponentially with the task scale and resource allocation, which can easily lead to problems such as explosion of training sample dimensions, difficulty in convergence of policy networks, and low sample utilization. In order to effectively alleviate the computational burden and learning barriers brought about by high-dimensional action space, this embodiment introduces an action space abstract modeling method based on heuristic strategies. Specifically, the original action space is compressed and mapped into the following representative task allocation rules. Each strategy represents a typical scheduling logic, thereby achieving effective compression of action dimensions and enhancement of semantic expression.

[0063] In practical applications, the longest execution time priority in S2 means that tasks that are expected to take the longest time are scheduled first, aiming to reduce system congestion and release high-occupancy resources; the shortest execution time priority means that short tasks are assigned first to quickly fill idle workstations and improve the scheduling response frequency per unit time; the least number of predecessor tasks priority means that tasks with the least dependencies are scheduled first to accelerate the expansion of the task network; the largest number of successor tasks priority means selecting tasks with the greatest potential impact, giving priority to their upstream nodes, and reducing the scheduling depth; the best cycle time matching means giving priority to task allocation schemes to make the target workstation load close to the theoretical cycle time and improve rhythm balance; the random strategy means randomly selecting tasks for allocation, which is used to explore new strategies or as a benchmark for control experiments.

[0064] It should be noted that the introduction of this abstract action space not only significantly compresses the policy learning dimension, improving training convergence speed and sample utilization efficiency, but also enhances the interpretability and versatility of policy behavior through the semantic encoding of different task allocation rules. During training, the policy network perceives the current system state, learns to judge the performance of various policies in different scenarios, and dynamically selects the optimal allocation logic, thereby achieving environment-driven scheduling strategy evolution and real-time switching.

[0065] In the multi-objective reinforcement learning modeling of product disassembly line optimization problems, the design of the objective function is crucial. In order to achieve comprehensive optimization of the disassembly system in terms of resource utilization, economic cost and environmental impact, three core objectives are comprehensively considered: the number of workstations , dismantling costs and disassembly energy consumption Compared with single-objective optimization, the policy learning mechanism based on multi-objective reinforcement learning can simultaneously balance performance in multiple dimensions, improving the robustness and practical feasibility of the solution.

[0066] As a specific implementation method, due to the potential conflict between the three objective functions, this embodiment adopts the weighted linear combination method to model multi-objective reinforcement learning. In the weighted combination form, the objective function in S2 is expressed as:

[0067] ;

[0068] in, is the weight coefficient of each goal, reflecting the relative importance of each goal in decision-making and achieving collaborative optimization among different goals. Indicates the number of workstations, represents the dismantling cost, Indicates disassembly energy consumption.

[0069] Combine Figure 3 As shown, the multi-objective reinforcement learning model constructed in this embodiment includes an intelligent agent and a disassembly line balance environment, which continuously interact with each other. The intelligent agent in this embodiment adopts a deep Q network. The intelligent agent receives the state characteristics of the disassembly line balance environment, selects actions according to the action strategy and objective function, and the disassembly line balance environment returns rewards and updates the network parameters.

[0070] In practical applications, S3 trains a multi-objective reinforcement learning model, including combining multiple target values ​​into a vector Q-value function through the MO-DQN algorithm during training, and using different preference weights as conditional inputs during training to learn a generalized mapping from preferences to Q-values.

[0071] In each round of environment interaction, the sampling preference vector The deep Q network takes the concatenation of the state vector s and the preference vector ω as input, and outputs the multi-objective Q value vector of each action according to the disassembly priority relation constraint matrix. The final action selection is selected by combining the ε-greedy strategy with the NSGA-II algorithm to select the action strategy. The NSGA-II algorithm performs non-dominated sorting on the multi-objective Q value vectors corresponding to the action and calculates the crowding distance to determine the optimal action.

[0072] The decomposition precedence constraint matrix is ​​used to encode the dependencies between tasks (i.e., which tasks must be completed before other tasks). In this embodiment, it is integrated into the training process through state representation and action masks to ensure that the intelligent agent optimizes the multi-objective Q value while satisfying the dependencies.

[0073] As a specific implementation method, the architecture of the deep Q network includes an input layer, a hidden layer, and an output layer. The input layer concatenates the state vector s and the preference vector ω to form a mixed input with the dimensions of the state dimension plus the preference vector dimension, that is, the state dimension plus the target dimension; the hidden layer is a three-layer fully connected neural network, using the ReLU activation function and adding Dropout to prevent overfitting; the output layer outputs the multi-target Q vector corresponding to each action for subsequent action selection.

[0074] It should be noted that the following mechanisms are used to enhance algorithm performance:

[0075] 1. Experience replay mechanism:

[0076] During training, a five-tuple experience pool is constructed. Each sample contains: current state, action, vectorized reward, next state, and termination flag. The reward is a multi-dimensional vector representing the immediate feedback on each target dimension. The experience pool capacity is set to 100,000 samples and supports parallel asynchronous sampling operations. In each round of training, a certain number of trajectories are randomly sampled from the experience pool and combined with the preference vector to construct a training sample batch.

[0077] 2. Preference vector sampling mechanism:

[0078] To ensure the policy network's adaptability to all possible combinations of target weights, the preference vector ω is uniformly sampled from the unit hyperplane during training. For three-dimensional targets, a large number of preference samples are generated using a Dirichlet distribution or uniform sampling within an equilateral triangle, thereby training a deep Q-network with strong generalization across the entire space. This approach enables the trained model to output not only a single policy, but also customized action policies based on different weight preferences.

[0079] 3. Action selection strategy:

[0080] During the training process, an ε-greedy strategy is used to balance exploration and exploitation: (1) an action is randomly selected with a probability of ε, where ε is initially set to 0.5 and linearly decreases to 0.05 during training; (2) the action that maximizes the inner product of ω and Q vector is selected with a probability of 1-ε, that is, the optimal strategy is selected based on the current preference;

[0081] In the test phase, a specific preference weight vector ω is input, and the action that maximizes the weighted Q value is directly selected to achieve the optimal policy output under preference control;

[0082] 4. Preference adaptation and strategy generalization capabilities:

[0083] Because this model learns a generalized mapping relationship from preference vectors to strategies, it is extremely flexible in actual deployment: (1) users can directly input specific target weight combinations, and the system quickly outputs the corresponding strategies; (2) when the environment changes or the target weights are dynamically adjusted, there is no need for retraining; only ω needs to be adjusted to generate a new strategy, achieving real-time adaptation and high responsiveness.

[0084] In practical applications, S3 trains a multi-objective reinforcement learning model, including combining multiple objective values ​​into a vector Q-value function through the MO-DQN algorithm during training, and using different preference weights as conditional inputs during training to learn a generalized mapping from preferences to Q-values. This includes a deep Q network that takes the concatenation of the state vector s and the preference vector ω as input and outputs the Q-vector corresponding to each possible action. The final action selection uses the NSGA-II algorithm to perform non-dominated sorting on the multi-objective Q-value vectors corresponding to the action and calculate their crowding distance to determine the optimal action.

[0085] As a specific implementation method, the architecture of the deep Q network includes an input layer, a hidden layer, and an output layer. The input layer concatenates the state vector s and the preference vector ω to form a mixed input with the dimensions of the state dimension plus the preference vector dimension, that is, the state dimension plus the target dimension; the hidden layer is a three-layer fully connected neural network, using the ReLU activation function and adding Dropout to prevent overfitting; the output layer outputs the multi-target Q vector corresponding to each action for subsequent action selection.

[0086] In practical applications, training multi-objective reinforcement learning models in S3 involves sampling preference vectors in each round of environment interaction. , enter the current state and preferences To the deep Q network, according to the disassembly priority relationship constraint matrix, the multi-objective Q value vector of each action is output, and the action strategy is selected by using the ε-greedy strategy combined with the NSGA-II algorithm.

[0087] In practical applications, after training, the optimal strategy generated by the MO-DQN algorithm is applied to task allocation in the actual disassembly line.

[0088] Specifically, after training, the optimal policy generated by the MO-DQN algorithm was applied to task allocation in an actual disassembly line. Leveraging the dynamic nature of reinforcement learning, the system can respond in real time to uncertainties in the disassembly process, such as fluctuations in task execution time and uneven resource allocation, thereby optimizing the overall efficiency of the disassembly line. This dynamic adjustment not only optimizes the load on each workstation and robot, but also effectively reduces idle time during the disassembly process, improving resource utilization.

[0089] Compared to existing traditional methods, this application uses multi-objective reinforcement learning to solve the disassembly line balance problem. By introducing a multi-objective optimization framework, MO-DQN can dynamically balance different objectives in the disassembly task and adjust the strategy based on real-time feedback to maximize the overall system benefit. In addition, the dynamic adjustment capabilities based on reinforcement learning enable this method to effectively cope with the ever-changing tasks and resource allocation issues during the disassembly process, showing strong adaptability and flexibility.

[0090] See Figure 5 Based on the same inventive concept, the present application also provides a disassembly line balance optimization system based on multi-objective reinforcement learning, the system comprising:

[0091] The acquisition unit is used to obtain input data and preprocess it: obtain various types of data during the execution of the disassembly task and preprocess it;

[0092] A construction unit is used to construct a state space, an action space, and a reward function, and deploy a training environment: the state space includes several state features, including workstation load state, robot load state, task allocation state, predecessor task completion state, successor task inspiration state, task-station fitting state, and cycle offset state; the action space includes multiple strategies, including longest execution time priority, shortest execution time priority, least number of predecessor tasks priority, most number of successor tasks priority, optimal cycle time matching, and random strategy; a reward function is designed based on three core objectives: the number of workstations, disassembly cost, and disassembly energy consumption, to construct a multi-objective reinforcement learning model;

[0093] The training unit is used to train a multi-objective reinforcement learning model. The MO-DQN algorithm introduces a preference vector to dynamically adjust the objective weights, optimizing multiple objectives in S2. By combining the ε-greedy strategy of the NSGA-II algorithm, it ultimately converges to an optimal model that balances multiple objectives.

[0094] The optimization unit is used for task scheduling and resource allocation optimization: the trained multi-objective reinforcement learning model is applied to task allocation in the actual disassembly line.

[0095] This application combines multi-objective reinforcement learning with disassembly task scenario modeling to successfully construct a dynamic disassembly line balancing system. This system deeply perceives the state space of key resources such as tasks, robots, and workstations, and adjusts the allocation and execution order of disassembly tasks in real time, thereby improving the system's adaptability and scheduling efficiency. Unlike traditional single-objective reinforcement learning methods, MO-DQN considers multiple objectives such as the number of workstations, disassembly cost, and disassembly energy consumption by designing a multidimensional objective function and adopting a vectorized objective approach. It can also flexibly adjust the optimization strategy based on different weight requirements. This method is particularly suitable for disassembly tasks with high complexity and uncertainty. In the actual refrigerator disassembly task, MO-DQN demonstrated excellent performance. Experimental verification shows that MO-DQN not only effectively resolves conflicts between multiple objectives, but also achieves coordinated optimization of multiple dimensions such as resource utilization, disassembly energy consumption, and disassembly cost. Experimental results also show that the policy network can gradually converge during training and flexibly adjust the policy output based on the preferences of different objectives, demonstrating good learning efficiency and convergence characteristics. MO-DQN provides an efficient and intelligent multi-objective optimization solution for disassembly tasks. It is highly adaptable and real-time, capable of handling the dynamic changes in the actual disassembly environment. This approach not only provides theoretical support for the intelligent scheduling of industrial disassembly lines, but also lays the foundation for further extension to larger-scale industrial systems.

[0096] See Figure 4 The reward graph shows the changing trends of the reward values ​​of the three algorithms, MO-DQN, MO-A2C, and MO-DDPG, during the training process in a multi-objective reinforcement learning task. MO-A2C represents the multi-objective A2C algorithm (Advantage Actor-Critic algorithm) and MO-DDPG represents the multi-objective DDPG algorithm (Deep Deterministic Policy Gradient algorithm). Among them, the MO-DQN algorithm exhibits faster convergence speed and higher stability, and the final reward value is the highest, showing the best performance; the MO-A2C algorithm converges faster in the early stage but fluctuates in the middle stage, and its final performance is second; the MO-DDPG algorithm converges the slowest and fluctuates more, and the final cumulative reward value is the lowest. This shows that the learning efficiency and policy optimization effect of the MO-DQN algorithm are better than the other two algorithms in the disassembly line equilibrium environment.

[0097] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method described above is implemented.

[0098] The program product of the present application for implementing the above-mentioned method may be a portable compact disk read-only memory and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In the present application, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0099] It should be noted that a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0100] The above is only a preferred embodiment of the present application and does not constitute any form of limitation to the present application. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the present application can make some changes or modifications to equivalent embodiments with equivalent changes using the technical content suggested above without departing from the scope of the technical solution of the present application. The implementation schemes in the above embodiments can also be further combined or replaced. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application that do not depart from the content of the technical solution of the present application still fall within the scope of the solution of the present application.

Claims

1. A disassembly line balance optimization method based on multi-objective reinforcement learning, characterized by: The following steps are involved: S1. Get input data and preprocess it: Obtain various data during the disassembly task execution and perform preprocessing, record the disassembly priority relationship constraints, construct a disassembly priority relationship constraint matrix, obtain the hazard attributes and demand attributes of the disassembly task, and obtain the disassembly time, disassembly cost, and disassembly energy consumption of different disassembly tasks at manual workstations and robot workstations respectively; S2. Construct the state space, action space, and objective function, and deploy the training environment: The state space includes a plurality of state features, and the state features include at least one of the following states: workstation load state, robot load state, task allocation state, predecessor task completion state, successor task inspiration state, task-station fitting state and cycle offset state; the action space includes a plurality of action strategies, and the action strategies include at least one of the following strategies: longest execution time priority strategy, shortest execution time priority strategy, least predecessor task number priority strategy, most successor task number priority strategy, cycle time matching optimal strategy and random strategy; the objective function is designed based on the three objectives of the number of workstations, disassembly cost and disassembly energy consumption, and a multi-objective reinforcement learning model is constructed based on the MO-DQN algorithm; the objective function in S2 is expressed as: R = -(w1f1+w2f2-w3f3); Among them, w1, w2, and w3 are the weight coefficients of each goal, reflecting the relative importance of each goal in decision-making and achieving collaborative optimization among different goals. f1 represents the number of workstations, f2 represents the disassembly cost, and f3 represents the disassembly energy consumption. S3. Training multi-objective reinforcement learning model: The multi-objective reinforcement learning model introduces a preference vector through the MO-DQN algorithm to dynamically adjust the objective weights. Based on the disassembly data obtained in S1, the objective function is calculated, and multiple objectives in S2 are optimized. By combining the ε-greedy strategy and the NSGA-II algorithm to select the action strategy, it finally converges to an optimal model that can balance multiple objectives. During training, the MO-DQN algorithm combines multiple target values ​​into a vector-based Q-value function. It uses different preference weights as conditional inputs during training to learn a generalized mapping from preferences to Q-values. The NSGA-II algorithm performs non-dominated sorting on the multi-target Q-value vectors corresponding to actions and calculates the crowding distance to determine the optimal action. S4. Apply the optimal strategy generated by the trained multi-objective reinforcement learning model to task allocation in the actual disassembly line.

2. The method according to claim 1, characterized in that In S2, a state space is constructed based on the disassembly line balance optimization problem, including: the workstation load state represents the ratio of the total time consumed by the tasks currently assigned to each workstation to the theoretical cycle time, reflecting the relative load situation between each workstation; the robot load state represents the total time consumed by the tasks currently undertaken by each robot, which is used to measure the global distribution balance of multi-robot resources; the task allocation state uses a Boolean value to mark whether the task has been assigned, which is used to dynamically identify the current feasible task set and is an important prerequisite for the subsequent action selection strategy; the predecessor task completion state is used to dynamically record whether the predecessor task of each task has been completed to ensure the correctness of the task execution order; the successor task inspiration state is used to indicate the number of subsequent tasks activated after the current task is completed, reflecting the contribution of the task to the overall scheduling progress; the task-workstation fitting state is used to measure whether a specific task will exceed the theoretical cycle time of a workstation after being assigned to the workstation; the cycle offset state is used to record the deviation between the load of the heaviest-loaded workstation in the current system and the theoretical cycle time, which serves as a global performance indicator of the scheduling effect.

3. The method according to claim 1, characterized in that In S2, an action space is constructed based on the disassembly line balance optimization problem, including: the longest execution time priority strategy indicates that the task with the longest execution time among the currently executable tasks is prioritized; the shortest execution time priority strategy indicates that the task with the shortest execution time among the currently executable tasks is prioritized; the least number of predecessor tasks priority strategy indicates that the task with the least predecessor tasks among the currently executable tasks is prioritized; the largest number of successor tasks priority strategy indicates that the task with the greatest potential impact is selected, its upstream nodes are processed first, and the scheduling depth is reduced; the cycle time matching optimal strategy indicates that the task allocation scheme is prioritized to make the target workstation load close to the theoretical cycle time; the random strategy indicates that tasks are randomly selected for allocation, which is used to explore new strategies or as a benchmark for control experiments.

4. The method according to claim 1, wherein S1 obtains input data and preprocesses it, including: Sensors are deployed on each workstation and robot in the disassembly line to monitor various data involved in the disassembly task execution in real time, including task execution time, workstation load, robot load, and task priority. These data are collected and pre-processed.

5. The method according to claim 3, characterized in that The multi-objective reinforcement learning model includes an intelligent agent and a disassembly line balance environment, which continuously interact with each other. The intelligent agent uses a deep Q network. The intelligent agent receives the state characteristics of the disassembly line balance environment and selects actions based on the action strategy and objective function. The disassembly line balance environment returns rewards and updates network parameters. In each round of environmental interaction, the preference vector w is sampled, and the deep Q network takes the concatenation of the state vector s and the preference vector ω as input. According to the decomposition of the priority relationship constraint matrix, it outputs the multi-objective Q value vector of each action. The final action selection is selected by combining the ε-greedy strategy with the NSGA-II algorithm to select the action strategy.

6. The method according to claim 5, characterized in that The architecture of the deep Q network includes an input layer, a hidden layer and an output layer. The input layer concatenates the state vector s and the preference vector ω to form a mixed input with the dimensions of the state dimension plus the preference vector dimension, that is, the state dimension plus the target dimension; the hidden layer is a three-layer fully connected neural network, using the ReLU activation function and adding Dropout to prevent overfitting; the output layer outputs the multi-target Q vector corresponding to each action for subsequent action selection.

7. A disassembly line balance optimization system based on multi-objective reinforcement learning, characterized by: The optimization system comprises: The acquisition unit is used to obtain input data and preprocess it: it obtains various types of data during the execution of the disassembly task and preprocesses them, records the disassembly priority relationship constraints, constructs the disassembly priority relationship constraint matrix, obtains the hazard attributes and demand attributes of the disassembly task, and obtains the disassembly time, disassembly cost, and disassembly energy consumption of different disassembly tasks on manual workstations and robot workstations respectively; A construction unit is used to construct a state space, an action space and a reward function, and deploy a training environment: the state space includes a plurality of state features, and the state features include at least one of the following states: workstation load state, robot load state, task allocation state, predecessor task completion state, successor task inspiration state, task-station fitting state and cycle offset state; the action space includes a plurality of action strategies, and the action strategies include at least one of the following strategies: longest execution time priority strategy, shortest execution time priority strategy, least predecessor task number priority strategy, most successor task number priority strategy, cycle time matching optimal strategy and random strategy; the objective function is designed based on the three objectives of the number of workstations, disassembly cost and disassembly energy consumption, and a multi-objective reinforcement learning model is constructed based on the MO-DQN algorithm; The objective function is expressed as: R = -(w1f1+w2f2+w3f3); Among them, w1, w2, and w3 are the weight coefficients of each goal, reflecting the relative importance of each goal in decision-making and achieving collaborative optimization among different goals. f1 represents the number of workstations, f2 represents the disassembly cost, and f3 represents the disassembly energy consumption. A training unit is used to train a multi-objective reinforcement learning model: the multi-objective reinforcement learning model dynamically adjusts the objective weights by introducing a preference vector through the MO-DQN algorithm, calculates the objective function based on the disassembly data obtained in S1, optimizes multiple objectives in S2, and selects action strategies by combining the ε-greedy strategy and the NSGA-II algorithm, ultimately converging to an optimal model that can balance multiple objectives; During training, the MO-DQN algorithm combines multiple target values ​​into a vector-based Q-value function. It uses different preference weights as conditional inputs during training to learn a generalized mapping from preferences to Q-values. The NSGA-II algorithm performs non-dominated sorting on the multi-target Q-value vectors corresponding to actions and calculates the crowding distance to determine the optimal action. The optimization unit is used for task scheduling and resource allocation optimization: the optimal strategy generated by the trained multi-objective reinforcement learning model is applied to task allocation in the actual disassembly line.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 6.

Images