A mixed flow assembly line collaborative optimization method based on deep reinforcement learning

Abstract

The application discloses a mixed flow assembly line collaborative optimization method based on deep reinforcement learning, three key problems are systematically and collaboratively optimized by constructing a double-layer multi-following optimization model of upper line balancing and lower buffer configuration and production sequencing; the upper layer optimizes the line balancing scheme by using a genetic algorithm, the lower layer optimizes the buffer configuration by using knowledge-enhanced deep reinforcement learning, and the production sequencing is solved in combination with the genetic algorithm. The method integrates a discrete event simulation system to realize high-fidelity performance evaluation, and finally outputs a high-quality and robust joint optimization scheme.

Description

Technical Field

[0001] This application belongs to the field of intelligent manufacturing and optimization control technology, and specifically relates to a collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning. Background Technology

[0002] In discrete manufacturing industries such as automobiles and electronics, mixed-flow assembly lines are widely used because they can flexibly assemble multiple models of products on the same production line. However, especially for mixed-flow assembly lines that are prone to random disturbances in actual operation and often use parallel workstations to increase capacity, they face the following three interrelated core challenges in actual operation, which seriously restrict their production efficiency and stability: (1) Bottleneck effect caused by uneven workstation load: Due to the differences in the task time of different products, and the greater difficulty in load balancing in complex layouts containing parallel workstations, the use of a fixed task allocation scheme (line balancing) will lead to uneven loads at each workstation. Some workstations are heavily loaded, becoming a bottleneck that limits the output of the entire line, while other workstations have a lot of idle time, resulting in a hidden waste of capacity. (2) Propagation interruption caused by frequent random disturbances: Random events such as equipment micro-faults, material supply delays, and operational fluctuations in the production process are difficult to avoid. These local disturbances will propagate between workstations and be amplified in the parallel workstation structure, causing "blockage" in upstream workstations and "starvation" in downstream workstations, which seriously damages the stability and availability of the production line and results in actual output being lower than the theoretical value. (3) Workstation overload risk caused by instantaneous load fluctuations; different products are mixed into production, causing the actual load of each workstation to change dynamically over time. When multiple products containing long-duration tasks of the workstation are continuously put into production, the workstation may face the risk of overload in the short term, which may not only affect the quality of the current product, but also disrupt the production rhythm and reduce the capacity.

[0003] Currently, existing technologies typically employ a "divide and conquer" strategy, breaking down the optimization problem of mixed-flow assembly lines into three independent sub-problems for research and solution: (1) Mixed-flow assembly line balancing problem (MALBP): aiming to alleviate bottlenecks by rationally allocating assembly tasks to make the operation time of each station as close as possible. (2) Buffer configuration problem (BAP): aiming to determine the location and capacity of buffers to absorb random disturbances and block the propagation of congestion and starvation. (3) Production sequencing problem (SP): aiming to adjust the product input sequence and smooth the dynamic load fluctuations of each station. However, there is a profound and complex hierarchical coupling relationship between line balancing, buffer configuration, and production sequencing. This relationship is more complex and critical in an environment where random disturbances and parallel workstations coexist: the line balancing scheme determines the load distribution and bottleneck location of the stations, which directly affects the demand for buffer configuration. (For example, a larger buffer capacity may be needed before and after the bottleneck station to alleviate congestion). Buffer configuration maintains system stability by absorbing disturbances, and its effect directly affects the actual efficiency of the given line balancing scheme, while also restricting the flexibility of production sequencing. (For example, small buffers limit the degrees of freedom in sorting.) Production sorting, by dynamically changing the workstation load, may disrupt the static equilibrium established by the line balancing scheme, inducing new instantaneous bottlenecks. This challenges the robustness of the line balancing scheme and also tests the effectiveness of the buffer configuration in real time.

[0004] The specific shortcomings of the existing technology are analyzed as follows: (1) Fragmented optimization model, ignoring system coupling: Most existing research and practice use step-by-step or independent optimization to deal with MALBP, BAP and SP. This method artificially severs the organic connection between the three sub-problems, and in particular fails to cope with the more dynamic and complex interaction between the three after the introduction of random perturbations and parallel workstations, resulting in optimization results that are only local optima, not global optima. For example, an "optimal" line balancing scheme that does not consider the impact of buffers may perform poorly after adding buffers; an "optimal" sorting scheme that does not consider line balancing may cause serious bottlenecks under certain task allocations.

[0005] (2) The problem modeling is too simplistic and fails to reflect hierarchical relationships: Even when studies attempt joint optimization, they usually model all three at the same decision level (single-level optimization), which is inconsistent with the hierarchical decision-making logic in real-world problems, where "line balance determines the system structure, and buffers and sorting are optimized based on this." Furthermore, existing modeling methods are generally based on deterministic models and the assumption of serial workstations, failing to accurately characterize the dynamic effects of random disturbances and the structural characteristics of parallel workstations. This imprecise modeling limits the quality of the solution and its practical guiding significance.

[0006] (3) Inefficient solution methods and high computational resource consumption: Due to the complexity of joint optimization problems, their solutions usually require huge computational resources. When the problem size is expanded by parallel workstations and a large number of simulations are needed to evaluate the performance under random perturbations, existing algorithms either take too long to solve and cannot meet the actual decision-making needs, or they have to terminate prematurely due to computational resource limitations, and can only obtain suboptimal solutions.

[0007] (4) Lack of prior guidance and blind search process: When solving complex subproblems such as buffer configuration, traditional optimization algorithms (such as genetic algorithms) need to perform a lot of random search and simulation evaluation, which is inefficient. Blind search is inefficient, especially in the early stages of training or searching reinforcement learning algorithms. The lack of effective prior knowledge to guide the search direction leads to a lot of computational resources being wasted on ineffective exploration.

[0008] Existing technologies, which optimize the three sub-problems separately, fail to fully consider and coordinate the inherent coupling relationships among them, especially lacking effective integration and optimization of the key element of parallel workstations under random perturbation environments. This independent optimization approach is prone to getting trapped in local optima, failing to obtain a globally optimal joint solution (line balancing, buffer configuration, and sorting), thus limiting further performance improvements in mixed-flow assembly lines in real industrial environments. Furthermore, although a few studies have attempted to model the three aspects in a unified manner, they mostly treat it as a single-layer optimization problem, failing to accurately characterize its leader-follower hierarchical structure, and the solution methods consume huge computational resources, making them difficult to apply to large-scale practical problems. Summary of the Invention

[0009] To address the technical problems of existing technologies, and specifically for mixed-flow assembly lines with random perturbations and parallel workstations, this invention provides a collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning. By constructing a bilevel multi-follower optimization (BLMF) model involving upper-level line balancing and lower-level buffer configuration and production sequencing, it achieves systematic collaborative optimization of three key issues. This method can systematically and simultaneously handle line balancing, buffer configuration, and production sequencing problems, fully coordinating their inherent hierarchical coupling relationships, and obtaining a high-quality, implementable global optimization solution within an acceptable computational cost. The upper layer utilizes a genetic algorithm to optimize the line balancing scheme, while the lower layer utilizes knowledge-enhanced deep reinforcement learning to optimize the buffer configuration, and combines this with a genetic algorithm to solve for production sequencing. This method integrates a discrete event simulation system (ALS) to achieve high-fidelity performance evaluation, ultimately outputting a high-quality, robust joint optimization solution; thereby significantly improving the production efficiency, stability, and economy of this type of mixed-flow assembly line in a real-world operating environment.

[0010] The technical solution adopted in this application is as follows: This application proposes a collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning, comprising the following steps: S101. Obtain basic information about the mixed-flow assembly line to establish an assembly line model that includes task time, priority relationship, parallel workstation structure, cycle time constraints and disturbance characteristics. At the same time, based on historical production data and / or domain knowledge, define and quantify the random disturbance characteristics of the system. S102. Establish a mixed-flow assembly line problem model for inputting into a deep reinforcement learning-based optimization algorithm; the problem model is an optimization model including a hierarchical structure, decision variables, objective function, and constraints for optimizing line balancing, buffer configuration, and production scheduling; the deep reinforcement learning-based optimization algorithm includes an upper-level solver, a first lower-level solver, and a second lower-level solver, used to collaboratively optimize the line balancing, buffer configuration, and production scheduling problems; S103. Construct a two-layer multi-follower optimization model, with the upper layer being a line balancing optimization model and the lower layer being a buffer configuration model and a production sorting optimization model. S104. Design a hybrid solution algorithm based on deep reinforcement learning and genetic algorithm, construct an optimization algorithm based on a two-layer multi-follower model of deep reinforcement learning, optimize the upper layer problem through the line balance optimization model, optimize the lower layer problem through the buffer configuration model and the production sorting optimization model, and obtain a preliminary optimization scheme. S105. Calculate the objective function value of the overall optimization scheme using a discrete event-based simulation system, and perform performance evaluation and feedback optimization on the preliminary optimization scheme. S106. Repeat the evolution and learning process multiple times until the optimal solution for output line balancing, buffer configuration, and production sorting is found.

[0011] The upper-layer solver uses a genetic algorithm to perform a global search for line balancing schemes. The first lower-layer solver uses a deep reinforcement learning network based on a knowledge enhancement strategy to train the agent and quickly decide on the buffer configuration scheme. The second lower-layer solver uses a genetic algorithm to optimize the production ranking scheme. Specifically, after the upper-layer solver's GA generates the preliminary optimization scheme for line balancing, it calls the DDQN agent of the lower-layer solver and the GA of the lower-layer solver to solve the corresponding preliminary optimization schemes for buffer configuration and production ranking.

[0012] The two-layer multi-follower optimization model is used to accurately describe the hierarchical coupling relationship between the problems, as follows: The upper-level model is used for balancing optimization of mixed-flow assembly lines. The upper-level problem is to determine the structural design of the production line from a strategic perspective, and to determine the number of work centers, the number of workstations in each work center, and the assigned tasks in the assembly line, i.e., the line balancing scheme, under the premise of satisfying task priority constraints. The optimization objective is to minimize the total cost while meeting the specified average throughput. Its mathematical model is as follows: Objective function: (1), (2), Constraints: (3), (4), (5), (6), Equation (1) is the objective function, where DC is the annual design production cost of the production line, NDC is the annual actual total production cost of the production line, and ct is the production cycle time of the given production line. The average cycle time is obtained from the ALS simulation software. As a higher-level penalty factor; Equation (2) represents the total design and production cost. Let be the number of workstations in work center k, and AC be the sum of the annual costs for a single assembly workstation and a single operator. Let k be the number of tasks in work center k, and CE be the annual cost of a single device. The buffer capacity between assembly center k-1 and assembly center k, where CB is the annual cost per unit buffer; when the scheme satisfies Only considering design cost DC; when The additional cost of efficiency loss is reflected through a secondary penalty term, and the penalty intensity increases significantly with the increase of the overtime ratio. Constraint (3) represents the task assignment integrity constraint. This indicates that if task n is assigned to work center k, the value is 1; otherwise, it is 0. This ensures that each task must be assigned to one and only one work center, avoiding task omissions or duplicate assignments. Constraint (4) represents the work center time constraint. This represents the weighted average processing time of task n, ensuring that the theoretical workload of each work center does not exceed the production cycle time. It is a static balance constraint based on the average time. Constraint (5) represents the task priority relationship constraint, k represents the work center index, index k=1,…,K, ensuring that for any task pair (i,h) with priority relationship, where i is the predecessor of h, task i must be assigned before h or to the same work center, thus guaranteeing the assembly process logic; Constraint (6) represents the constraint on the decision variable, defined as follows: Variables are 0-1; The buffer configuration model is used for buffer configuration optimization. The first lower-level problem is to optimize the buffer resource configuration from a tactical level based on a given line balancing scheme. The buffer configuration affects the production line's ability to resist random disturbances. The optimization objective is to minimize the buffer configuration cost while ensuring that the production line achieves the specified average throughput. Objective function: (7), (8), Constraints: (9), Equation (7) is the objective function of the lower-level buffer configuration optimization problem, TDB represents the lower-level objective function, and buffer cost. This objective minimizes the buffer cost while ensuring production line efficiency. When the actual cycle time meets the requirements, only the buffer cost is considered. When the cycle time exceeds the limit, a penalty term is used to prompt the agent to find a buffer configuration that can meet the efficiency requirements and ensure a lower cost. is the penalty factor for the first lower-level solver, and its value is a large constant; Equation (8) is the total cost of the buffer, DB represents the total capacity of the production line buffer, and the total cost of the buffer is proportional to the total capacity of the buffer, assuming that the unit buffer cost is the same; constraint (9) is the upper and lower bound constraints of the buffer capacity. This indicates the buffer capacity between assembly center K-1 and assembly center K. =0 means no buffer, and =5 is based on actual space constraints to ensure that the configuration plan is feasible; The production scheduling optimization model is used for production scheduling optimization. The second lower-level problem is to optimize the production scheduling problem from the operational level based on the given line balance and buffer scheme. The scheduling problem involves determining the processing sequence of product models entering the assembly line. This decision is closely related to the production line balance, equipment utilization rate and overall capacity. The optimization objective is to minimize the production cycle time, so as to achieve capacity improvement and cost control with the most efficient resource allocation. And achieve increased production capacity and controllable costs through the most efficient allocation of resources; Objective function: (10) Constraints: (11), (12) Wherein, equation (10) is the objective function of the lower-level sorting optimization problem. The average cycle time obtained from the ALS simulation software is directly minimized to reflect the actual cycle time obtained through high-fidelity simulation, thus reflecting the true efficiency of the scheme under random perturbations. The objective is clear and direct; constraint (11) represents the sequence position allocation constraint. This indicates that if sequence p is assigned to product m, then it is 1; otherwise, it is 0, ensuring that each position in the MPS must be assigned a specific product model; constraint (12) indicates that the product demand satisfies the constraint. This represents the production quantity of model m in the MPS, i.e., the production proportion of m. It ensures that the production quantity of each product model strictly conforms to the MPS requirements, meaning each model m will occupy a certain position in the sequence. One position.

[0013] The optimization algorithm of the deep reinforcement learning-based two-layer multi-follower model includes three core solvers: Upper-Level-GA, Lower-Level-DDQN, and Lower-Level-GA. These solvers are used to collaboratively optimize line balancing, buffer configuration, and production sequencing problems in mixed-flow assembly lines. The implementation process includes: S201. Input and Initialization The system reads the input data of the mixed-flow assembly line, including assembly task data, cycle time constraints, parallel station structure, buffer parameters and product mixing ratio, establishes an optimization model, and initializes the population of the upper-layer genetic algorithm, the lower-layer deep reinforcement learning agent and the discrete event simulation system. S202, Upper-layer line balancing optimization The upper-level optimization module uses a genetic algorithm to generate line balancing schemes. The specific process is as follows: S2021. Initialize the Upper-Level-GA algorithm parameters and randomly generate initial solutions (i.e., initial population), each solution representing a candidate line balance scheme.

[0014] S2022. For each individual in the population, perform selection, crossover, and mutation operations to generate different task assignment structures.

[0015] S2023. Before evaluating the fitness of each individual, it is necessary to call the two lower-level modules (buffer configuration optimization and production sorting optimization) to obtain the corresponding optimal lower-level solution.

[0016] S203, Lower-level buffer configuration optimization The first lower-level module uses a knowledge-enhanced deep double-Q network (DDQN) algorithm to optimize the buffer configuration: The intelligent agent takes workstation load, location, and capacity as state inputs; The action output is to adjust the buffer capacity. Construct a reward function by improving cycle time and cost constraints; Fast convergence is achieved by introducing regularization constraints based on the "inverted bowl" prior law, and an optimized buffer capacity distribution scheme is output.

[0017] S204, Optimization of Lower-Level Production Sorting The second lower-level module uses a genetic algorithm to optimize the production sorting: Based on the line balancing scheme provided by the upper layer and the buffer configuration scheme output by the first module of the lower layer, multiple product production sequences are generated; The cycle time and stability index of each sorting scheme are calculated using a discrete event simulation system. S205, Simulation Evaluation and Feedback The ALS simulation evaluation module performs discrete event simulations on a joint scheme consisting of a line balancing scheme, a buffer configuration scheme, and a production sequencing scheme. The following performance indicators were simulated and calculated: average cycle time, station idle rate, production line throughput, and balance rate. The performance metrics obtained from the simulation are used as feedback signals and transmitted to the upper-layer genetic algorithm and the lower-layer reinforcement learning module, respectively, for calculating the fitness value of the upper-layer algorithm and the DDQN agent of the lower-layer algorithm. Strategy update; S206, Evolutionary Iteration and Output The system continuously improves the quality of the solution through multiple rounds of evolutionary iteration: S2061. Use the initial population to perform evolutionary operations (selection, crossover, mutation) to generate offspring solutions.

[0018] S2062. For each newly generated child solution, call Lower-Level-DDQN and Lower-Level-GA again to obtain its corresponding lower-level optimal solution and calculate its upper-level objective function value (fitness).

[0019] S2063. Through the environment selection operation, select individuals from the child and parent solutions whose fitness reaches the convergence threshold or the maximum number of iterations to form the next generation population.

[0020] S2064. Repeat the above iterative steps until the stopping condition of the upper-level genetic algorithm is met. The algorithm finally outputs the joint optimal solution, including the optimal line balance solution, the optimal buffer configuration solution, and the optimal production sorting solution.

[0021] The Upper-Level-GA solver is used to optimize the upper-level line balance problem. This solver employs an improved genetic algorithm, globally searching for line balance schemes through population evolution. It maps chromosomes to feasible line balance schemes using a specific encoding / decoding strategy. The solution process includes: S301, System Initialization and Encoding; The system encodes assembly tasks and their priority constraints, with each individual representing a candidate task allocation scheme or workstation partitioning structure. The algorithm generates a diverse set of line balancing schemes by randomly generating an initial population. S302, Individual Decoding and Fitness Assessment; For each individual task, the task time and cycle time constraints for each workstation are calculated, and the performance is evaluated by calling the lower-level modules (including buffer configuration and production scheduling optimization) and the Discrete Event Simulation System (ALS). S3021. Call the lower-level dual-depth Q-network solver, input the current line balancing scheme, and output the optimal buffer configuration scheme; S3022: Call the lower-level genetic algorithm solver, input the current line balance scheme, and output the optimal production sorting scheme; S3023. Call the discrete event simulation system to perform performance evaluation and obtain indicators such as balance rate, idle rate and number of workstations; S3024. Calculate the fitness value of the upper layer: Based on the evaluation results, with the optimization objectives of minimizing the number of workstations and cycle time and maximizing the balance rate, calculate the fitness value corresponding to the line balancing scheme. S303, Evolutionary Iterative Optimization; Repeat the following evolutionary process until the fitness continues to improve or the maximum number of iterations is reached: S3031. Selection Operation: Use tournament selection method to select high-quality individuals as parents; S3032, Crossover operation: The selected parent individuals are subjected to an order crossover operation to exchange some chromosome segments to generate new offspring solutions; S3033, Mutation Operation: Perform a task exchange mutation operation on offspring individuals with a preset probability, randomly swapping some gene positions to maintain population diversity and explore better solutions; S3034, Offspring Fitness Assessment: After the newly generated individuals are decoded and simulated on the platform (i.e., steps S3021-3024 are re-executed), their fitness values ​​are calculated. S3035, Environmental Selection: Select individuals with higher fitness values ​​from the new offspring and the old population to form the next generation of the population; S304, Output the results.

[0022] The Upper-Level-GA uses specific encoding and decoding: Encoding scheme: The upper-level solution is represented as a chromosome, i.e., a task sequence. ,in This indicates the task number, and the sequence order reflects the priority of task assignment; Decoding process: Input: Task sequence Priority relationships between tasks, assembly line cycle time (CT) Step 1: According to the Maximum Weighted Value (LWV) rule, the solution... Convert to a candidate set of task sequences ; Step 2: Initialize the first work center k=1; Step 3: According to The tasks are sorted and assigned to the current work center one by one. If the total working time of the current work center plus the working time of the next task to be assigned exceeds the cycle time ct, task assignment to that work center is stopped. Step 4: Remove the assigned task from the task sequence. If there are unassigned tasks, create a new work center k=k+1 and return to step 3; otherwise, proceed to step 5. Step 5: Calculate the sum of idle times of adjacent work centers. If the sum of idle times of adjacent work centers is greater than the cycle time ct, then merge the two work centers into one, and reduce the number of workstations accordingly. Step 6: Complete the decoding operation and output a feasible line balancing scheme AL (set of work centers, task allocation for each work center, and total number of workstations).

[0023] The Lower-Level-DDQN solver is used to optimize the buffer configuration of the first lower-level module. This solver trains the agent based on a two-layer deep Q-network, accelerates convergence through experience replay and stable training of the target network, and incorporates knowledge enhancement strategies. This enables it to quickly generate the optimal buffer capacity configuration scheme based on a given line balancing scheme. The optimization process includes: S601, Start: Initiate the buffer configuration optimization process; S602, Input line balancing scheme AL: Receives the line balancing scheme transmitted from the upper layer as input; S603. Calculate the number of assembly centers k and their load based on LB: Based on the input line balancing scheme, calculate the number of assembly centers k and the load of each assembly center. S604. Determine if all buffer configurations are complete: Check if all buffer configurations are complete: If not, proceed to step S605; if yes, proceed to step S208. S605. Determine the current buffer position: Determine the specific position of the buffer that needs to be configured in the production line; S606, Agent determines the capacity (Bi) of the i-th buffer based on the state: The agent decides the current buffer capacity based on the state space information through the DDQN network; S607. Update buffer configuration x; update the currently determined buffer capacity to the configuration scheme, and return to step S604 to continue configuring the next buffer; S608. Determine if the termination condition is met: Verify whether the current buffer configuration scheme meets the algorithm termination condition; S609. Convert the configuration result into an assembly line design scheme T: Convert the optimized buffer configuration result into an assembly line design scheme. S610, Output Assembly Line Design Scheme T and Buffer Configuration Scheme x: Output the final assembly line design scheme and the corresponding buffer configuration scheme; S611, End: The buffer configuration optimization process is complete.

[0024] The first lower-level module buffer configuration optimization process is implemented by a solver based on a deep dual-Q network (DDQN), and its operation includes: Step 1, State Modeling: The system establishes a reinforcement learning environment based on the upper-level line balance results, and constructs a state vector from the relevant features of each buffer. The DDQN state space is designed with 6-dimensional state features: Previous station load: The load of the station preceding the current buffer position. ; Next station load: The load of the station following the current buffer position; Absolute position: The absolute position of the current buffer within the entire production line; Relative position: The relative position of the current buffer within the entire production line; Peak position: Assuming the buffer zone before the peak load station is located in the relative position of the entire production line; Current capacity: The size of the currently allocated buffer; Step 2, Action Selection: The agent selects different capacity levels from the discrete action space for adjustment, using buffer capacity adjustment as the action. Step 3, Environment Interaction and Reward Calculation: For each action executed, the system calls ALS to evaluate the new production takt time, idle rate and cost changes, and constructs a reward function based on the takt time improvement and buffer cost; The reward function is calculated according to equation (13), where a decrease in pace yields a positive reward, while an increase in cost incurs a negative penalty: (13), In equation (13): β is a coefficient used to adjust the merits of different buffer configuration schemes; β is a coefficient used to adjust the merits of different buffer configuration schemes; β=1, if cts ≤ ct, the configuration is valid; otherwise, β=0. When β = 1, α = 1; otherwise, α = -1. Step 4: Knowledge Enhancement and Network Updates In the early stages of training, the agent is strongly guided to learn domain knowledge, while in the later stages it gradually relies on data-driven learning to balance prior knowledge and experience learning. Knowledge enhancement is introduced, and the distribution pattern of the "inverted bowl" buffer is embedded into the loss function in the form of regularization constraints to guide the agent to learn a reasonable capacity distribution trend and avoid blind exploration; its calculation method is shown in Equation 14: (14), in, λ is the standard DDQN loss function, which includes Q learning loss and regularization term; λ is the decaying weight coefficient, whose value decays as training rounds progress; L is the fitting function for embedded knowledge, which measures the difference between the current policy and the "inverted bowl" distribution, and its value is the regularization loss of the "inverted bowl" pattern and the optimal action of the current agent state. Step 5, Policy Convergence and Output: Through multiple rounds of training, the agent gradually learns the optimal capacity decision-making strategy under different production states, so as to minimize the overall cycle time and optimize the production smoothness. The system outputs the optimal capacity configuration scheme for each buffer and passes the simulation feedback results to the upper-level genetic algorithm for fitness updates in global optimization, realizing a closed loop of collaborative optimization between buffer configuration and line balancing.

[0025] The Lower-Level-GA solver is used to solve the production sequencing problem. This solver employs a genetic algorithm to optimize the product production sequence and evaluates the quality of the solution through ALS simulation. The production sequencing optimization process includes: S701, Start: Initiate the production sequencing optimization process; S702, Input Assembly Line Design Scheme T: Receives the upper-level line balancing scheme and the lower-level buffer configuration result as input; S703. Generate an initial population: Randomly generate an initial random population of size NL, where each individual corresponds to a specific production sequence, satisfying product ratio and production batch constraints. S704. Determine if all individuals have completed the assessment: Check if all individuals in the population have completed the assessment: If not, continue the assessment process; if yes, proceed to step S705. S705, Environmental Selection: Based on fitness results, select high-quality individuals to enter the next generation; S706. Determine if the stopping condition is met: Verify if the algorithm termination condition is met: If not, proceed to step S707; if yes, jump to step S710. S707, Genetic Operations: Perform selection, crossover, and mutation operations, including: Selection strategy: Prioritize retaining high-quality individuals using roulette or tournament methods; Crossover operations: Employ partial mapping crossover or sequential crossover to preserve product order characteristics; Mutation operations: breaking local patterns through swapping or insertion operations to maintain population diversity; S708. Generating offspring population: Creating a new offspring population through genetic operations; S709. Evaluate the offspring population: For each newly generated individual, call the discrete event simulation system to perform performance evaluation and calculate the corresponding average cycle time, production line volatility, blockage rate, and product changeover smoothness, etc. S710. Call the discrete event simulation system to simulate and evaluate the optimal production sequencing scheme and its simulation time: perform simulation evaluation on the final optimal scheme and obtain its performance indicators. S711, Output the optimal production sequencing scheme and its simulation time: Output the optimal product production sequence scheme and the corresponding cycle time and capacity performance; S712, End: Complete the production sorting optimization process.

[0026] The basic information of the mixed-flow assembly line includes: all assembly tasks and their processing time under different product models, priority relationship between tasks, possibility of parallel configuration of workstations, production cycle time, and product demand ratio; the random disturbance characteristics refer to uncertain events that cause the actual operation time to deviate from the standard value or cause production interruption, and their types include equipment failure, material supply delay, and fluctuation in operator efficiency.

[0027] The beneficial effects of this invention are as follows: Compared with existing technologies, this application has the following advantages: By constructing a two-layer multi-follower optimization model consisting of upper-layer line balancing and lower-layer buffer configuration and production sequencing, it achieves systematic collaborative optimization of three key issues. The upper layer utilizes a genetic algorithm to optimize the line balancing scheme, while the lower layer utilizes knowledge-enhanced deep reinforcement learning to optimize the buffer configuration, and combines this with a genetic algorithm to solve the production sequencing. This method integrates a discrete event simulation system to achieve high-fidelity performance evaluation, ultimately outputting a high-quality, robust joint optimization scheme, thereby significantly improving the production efficiency, stability, and economy of this type of mixed-flow assembly line in a real operating environment. Attached Figure Description

[0028] Figure 1 is a flowchart illustrating a collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning provided in an embodiment of this application. Figure 2 is a schematic diagram of the RL-BLMF algorithm provided in an embodiment of this application; Figure 3 is a schematic diagram of the Upper-Level-GA solver provided in the embodiments of this application for solving the upper-level line balance optimization problem; Figure 4 is an example diagram of the decoding priority relationship provided in the embodiments of this application; Figure 5 is an example diagram of the post-decoding line balancing scheme provided in the embodiments of this application; Figure 6 is a schematic diagram of the optimization process of the Lower-Level-DDQN solver provided in the embodiment of this application in the first lower-level buffer configuration;

[0029] Figure 7 is a flowchart illustrating the process of solving the Lower-Level-GA solver in the second lower-level production sequencing process provided in the embodiments of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0031] The following section, with reference to Figures 1-7, details a collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning provided in the embodiments of this application.

[0032] First, referring to Figure 1, the collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning in this application embodiment will be described in detail. As shown in Figure 1, the collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning provided in this application embodiment includes the following steps: S101. Obtain basic information about the mixed-flow assembly line to establish an assembly line problem model that includes task time, priority relationship, parallel workstation structure, cycle time constraints and disturbance characteristics. Simultaneously, based on historical production data or domain knowledge, the random disturbance characteristics of the system are defined and quantified; the random disturbance characteristics refer to uncertain events that cause actual working time to deviate from the standard value or trigger production interruptions, and their types include equipment failure, material supply delays, and fluctuations in operator efficiency.

[0033] The basic information of the mixed-flow assembly line includes, but is not limited to, all assembly tasks and their processing time under different product models, priority relationships between tasks, the possibility of parallel workstation configuration, production cycle time, and product demand ratio. The basic information of the mixed-flow assembly line is obtained to establish an assembly line problem model that includes task time, priority relationships, parallel workstation structure, cycle time constraints, and disturbance characteristics.

[0034] S102. Establish a mixed-flow assembly line problem model for input into a deep reinforcement learning-based optimization algorithm. The problem model is a mathematical optimization model that includes a hierarchical structure, decision variables, objective function, and constraints for optimizing the line balancing, buffer configuration, and production scheduling of the mixed-flow assembly line. The deep reinforcement learning-based optimization algorithm includes an upper-level solver, a first lower-level solver, and a second lower-level solver, used to collaboratively optimize the line balancing, buffer configuration, and production scheduling of the mixed-flow assembly line. S103. Construct a two-layer multi-follower optimization model, with the upper layer being a line balancing optimization model and the lower layer being a buffer configuration and production sorting optimization model.

[0035] Understandably, constructing a two-layer multi-follower optimization model decomposes the complex collaborative optimization problem into one upper-level problem and two lower-level problems. Specifically, the upper-level problem is mixed-flow assembly line balancing optimization. The decision variable is the task allocation scheme to work centers. The objective is to minimize the total design production cost, consisting of workstation cost, equipment cost, and buffer base cost, while satisfying production cycle time and task priority, and considering the penalty cost due to actual cycle timeouts. The first lower-level problem is buffer configuration optimization. Based on the upper-level line balancing scheme, the capacity of the buffers between work centers is determined. The objective is to minimize the total buffer cost while satisfying production cycle time, also considering cycle timeout penalties.

[0036] The second lower-level problem is production sequencing optimization. Given the upper-level line balancing scheme and the lower-level buffer configuration scheme, the production sequence of product models is determined, with the goal of minimizing the actual production cycle time obtained through high-fidelity simulation.

[0037] S104. Design a hybrid solution algorithm based on deep reinforcement learning and genetic algorithm, construct an optimization algorithm based on a two-layer multi-follower model of deep reinforcement learning, optimize the upper layer problem and the lower layer problem respectively, and obtain a preliminary optimization scheme.

[0038] The upper-layer solver uses a genetic algorithm (GA) to perform a global search for line balancing schemes, while the first lower-layer solver uses a deep reinforcement learning network based on a knowledge enhancement strategy. (DDQN) trains the agent, quickly decides the buffer configuration scheme, and the second lower-level solver uses a genetic algorithm (GA) to optimize the production sorting scheme.

[0039] Specifically, after the upper-level solver generates the initial optimization scheme for line balancing in the GA generation, it calls the initial optimization schemes for the buffer configuration and production sorting of the lower-level solver's DDQN agent and the GA solution corresponding to the lower-level solver.

[0040] S105. Calculate the objective function value of the overall optimization scheme using a discrete event-based simulation system (ALS), and perform performance evaluation and feedback optimization on the preliminary optimization scheme.

[0041] The core of the performance evaluation is the integration of a discrete event-based assembly line simulator to accurately evaluate the actual performance of the initial optimization scheme under random disturbances, ensuring the robustness and practicality of the optimization results.

[0042] S106. Repeat the evolution and learning process multiple times until a high-quality joint optimal solution for line balancing, buffer configuration, and production sorting is output.

[0043] Furthermore, the hybrid solution algorithm based on deep reinforcement learning and genetic algorithm in this application embodiment uses a genetic algorithm for global search in the upper-layer line balance optimization, with the goal of minimizing the total production cost, including workstation cost, equipment cost and buffer cost; the lower-layer buffer configuration is solved using a knowledge-enhanced deep double Q network (DDQN) algorithm, and the reward function combines production cycle improvement and buffer cost.

[0044] Specifically, this embodiment constructs a two-layer multi-follower optimization mathematical model to accurately describe the hierarchical coupling relationship between the three sub-problems. The specific model assumptions are as follows, and the parameters and variable symbols used are explained below: Parameter description: N: Number of tasks (index n = 1, …, N); M: Product quantity (index m = 1, …, M); k: Work center index (index k = 1, …, K); : The processing time required for task n in product m; The weighted average processing time of task n. = Σ = 1 smtmn / NP; ct: The production cycle time of the given production line; : Average cycle time obtained from ALS simulation software; : Decision variable, which is 1 if task n is assigned to work center k, and 0 otherwise; : The buffer capacity between assembly center k-1 and assembly center k; : Number of workstations in work center k; The number of tasks in work center k. ; P: Product production sequence, i.e., the smallest integer set MPS (index p = 1, …, NP) obtained according to the production demand ratio of different models; The production quantity of model m in MPS, i.e., the production proportion of m; If sequence p is assigned to product m, then the value is 1; otherwise, it is 0. AC: The sum of the annual costs for a single assembly workstation and a single operator; CE: Annual cost per unit; CB: Annual cost per unit buffer zone; DC: Annual design and production cost of the production line; NDC: The upper-level objective function, representing the total actual annual production cost of the production line; DB: Total capacity of the production line buffer; TDB: Lower-level objective function, buffer cost; θ: Penalty factor (upper level is...) The lower layer is ); T: Assembly line design scheme, including line balancing and buffer zone configuration scheme; NU: Population size of the upper-level algorithm; NL: Population size of the lower-level algorithm; The solution of the upper-level optimization algorithm; The solution of the lower-level optimization algorithm; AL: Assembly line layout obtained by decoding the upper-level algorithm, i.e., line balancing scheme; FE: Number of fitness evaluations for the algorithm; FEs: Total number of evaluations. Upper-level model: Mixed-flow assembly line balancing optimization (MALBP).

[0045] The upper-level problem (i.e., the line balancing problem) determines the structural design of the production line at the strategic level. Under the premise of satisfying task priority constraints, it determines the number of work centers in the assembly line, the number of workstations in each work center, and the assigned tasks, i.e., the line balancing scheme. The optimization objective is to minimize the total cost while meeting a specified average throughput. Its mathematical model is as follows: Objective function: (1), (2), Constraints: (3), (4), (5), (6), Equation (1) is the objective function, aiming to minimize the total actual production cost. Equation (2) is the total designed production cost, which includes workstation maintenance and labor costs, equipment costs, and buffer costs. When the scheme satisfies... We only consider the design cost DC. It's worth noting that when the actual production cycle time exceeds the given average cycle time, production efficiency decreases, increasing actual costs. Therefore, when the actual production cycle time (simulation time) cts > ct, a secondary penalty term reflects the additional cost caused by efficiency loss, with the penalty intensity significantly increasing as the overtime ratio increases.

[0046] Constraint (3) represents the task allocation integrity constraint, ensuring that each task must be assigned and can only be assigned to one work center, avoiding task omissions or duplicate assignments. Constraint (4) represents the work center time constraint, ensuring that the theoretical workload of each work center does not exceed the production cycle time, which is a static balance constraint based on average time. Constraint (5) represents the task priority relationship constraint, ensuring that for any task pair (i, h) with priority relationship (where i is the predecessor of h), task i must be assigned before h or to the same work center, guaranteeing the assembly process logic. Constraint (6) represents the decision variable constraint, defined as follows: It is a 0-1 variable.

[0047] First lower-level model: Buffer Configuration Optimization (BAP) The first lower-level problem (i.e., the buffer configuration problem) optimizes the allocation of buffer resources at the tactical level based on a given production line balancing scheme. The buffer configuration affects the robustness of the production line (the production line's ability to resist random disturbances). The optimization objective is to minimize the buffer configuration cost while ensuring the production line reaches the specified average throughput, thus not only achieving the specified capacity but also controlling costs as much as possible.

[0048] Objective function: (7), (8), Constraints: (9), Equation (7) is the objective function of the lower-level buffer configuration optimization problem, where TDB represents the lower-level objective function and the buffer cost. This objective aims to minimize the buffer cost while ensuring production line efficiency. When the actual cycle time meets the requirements, only the buffer cost is considered. When the cycle time exceeds the limit, a penalty term is used to prompt the agent to find a buffer configuration that can meet the efficiency requirements and ensure a lower cost. is the penalty factor for the first lower-level solver, and its value is a large constant; Equation (8) is the total cost of the buffer, DB represents the total capacity of the production line buffer, and the total cost of the buffer is proportional to the total capacity of the buffer, assuming that the unit buffer cost is the same; constraint (9) is the upper and lower bound constraints of the buffer capacity. This indicates the buffer capacity between assembly center K-1 and assembly center K. =0 means no buffer, and =5 is set to ensure the configuration scheme is feasible based on actual space constraints.

[0049] Second lower-level model: Production sequencing optimization (SP); The second lower-level problem (i.e., the production sequencing problem) optimizes production scheduling at the operational level, based on a given production line balancing and buffer scheme. The sequencing problem involves determining the processing sequence of product models entering the assembly line; this decision is closely related to production line balance, equipment utilization, and overall capacity. The optimization objective is to minimize the production takt time, thereby achieving capacity improvement and cost control with the most efficient resource allocation.

[0050] Objective function: (10) Constraints: (11), (12) Equation (10) is the objective function of the lower-level sorting optimization problem. The average cycle time obtained from the ALS simulation software is directly minimized to reflect the actual cycle time obtained through high-fidelity simulation, thus reflecting the true efficiency of the scheme under random perturbations. The objective is clear and direct; constraint (11) represents the sequence position allocation constraint. This indicates that if sequence p is assigned to product m, then it is 1; otherwise, it is 0, ensuring that each position in the MPS must be assigned a specific product model; constraint (12) indicates that the product demand satisfies the constraint. This represents the production quantity of model m in the MPS, i.e., the production proportion of m. It ensures that the production quantity of each product model strictly conforms to the MPS requirements, meaning each model m will occupy a certain position in the sequence. One position.

[0051] The above, in conjunction with Figure 1, introduces a collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning provided by an embodiment of this application. The following section, in conjunction with... Figure 2-7 This paper describes the specific implementation process of the deep reinforcement learning-based two-layer multi-follower optimization (RL-BLMF) algorithm in the embodiments of this application.

[0052] Figure 2 is a schematic diagram of the RL-BLMF algorithm provided in an embodiment of this application. Figure 2As shown, it includes three core solvers: Upper-Level-GA (upper-level genetic algorithm), Lower-Level-DDQN (lower-level deep double-Q network), and Lower-Level-GA (lower-level genetic algorithm). This RL-BLMF algorithm aims to collaboratively optimize line balancing, buffer configuration, and production sequencing problems in mixed-flow assembly lines. Its overall implementation process comprises six main stages: input and initialization, upper-level line balancing optimization, lower-level buffer configuration optimization, lower-level production sequencing optimization, simulation evaluation and feedback, and evolutionary iteration output. A detailed description of the overall implementation process is as follows: S201, Input and Initialization; The system reads the input data of the mixed-flow assembly line, including assembly task data, cycle time constraints, parallel station structure, buffer parameters and product mixing ratio, establishes an optimization model, and initializes the population of the upper-layer genetic algorithm, the lower-layer deep reinforcement learning agent and the discrete event simulation system.

[0053] S202, Upper-layer line balance optimization; The upper-level optimization module uses a genetic algorithm to generate line balancing schemes. The specific process is as follows: S2021. Initialize the Upper-Level-GA algorithm parameters and randomly generate initial solutions (i.e., initial population), each solution representing a candidate line balance scheme.

[0054] S2022. For each individual in the population, perform selection, crossover, and mutation operations to generate different task assignment structures.

[0055] S2023. Before evaluating the fitness of each individual, it is necessary to call the two lower-level modules (buffer configuration optimization and production sorting optimization) to obtain the corresponding optimal lower-level solution.

[0056] S203, Optimization of lower-level buffer configuration; The first module at the lower layer uses a knowledge-enhanced deep double-Q network (DDQN) algorithm to optimize the buffer configuration: The intelligent agent takes workstation load, location, and capacity as state inputs; The action output is to adjust the buffer capacity. Construct a reward function by improving cycle time and cost constraints; Fast convergence is achieved by introducing regularization constraints based on the "inverted bowl" prior law, and an optimized buffer capacity distribution scheme is output.

[0057] S204, Optimization of lower-level production sequencing; The second module at the lower level uses a genetic algorithm to optimize the production order: Based on the line balancing scheme provided by the upper layer and the buffer configuration scheme output by the first module of the lower layer, multiple product production sequences are generated; The cycle time and stability index of each sorting scheme are calculated using a discrete event simulation system.

[0058] S205, Simulation Evaluation and Feedback; The ALS simulation evaluation module performs discrete event simulations on a joint scheme consisting of a line balancing scheme, a buffer configuration scheme, and a production sequencing scheme.

[0059] The following performance indicators were simulated and calculated: average cycle time, station idle rate, production line throughput, and balance rate. The performance metrics obtained from the simulation are used as feedback signals and transmitted to the upper-layer genetic algorithm and the lower-layer reinforcement learning module, respectively, for calculating the fitness value of the upper layer and updating the policy of the lower-layer DDQN agent.

[0060] S206, Evolutionary Iteration and Output; The system continuously improves the quality of the solution through multiple rounds of evolutionary iteration: S2061. Use the initial population to perform evolutionary operations (selection, crossover, mutation) to generate offspring solutions.

[0061] S2062. For each newly generated child solution, call Lower-Level-DDQN and Lower-Level-GA again to obtain its corresponding lower-level optimal solution and calculate its upper-level objective function value (fitness).

[0062] S2063. Through the environment selection operation, select individuals from the child and parent solutions whose fitness reaches the convergence threshold or the maximum number of iterations to form the next generation population.

[0063] S2064. Repeat the above iterative steps until the stopping conditions of the upper-level genetic algorithm are met (such as reaching the maximum number of iterations or the quality of the solution converges). The algorithm finally outputs the joint optimal solution, including: the optimal line balance solution, the optimal buffer configuration solution, and the optimal production sorting solution.

[0064] The entire algorithm forms a closed-loop optimization system consisting of upper-level global search, lower-level adaptive learning, and high-fidelity simulation feedback. It achieves coordinated optimization of mixed-flow assembly line balancing, buffer configuration, and production sequencing, thereby improving the quality, robustness, and computational efficiency of the solution.

[0065] The solution process of the upper-level solver will be described in detail below with reference to Figure 3: The upper-level line balance optimization problem is solved using the Upper-Level-GA solver, which employs an improved genetic algorithm (GA). This solver searches for line balance schemes globally through population evolution and maps chromosomes to feasible line balance schemes through a specific encoding and decoding strategy. The overall solution process includes steps such as encoding initialization, individual decoding, fitness evaluation, genetic operations, and convergence judgment.

[0066] S301, System Initialization and Coding.

[0067] The system encodes assembly tasks and their priority constraints, with each individual representing a candidate task allocation scheme or workstation partitioning structure. The algorithm generates a diverse set of line balancing schemes by randomly generating an initial population.

[0068] S302, Individual Decoding and Fitness Assessment.

[0069] For each individual task, the task time and cycle time constraints for each workstation are calculated, and the performance is evaluated by calling the lower-level modules (including buffer configuration and production scheduling optimization) and the Discrete Event Simulation System (ALS). S3021. Call the lower-level dual-depth Q-network solver, input the current line balancing scheme, and output the optimal buffer configuration scheme.

[0070] S3022: Call the lower-level genetic algorithm solver, input the current line balance scheme, and output the optimal production sorting scheme.

[0071] S3023. Call the discrete event simulation system to perform performance evaluation and obtain indicators such as balance rate, idle rate and number of workstations.

[0072] S3024. Calculate the fitness value of the upper layer: Based on the evaluation results, with the optimization objectives of minimizing the number of workstations and cycle time and maximizing the balance rate, calculate the fitness value corresponding to the line balancing scheme.

[0073] S303, Evolutionary Iterative Optimization.

[0074] Repeat the following evolutionary process until the fitness continues to improve or the maximum number of iterations is reached: S3031. Selection Operation: Use tournament selection method to select high-quality individuals as parents.

[0075] S3032, Crossover operation: The selected parent individuals are subjected to an order crossover operation to exchange some chromosome segments to generate new offspring solutions.

[0076] S3033, Mutation Operation: Perform a task exchange mutation operation on offspring individuals with a preset probability, randomly exchanging some gene positions to maintain population diversity and explore better solutions.

[0077] S3034, Offspring Fitness Assessment: After the newly generated individuals are decoded and simulated on the platform (i.e., steps S3021-3024 are re-executed), their fitness values ​​are calculated.

[0078] S3035, Environmental Selection: Selecting elite individuals (individuals with higher fitness values) from the new offspring and the old population to form the next generation of the population.

[0079] S304, Output the results.

[0080] The entire evolutionary process terminates when the fitness continuously improves or the maximum number of iterations is reached, ultimately outputting the optimal line balancing scheme and the corresponding workstation partitioning results, providing the input basis for the lower-level reinforcement learning and production ranking modules. This process forms an adaptive genetic optimization loop based on simulation feedback, realizing the global search and optimal solution convergence of the upper-level line balancing problem.

[0081] Furthermore, the upper-level genetic algorithm employs specific encoding and decoding methods: Encoding scheme: The upper-level solution is represented as a chromosome, i.e., a task sequence. ,in This indicates the task number, and the sequence order reflects the priority of task assignment; Decoding process: Input: Task sequence Priority relationships between tasks, assembly line cycle time (CT) Step 1: According to the Maximum Weighted Value (LWV) rule, the solution... Convert to a candidate set of task sequences ; Step 2: Initialize the first work center k=1; Step 3: According to The task is sorted and assigned to the current work center one by one. If the total working time of the current work center plus the working time of the next task to be assigned exceeds the cycle time ct, the assignment of tasks to that work center is stopped. Step 4: Remove the assigned task from the task sequence. If there are unassigned tasks, create a new work center k=k+1 and return to step 3; otherwise, proceed to step 5. Step 5: Calculate the sum of idle times of adjacent work centers. If the sum of idle times of adjacent work centers is greater than the cycle time ct, then merge the two work centers into one, and reduce the number of workstations accordingly. Step 6: Complete the decoding operation and output a feasible line balancing scheme AL (set of work centers, task allocation for each work center, and total number of workstations).

[0082] The following example uses a cycle time of 10 seconds to illustrate the line balancing scheme obtained by the upper-level solver. Figure 4 shows the task priority relationship in the example; the numbers inside the circles represent the task sequence, and the numbers outside the circles represent the weighted average time corresponding to that task. Table 1 details the decoding process. Figure 5 shows an example of the line balancing scheme design after decoding. Table 1: Example Decoding Process

[0083] In the RL-BLMF algorithm proposed in this embodiment, the first lower-level module is responsible for buffer configuration optimization, and its specific implementation is as follows: First lower-level solver: Buffer configuration based on knowledge-enhanced deep reinforcement learning (Lower-Level-DDQN) The buffer configuration of the first lower-level module is optimized using the Lower-Level-DDQN solver. This solver trains the agent based on a two-layer deep Q-network (DDQN), and accelerates convergence through experience replay and target network stabilization training, combined with knowledge enhancement strategies, enabling it to quickly generate the optimal buffer capacity configuration scheme based on a given line balancing scheme.

[0084] like Figure 6 As shown, the steps of the buffer configuration optimization process are as follows: S601, Begin; Initiate the buffer configuration optimization process.

[0085] S602, Input line balancing scheme AL; It receives the line balancing scheme passed from the upper layer as input.

[0086] S603. Calculate the number of assembly centers k and its load based on LB. Based on the input line balancing scheme, calculate the number of assembly centers k and the load status of each assembly center.

[0087] S604. Determine if all buffers have been configured.

[0088] Check if all buffer configurations have been completed: if not, proceed to step S605; if yes, proceed to step S208. S605. Determine the current buffer position.

[0089] Determine the specific location of the buffer that needs to be configured in the production line.

[0090] S606, Agent determines the capacity (Bi) of the i-th buffer based on the state.

[0091] The agent determines the current buffer size based on state space information through the DDQN network.

[0092] S607, Update buffer configuration x; Update the currently determined buffer capacity to the configuration scheme, and return to step S604 to continue configuring the next buffer.

[0093] S608. Determine whether the termination condition is met; Verify whether the current buffer configuration scheme meets the algorithm termination condition.

[0094] S609. Convert the configuration results into an assembly line design scheme T; The optimized buffer configuration results are converted into an assembly line design scheme.

[0095] S610, output assembly line design scheme T and buffer configuration scheme x; Output the final assembly line design scheme and the corresponding buffer configuration scheme.

[0096] S611, End.

[0097] Complete the buffer configuration optimization process.

[0098] The specific implementation of the reinforcement learning optimization process is as follows: The first lower-level module buffer configuration optimization process is implemented by a solver based on a deep dual-Q network (DDQN). Its operation process includes steps such as state modeling, action selection, environment interaction, reward calculation, network update and policy convergence.

[0099] Step 1, State Modeling: The system establishes a reinforcement learning environment based on the upper-level line balance results, and constructs a state vector from the relevant features of each buffer.

[0100] The DDQN state space is designed with 6-dimensional state features: Previous station load: The load of the station preceding the current buffer position ( ) ; Next station load: The load of the station following the current buffer position; Absolute position: The absolute position of the current buffer within the entire production line; Relative position: The relative position of the current buffer within the entire production line; Peak position: Assuming the buffer zone before the peak load station is located in the relative position of the entire production line; Current capacity: The size of the currently allocated buffer.

[0101] Step 2, Action Selection: The agent adjusts the buffer capacity as an action, selecting different capacity levels from the discrete action space. The action space is defined as the size of the buffer, with a total of 6 actions, and the discrete set is A = {0, 1, 2, 3, 4, 5}, representing buffer sizes from 0 to 5. An ε-greedy strategy is used in the action selection process.

[0102] Step 3, Environmental Interaction and Reward Calculation: For each action performed, the system evaluates the new production takt time, idle rate, and cost changes by calling the Discrete Event Simulation System (ALS), and constructs a reward function based on takt time improvement and buffer cost. The reward function is calculated according to equation (13), where a decrease in takt time results in a positive reward, and an increase in cost results in a negative penalty: The formula uses ALS to evaluate its simulation time. When the buffer configuration can effectively alleviate workstation overload (reduce production cycle time) and the buffer capacity setting is lower, the reward value is larger, which is -0.05(Bk + 1) + 1; conversely, it is negative, which is -(Bk + 1) - 1, and the buffer capacity setting is larger.

[0103] (13) In equation (13): β is a coefficient used to adjust the merits of different buffer configuration schemes; β=1, if cts ≤ ct, the configuration is valid; otherwise, β=0. When β = 1, α = 1; otherwise, α = -1. Step 4, Knowledge Augmentation and Network Update: The knowledge augmentation strategy strongly guides the agent to learn domain knowledge in the early stages of training, and gradually relies on data-driven learning in the later stages, balancing prior knowledge and experience-based learning. Knowledge augmentation is introduced by embedding the "inverted bowl" buffer distribution pattern into the loss function as a regularization constraint, guiding the agent to learn a reasonable capacity distribution trend and avoiding blind exploration. This method guides the agent's update direction in the early stages of training by introducing prior knowledge into the loss function as a regularization term. Its calculation method is shown in Equation 14.

[0104] (14) in, λ is the standard DDQN loss function, which includes Q learning loss and regularization term; λ is the decaying weight coefficient, whose value decays as training rounds progress; L is the knowledge embedding fitting function, which measures the difference between the current policy and the "inverted bowl" distribution, and its value is the regularization loss of the "inverted bowl" pattern and the optimal action of the current agent state.

[0105] Experience replay stores the interaction samples of the agent. DDQN uses the main Q network and the target Q network to update alternately and synchronize parameters periodically to improve training stability.

[0106] Step 5, Policy Convergence and Output: Through multiple rounds of training, the agent gradually learns the optimal capacity decision-making strategy under different production states, minimizing the overall cycle time and maximizing production smoothness. Finally, the system outputs the optimal capacity configuration scheme for each buffer and passes the simulation feedback results to the upper-level genetic algorithm for global optimization and fitness updates, achieving a collaborative optimization closed loop between buffer configuration and line balancing.

[0107] The agent is trained using a stepwise training method. Specifically, first, a line balancing scheme for the assembly line is randomly generated. Then, the designs of the first two workstations (WCs) are used to train the optimal configuration scheme for the first buffer. After training, the designs of the first three WCs and the first buffer configuration are used to train the optimal configuration scheme for the second buffer, and so on, until all buffer configurations are trained. It is worth noting that the line balancing scheme for the randomly generated assembly line is set as follows: the number of assembly centers is K, each assembly center has only one assembly workstation, and only one task is assigned. Then, based on the given cycle time, K job task times are randomly generated from [0.05ct.ct] and assigned to the assembly centers one by one.

[0108] In the RL-BLMF algorithm proposed in this embodiment, the second lower-level module is responsible for production sorting optimization, and the specific implementation is as follows: Second lower-level solver: based on genetic algorithm (Lower-Level-GA); The lower-level production sequencing is solved using a Lower-Level-GA solver. This solver employs a genetic algorithm (GA) to optimize the product production sequence, and the quality of the solution is evaluated using ALS simulation. Its process is similar to that of the upper-level solver, including initialization, evolution, evaluation, and selection. The difference lies in that it requires input from the upper-level solution and the solution from the first lower-level solver before the fitness value of the generated production sequencing scheme can be evaluated and the performance of the solution can be assessed using ALS simulation.

[0109] As shown in Figure 7, the steps of this production scheduling optimization process are as follows: S701, Begin.

[0110] Initiate the production scheduling optimization process.

[0111] S702, Input assembly line design scheme T.

[0112] It receives the upper-layer line balancing scheme and the lower-layer buffer configuration results as input.

[0113] S703, Generate the initial population.

[0114] An initial random population Pl of size NL is randomly generated, with each individual corresponding to a specific production sequence that satisfies the product ratio and production batch constraints.

[0115] S704. Determine whether all individual assessments have been completed.

[0116] Check if any member of the population has completed the evaluation: if not, continue the evaluation process; if yes, proceed to step S705. S705, Environmental Selection.

[0117] Based on the fitness results, high-quality individuals are selected to enter the next generation.

[0118] S706. Determine whether the stopping condition is met.

[0119] Verify whether the algorithm termination condition has been met: if not, proceed to step S707; if yes, proceed to step S710.

[0120] S707, Genetic Operations.

[0121] Perform selection, crossover, and mutation operations: Selection strategy: Prioritize retaining high-quality individuals using roulette or tournament methods; Crossover operations: Use partial mapping crossover (PMX) or sequential crossover (OX) to maintain product order characteristics; Mutation operations: breaking local patterns through swapping or insertion operations to maintain population diversity. S708, generating offspring population Ol.

[0122] New offspring populations are generated through genetic manipulation.

[0123] S709. Evaluate the offspring population.

[0124] For each newly generated individual, the Discrete Event Simulation System (ALS) is invoked to perform performance evaluation, and the corresponding average cycle time, production line volatility, blockage rate, and product changeover smoothness are calculated.

[0125] S710 calls ALS to simulate and evaluate the optimal production sequencing scheme and its simulation time.

[0126] The optimal solution is then evaluated through simulation to obtain its performance metrics.

[0127] S711 outputs the optimal production scheduling scheme and its simulation time.

[0128] Output the optimal product launch sequence plan and the corresponding cycle time and capacity performance.

[0129] S712, End.

[0130] Complete the production scheduling optimization process.

[0131] The optimized process is implemented as follows: The second-level production sorting optimization process is implemented by a genetic algorithm (Lower-LevelGASolver), whose main steps include encoding initialization, fitness evaluation, genetic operations and convergence determination.

[0132] First, the system encodes the production sequence of different product models based on the upper-level line balancing scheme and the lower-level buffer configuration results. Each individual corresponds to a specific production sequence, satisfying product ratio and production batch constraints. The algorithm randomly generates an initial population, forming multiple production sequencing schemes.

[0133] Subsequently, the discrete event simulation system (ALS) was invoked for performance evaluation of each individual, and the corresponding indicators such as average cycle time, production line volatility, blockage rate and product changeover smoothness were calculated. The fitness function was constructed with the goal of minimizing average cycle time and production volatility.

[0134] Based on the fitness results, the algorithm performs selection, crossover, and mutation operations. Selection uses a roulette wheel or tournament approach to prioritize the retention of high-quality individuals. Crossover uses partially mapped crossover (PMX) or sequential crossover (OX) to maintain product order characteristics. Mutation breaks local patterns through swapping or insertion operations to maintain population diversity.

[0135] Newly generated individuals are decoded and evaluated through simulation before being added to the next generation of the population, forming an evolutionary cycle based on performance feedback. After multiple iterations, when the fitness converges or the maximum number of iterations is reached, the algorithm outputs the optimal product deployment sequence and feeds its corresponding cycle time and production capacity performance back to the upper-level genetic algorithm for joint fitness updates.

[0136] Through this process, the genetic algorithm can achieve global optimization of production scheduling under multi-objective constraints, effectively improving the balance and operating efficiency of mixed-flow production lines.

[0137] This application provides a two-layer multi-follower optimization model that integrates the balancing, buffer allocation, and production sequencing problems of mixed-flow assembly lines considering random disturbances and parallel workstations into a unified model. The upper layer handles the line balancing problem, while the lower layer simultaneously handles the two follower problems of buffer allocation and production sequencing. This model accurately describes the hierarchical coupling relationship under uncertain conditions, laying the foundation for true collaborative optimization. Furthermore, a two-layer multi-follower optimization algorithm based on deep reinforcement learning is proposed, combining the global search capability of genetic algorithms with the rapid decision-making capability of deep reinforcement learning. By replacing the traditional nested optimization loop with a trained reinforcement learning agent to solve for buffer allocation, the computational cost of evaluation in stochastic environments is significantly reduced, achieving a good balance between solution quality and efficiency. To further improve the training efficiency of the reinforcement learning agent in complex stochastic environments, this application embeds prior knowledge from the field of assembly line optimization—the "inverted bowl" buffer allocation rule (i.e., buffers in the middle of the production line are usually more important than those at the ends)—into the agent's loss function as a regularization term. This strategy effectively guides the agent's update direction in uncertain environments during the early stages of training, avoiding blind exploration and accelerating the convergence process. Throughout the optimization process, a discrete event-based assembly line simulator is integrated to evaluate the actual performance (such as simulation cycle time) of candidate schemes (line balancing, buffering, and sorting) under random disturbances. This makes the optimization results closer to actual production, fully considers dynamic factors such as randomness and parallel structures, and improves the robustness and practicality of the schemes.

[0138] It is understood that the above specific description of the present invention is only for illustrating the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention to achieve the same technical effect; as long as the use needs are met, they are all within the protection scope of the present invention.

Claims

1. A collaborative optimization method for mixed-flow assembly lines based on deep reinforcement learning, characterized in that: Includes the following steps: S101. Obtain basic information about the mixed-flow assembly line to establish an assembly line model that includes task time, priority relationship, parallel workstation structure, cycle time constraints and disturbance characteristics. At the same time, based on historical production data and / or domain knowledge, define and quantify the random disturbance characteristics of the system. S102. Establish a mixed-flow assembly line problem model for inputting into a deep reinforcement learning-based optimization algorithm; the problem model is an optimization model including a hierarchical structure, decision variables, objective function, and constraints for optimizing line balancing, buffer configuration, and production scheduling; the deep reinforcement learning-based optimization algorithm includes an upper-level solver, a first lower-level solver, and a second lower-level solver, used to collaboratively optimize the line balancing, buffer configuration, and production scheduling problems; S103. Construct a two-layer multi-follower optimization model, with the upper layer being a line balancing optimization model and the lower layer being a buffer configuration model and a production sorting optimization model. S104. Design a hybrid solution algorithm based on deep reinforcement learning and genetic algorithm, construct an optimization algorithm based on a two-layer multi-follower model of deep reinforcement learning, optimize the upper layer problem through the line balance optimization model, optimize the lower layer problem through the buffer configuration model and the production sorting optimization model, and obtain a preliminary optimization scheme. S105. Calculate the objective function value of the overall optimization scheme using a discrete event-based simulation system, and perform performance evaluation and feedback optimization on the preliminary optimization scheme. S106. Repeat the evolution and learning process multiple times until the optimal solution for output line balancing, buffer configuration, and production ordering is found. The upper-level solver is an upper-level assembly line balancing solver, which uses a genetic algorithm for global search of line balancing schemes. The first lower-level solver is a lower-level buffer configuration solver, which uses a deep reinforcement learning network based on a knowledge enhancement strategy to train the agent and quickly decide on the buffer configuration scheme. The second lower-level solver is a lower-level production ordering solver, which uses a genetic algorithm to optimize the production ordering scheme, specifically: After the GA generation line balancing preliminary optimization scheme of the upper-level solver is completed, the DDQN agent of the first lower-level solver and the GA solver of the second lower-level solver are respectively called to perform the preliminary optimization scheme of buffer configuration and production sorting. The optimization algorithm based on a deep reinforcement learning-based two-layer multi-follower model comprises three core solvers: an upper-level assembly line balancing solver (Upper-Level-GA), a lower-level buffer configuration solver (Lower-Level-DDQN), and a lower-level production sequencing solver (Lower-Level-GA). These solvers are used to collaboratively optimize the line balancing, buffer configuration, and production sequencing problems of mixed-flow assembly lines. The implementation process includes: S201. Input and Initialization; The system reads the input data of the mixed-flow assembly line, including assembly task data, cycle time constraints, parallel station structure, buffer parameters and product mixing ratio, establishes an optimization model, and initializes the population of the upper-layer genetic algorithm, the lower-layer deep reinforcement learning agent and the discrete event simulation system. S202, Upper-layer line balance optimization; The upper-level optimization module uses a genetic algorithm to generate line balancing schemes. The specific process is as follows: S2021. Initialize the parameters of the Upper-Level-GA algorithm and randomly generate initial solutions, i.e., the initial population, where each solution represents a candidate line balancing scheme. S2022. For each individual in the population, perform selection, crossover, and mutation operations to generate different task assignment structures; S2023. Before evaluating the fitness of each individual, it is necessary to call the two lower-level modules: buffer configuration optimization and production sorting optimization, to obtain the corresponding optimal lower-level solution. S203, Optimization of lower-level buffer configuration; The first lower-level module uses a knowledge-enhanced deep double-Q network algorithm to optimize the buffer configuration: The intelligent agent takes workstation load, location, and capacity as state inputs; The action output is the adjustment of the buffer capacity; Construct a reward function by improving cycle time and cost constraints; Fast convergence is achieved by introducing regularization constraints based on the "inverted bowl" prior law, and the optimized buffer capacity distribution scheme is output. S204, Optimization of the lower-level production sequence; The second lower-level module uses a genetic algorithm to optimize the production sorting: Based on the line balancing scheme provided by the upper layer and the buffer configuration scheme output by the first module of the lower layer, multiple product production sequences are generated; The cycle time and stability index of each sorting scheme are calculated using a discrete event simulation system. S205, Simulation Evaluation and Feedback; The ALS simulation evaluation module performs discrete event simulations on a joint scheme consisting of a line balancing scheme, a buffer configuration scheme, and a production sequencing scheme. The following performance indicators were simulated and calculated: average cycle time, station idle rate, production line throughput, and balance rate. The performance metrics obtained from the simulation are used as feedback signals and transmitted to the upper-layer genetic algorithm and the lower-layer reinforcement learning module, respectively, for calculating the fitness value of the upper layer and updating the policy of the lower-layer DDQN agent; S206, Evolutionary Iteration and Output; The system continuously improves the quality of the solution through multiple rounds of evolutionary iteration: S2061. Use the initial population to perform evolutionary operations to generate offspring solutions; S2062. For each newly generated child solution, the Lower-Level DDQN and Lower-Level GA solvers are called again to obtain the corresponding lower-level optimal solution and calculate the objective function value of the upper level, i.e., the fitness value. S2063. Through the environment selection operation, select individuals whose fitness reaches the convergence threshold or the maximum number of iterations from the child and parent solutions to form the next generation population. S2064. Repeat iterative steps S2061-2063 until the stopping condition of the upper-level genetic algorithm is met. The algorithm finally outputs the joint optimal solution, including the optimal line balance solution, the optimal buffer configuration solution, and the optimal production sorting solution.

2. The method according to claim 1, characterized in that: The two-layer multi-follower optimization model is used to accurately describe the hierarchical coupling relationship between problems, as follows: The upper-level model is used for balancing optimization of mixed-flow assembly lines. The upper-level problem is to determine the structural design of the production line from a strategic perspective, and to determine the number of work centers, the number of workstations in each work center, and the assigned tasks in the assembly line, i.e., the line balancing scheme, under the premise of satisfying task priority constraints. The optimization objective is to minimize the total cost while meeting the specified average throughput. Its mathematical model is as follows: Objective function: （1）， （2）， Constraints: （3）， （4）， （5）， （6）， Equation (1) is the objective function, where DC is the annual design production cost of the production line, NDC is the annual actual total production cost of the production line, and ct is the production cycle time of the given production line. The average cycle time is obtained from the ALS simulation software. As a higher-level penalty factor; Equation (2) represents the total design and production cost. Let be the number of workstations in work center k, and AC be the sum of the annual costs for a single assembly workstation and a single assembly operator. Let k be the number of tasks in work center k, and CE be the annual cost of a single device. Let CB be the buffer capacity between work center k-1 and work center k, and CB be the annual cost per unit buffer; when the scheme satisfies At that time, only the annual design production cost (DC) of the production line is considered; when At that time, the additional cost of efficiency loss is reflected through a secondary penalty term, and the penalty intensity increases significantly as the overtime ratio increases; Constraint (3) represents the task allocation integrity constraint. This indicates that if task n is assigned to work center k, the value is 1; otherwise, it is 0. This ensures that each task must be assigned to one and only one work center, avoiding task omissions or duplicate assignments. Constraint (4) represents the work center time constraint. This represents the weighted average processing time of task n, ensuring that the theoretical workload of each work center does not exceed the production cycle time. It is a static balance constraint based on the average time. Constraint (5) represents the task priority relationship constraint, k represents the work center, k=1,2,...,K, ensuring that for any task pair (i,h) with priority relationship, where i is the predecessor of h, task i must be assigned before h or to the same work center, thus guaranteeing the assembly process logic; Constraint (6) represents the decision variable constraint, defined as follows: Variables are 0-1; The buffer configuration model is used for buffer configuration optimization. The first lower-level problem is to optimize the buffer resource configuration from a tactical level based on a given line balancing scheme. The buffer configuration affects the production line's ability to resist random disturbances. The optimization objective is to minimize the buffer configuration cost while ensuring that the production line achieves the specified average throughput. Objective function: （7）， （8）， Constraints: （9）， Equation (7) is the objective function of the lower buffer configuration optimization problem, TDB represents the lower objective function. The lower buffer configuration objective is to minimize the buffer cost while ensuring production line efficiency. When the actual cycle time meets the requirements, only the buffer cost is considered. When the cycle time exceeds the limit, the penalty term prompts the agent to find a buffer configuration that can meet the efficiency requirements and ensure a lower cost. is the penalty factor for the first lower-level solver, and its value is a large constant; Equation (8) is the total cost of the buffer, DB represents the total capacity of the production line buffer, and the total cost of the buffer is proportional to the total capacity of the buffer, assuming that the unit cost of the buffer is the same; Constraint (9) is the upper and lower bound constraints of the buffer capacity. This indicates the buffer capacity between assembly center K-1 and assembly center K. =0 means no buffer, and =5 is based on actual space constraints to ensure that the configuration plan is feasible; The production sequencing optimization model is used for production sequencing optimization. The second lower-level problem is to optimize the production sequencing and scheduling problem from the operational level based on the given line balance and buffer scheme. The sequencing problem involves determining the processing sequence of product models entering the assembly line, and the optimization objective is to minimize the production cycle time. Objective function: （10）， Constraints: （11）， （12）， Wherein, equation (10) is the objective function of the lower-level sorting optimization problem. The average cycle time obtained from the ALS simulation software is directly minimized to reflect the actual cycle time obtained through high-fidelity simulation, thus reflecting the true efficiency of the scheme under random perturbation. The objective is clear and direct. Constraint (11) represents the sequence position allocation constraint. This indicates that if sequence p is assigned to product m, then it is 1; otherwise, it is 0, ensuring that each position in the MPS must be assigned a specific product model; constraint (12) indicates that the product demand satisfies the constraint. This represents the production quantity of model m in the MPS, i.e., the production proportion of m. It ensures that the production quantity of each product model strictly conforms to the MPS requirements, meaning each model m will occupy a certain position in the sequence. One position.

3. The method according to claim 1, characterized in that: The Upper-Level Assembly Line Balancing Solver (Upper-Level-GA) is used to optimize the upper-level assembly line balancing problem. This solver employs an improved genetic algorithm, globally searching for line balancing schemes through population evolution. It maps chromosomes to feasible line balancing schemes using a specific encoding / decoding strategy. The solution process includes: S301, System Initialization and Coding; The system encodes the assembly tasks and their priority constraints, with each individual representing a candidate task allocation scheme or workstation partitioning structure; the algorithm generates a diverse set of line balancing schemes by randomly generating an initial population. S302, Individual Decoding and Fitness Assessment; For each individual task, the task time and cycle time constraints for each workstation are calculated. Then, the lower-level modules, including buffer configuration and production scheduling optimization, as well as the discrete event simulation system, are called to perform performance evaluation. S3021. Call the lower-level dual-depth Q-network solver, input the current line balancing scheme, and output the optimal buffer configuration scheme; S3022: Call the lower-level genetic algorithm solver, input the current line balance scheme, and output the optimal production sorting scheme; S3023. Call the discrete event simulation system to perform performance evaluation and obtain the balance rate, idle rate and number of workstations; S3024. Calculate the fitness value of the upper layer: Based on the evaluation results, with the optimization objectives of minimizing the number of workstations and cycle time and maximizing the balance rate, calculate the fitness value corresponding to the current line balancing scheme. S303, Evolutionary Iterative Optimization; Repeat the following evolutionary process until the fitness continues to improve or the maximum number of iterations is reached: S3031. Selection Operation: Use tournament selection method to select high-quality individuals as parents; S3032, Crossover operation: A sequential crossover operation is performed on the selected parent individuals to exchange some chromosome segments to generate new offspring solutions; S3033, Mutation Operation: Perform task exchange mutation operation on offspring individuals with a preset probability, randomly exchange some gene positions to maintain population diversity and explore better solutions; S3034, Offspring Fitness Assessment: After the newly generated individuals are decoded and simulated on the platform, steps S3021-3024 are repeated to calculate their fitness values. S3035, Environmental Selection: Select individuals with higher fitness values ​​from the new offspring and the old population to form the next generation of the population; S304, Output the results.

4. The method according to claim 3, characterized in that: The Upper-Level GA solver employs specific encoding and decoding: Encoding scheme: The upper-level solution is represented as a chromosome, i.e., a task sequence. ,in This indicates the task number, and the sequence order reflects the priority of task assignment; Decoding process: Input: Task sequence Priority relationships between tasks, assembly line cycle time (CT) Step 1: According to the Maximum Weighted Value (LWV) rule, the solution... Convert to a candidate set of task sequences ; Step 2: Initialize the first work center k=1; Step 3: According to The task is sorted and assigned to the current work center one by one. If the total working time of the current work center plus the working time of the next task to be assigned exceeds the cycle time ct, the assignment of tasks to that work center is stopped. Step 4: Remove the assigned task from the task sequence. If there are unassigned tasks, create a new work center k=k+1 and return to step 3; otherwise, proceed to step 5. Step 5: Calculate the sum of idle times of adjacent work centers. If the sum of idle times of adjacent work centers is greater than the cycle time ct, then merge the two work centers into one, and reduce the number of workstations accordingly. Step 6: Complete the decoding operation and output a feasible line-balanced scheme AL.

5. The method according to claim 1, characterized in that: The Lower-Level-DDQN solver is used to optimize the buffer configuration of the first lower-level module. This solver trains the agent based on a two-layer deep Q-network, and accelerates convergence through experience replay and stable training of the target network, combined with knowledge enhancement strategies. This enables it to quickly generate the optimal buffer capacity configuration scheme based on a given line balancing scheme. The optimization process includes: S601, Start: Initiate the buffer configuration optimization process; S602, Input line balancing scheme AL: Receives the line balancing scheme transmitted from the upper layer as input; S603. Calculate the number of assembly centers k and their load based on AL: Based on the input line balancing scheme, calculate the number of assembly centers k and the load of each assembly center. S604. Determine if all buffer configurations are complete: Check if all buffer configurations are complete: If not, proceed to step S605; if yes, proceed to step S608. S605. Determine the current buffer position: Determine the specific position of the buffer that needs to be configured in the production line; S606. The buffer configuration agent determines the capacity (Bi) of the i-th buffer based on the state: The agent decides the current buffer capacity based on the state space information through the DDQN network. S607. Update buffer configuration x; Update the currently determined buffer capacity to the configuration scheme, and return to step S604 to continue configuring the next buffer; S608. Determine if the termination condition is met: Verify whether the current buffer configuration scheme meets the algorithm termination condition; S609. Convert the configuration result into an assembly line design scheme T: Convert the optimized buffer configuration result into an assembly line design scheme. S610, Output Assembly Line Design Scheme T and Buffer Configuration Scheme x: Output the final assembly line design scheme and the corresponding buffer configuration scheme; S611, End: The buffer configuration optimization process is complete.

6. The method according to claim 5, characterized in that: The first lower-level module buffer configuration optimization process is implemented by a solver based on a deep double-Q network, and its operation includes: Step 1, State Modeling: The system establishes a reinforcement learning environment based on the upper-level line balance results, and constructs a state vector from the relevant features of each buffer. The DDQN state space is designed with 6-dimensional state features: Previous station load: The load of the station preceding the current buffer position. ; Next station load: The load of the station following the current buffer position; Absolute position: The absolute position of the current buffer within the entire production line; Relative position: The relative position of the current buffer within the entire production line; Peak position: Assuming the buffer zone before the peak load station is located in the relative position of the entire production line; Current capacity: The size of the currently allocated buffer; Step 2, Action Selection: The agent selects different capacity levels from the discrete action space for adjustment, using buffer capacity adjustment as the action. Step 3, Environment Interaction and Reward Calculation: For each action executed, the system calls ALS to evaluate the new production takt time, idle rate and cost changes, and constructs a reward function based on the takt time improvement and buffer cost; The reward function is calculated according to equation (13), where a decrease in pace yields a positive reward, while an increase in cost incurs a negative penalty: （13）， In equation (13): β is a coefficient used to adjust the merits of different buffer configuration schemes; β=1, if ct s ≤ ct indicates that the configuration is valid; otherwise, β=0. α = (-1) 1+β When β = 1, α = 1; otherwise, α = -1. Step 4: Knowledge Enhancement and Network Updates In the early stages of training, the agent is strongly guided to learn domain knowledge, while in the later stages it gradually relies on data-driven learning to balance prior knowledge and experience learning. Knowledge enhancement is introduced, and the distribution pattern of the "inverted bowl" buffer is embedded into the loss function in the form of regularization constraints to guide the agent to learn a reasonable capacity distribution trend and avoid blind exploration; its calculation method is shown in Equation (14): （14）， in, λ is the standard DDQN loss function, which includes Q learning loss and regularization term; λ is the decaying weight coefficient, whose value decays as training rounds progress; L is the fitting function for embedded knowledge, which measures the difference between the current policy and the "inverted bowl" distribution, and its value is the regularization loss of the "inverted bowl" pattern and the current buffer configuration agent state optimal action. Step 5, Policy Convergence and Output: Through multiple rounds of training, the agent gradually learns the optimal capacity decision-making strategy under different production states, so as to minimize the overall cycle time and optimize the production smoothness. The system outputs the optimal capacity configuration scheme for each buffer and passes the simulation feedback results to the upper-level genetic algorithm for fitness updates in global optimization, realizing a closed loop of collaborative optimization between buffer configuration and line balancing.

7. The method according to claim 1, characterized in that: The Lower-Level GA solver is used to solve the production sequencing problem. This solver employs a genetic algorithm to optimize the product production sequence and evaluates the quality of the scheme through ALS simulation. The production sequencing optimization process includes: S701, Start: Initiate the production sequencing optimization process; S702, Input Assembly Line Design Scheme T: Receives the upper-level line balancing scheme and the lower-level buffer configuration result as input; S703. Generate an initial population: Randomly generate an initial random population of size NL, where each individual corresponds to a specific production sequence, satisfying product ratio and production batch constraints. S704. Determine if all individuals have completed the assessment: Check if all individuals in the population have completed the assessment: If not, continue the assessment process; if yes, proceed to step S705. S705. Environmental selection: Based on fitness results, select high-quality individuals to enter the next generation; S706. Determine if the stopping condition is met: Verify if the algorithm termination condition is met: If not, proceed to step S707; if yes, jump to step S710. S707, Genetic Operations: Perform selection, crossover, and mutation operations, including: Selection strategy: Prioritize retaining high-quality individuals using roulette or tournament methods; Crossover operations: Employ partial mapping crossover or sequential crossover to preserve product order characteristics; Mutation operations: breaking local patterns through swapping or insertion operations to maintain population diversity; S708. Generating offspring population: Creating a new offspring population through genetic operations; S709. Evaluate the offspring population: For each newly generated individual, call the discrete event simulation system to evaluate its performance and calculate the corresponding average cycle time, production line volatility, blockage rate, and product switching smoothness. S710. Call the discrete event simulation system to simulate and evaluate the optimal production sequencing scheme and its simulation time: perform simulation evaluation on the final optimal scheme and obtain its performance indicators. S711, Output the optimal production sequencing scheme and its simulation time: Output the optimal product production sequence scheme and the corresponding cycle time and capacity performance; S712, End: Complete the production sorting optimization process.

8. The method according to claim 1, characterized in that: The basic information of the mixed-flow assembly line includes: all assembly tasks and their processing time under different product models, priority relationship between tasks, possibility of parallel configuration of workstations, production cycle time, and product demand ratio; the random disturbance characteristics refer to uncertain events that cause the actual operation time to deviate from the standard value or cause production interruption, and their types include equipment failure, material supply delay, and fluctuation in operator efficiency.

Images