A production system merging strategy optimization method considering workpiece quality

Abstract

The application provides a production system merging strategy optimization method considering workpiece quality, Markov chain is used to describe production system state transition, a production system performance evaluation model is constructed, and a production system performance index is quantified, which is used as a basis for a merging strategy optimization model; a merging strategy optimization model is constructed based on a Markov decision process, and a deep reinforcement learning algorithm is used to solve the model to obtain an effective merging strategy. Through simulation experiments, the obtained merging strategy and different heuristic strategies are respectively interacted with a production system simulation environment, relevant performance indexes are calculated, and the effectiveness of the proposed merging strategy optimization method is verified.

Description

Technical Field

[0001] This invention relates to the field of production system merging strategy optimization, specifically a production system merging strategy optimization method that considers workpiece quality. Background Technology

[0002] Remanufacturing is an advanced manufacturing process that uses high-tech methods to repair and remodel recyclable products. By disassembling and reassembling these products, it aims to conserve natural resources and reduce emissions. The raw materials in a remanufacturing system are finished parts that have been in service for a long time and are now scrapped. Due to wear, corrosion, fatigue, vibration, and impact, these raw materials often exhibit varying degrees of failure on their surface and internal structure, such as residual stress, internal cracks, and deformation. Therefore, the quality levels of workpieces in a remanufacturing system are complex, variable, and difficult to control, resulting in a highly volatile remanufacturing process. In remanufacturing systems, the quality levels of workpieces fluctuate greatly, and automated workstations cannot process certain low-quality workpieces. Compared to automated workstations, manual workstations can not only handle low-quality workpieces that automated workstations cannot process but also help automated workstations share some of the processing tasks. Therefore, a human-machine collaborative production model can leverage both the high efficiency of automated workstations and the flexibility of humans, thereby improving the production efficiency of the production system, reducing production costs, and improving product quality.

[0003] Merging strategy refers to a strategy where merging stations select workpieces from multiple upstream parallel buffers for processing. In remanufacturing systems, merging stations are often manual stations, and their processing efficiency depends on the quality grade of the workpieces. Processing lower-quality workpieces often requires a long processing time. If manual stations only select high-quality workpieces and ignore the work-in-process levels in other upstream buffers, it can cause congestion at upstream stations, leading to production losses. Furthermore, when manual stations switch production between different buffers, varying operator proficiency with different processes can limit the overall efficiency of the production system. Therefore, it is necessary to optimize the merging strategy to maintain high utilization rates for each station in the production system while minimizing the work-in-process levels in the buffers and the frequency of production switching, thereby increasing the production capacity and maximizing the system's profits.

[0004] Production system performance evaluation, as the foundation of merging strategy optimization, refers to using mathematical tools such as stochastic processes to characterize the interaction between material flow and buffer capacity, calculate the performance indicators of the production system, and guide the management and decision-making of the production system. Remanufacturing systems face problems such as workpiece quality grade fluctuations, automated workstation malfunctions, and limited buffer capacity. These problems not only affect the output of automated workstations but also propagate along the upstream and downstream of the system, affecting the operation of other automated workstations. This results in a complex nonlinear relationship between system input and output. Furthermore, the coupling between the processing efficiency of manual workstations and workpiece quality further complicates this nonlinear relationship, making it difficult to evaluate the performance of the production system. Summary of the Invention

[0005] The technical problem to be solved:

[0006] Existing research on production system performance evaluation, particularly under dynamic merging strategy control, is limited. Furthermore, research on merging strategies in production systems primarily focuses on single-product and multi-product production systems without considering workpiece quality. Therefore, this patent proposes an optimization method for merging strategies in production systems that consider workpiece quality, guided by the system's real-time operating status and driven by a performance evaluation model. It analyzes the impact of workpiece quality fluctuations and merging strategies on the production system, explores dynamic merging strategy optimization schemes, and provides methodological support and decision-making basis for merging decisions in production systems that consider workpiece quality.

[0007] The technical solution of this invention is as follows:

[0008] A method for optimizing the merging strategy of a production system considering workpiece quality includes the following steps:

[0009] Step 1: Establish a model of the split-flow and merge-flow production system that considers workpiece quality, including

[0010] Step 1.1: Define the basic parameters of the production system, including workpiece quality grade, buffer capacity and arrangement rules within the buffer, workstation reliability model, processing pass rate, splitting and merging function, and the functional relationship between the Bernoulli parameter of the merging workstation and the quality grade of the defective workpiece.

[0011] Step 1.2: Define the workstation status and buffer status at each time point;

[0012] Step 1.3: Define the production system operation logic, including the interaction between workstations, workpiece flow, and buffer zones;

[0013] Step 2: Construct a production system performance evaluation model, including

[0014] Step 2.1: Use buffer states to represent the production system state, and use Markov chains to characterize the state transition process of the workstation;

[0015] Step 2.2: Based on the characteristics of the Bernoulli station, derive the state transition process of the production system using the station's state transition process, and calculate the state transition process of the production system from state to state. Transition to state Based on all transition probabilities, the update scheme for the workstation state and buffer state at each moment is derived, and the computational scale of the production system state transition probability is analyzed.

[0016] Step 2.3: Quantify the transient performance indicators of the production system, including workstation output and work-in-process inventory in the buffer zone; based on the state transition process of the production system, derive the steady-state performance indicators based on the transient performance indicators of the production system, including total workstation output, average work-in-process inventory level in the buffer zone, total work-in-process inventory level, congestion rate, and starvation rate; use the production system performance indicators as the basis for improving the merging strategy optimization model.

[0017] Step 3: Construct a merging strategy optimization model, including

[0018] Step 3.1: Based on the state transition process of the production system, construct a simulation environment for the production system, use it as an environmental element in the Markov decision process, and use the merging workstation as an agent element in the Markov decision process;

[0019] Step 3.2: Design a reward function based on the unit output profit, unit work-in-process cost, and unit production switching cost of the production system. After the agent makes a decision at each moment, the environment provides an immediate reward to the agent based on the agent's action and moves to the next state.

[0020] Step 3.3: Based on the Bellman optimal equation, with the goal of maximizing the optimal action value function in the Bellman optimal equation, the process of the agent continuously interacting with the production system simulation environment to obtain rewards is constructed as a merging strategy optimization model.

[0021] Step 4: Solve the merging strategy optimization model using the D3QN algorithm;

[0022] Step 4.1: Use a duel neural network to fit the optimal action value function in the Bellman optimal equation, and decompose the optimal action value approximation function into the sum of the optimal state value approximation function and the optimal advantage approximation function;

[0023] Step 4.2: Approximate the expectation of the Bellman optimal equation using the Monte Carlo method;

[0024] Step 4.3: Adopt a dual-network structure and use the TD algorithm to train the neural network so that the neural network prediction value continuously approaches the true value, thereby maximizing the optimal action value function and optimizing the merging strategy.

[0025] In addition, the following steps were used to verify the effectiveness of the merging strategy optimization method:

[0026] Step 5.1: Initialize the simulation environment parameters and D3QN algorithm parameters of the production system;

[0027] Step 5.2: For the initialized production system simulation environment, based on the production system performance evaluation model, construct a merging strategy optimization model, and use the D3QN algorithm to solve the merging strategy optimization model to obtain strategy π. * ;

[0028] Step 5.3: Transfer strategy π * Heuristic strategies different from those in production systems π buffer π priority π random To make a comparison, the system interacted with the production system simulation environment, and the transient performance indicators of each interaction were statistically analyzed. The steady-state performance indicators were then calculated based on the transient performance indicators, which served as the criteria for evaluating the merits of the strategies.

[0029] Step 5.4: Analyze the performance indicators of the production system under different control strategies to verify strategy π. * The effectiveness.

[0030] Beneficial effects

[0031] This invention proposes an optimization method for merging strategies in production systems that considers workpiece quality, mainly addressing the following issues:

[0032] (1) Construction of production system performance evaluation model

[0033] For a split-flow and merging production system considering workpiece quality, this invention defines a basic model of the production system, uses Markov chains to characterize the state transition process, analyzes the interaction between workpiece quality level, merging strategy, and the production system, dissects the operational rules of the production system, constructs a production system performance evaluation model, and quantifies the performance indicators of the production system. The first inventive point of this invention is to construct the production system performance evaluation model, which serves as the basis for the merging strategy optimization model.

[0034] (2) Construction of Optimization Model for Production System Merging Strategy

[0035] For the optimization model of merging strategies in a production system, based on Markov decision processes, a reward function is designed by comprehensively considering the unit output profit, unit work-in-process cost, and unit production changeover cost of the production system. A production system simulation environment is constructed according to the state transition process of the production system. Merging workstations are used as agent elements in the Markov decision process, and the production system simulation environment is used as the environmental element in the Markov decision process. Based on the optimal action value function in the Bellman optimal equation, the process of merging workstations continuously interacting with the production system simulation environment to obtain rewards is constructed as the merging strategy optimization model. Constructing a production system merging strategy optimization model based on Markov decision processes is the second inventive point of this invention.

[0036] (3) Solution and verification of the merging strategy optimization model based on D3QN algorithm

[0037] In solving and validating the merging strategy optimization model, the D3QN (Dueling Double Deep Q-Learning Network) algorithm based on Q-value learning in Deep Reinforcement Learning is used to solve the merging strategy optimization model. An optimal action value function in the Bellman equation is fitted using a Dueling Network. The merging station continuously interacts with the production system simulation environment based on the optimal action value function to obtain a series of four-tuple state data, which are then stored in an experience pool. Four-tuple data are randomly batch-sampled, and a Double Deep Q-Learning Network structure is used. The neural network is trained using the Temporal Difference (TD) algorithm, while mini-batch gradient descent is performed on the loss function to update the neural network parameters. Finally, based on the optimal action value output by the neural network, a greedy search algorithm is used to search for the effective merging strategy π. * . Strategy π * Heuristics different from those in production systems, such as the buffer occupancy-based strategy π. buffer 1. Buffer priority strategy π priority Random strategy π random To compare the performance of the production system under different control strategies, the systems were interacted with in a production system simulation environment. The results were analyzed to verify the effectiveness of strategy π. * The effectiveness of the strategy is verified by optimizing the merging strategy using the D3QN algorithm and then validating the effectiveness of the optimized strategy. This is the third inventive aspect of the present invention.

[0038] As shown in the parameter configuration of the production system and optimization algorithm in the embodiment, the strategy π based on buffer priority will be implemented. priority Initialize to policy π b3π b1 , respectively, indicate that the merging workstation preferentially selects workpieces from buffer zones b3 and b1 for processing, and the strategy π * π b3 π b1 π random π buffer The simulation was conducted with each workstation, with a processing cycle of 1 minute. Each strategy interacted with the production system for 1000 rounds, with each round lasting 10,000 minutes. Performance metrics of the production system under each strategy were calculated and compared to validate the obtained merging strategy π. * The effectiveness of the measures is shown in Table 1, which compares the performance indicators of the production system under different merging strategies.

[0039] Table 1 Comparison of production system performance indicators under different merging strategies

[0040]

[0041]

[0042] Simulation results show that strategy π * It comprehensively considers unit production system output profit, unit work-in-process cost, and unit production changeover cost, weighing the relationship between the three and selecting effective actions. While ensuring production system output, it effectively reduces the total work-in-process inventory level and the production changeover frequency of merging stations, compared to Strategy π. b3 π b1 π random π buffer The total revenue increased by 5.9%, 9.2%, 9.3%, and 9.9% respectively, achieving the goal of maximizing the total revenue of the production system, thus verifying the effectiveness of the merging strategy optimization method proposed in this patent.

[0043] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0045] Figure 1 A split-flow and merge-flow production system that takes into account workpiece quality.

[0046] Figure 2 A battle of network structures.

[0047] Figure 3 Changes in the size of the loss function.

[0048] Figure 4 The changes in the rewards received by the Convergence Workstation each round.

[0049] Figure 5 Comparison of workstation output under different control strategies.

[0050] Figure 6 Comparison of workstation blocking rates under different control strategies.

[0051] Figure 7 Comparison of workstation starvation rates under different control strategies.

[0052] Figure 8 Comparison of total work-in-process inventory levels under different control strategies.

[0053] Figure 9 Comparison of production switching times at merging workstations under different control strategies. Detailed Implementation

[0054] This patent addresses the issue of splitting and merging production systems that consider workpiece quality. It constructs a production system performance evaluation model, proposes a merging strategy optimization method, and verifies the effectiveness of the proposed optimization method through simulation experiments.

[0055] The specific process is as follows:

[0056] First, a production system model considering workpiece quality is defined. A Markov chain is used to characterize the state transition process of the production system, analyzing the interaction between workpiece quality level, merging strategy, and the production system. The operational rules of the production system are dissected, a performance evaluation model is constructed, and the performance indicators of the production system are quantified. Second, considering the unit output profit, unit work-in-process cost, and unit production changeover cost of the production system, a reward function is designed, and a production system simulation environment is constructed based on the state transition process. The merging workstation and the production system simulation environment are respectively used as agent elements and environment elements in the Markov decision process. The process of the merging workstation continuously interacting with the production system simulation environment to obtain rewards is constructed as a merging strategy optimization model. Finally, the D3QN algorithm based on Q-value learning in deep reinforcement learning is used to optimize the merging strategy, and simulation experiments are designed to verify the effectiveness of the proposed merging strategy optimization method.

[0057] (I) Definition of a split-flow and merge-flow production system model considering workpiece quality

[0058] This invention relates to a split-flow and merging production system that considers workpiece quality. A schematic diagram of the production system model is shown below. Figure 1As shown. The research objects include one merging station, represented by a shaded circle; two automated stations, represented by blank circles; three work-in-process buffer zones, represented by rectangles; and workpieces of different quality grades in the buffer zones, represented by shaded triangles. The darker the shade, the higher the quality grade of the workpiece and the easier it is to process; the lighter the shade, the lower the quality grade of the workpiece and the more difficult it is to process.

[0059] The production system is divided into two stages: The first stage is handled by an automated workstation M1. Workstation M1 produces qualified workpieces with probability p1 and sends them to buffer zone b2; it produces unqualified workpieces with probability 1-p1 and sends them to buffer zone b1. The second stage consists of a merging workstation M2 and an automated workstation M3. Automated workstation M3 produces qualified workpieces with probability p3 and sends them downstream in the production system; it produces unqualified workpieces with probability 1-p3 and sends them to buffer zone b3. The processing efficiency of the merging workstation M2 varies depending on the quality level of the workpieces in buffer zones b1 and b3, and it needs to select workpieces from either buffer zone b1 or b3 for processing.

[0060] The model definition of a split-flow and merge-flow production system considering workpiece quality is as follows:

[0061] Workpiece:

[0062] 1) The set of workpiece quality grades Q = {q1, q2, ..., q} thre ,…,q num}, where num is the number of quality grades, and thre represents the quality grade threshold for acceptable workpieces. q + ∈Q + ={q thre+1 ,…,q num} represents the quality grade of a qualified workpiece, q - ∈Q - ={q1,q2,…,q thre} indicates the quality grade of a defective workpiece. This invention assumes that a qualified workpiece has a unique quality grade, while defective workpieces have different quality grades.

[0063] Buffer:

[0064] 2) Buffer b i If 1 ≤ i ≤ 3, the buffer capacity is B. i The queue arrangement rule in the buffer is (First In, First Out, FIFO). b represents the buffer at time t. i Sequence information of workpieces of different quality grades.

[0065] b2(t) represents the number of workpieces in buffer b2 at time t.

[0066] Workstation:

[0067] 3) M1 and M3 represent automated workstations, and M2 represents a merging workstation. α i (t) represents workstation M i The state of i = 1, 2, 3 at time t.

[0068] 4) The processing cycle time (τ) of a workstation represents the time required to process one workpiece. It is assumed that the processing cycle time τ is the same for all workstations in the production system, and that a piecewise time axis model is used, dividing the time into segments with one processing cycle as the unit length. The workstation's state α... i (t) The state of the buffer is determined at the beginning of each processing cycle. This is determined at the end of each processing cycle.

[0069] 5) Workstation M2 is a merging workstation, which needs to make a decision to extract a workpiece from a buffer in the upstream parallel process for processing. At the beginning of each processing cycle, the decision variable of workstation M2 is a(t).

[0070]

[0071] 6) Workstations M1 and M3 follow a Bernoulli reliability model with Bernoulli parameter p. mi Let i = 1, 3, representing the probability that a workstation operates normally within a processing cycle τ. Workstations M1 and M3 are automated workstations, subject to processing quality issues and following a Bernoulli quality model, meaning the probability of producing a qualified workpiece is p. i For i = 1, 3, the probability of producing defective workpieces is 1 - p. i .

[0072] 7) Workstation M2 follows a Bernoulli reliability model with Bernoulli parameter p. m2 (t) represents the probability that a selected defective workpiece will be processed within a processing cycle τ. The Bernoulli parameter of workstation M2 will vary depending on the quality level of the defective workpiece.

[0073] 8) The state α of workstations M1 and M3 at time t i (t), i = 1, 3 can be obtained by using Bernoulli parameter p mi p i The Bernoulli random experiment is used to determine this. For example, at the beginning of each processing cycle, station M1 is determined with probability 1-p. m1 In fault state

[0074] α1(t) = 0; with probability p m1When in working condition, a workpiece of acceptable quality (α1(t) = 1) will be produced with probability p1; when in working condition, a workpiece of unacceptable quality (α1(t) = q) will be produced with probability 1-p1, and the unacceptable workpieces contain a certain quality grade. - .

[0075]

[0076] 9) The state α2(t) of station M2 at time t can be determined by the decision variable a(t) and the Bernoulli parameter p. m2 The Bernoulli random experiment is used to determine (t). For example, at the beginning of each processing cycle, the quality grade of the currently processed defective workpiece is determined based on the decision a(t) of station M2, and the Bernoulli parameters of station M2 are updated accordingly.

[0077] And based on the updated Bernoulli parameter p m2 The state α2(t) of station M2 at time t is determined by the decision a(t) and the decision a(t). a(t) represents the quality grade of the last defective workpiece in the buffer corresponding to a(t), f(·) represents the functional relationship between the Bernoulli parameter of workstation M2 and the quality grade of the defective workpiece, p m2 It is the initial Bernoulli parameter of workstation M2, which is a constant of 1.

[0078]

[0079] 10) Workstation flow splitting (merging) function h(·): The index corresponding to the buffer zone for workpiece inflow (outflow). For workstation M1,

[0080] M3 has a processing pass rate p i The downstream workpieces will be diverted, and h(·) indicates that workstations M1 and M3 will send the workpieces to buffer zone b at time t. h(·) For example, h(α1(t)=q - This indicates that workstation M1 has produced a defective workpiece and has sent it to the buffer zone. For workstation M2, a certain merging strategy is followed to select workpieces from the upstream buffer for processing. The upstream workpieces will merge. h(·) represents the number of workstations M2 select from buffer b at time t. h (·) Select a workpiece for processing, such as h(a(t)=1) indicating that workstation M2 starts from buffer b. h(a(t)＝1) =b3 Select the workpiece for processing.

[0081] 11) Blocked Before Service (BBS): During a processing cycle, if workstation M...i If a workstation is in operation, but its downstream buffer was full at the end of the previous processing cycle, and the downstream workstation cannot select a workpiece from the buffer for processing at the start of this processing cycle, then workstation M... i If a processing cycle is blocked and processing is impossible, it is assumed that the downstream station will not be blocked. Specifically, if station M1 is blocked in buffer b1 or b2 at the start of the current processing cycle, it is necessary to determine whether station M1 is still blocked in buffer b1 or b2 at the start of the next processing cycle; if station M3 is blocked in buffer b3 at the start of the current processing cycle, it is necessary to determine whether station M3 is still blocked in b3 at the start of the next processing cycle.

[0082] 12) Within one processing cycle, workstation M i If workstation M is in a working state and its upstream buffer was empty at the end of the previous processing cycle, then workstation M... i During this processing cycle, the system is in a state of starvation, assuming that the upstream station will not be starved. Specifically, station M1 is at the upstream of the production system and will not be starved; at the start of the current processing cycle, there are no workpieces in buffers b1 and b3, so station M2 is starved; at the start of the current processing cycle, there are no workpieces in buffer b2 and station M3 is working normally, so station M3 is starved.

[0083] The meanings and explanations of production system symbols are shown in Table 2:

[0084] Table 2. Meaning and Explanation of Symbols in the Production System

[0085]

[0086]

[0087] (II) Constructing a Production System Performance Evaluation Model

[0088] • Analyze the state transitions of the production system

[0089] During the operation of the production system, changes in the buffer state are caused by changes in the workstation state. For buffer b1, when upstream workstation M1 works and produces a workpiece that is substandard, the quantity of buffer b1 increases. When downstream workstation M2 selects a substandard workpiece from buffer b1 and successfully processes it, the quantity of buffer b1 decreases. Whether the workpiece processed by the upstream workstation is qualified or not is random, and the decisions made by downstream workstation M2 are constantly changing. The interaction between upstream and downstream workstations causes changes in the buffer state. Because Bernoulli workstations are memoryless, meaning that the state of a workstation at any given time is independent of its previous state, the state of the production system can be represented by the buffer state, and the state transition probability of the production system can be determined by the state transition probabilities of all workstations.

[0090] Based on the definition of the production system model and the characteristics of Bernoulli's workstation, a Markov chain is used to characterize the state transition process of the production system. The state of the production system at time t consists of the sequence of workpieces of different quality grades in buffers b1 and b3, and the number of workpieces in buffer b2, i.e.

[0091] The status of all workstations at time t is Specifically, if at time t-1, workstation M2 completes the processing of the defective workpiece, i.e., α2(t-1) = 1, 2, then at time t, workstation M2 can make a decision a(t), selecting a workpiece with a certain quality level from the target buffer according to the FIFO rule for processing, thereby updating the Bernoulli parameter of workstation M2 at time t, and based on the updated Bernoulli parameter p... m2 (t) yields the state α2(t) of workstation M2 at time t. If workstation M2 has not finished processing the defective workpiece at time t-1, i.e., α2(t-1) = 0, then workstation M2 cannot make a decision at time t, a(t) = a(t-1), and repeats the previous pick-up action. Workstation M2 continues to process the workpiece that was not finished at time t-1, and the Bernoulli parameter is not updated. p m2 (t)=p m2 (t-1), and the Bernoulli parameter p at the previous time step. m2 The state α2(t) of workstation M2 at time t is determined by (t-1). Based on this loop logic, the state α2(t) of workstation is updated. The range of values ​​for the workstation state parameters is shown in Table 3.

[0092] Table 3. Range of values ​​for workstation status parameters

[0093] Workstation status <![CDATA[α1(t)]]> <![CDATA[α2(t)]]> <![CDATA[α3(t)]]> Range of values <![CDATA[0,1,q - ]]> 0,1,2 <![CDATA[0,1,q - ]]>

[0094] At time t, all workstations transition to state. The probability is:

[0095]

[0096] p(α2(t)|α2(t-1),a(t))=g1(p m2 (t)) (2)

[0097] p(α i (t)|α i (t-1))=p(α i (t))=g2(p mi ,p i ), i = 1, 3 (3)

[0098] Where g1(·) represents the functional relationship between the state transition probability of workstation M2 and the Bernoulli parameter, and g2(·) represents the functional relationship between the state transition probabilities of workstations M1 and M3 and the Bernoulli parameter and the processing qualification rate.

[0099] The production system status is changed by Transition to state The probability is:

[0100]

[0101] When the production system is in a state At time t+1, the production system transitions to state . The total number of possible states is 2 × (thre + 1). 2 2 represents the maximum number of state transitions for workstation M2, and thre+1 represents the maximum number of state transitions for workstations M1 and M3, which is 2×(thre+1). 2 This represents the total number of possible combinations of state transitions at the workstation. Calculating the state transition probability of the production system at each time step requires a maximum of 2×(thre+1) steps. 2 This operation. Based on the state. Calculate the transition to the state The algorithm for determining the probability and updating the state of the workstation and production system at each time step is as follows:

[0102]

[0103]

[0104] • Computing system performance indicators

[0105] The performance indicators of the production system are analyzed based on the state transition process of the production system. The performance indicators involved include both transient and steady-state performance indicators, as detailed below:

[0106] 1) Output (Throughput, TH): At time t, the output of station M i The number of workpieces produced is TH i (t), i = 1, 2, 3; Workstation M i The number of workpieces produced by a certain branch is Workstation M i Total output is

[0107]

[0108] 2) Buffer b i Average Work in Process (WIP) i ): At time t, buffer b iThe number of work-in-process inventory (WIP) is i (t); Under steady-state operation of the production system, buffer b i The average work-in-process level is Where T is the total operating time of the production system.

[0109] 3) Total Work in Process (WIP): The average total occupancy of all buffer zones under steady-state operation of the production system.

[0110] 4) Workstation M i Blocking rate (Blockage, BL) i ): The probability that a workstation is in a working state, the downstream buffer is full, and the downstream workstation cannot select a workpiece under steady-state operation of the production system.

[0111] 5) Workstation M i Starvation (ST) i ): The probability that a workstation is in a working state and the upstream buffer is empty under steady-state operation of the production system.

[0112] Analyze the output of the production system:

[0113] The output TH1(t) of station M1 at time t depends on the output branch of station M1. and the remaining capacity of the downstream buffer for each output branch.

[0114]

[0115] The output TH2(t) of station M2 at time t depends on the output branch of station M2. and the capacity WIP of the upstream buffer for each output branch. h(a(t)) (t-1):

[0116]

[0117] The output TH3(t) of station M3 at time t depends on the output branch of station M3. The upstream buffer capacity WIP2(t-1) and the remaining capacity of the downstream buffer for each output branch.

[0118]

[0119] The output TH(t) of the production system at time t is the output TH2(t) of station M2 and the output branch TH of station M3. 31 The sum of (t):

[0120] TH(t)=TH2(t)+TH 31 (t) (8)

[0121] Analyze the work-in-process level in the buffer zone of the production system:

[0122] For buffer b1, the work-in-process quantity WIP1(t) at time t depends on the output branch TH of station M1. 1q- (t) and the output branch TH of station M2 20 (t):

[0123] WIP1(t)=WIP1(t-1)+TH 1q -(t)-TH 21 (t) (9)

[0124] For buffer b2, the work-in-process quantity WIP2(t) at time t depends on the output branch TH of station M1. 11 (t) and the output TH3(t) of station M3:

[0125] WIP2(t)=WIP2(t-1)+TH 11 (t)-TH3(t) (10)

[0126] For buffer b3, the work-in-process quantity WIP3(t) at time t depends on the output branch TH of station M2. 22 (t) and the output branch TH of station M3 3q- (t):

[0127] WIP3(t)=WIP3(t-1)+TH 22 (t)-TH 3q -(t) (11)

[0128] Therefore, the number of buffers (WIP) in the production system at time t is... i The expression for (t) is as follows:

[0129]

[0130] Analyze the blocking rate and starvation rate of the workstations:

[0131] For workstation M i Given the blocking rate, station M1 may block in buffer b1 or possibly in buffer b2; station M2, being at the very bottom of the production system, will not block; station M3 may block in buffer b3. Therefore, at time t, station M... i i = 1, 3 in the buffer The indicator of whether it is blocked is Where 0 and 1 represent workstation M respectively.i In the buffer Not blocked, blocked. Under steady-state operation of the production system, workstation M... i The blocking rate is:

[0132]

[0133] For workstation M i The starvation rate is such that station M1, being at the top of the production system, will not experience starvation; station M2 will experience starvation when it is in a working state and buffers b1 and b3 are empty; station M3 will experience starvation when it is in a working state and buffer b2 is empty. Therefore, at time t, station M... i The sign of hunger is ST. i (t)∈{0,1},i=2,3, where 0 and 1 represent workstation M respectively. i Not hungry, hungry. Under steady-state operation of the production system, workstation M i The hunger rate is:

[0134] ST i =1 / T×∑ t ST i (t) (14)

[0135] (III) Constructing an optimization model for merging strategies

[0136] The state of the production system consists of the sequence of workpieces of different quality grades in buffers b1 and b3, and the number of workpieces in buffer b2. During the operation of the production system, the current state of the production system, the state of the workstations, and the merging decision actions of the workstations jointly determine the state of the production system at the next moment. After workstation M2 makes a decision, the production system will transition to the next state according to a certain transition probability. The transition probability depends only on the current state of the system, the state of the workstations, and the merging decision actions, and is independent of past states; the system's state transitions have one-step memory. Therefore, the merging decision problem of a production system considering workpiece quality exhibits the characteristics of sequential decision-making and can be described as a Markov decision process.

[0137] The inputs to a Markov decision process are the set of states of the production system, the set of merging decision actions, the set of transition probabilities of the production system, the reward function related to the state of the production system and the merging decision actions, and a discount factor considering future rewards. The output is the decision basis of station M2 at each time step. Based on the state transition process of the production system and Markov decision process theory, a discrete-time discrete-state finite-term Markov decision process model (S,A,P,R,γ) is constructed.

[0138] 1) S represents the set of system states:

[0139]

[0140] The state of the system at each moment

[0141] 2) A represents the set of merging decision actions:

[0142] A = {0, 1}, a(t) ∈ A, indicating that workstation M2 selects a workpiece from buffer b1 or b3 for processing.

[0143] 3) P represents the state transition probability of the production system:

[0144] At time t, the system state is When the action a(t) is taken, the transition occurs. probability

[0145]

[0146] 4) R represents the reward function:

[0147] At time t, the system state is The immediate reward after taking action a(t)

[0148]

[0149] 5) Gamma Discount Factor: The discount factor is used to discount the value of future rewards and to evaluate the value of future rewards relative to current rewards.

[0150] At time t, the decisions made by station M2 directly affect the work-in-process level in the production system, thus impacting the operational efficiency of each station. Furthermore, station M2 incurs switching costs when changing production processes. Therefore, the reward function... Profit per unit output of the production system Buffer unit work-in-process level cost Unit production switching cost of merging workstations The components are as follows:

[0151]

[0152] This represents the profit per unit of output, specifically the profit a production system makes from producing one workpiece within one processing cycle. This represents the unit cost of work-in-process, which is the cost of a single workpiece within each processing cycle of the production system. This represents the unit production changeover cost. It is the production changeover cost incurred when the decision made by workstation M2 in one processing cycle differs from the decision made in the previous processing cycle. TH(t) represents the output of the production system at each moment; WIP(t) represents the sum of all buffer work-in-process levels at each moment; I(t)∈{0,1} indicates whether workstation M2 performs a production changeover at each moment, where 0 indicates no changeover and 1 indicates a changeover.

[0153] A finite-duration episode contains T discrete moments. Workstation M2 continuously interacts with the production system simulation environment, receiving a series of rewards r(1),…,r(t),…,r(T) from the start to the end of the interaction. Based on the rewards of an episode, the sum of discounted rewards U from moment t onwards can be calculated. t = r(t) + γr(t+1) + γ 2 r(t+2)+…+γ T-t r(T), where γ represents the discount factor.

[0154] A Markov decision model outputs a policy function that maps states to actions. The value function It is a discount reward U t The expected outcome. The value function and policy together guide the agent in each state. Which action a(t) should be taken to maximize the cumulative reward? The Bellman equation describes the recursion of the value function, where the value of the current state equals the value of the next state plus the immediate reward of the current state, as shown in the following equation. Using the recursion of the Bellman equation, a Markov decision process model can be solved using dynamic programming.

[0155]

[0156] When the production system does not consider the workpiece quality grade, the system state can be simplified to: Where b i (t) represents the number of workpieces in each buffer. The system's state space contains (B1+1)(B2+1)(B3+1) possibilities. The problem size is relatively small, and dynamic programming can be used to solve this small-scale problem and obtain an effective merging decision scheme π. * When the production system considers the quality grades of defective workpieces, the system state space increases dramatically, encompassing... The transition probabilities of high-dimensional data are complex and difficult to solve. Deep reinforcement learning is a technique that combines deep learning and reinforcement learning. It uses deep neural networks to approximate value functions or policy functions, can handle high-dimensional and complex input data, is suitable for continuous and nonlinear environments, and can discover good policies in unknown state spaces, effectively solving the "curse of dimensionality" problem.

[0157] (iv) Solving Deep Reinforcement Learning Algorithms

[0158] The model constructed in this invention is a Markov decision process with discrete states and discrete actions. The D3QN algorithm, based on Q-learning in deep reinforcement learning, is used to solve the model. Through continuous interaction between the agent and the production system simulation environment, the network parameters are adaptively adjusted to learn the optimal action value function Q. * And the optimal strategy π is derived using the greedy search method. * .

[0159] Q-learning is a process of iteratively solving the Bellman optimal equation, with the aim of learning the optimal action-value function. This determines which action the agent should take in each state, thereby maximizing the cumulative reward. The specific expression for the optimal action value function is as follows:

[0160]

[0161] DQN uses neural networks to fit the optimal action value function. The expectation in the Bellman optimal equation is approximated using the Monte Carlo method, and its specific expression is as follows:

[0162]

[0163] in Describes the optimal action value function Q * An approximation, representing the state of the production system. The value of time station M2 after taking action a(t) This represents the parameters of each layer in the neural network. The TD target value of DQN. The following formula represents the data labels used when training the network.

[0164]

[0165] The DDQN dual-network structure, proposed by DeepMind in 2013, breaks the bootstrapping mechanism of Q-learning and decouples action selection from Q-value estimation, thus solving the problem of uneven Q-value overestimation. D3QN combines the advantages of DDQN with adversarial networks, approximating the optimal action value function. Decomposed into an approximation of the optimal state value function Approximation of the optimal advantage function The sum. The network structure of the showdown is as follows: Figure 2 As shown, the expression for fitting the optimal action value function using a duel network is as follows. Introducing a duel network can improve training stability and better explore the state space, thereby improving learning performance.

[0166]

[0167] The key to optimizing the merging strategy of a split-and-merge production system based on the D3QN algorithm is to utilize a neural network as the decision-making basis for workstation M2, continuously interacting with the production system simulation environment to obtain the data needed to train the neural network. Specifically, workstation M2 is treated as an intelligent agent, and at each moment, it determines the decision-making basis based on the state of the production system. The merging decision action a(t) is applied to the environment, and the environment provides feedback to station M2 with the immediate reward r(t) and the state at the next moment. Workstation M2 continuously interacts with the environment to obtain quadruple data. The experience replay technique continuously stores quadruple data into the experience pool. This technique breaks the correlation between data sequences, enables the neural network to converge quickly, and has a strong generalization ability.

[0168] For the neural network training process, an evaluation network is used to select actions based on the system state, and a target network is used to calculate the TD target value of D3QN, thus constructing a loss function. During training, data from four-tuples are sampled in batches and applied to the loss function. Mini-batch gradient descent is used to update the neural network parameters, thereby improving the evaluation of the network's predictions. Continuously approaching the true value Ultimately, the goal is to optimize the strategy. Specifically:

[0169]

[0170]

[0171] In the formula, Indicates the evaluation of network parameters, Represents the target network parameters, a * It utilizes the action selected by the evaluation network. The merging strategy optimization algorithm based on D3QN is as follows:

[0172]

[0173]

[0174] (V) Experimental Simulation and Result Analysis

[0175] Simulation experimental environment configuration

[0176] 1) Operating System: Windows 10 Professional (64-bit);

[0177] 2) Processor configuration: Intel(R) Core(TM) i7-6700 CPU (3.40GHz);

[0178] 3) System memory: 8GB;

[0179] 4) Development tools: PyCharm 2018.2;

[0180] 5) Programming language: Python 3.6;

[0181] 6) Python libraries: numpy, random, torch 1.10.2, etc.

[0182] Simulation experiment parameter configuration

[0183] Numerical simulation experiments were conducted to study the π merging strategy. * To verify the effectiveness, the specific parameters for the experiment were set as follows:

[0184] 1) The defective workpieces include 6 different quality grades, namely q - ∈{2,3,…,7} and follow a uniform distribution;

[0185] 2) The processing cycle for all workstations is 1 minute, each simulation round lasts 10,000 minutes, and each strategy interacts with the production system for 1,000 rounds. Specific parameter settings for the production system are shown in Tables 4-6.

[0186] Table 4 Processing parameters of the workstation

[0187]

[0188] Table 5 Buffer Capacity Parameters

[0189]

[0190] Table 6 Revenue Parameters

[0191]

[0192] The D3QN algorithm's dueling neural network includes a dominance function network. and a value function network The advantage function network and the value function network share the same input and output layers, and both have three hidden layers. The specific number of neurons in each layer is either 64, 64, 32, 2, 2 or 64, 64, 32, 1, 2. The ReLU activation function is used. The parameter settings for training the neural network using the D3QN algorithm are shown in Table 7.

[0193] Table 7 D3QN Algorithm Parameter Settings

[0194]

[0195] To verify the merging strategy π * The effectiveness of combining heuristic merging strategies in the production system with the proposed merging strategy π. * A comparison is made. The specific details of the heuristic confluence strategy are as follows:

[0196] 1)π buffer Workstation M2 makes decisions based on the occupancy of buffers b1 and b3, selecting the workpiece for processing from the buffer with the higher occupancy.

[0197] 2)π b1 Workstation M2 first selects a workpiece from buffer b1 for processing. When there is no workpiece in buffer b1, workstation M2 then selects a workpiece from buffer b3 for processing.

[0198] 3)π b3 Workstation M2 prioritizes selecting workpieces from buffer b3 for processing. When there are no workpieces in buffer b3, workstation M2 then selects workpieces from buffer b1 for processing.

[0199] 4)π random Workstation M2 randomly selects workpieces from buffer zones b1 and b3 for processing;

[0200] Simulation results and performance index analysis

[0201] Figure 3 This indicates the trend of the loss function when optimizing the merging strategy using the D3QN algorithm. As the training process progresses, the parameters of the neural network are continuously updated, the loss function gradually decreases and converges, and the predicted optimal action value gradually approaches the true optimal action value. Figure 4 This indicates the trend of reward changes for workstation M2 each round when optimizing the merging strategy using the D3QN algorithm. Workstation M2 receives a reward each round it interacts with the production system simulation environment. As the predictive ability of the neural network improves, workstation M2 continuously enhances its decision-making level, effectively balancing the relationship between profit and cost, thereby increasing the reward received each round and eventually converging to a certain level.

[0202] Table 1 shows the production system performance indicators obtained by interacting with different merging strategies. Experimental results show that π * Policy compared to policy π b3 π b1 π random π buffer Total revenue increased by 5.9%, 9.2%, 9.3%, and 9.9%, respectively. Figures 5-9These figures represent a comparison of workstation output, workstation blockage rate, workstation starvation rate, total work-in-process inventory level, and workstation M2 production switchover times in the production system under different heuristic control strategies. Different heuristic strategies have varying advantages and disadvantages in their impact on the production system, while π... * A strategy can balance the relationship between profit and cost, and take effective actions to maximize the total revenue of the production system. Specifically:

[0203] π buffer The strategy uses the upstream buffer occupancy level as the decision-making basis, prioritizing the selection of workpieces for processing from upstream buffers with high occupancy. This effectively prevents the upstream buffer from being full, alleviating congestion at workstation M2 in buffer b1 and at workstation M3 in buffer b2. As the congestion at workstation M3 is alleviated, its utilization rate increases, thereby reducing the work-in-process level in buffer b2 and indirectly alleviating congestion at workstation M2 in buffer b2. π buffer This directly alleviated the congestion of station M1 at b1, and indirectly alleviated the congestion of station M1 at b2. Since station M2 is at the upstream of the production system, its congestion was effectively alleviated, thus increasing the output of the production system. However, π buffer Such strategies can lead to frequent production changes and high total work-in-process inventory levels, ultimately resulting in lower overall revenue for the production system.

[0204] π b1 The strategy prioritizes buffer selection, prioritizing workpieces from buffer b1 for processing. This minimizes the work-in-process level in buffer b1, alleviating congestion at workstation M1. However, when buffer b1 contains workpieces, buffer b3 becomes uncontrolled, increasing its work-in-process level and exacerbating congestion at workstation M3, thus reducing its utilization. Consequently, the work-in-process level in upstream buffer b2 also increases, further worsening congestion at workstation M1. Simulation experiments show that π b1 Strategies can also enable production systems to achieve high output. However, π b1 Such strategies can lead to frequent production changes and high total work-in-process inventory levels, resulting in lower overall revenue for the production system.

[0205] π b3 The strategy prioritizes buffer selection, prioritizing workpieces from buffer b3 for processing. This minimizes the work-in-process level in buffer b3, alleviating congestion in workstation M3 and maximizing its utilization, while reducing the work-in-process level in buffer b2. π b3The strategy directly affects the work-in-process level in buffer b3 and indirectly affects the work-in-process level in buffer b2, thereby reducing the total work-in-process inventory level. However, when there are workpieces in buffer b3, buffer b1 becomes uncontrolled, and its work-in-process level continues to rise, increasing the congestion of workstation M1 in buffer b1, resulting in lower output. Simultaneously, π... b3 The strategy may also involve frequent production switching, resulting in a smaller overall benefit for the production system.

[0206] π * The strategy comprehensively considers the unit output profit, unit work-in-process cost, and unit production changeover cost of the production system. Based on the real-time status of the production system, it weighs the relationship between these three factors and selects effective actions. While ensuring the output of the production system, it reduces the total work-in-process inventory level and the production changeover frequency of workstation M2, thereby maximizing the total revenue of the production system.

[0207] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for optimizing the merging strategy of a production system considering workpiece quality, characterized in that: Includes the following steps: Step 1: Establish a model of the split-flow and merge-flow production system that considers workpiece quality, including Step 1.1: Define the basic parameters of the production system, including workpiece quality grade, buffer capacity and arrangement rules within the buffer, workstation reliability model, processing pass rate, splitting and merging function, and the functional relationship between the Bernoulli parameter of the merging workstation and the quality grade of the defective workpiece. Step 1.2: Define the workstation status and buffer status at each time point; Step 1.3: Define the production system operation logic, including the interaction between workstations, workpiece flow, and buffer zones; Step 2: Construct a production system performance evaluation model, including Step 2.1: Use buffer states to represent the production system state, and use Markov chains to characterize the state transition process of the workstation; Step 2.2: Based on the characteristics of the Bernoulli station, derive the state transition process of the production system using the station's state transition process, and calculate the state transition process of the production system from state to state. Transition to state Based on all transition probabilities, the update scheme for the workstation state and buffer state at each moment is derived, and the computational scale of the production system state transition probability is analyzed. Step 2.3: Quantify the transient performance indicators of the production system, including workstation output and work-in-process inventory in the buffer zone; Based on the state transition process of the production system, steady-state performance indicators are derived from the transient performance indicators of the production system, including total output of workstations, average work-in-process level in buffer zones, total work-in-process inventory level, congestion rate, and starvation rate; the production system performance indicators are used as the basis for improving the merging strategy optimization model. Step 3: Construct a merging strategy optimization model, including Step 3.1: Based on the state transition process of the production system, construct a simulation environment for the production system, use it as an environmental element in the Markov decision process, and use the merging workstation as an agent element in the Markov decision process; Step 3.2: Design a reward function based on the unit output profit, unit work-in-process cost, and unit production switching cost of the production system. After the agent makes a decision at each moment, the environment provides an immediate reward to the agent based on the agent's action and moves to the next state. Step 3.3: Based on the Bellman optimal equation, with the goal of maximizing the optimal action value function in the Bellman optimal equation, the process of the agent continuously interacting with the production system simulation environment to obtain rewards is constructed as a merging strategy optimization model. Step 4: Solve the merging strategy optimization model using the D3QN algorithm; Step 4.1: Use a duel neural network to fit the optimal action value function in the Bellman optimal equation, and decompose the optimal action value approximation function into the sum of the optimal state value approximation function and the optimal advantage approximation function; Step 4.2: Approximate the expectation of the Bellman optimal equation using the Monte Carlo method; Step 4.3: Adopt a dual-network structure and use the TD algorithm to train the neural network so that the neural network prediction value continuously approaches the true value, thereby maximizing the optimal action value function and optimizing the merging strategy.

2. The method for optimizing the merging strategy of a production system considering workpiece quality according to claim 1, characterized in that: It also includes steps to verify the effectiveness of the merging strategy optimization method: Step 5.1: Initialize the simulation environment parameters and D3QN algorithm parameters of the production system; Step 5.2: For the initialized production system simulation environment, based on the production system performance evaluation model, construct a merging strategy optimization model, and use the D3QN algorithm to solve the merging strategy optimization model to obtain strategy π. * ; Step 5.3: Transfer strategy π * Heuristic strategies different from those in production systems π buffer π priority π random To make a comparison, the system interacted with the production system simulation environment, and the transient performance indicators of each interaction were statistically analyzed. The steady-state performance indicators were then calculated based on the transient performance indicators, which served as the criteria for evaluating the merits of the strategies. Step 5.4: Analyze the performance indicators of the production system under different control strategies to verify strategy π. * The effectiveness.

3. The method for optimizing the merging strategy of a production system considering workpiece quality according to claim 1, characterized in that: In step 1.1, the production system includes a merging station, an automated station, and a buffer zone. The automated station is capable of processing qualified workpieces and producing qualified workpieces with probability p and sending them to the first buffer zone or the downstream of the production system, and producing unqualified workpieces with probability 1-p and sending them to the second buffer zone. The merging station is capable of processing both qualified and unqualified workpieces and producing qualified workpieces and sending them to the downstream of the production system.

4. The method for optimizing the merging strategy of a production system considering workpiece quality according to claim 3, characterized in that: In step 1.1, the automated workstation follows the Bernoulli reliability model, and the corresponding Bernoulli parameter represents the probability that the workstation will work normally within a processing cycle; the automated workstation has processing quality problems and production follows the Bernoulli quality model; the merging workstation follows the Bernoulli reliability model, and the corresponding Bernoulli parameter represents the probability that the selected unqualified workpiece will be processed within a processing cycle.

5. The method for optimizing the merging strategy of a production system considering workpiece quality according to claim 4, characterized in that: There are several quality grades of defective workpieces, and the Bernoulli parameter of the merging station varies depending on the quality grade of the defective workpiece.

6. The method for optimizing the merging strategy of a production system considering workpiece quality according to claim 3, characterized in that: In step 3, the inputs of the Markov decision process are the set of states of the production system, the set of merging decision actions, the set of transition probabilities of the production system, the reward function related to the state of the production system and the merging decision actions, and the discount factor considering future rewards. The output is the decision basis of the merging station at each moment.

Images