Encapsulation test workshop dynamic feeding and production scheduling integrated optimization method based on heterogeneous double-agent collaboration

Abstract

The invention provides an encapsulation test workshop dynamic feeding and production scheduling integrated optimization method based on heterogeneous double-agent collaboration, and the method comprises the steps: firstly building a multi-target mathematical model of an encapsulation test workshop, and taking the minimization of the maximum flow time, the total mold change time and the total tardiness time as optimization targets; and then designing a feeding agent adopting a TD3 algorithm and a scheduling agent adopting an IDDQN algorithm, and realizing closed-loop feedback of dynamic feeding and production scheduling through collaborative optimization of the double agents. The feeding intelligent agent is responsible for dynamically adjusting the feeding batch and feeding interval of each product, and the scheduling intelligent agent realizes self-adaptive matching of workpieces and machines based on a composite scheduling rule. The system improves the convergence efficiency and the stability of the algorithm through state feature normalization, step-by-step reward design and a parameter migration mechanism. Compared with a traditional static scheduling method, the method can effectively cope with process coupling, frequent die change and equipment maintenance constraints in the production process, achieves efficient collaboration of production resources and intelligent optimization of the overall performance of a workshop, and has good expandability and industrial application value.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing in semiconductor packaging and testing workshops, specifically to an integrated optimization method for dynamic material feeding and production scheduling in packaging and testing workshops based on heterogeneous dual-agent collaboration. Background Technology

[0002] Semiconductor packaging and testing workshops are typically characterized by a wide variety of products, small batch sizes, complex processes, and tight delivery schedules. Frequent equipment changeovers and maintenance operations during production lead to a highly dynamic scheduling environment. Existing workshop scheduling methods mainly include rule-based heuristic algorithms and model-based mathematical programming methods. The former relies on human experience and is difficult to adapt to complex and changing production conditions; the latter involves large computational loads when solving high-dimensional nonlinear problems and struggles to respond to dynamic events in real time. In recent years, reinforcement learning technology has been introduced into the field of production scheduling, enabling it to automatically learn optimal strategies through interaction with the environment. However, existing research largely focuses on static scheduling or single-layer decision-making, failing to consider the coordinated optimization of material feeding and scheduling plans simultaneously, resulting in low resource utilization and frequent changeovers. Especially in the packaging and testing process, the changeover times between different products vary significantly, and equipment maintenance constraints are complex, making it difficult for single-level scheduling optimization to achieve global optimum. Therefore, it is necessary to propose an intelligent method that can achieve coordinated optimization of material feeding and scheduling in a dynamic environment to improve the overall efficiency and stability of the system. Summary of the Invention

[0003] Based on the above background, the purpose of this invention is to propose an integrated optimization method for dynamic material feeding and production scheduling in packaging and testing workshops based on heterogeneous dual-agent collaboration. This method aims to solve problems such as the disconnect between material feeding plans and production scheduling, frequent mold changes, and low equipment utilization in existing packaging and testing workshops. This method comprehensively considers actual operating conditions such as product feeding lot volume, feeding interval, machine mold change time, and equipment maintenance. It establishes a multi-objective scheduling model for the packaging and testing workshop with the optimization objectives of minimizing maximum flow time, total mold change time, and total delay time. By introducing a material feeding agent based on the TD3 algorithm and a scheduling agent based on the IDDQN algorithm, collaborative optimization between the material feeding layer and the scheduling layer is achieved. A state space, action space, and reward function are designed to guide the agents to autonomously learn the optimal strategy, and a parameter migration mechanism is adopted to improve the algorithm's convergence speed and solution stability. This method can achieve closed-loop feedback and adaptive adjustment of material feeding plans and scheduling decisions in complex dynamic environments, significantly improving the resource utilization and production efficiency of the packaging and testing workshop.

[0004] To achieve the above-mentioned technical features, the objective of this invention is as follows: A method for integrated optimization of dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration, characterized by comprising the following steps: Step 1: Design the feeding agent and use the TD3 reinforcement learning algorithm to generate a feeding plan for the continuous action space, including the adjustment value of the feeding lot amount and the adjustment value of the feeding interval for each product. Step 2: Design a scheduling agent and use the IDDQN reinforcement learning algorithm to select composite scheduling rules for the discrete action space to realize workpiece allocation and machine scheduling decisions; Step 3: The feeding agent generates an initial feeding plan and transmits it to the scheduling agent; Step 4: The scheduling agent performs multiple rounds of scheduling iterations based on the current material feeding plan, selects the optimal scheduling scheme and the corresponding workshop status, and feeds it back to the material feeding agent. Step 5: The feeding intelligent agent corrects the feeding plan based on the feedback information, forming a closed-loop optimization of feeding and scheduling; Step 6: Use a parameter transfer mechanism between adjacent feeding plans to transfer the optimal weights of the trained agent to the initial network of the next training, thereby improving the learning efficiency of the agent and the quality of the scheduling solution.

[0005] Preferably, the specific process of Step 1 is as follows: 1) Obtain state features Then, the state features are input into the TD3 online Actor network, and... Greedy strategy selects the optimal action ; 2) Rewards for interacting with the environment and new status and form an experience group ; 3) During the training phase of the TD3 algorithm, priority is calculated based on the temporal difference error of each sample in the empirical replay pool. And based on this, determine the probability of it being sampled. , specifically ,in To adjust the hyperparameters that influence the priority level, The total number of samples in the experience replay pool; 4) Sample a batch of samples from the empirical replay pool according to this probability and input them into the online system. In a network, calculate the target Values; two independent values ​​were introduced in TD3. network and And take the minimum of the two as the target. The value is calculated using the following formula:

[0006] in As a discount factor, For the goal network; 5) The network updates using the minimum mean square error as the loss. And whenever Network Update After that, The network updates once; 6) A soft update strategy is adopted, and its calculation formula is as follows:

[0007]

[0008] in This is the weighting factor.

[0009] Preferably, the specific process of Step 2 is as follows: 1) Obtain state features Then, the state features are input into the online network of IDDQN, which continuously learns and explores through the neural network to achieve... Greedy strategy selects the optimal action ; 2) Based on the optimal action Select the corresponding composite scheduling rule from the rule pool, and arrange the appropriate workpiece to the idle machine for processing according to the composite scheduling rule; 3) Receive instant rewards and new status and form an experience group Stored in the experience replay pool; 4) During the training phase of IDDQN, random samples are drawn from the experience replay pool. A sample is input into an online network to calculate the target. Value; the IDDQN algorithm uses a two-layer depth Network: A stable network generated from the target network Value estimation, while the online network is used for policy updates, thereby enabling the estimation of values. The dual constraints of value; its objective The value is calculated as follows:

[0010] in, and These represent the parameters of the online network and the target network, respectively. Discount factor; 5) The algorithm minimizes the prediction Values ​​and Objectives Mean square error of the value To update network parameters.

[0011] Preferably, the specific process of Step 4 is as follows: 1) Define the state space of the scheduling agent, including the following two states: ① Machine status: Includes the average utilization rate and standard deviation of all machines, the average mold change utilization rate and standard deviation of all machines, and the average idle rate and standard deviation of all machines; ② Workpiece status: Includes the average completion rate and standard deviation of all workpieces, and the estimated and actual delay rates of all workpieces; 2) Define the state space of the scheduling agent, including the following six composite scheduling rules: First, the machining center with workpieces to be processed is identified. Then, a suitable machine is selected according to the machine rule MR. Next, the delay status of all workpieces in the buffer is calculated. If there are severely delayed workpieces, the corresponding workpieces are selected for processing according to the urgency level ED. Otherwise, the workpieces are selected for processing according to the mold change cost-effectiveness RH. Among them, the machine rule includes MR1 ​​and MR2, and the urgency level includes ED1, ED2 and ED3. This constitutes the action space of the scheduling agent, namely 6 composite scheduling rules: MR1+ED1+RH, MR1+ED2+RH, MR1+ED3+RH, MR2+ED1+RH, MR2+ED2+RH, MR2+ED3+RH. 3) Design the reward function for the scheduling agent, including step-by-step rewards and a final reward: ① The step-by-step reward is designed based on four dimensionless indicators: actual workpiece delay rate, workpiece assessment delay rate, machine idle rate, and machine mold change rate. Improvement of any one of these indicators will generate a positive reward, while deterioration of any one indicator will generate a negative reward. If there is no change in any indicator, the step-by-step reward will be zero. ② The final reward is designed based on the optimization objectives of maximum flow time, total mold change time, and total delay time. For multi-objective optimization problems, they are first decomposed into multiple sub-problems, and then aggregated using a weighted sum to form the final reward.

[0012] Preferably, the specific process of Step 5 is as follows: 1) Define the state space of the feeding agent, including the following four states: ① Machine status: Includes the average utilization rate and standard deviation of all machines, the average mold change utilization rate and standard deviation of all machines, and the average idle rate and standard deviation of all machines; ② Workpiece status: Includes the average completion rate and standard deviation of all workpieces, and the estimated and actual delay rates of all workpieces; ③ Workshop congestion level: including the average waiting time for workpieces; ④ Overall scheduling performance: includes the maximum flow time under the current material feeding plan, the total mold change time under the current material feeding plan, and the total delay time under the current material feeding plan.

[0013] 2) Define the state space of the feeding agent, which consists of the following two dimensions: ① Adjustment value for feed lot quantity; ② Feeding interval adjustment value.

[0014] 3) Design the reward function for the feeding agent: Four sub-reward indicators are set for maximum flow time, total mold change time, total delay time, and material feeding timeout, and index normalization is used to ensure adaptability in different scenarios. Finally, the total reward is the equal-weighted average of the four indicators to comprehensively evaluate the overall quality of the material feeding plan.

[0015] Preferably, Step 6 specifically involves the following process: In the The optimal weight parameters obtained during training under the next feeding plan Migrate to the Initial network parameters for the next feeding plan This improves the learning efficiency of the agent and the quality of the scheduling solution.

[0016] The present invention has the following beneficial effects: This invention addresses the characteristics of packaging and testing workshops—multiple product types, small batches, and frequent mold changes—by proposing an integrated optimization method for dynamic material feeding and production scheduling based on heterogeneous dual-agent collaboration. This method achieves closed-loop optimization of material feeding and scheduling plans by constructing a collaborative framework between a material feeding agent and a scheduling agent, effectively improving the overall efficiency of workshop operations. Employing multi-objective optimization modeling, a hierarchical reward mechanism, and a parameter migration strategy enhances the algorithm's convergence speed and stability. Compared to traditional scheduling methods, this invention significantly reduces maximum flow time, total mold change time, and total delay time, improving equipment utilization and demonstrating significant engineering application value. Attached Figure Description

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Figure 1 Layout of the packaging and testing workshop.

[0019] Figure 2 TD3-IDDQN algorithm framework.

[0020] Figure 3 Overall framework diagram of the TD3 algorithm.

[0021] Figure 4 Flowchart of the operation of composite scheduling rules.

[0022] Figure 5 The calculation results of different algorithms on twelve examples. Detailed Implementation

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0024] Example 1: Please see Figure 1 This embodiment addresses the integrated scheduling problem of dynamic material feeding and production scheduling in a packaging and testing workshop. Specifically, the packaging and testing workshop consists of multiple processing centers, each containing multiple parallel machines that need to process multiple products simultaneously. Each product is composed of several lots, and the feeding batch and interval for each product must be determined during scheduling. The number of workpieces for each product is determined by the total number of lots and the number of lots in a single batch. All workpieces must complete multiple processes sequentially, and the processing time of each process is related to the product batch size and is subject to delivery date constraints; exceeding the deadline will result in a late payment penalty. Switching between different products requires mold changing operations, the time of which depends on the types of products being processed before and after; frequent mold changing will lead to a decrease in equipment utilization. At the same time, each machine also requires regular maintenance; if the processing or mold changing time overlaps with the maintenance window, the task must be postponed. Throughout the entire production scheduling process, this method mainly considers three optimization objectives: minimizing the maximum flow time. Total mold change time Total delay time .

[0025] To achieve the above-mentioned technical features, the objective of this invention is as follows: a method for integrated optimization of dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration, the method comprising the following steps: Step 1: Construct a feeding agent and use the TD3 reinforcement learning algorithm to generate a feeding plan for the continuous action space, including the adjustment value of the feeding lot amount and the adjustment value of the feeding interval for each product. Step 2: Construct a scheduling agent and use the IDDQN reinforcement learning algorithm to select composite scheduling rules for the discrete action space to realize workpiece allocation and machine scheduling decisions; Step 3: The feeding agent generates an initial feeding plan and transmits it to the scheduling agent; Step 4: The scheduling agent performs multiple rounds of scheduling iterations based on the current material feeding plan, selects the optimal scheduling scheme and the corresponding workshop status and feeds it back to the material feeding agent; Step 5: The feeding intelligence optimizes the feeding plan based on feedback information, forming a closed-loop optimization of feeding and scheduling; Step 6: Implement a parameter migration mechanism between adjacent feeding plans to transfer the optimal weights of the trained agent to the initial network of the next training, thereby improving the learning efficiency of the agent and the quality of the scheduling solution.

[0026] The specific process of Step 1 is as follows: 1) Obtain state features Then, the state features are input into the TD3 online Actor network, and... Greedy strategy selects the optimal action ; 2) Interact with the environment to receive instant rewards and new status and form them into empirical tuples ; 3) During the training phase of TD3, priority is calculated based on the temporal difference error of each sample in the empirical replay pool. And based on this, determine the probability of it being sampled. , specifically ,in To adjust the hyperparameters that influence the priority level, The total number of samples in the experience replay pool; 4) Sample a batch of samples from the empirical replay pool according to this probability and input them into the online system. In a network, calculate the target Value. To avoid To address the issue of overestimation, TD3 introduced two independent [measures / solutions]. network and And take the minimum of the two as the target. The value is calculated using the following formula:

[0027] in As a discount factor, For the goal network.

[0028] 5) The network updates using the minimum mean square error as the loss. , and when Network Update After that, The network updates once; 6) To ensure smooth convergence of the target network, TD3 adopts a soft update strategy, the calculation formula of which is as follows:

[0029]

[0030] in This is the weighting factor.

[0031] The specific process of Step 2 is as follows: 1) Obtain state features Then, the state features are input into the online network of IDDQN, which continuously learns and explores through the neural network to achieve... Greedy strategy selects the optimal action ; 2) Based on the optimal action Select the corresponding composite scheduling rule from the rule pool, and arrange the appropriate workpiece to the idle machine for processing according to the composite scheduling rule; 3) Receive instant rewards and new status and form an experience group Stored in the experience replay pool; 4) During the training phase of IDDQN, random samples are drawn from the experience replay pool. A sample is input into an online network to calculate the target. Value. To avoid To address the issue of overestimation, the IDDQN algorithm employs a two-layer depthwise approach. Network: A stable network generated from the target network Value estimation, while the online network is used for policy updates, thereby enabling the estimation of values. The dual constraint of value. Its objective. The value is calculated as follows:

[0032] in, and These represent the parameters of the online network and the target network, respectively. This is the discount factor.

[0033] 5) The algorithm minimizes the prediction Values ​​and Objectives Mean square error of the value To update network parameters.

[0034] The specific process of Step 4 is as follows: 1) Define the state space of the scheduling agent, including the following two states: ① Machine status: Includes the average utilization rate and standard deviation of all machines, the average mold change utilization rate and standard deviation of all machines, and the average idle rate and standard deviation of all machines; ② Workpiece status: Includes the average completion rate and standard deviation of all workpieces, and the estimated and actual delay rates of all workpieces.

[0035] 2) Define the state space of the scheduling agent, including the following six composite scheduling rules: First, the machining centers with workpieces awaiting processing are identified. Then, suitable machines are selected according to machine rules (MR). Next, the delay status of all workpieces in the buffer is calculated. If there are severely delayed workpieces, the corresponding workpieces are selected for processing according to their urgency level (ED). Otherwise, workpieces are selected for processing according to the mold change cost-effectiveness (RH). The machine rules include MR1 and MR2, and the urgency levels include ED1, ED2, and ED3. This constitutes the action space of the scheduling agent, namely six composite scheduling rules: MR1+ED1+RH, MR1+ED2+RH, MR1+ED3+RH, MR2+ED1+RH, MR2+ED2+RH, and MR2+ED3+RH.

[0036] 3) Design the reward function for the scheduling agent, including step-by-step rewards and a final reward: ① The step-by-step reward is designed based on four dimensionless indicators: actual workpiece delay rate, workpiece assessment delay rate, machine idle rate, and machine mold change rate. Improvement (decrease) in any of these indicators will generate a positive reward, while deterioration (increase) in any of these indicators will generate a negative reward. If there is no change in any indicator, the step-by-step reward is zero.

[0037] ② The final reward is designed based on the optimization objectives of maximum flow time, total mold change time, and total delay time. For multi-objective optimization problems, they are first decomposed into multiple sub-problems, and then aggregated using a weighted sum to form the final reward.

[0038] The specific process of Step 5 is as follows: 1) Define the state space of the feeding agent, including the following four states: ① Machine status: Includes the average utilization rate and standard deviation of all machines, the average mold change utilization rate and standard deviation of all machines, and the average idle rate and standard deviation of all machines; ② Workpiece status: Includes the average completion rate and standard deviation of all workpieces, and the estimated and actual delay rates of all workpieces; ③ Workshop congestion level: including the average waiting time for workpieces; ④ Overall scheduling performance: includes the maximum flow time under the current material feeding plan, the total mold change time under the current material feeding plan, and the total delay time under the current material feeding plan.

[0039] 2) Define the state space of the feeding agent, which consists of the following two dimensions: ① Adjustment value for feed lot quantity; ② Feeding interval adjustment value.

[0040] The specific process of Step 6 is as follows: In the The optimal weight parameters obtained during training under the next feeding plan Migrate to the Initial network parameters for the next feeding plan This improves the learning efficiency of the agent and the quality of the scheduling solution.

[0041] Example 2: See Figure 1-5 First, for ease of explanation, the following table will provide a unified explanation of the symbols introduced in this example: Model assumptions: Considering the characteristics and manageability of the actual problem, the following reasonable assumptions are made about the problem: (1) Each machine can only process one process at a time, and workpieces are not allowed to preemptively occupy the machine.

[0042] (2) At time zero, all machines are available and the initial processing attributes of the machines are the same as the product type attributes of the first processed machine.

[0043] (3) The mold change time for different products on all machines is known (the mold change time for some processes is not zero), and the mold change operation of the equipment is not allowed to be interrupted.

[0044] (4) The maintenance plans for all machines are known, and maintenance activities are not allowed to be interrupted once they begin.

[0045] (5) The total production volume and single lot capacity of all products are known.

[0046] (6) In order to facilitate the normal entry of workpieces into the processing workshop, the mold change time of the first processing center is set to zero.

[0047] (7) The buffer zone in front of all machining centers has a large enough capacity, that is, the workpiece can immediately enter the buffer zone in front of the next machining center after the machining is completed.

[0048] (8) The processing time of all product workpieces on different machining center machines is known, and the processing of workpieces is not allowed to be interrupted.

[0049] Symbol definition:

[0050] Scheduling model: The semiconductor packaging and testing workshop studied in this embodiment (workshop layout as follows) Figure 1The dynamic material feeding and production scheduling optimization problem (as shown) can be defined as follows: The workshop consists of... One machining center Composition, each machining center is equipped with parallel machine ( ), total This product requires processing ( Each product of Number of During scheduling, the feed lot quantity for each product must first be determined. and feeding interval This allows for the gradual release of workpieces into the production system. Therefore, the product... The number of workpieces is The total number of workpieces is All workpieces must be completed. In the first process, when the workpiece's first... After the first process is completed, you can proceed directly to the next step. A buffer zone for each process step. Meanwhile, each product individually... Processing time Given the processing time of the corresponding workpiece In addition, each workpiece Delivery period is specified for all. If the completion time of the workpiece Exceed If a delay occurs, a penalty will be imposed; otherwise, there will be no penalty.

[0051] Compared to classic production workshops, packaging and testing workshops offer a wider variety of products in smaller batches with tighter lead times, resulting in higher mold changeover frequencies. Since different products require different packaging and testing molds, a mold changeover operation is necessary if the product type changes before processing. This process typically involves fixture disassembly and assembly, equipment calibration, and other steps, which is time-consuming and incurs manpower and material costs. Frequent mold changes significantly reduce equipment utilization and production capacity, becoming a major bottleneck in workshop scheduling. Mold changeover time is a significant factor. Depends on the machine Product types for pre- and post-processing and Furthermore, each machine requires maintenance after a period of operation, and each machine... exist A maintenance activity, if the machine The machining or mold changing process and the machine's maintenance time window If there is overlap, the process must be postponed until maintenance is complete.

[0052] In summary, the integration problem of dynamic material feeding and production scheduling in the packaging and testing workshop studied in this embodiment mainly includes the following four sub-problems: (1) Dynamic material feeding, based on the total amount of each product (1) Task volume: Determine the appropriate amount of material to be fed and the feeding interval; (2) Workpiece selection: Assign appropriate workpieces to be processed to idle machines; (3) Equipment maintenance: Adjust the production schedule according to the maintenance plan to ensure equipment performance; (4) Machine mold change: Determine the mold change time required before and after the machine processes different products.

[0053] Objective function:

[0054] in,

[0055]

[0056]

[0057] Constraints:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] Equation (9) defines the first The number of workpieces for a product; Equation (10) defines the number of workpieces. Complete the first The end time of each process step; Equation (11) is used for calculating the machine. The total mold-changing time; Equation (12) defines the first... Workpieces in this type of product The This process is done on the machine. The relationship between the start and end times of processing; Equation (13) defines the first Workpieces in this type of product The The start time of the processing step on this machine is related to the machine's... The relationship between the earliest available time, mold change time, and maintenance time; Equation (14-16) is used to determine the relationship between the earliest available time, mold change time, and maintenance time; Workpieces in this type of product The This process is done on the machine. Whether the processing is affected by maintenance interruptions depends on the following three scenarios: Scenario 1: If the machine The mold changing activity and the machine's first If two maintenance activities overlap in time, perform maintenance first, then mold replacement, and then workpiece processing; Scenario 2: If the machine The mold changing activity and the machine's first Although the maintenance activities do not overlap, the workpiece The This process is done on the machine. The processing procedure and the first step of the machine If maintenance activities overlap in time, the mold should be replaced first, followed by maintenance, and then workpiece processing; Case 3: If neither of the above two cases is met, the machine should replace the mold first and then process the workpiece; Equation (17) is used for calculating the machine The The end time of the maintenance activity; Equation (18) describes the workpiece The total delay time; Equation (19) defines the product The range of the amount of material to be fed; Equation (20) defines the boundary of the feeding interval; Equation (21) indicates that each workpiece can only belong to one product type; Equation (22) indicates that each process can only be processed by one machine at a time; Equation (23) indicates that one machine can only process one process at a time; Equation (24) defines the binary decision variables; Equation (25) defines the range of values ​​for the key parameters.

[0075] ①TD3-IDDQN Algorithm Framework Based on the problem to be solved in this embodiment, the following design is provided: Figure 2 The TD3-IDDQN algorithm is shown. This algorithm consists of a top-level agent (the feeding agent) and a bottom-level agent (the scheduling agent), and its operating mechanism is as follows: First, the feeding agent generates an initial feeding plan and passes it to the scheduling agent; subsequently, the scheduling agent performs [further actions] based on this feeding plan. The scheduling process iterates through rounds, selecting the optimal scheduling result and corresponding workshop status, and feeding it back to the feeding agent. Finally, the feeding agent corrects the feeding plan based on the feedback information, forming a closed-loop optimization of feeding and scheduling.

[0076] ②Design and Operation Mechanism of Feeding Intelligent Agent In the TD3 algorithm (the overall framework is as follows) Figure 3 As shown), first obtain the state features. Then, the state features are input into the TD3 online Actor network, and... Greedy strategy selects the optimal action Then, interact with the environment to receive instant rewards. and new status and form an experience group During the training phase of the TD3 algorithm, the priority is first calculated based on the temporal difference error of each sample in the empirical replay pool. And based on this, determine the probability of it being sampled. , specifically ,in To adjust the hyperparameters that influence the priority level, This represents the total number of samples in the experience replay pool. Subsequently, a batch of samples is sampled from the experience replay pool according to this probability and input into the online... In a network, calculate the target Value. To avoid To address the issue of overestimation, TD3 introduced two independent [measures / solutions]. network and And take the minimum of the two as the target. The value is calculated using the following formula:

[0077] in, As a discount factor, For the goal network.

[0078] Immediately afterwards, The network updates using the minimum mean square error as the loss. And whenever Network Update After that, The network will be updated once.

[0079] Finally, to ensure the smooth convergence of the target network, a soft update strategy is adopted, and its calculation formula is as follows:

[0080]

[0081] in, This is the weighting factor.

[0082] ③ The state space of the feeding agent This embodiment constructs 14 statistical indicators to comprehensively characterize the operational status of the semiconductor packaging and testing workshop. Furthermore, to enhance the algorithm's versatility, each indicator is normalized. Specifically, status indicators 1-6 are designed based on machine status, characterizing equipment load levels and fluctuations; status indicators 7-10 are designed based on workpiece status, reflecting workpiece processing progress and delays; status indicator 11 is designed based on the average waiting time of workpieces, reflecting the workshop's congestion level; and status indicators 12-14 are designed based on historical scheduling performance, measuring the impact of current material feeding decisions on overall scheduling performance.

[0083] Status Indicator 1: Average Machine Utilization

[0084]

[0085] in, For machines The utilization rate is calculated using the following formula:

[0086] in, Indicates machine Machining workpieces The end time.

[0087] Status Index 2: Standard Deviation of Machine Average Utilization

[0088]

[0089] Status Indicator 3: Average Mold Change Rate of the Machine

[0090]

[0091] in, Indicates machine The mold change rate is calculated using the following formula:

[0092] in, Indicates machine Total mold change time at the current moment.

[0093] Status index 4: Standard deviation of the machine's average mold change rate

[0094]

[0095] Status Indicator 5: Average Machine Idle Rate

[0096]

[0097] in, For machines idle rate, For machines The maintenance rate is calculated using the following formula:

[0098] in, In the current Internal machine The number of times it has been maintained.

[0099] Status Indicator 6: Standard Deviation of Machine Average Idle Rate

[0100]

[0101] Status Index 7: Average Completion Rate of Workpieces

[0102]

[0103] in, For workpiece The completion rate is calculated using the following formula:

[0104] in, Indicates workpiece Number of completed processes.

[0105] Status Index 8: Standard Deviation of Average Completion Rate of Workpieces

[0106]

[0107] Status Index 9: Workpiece Assessment Delay Rate

[0108]

[0109] in, To assess the total number of overdue workpieces, To assess the total number of unfinished tasks. If the current time... If the sum of the average processing time of the remaining operations of a workpiece and its delivery date is greater than its delivery date, then the workpiece is considered a delayed workpiece; otherwise, it is not a delayed workpiece.

[0110] Status Index 10: Actual Delay Rate of Workpieces

[0111]

[0112] in, This represents the actual total number of overdue workpieces. This represents the total number of unfinished workpieces. If the end time of the current process is longer than its delivery date, the workpiece is considered a delayed workpiece; otherwise, it is not.

[0113] Status Indicator 11: Average waiting time of workpieces under the current material feeding plan

[0114]

[0115] in, Indicates the first The average waiting time of the workpiece under the material feeding plan in this iteration is calculated using the following formula:

[0116] in, Indicates workpiece In the The start time of each process step Indicates workpiece In the The end processing time of each step.

[0117] Status indicator 12: Maximum flow time under the current feeding plan

[0118]

[0119] in, Indicates the first The maximum flow time of the product under the next iteration of the feeding plan is calculated using the following formula:

[0120] Status indicator 13: Total mold change time under the current material feeding plan

[0121]

[0122] in, Indicates the first The total mold change time of the machine under the material feeding plan of the next iteration is calculated using the following formula:

[0123] Status indicator 14: Total delay time under the current material feeding plan

[0124]

[0125] in, Indicates the first The total delay time of the workpiece under the next iteration of the material feeding plan is calculated using the following formula:

[0126] ④ Action space of the feeding agent To dynamically optimize the feeding plan and take into account actual physical constraints, the action space of the feeding agent consists of two dimensions: the adjustment value of the feeding lot amount for each product and the adjustment value of the feeding interval. The specific definitions are as follows: 1) Feed lot quantity adjustment value: for each product Its feed lot adjustment value The range of values ​​is ,in This is a scaling factor, designed to prevent excessively large single adjustments from causing algorithm instability. When, it indicates that the amount of material added is increased based on the current lot quantity; When the current value is 0, it indicates a reduction in the amount of material fed; otherwise, the current lot quantity remains unchanged.

[0127] 2) Feeding interval adjustment value: All products adopt a uniform feeding interval, and its adjustment value is... The range of values ​​is .when When the time interval is extended, it indicates that the feeding interval has been extended. When the interval is short, it indicates that the feeding interval should be shortened; otherwise, the current feeding interval should remain unchanged.

[0128] ⑤ Reward Design for Feeding Agents The reward function evaluates the quality of an agent's actions in a given state, serving as environmental feedback to guide the agent in updating and optimizing its strategy. To guide the agent in learning the optimal strategy, the reward needs to be consistent with the optimization objective; that is, minimizing the objective corresponds to maximizing the reward. Furthermore, to improve the algorithm's adaptability to different scenarios, exponential normalization is applied to each metric. Simultaneously, to ensure the rationality of the material feeding plan, all workpieces must be fed within a given timeframe; otherwise, a timeout penalty will be incurred. The specific design is as follows: 1) Incentive Indicators , indicating the first Under the feeding plan of the next iteration, for the objective function The reward metric designed for (maximum flow time) is calculated using the following formula:

[0129] 2) Incentive Indicators , indicating the first Under the feeding plan of the next iteration, for the objective function The reward metric designed for (total mold change time) is calculated using the following formula:

[0130] 3) Incentive Indicators , indicating the first Under the feeding plan of the next iteration, for the objective function The incentive criteria designed for (total delay time) are calculated using the following formula:

[0131] 4) Incentive Indicators , indicating the first The reward index designed for workpiece feeding timeout under the next iteration of the feeding plan is calculated using the following formula:

[0132] in This indicates a penalty for exceeding the time limit. If all workpieces are fed within 12 hours (720 minutes), then... ,otherwise Furthermore, the penalties increase as the time limit for feeding materials increases.

[0133] 5) Total Rewards Since the four indicators are of equal importance, each indicator is assigned an equal weight in order to evaluate the overall performance of the system in a balanced manner.

[0134]

[0135] ⑥ Design and Operation Mechanism of Scheduling Agent In the IDDQN algorithm, the state features are first obtained. Then, the state features are input into the online network of IDDQN, where they are continuously learned and explored through the neural network. Greedy strategy selects the optimal action Then, based on the optimal action... Select the corresponding composite scheduling rule from the rule pool, and assign the appropriate workpiece to an idle machine for processing according to the composite scheduling rule; finally, receive an immediate reward. and new status and form an experience group Stored in the experience replay pool.

[0136] During the training phase of IDDQN, samples are first randomly drawn from the experience replay pool. A sample is taken, then fed into an online network, and the target is calculated. Value. To avoid To address the issue of overestimation, the IDDQN algorithm employs a two-layer depthwise approach. Network: A stable network generated from the target network Value estimation, while the online network is used for policy updates, thereby enabling the estimation of values. The dual constraint of value. Its objective. The value is calculated as follows:

[0137] in, and These represent the parameters of the online network and the target network, respectively. This is the discount factor. The algorithm minimizes the prediction. Values ​​and Objectives Mean square error of the value To update network parameters.

[0138] It is worth noting that in deep reinforcement learning, the action-value function is usually approximated by a neural network, so each problem will have corresponding network parameters trained. Since adjacent problems have similar state distributions and consistent optimization objectives, their optimal solutions are often close. Therefore, the network parameters of the already trained problems can be used to assist in solving the problem. Based on this, this embodiment adopts a neighborhood parameter transfer strategy: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] The optimal weight parameters obtained during training under the next feeding plan Migrate to the Initial network parameters for the next feeding plan This improves the learning efficiency of the agent and the quality of the scheduling solution.

[0139] ⑦ The state space of the scheduling agent This embodiment designs 10 normalized statistical indicators for the scheduling agent, covering both machine status and workpiece status. Specifically, machine status includes the following 6 indicators: 1) Average machine utilization. with standard deviation 2) Average mold change utilization rate of the machine with standard deviation 3) Average idle rate of the machine with standard deviation The workpiece status includes the following four indicators: 1) Average completion rate of the workpiece with standard deviation ;2) Delay rate in workpiece evaluation Compared with the actual delinquency rate .

[0140] ⑧ The action space of the scheduling agent Table 1 Basic Scheduling Rules

[0141] The scheduling problem in this embodiment can be decomposed into two sub-problems: machine selection and job allocation. Therefore, each scheduling rule needs to determine the matching method between machines and jobs based on the state. Specifically, as follows... Figure 4 As shown, first, the machining center with workpieces to be processed is identified. Then, a suitable machine is selected according to the machine rules (MR). Next, the delay status of all workpieces within the buffer is calculated. If there are severely delayed workpieces (i.e., ,in If the priority is MR1 ​​and MR2, then the corresponding workpiece will be selected for processing according to the urgency level (ED); otherwise, the workpiece will be selected for processing according to the mold change cost-effectiveness (RH). As shown in Table 1, the machine rules include MR1 and MR2, and the urgency level includes ED1, ED2 and ED3, which constitute the action space of the scheduling agent, namely 6 composite scheduling rules: 1) MR1+ED1+RH, 2) MR1+ED2+RH, 3) MR1+ED3+RH, 4) MR2+ED1+RH, 5) MR2+ED2+RH, 6) MR2+ED3+RH.

[0142] 9. Reward Design for Scheduling Agents This embodiment decomposes the objective function according to the scheduling result and process, realizes the step-by-step reward of the scheduling agent, and gives the final reward when the workpiece is completed.

[0143] 1) Step-by-step rewards Based on the actual delay rate of the workpiece Workpiece evaluation delay rate Machine idle rate and machine mold change rate The design incorporates four dimensionless indicators. Improvement (decline) in any indicator will generate a positive reward, while deterioration (increase) will generate a negative reward. No change in any indicator will result in zero reward. The specific calculation formula is as follows: ① Reward Indicators :

[0144] ② Reward Indicators :

[0145] ③ Reward Indicators :

[0146] ④ Reward Indicators :

[0147] ⑤ Step-by-step rewards :

[0148] 2) Final Reward Based on optimization objectives (Maximum flow time) (Total mold change time) and The design is based on (total delay time). For multi-objective optimization problems, it is first decomposed into multiple sub-problems, and then aggregated using a weighted sum to form the final reward. Specifically: ① Reward Indicators :

[0149] in, Indicates workpiece The time it takes for the water to flow through This represents the average processing time for all workpieces.

[0150] ② Reward Indicators :

[0151] in, Indicates that the workpiece The resulting machine mold change time, This represents the average mold change time for all machines.

[0152] ③ Reward Indicators :

[0153] ④ Final Reward :

[0154] Among them, weight This indicates that the three objectives are of equal importance, and given that each task generates [something] during the scheduling process... To highlight the dominant role of the final reward, the subsequent step-by-step rewards are distributed according to the number of processes during aggregation. Weighting is applied.

[0155] ⑩ Algorithm Example Comparison To verify the effectiveness of the strategy and algorithm proposed in this embodiment, test case sets for four quarters were randomly generated based on actual production conditions, totaling 12 cases: the first quarter (cases 1-3), the second quarter (cases 4-6), the third quarter (cases 7-9), and the fourth quarter (cases 10-12). Each quarter contains 3 sub-cases, denoted as "". The rules for generating the example are as follows: (1) The arrival time of the workpiece is determined by the material feeding plan; (2) The processing time of each process in a single lot is randomly selected from [10, 20]; (3) The processing time of a single workpiece is the processing time of the corresponding lot multiplied by the number of material feeding lots for that workpiece; (4) The delivery date is determined by the average processing time and the urgency level (i.e., (5) The mold changeover time between different products is within The process is randomly generated, and the mold changeover time is determined by the product type before and after the changeover. Each test case contains three products, each with different processing times. The number of completed workpieces is counted by recording the work logs of each buffer to clarify the completion status of each product.

[0156] This section selects the classic genetic algorithm and the IMDEA / IOD algorithm as comparison objects. By running all the above test examples, it was found (the specific calculation results are shown in Table 2): (1) In terms of IGD and HV index, the TD3-IDDQN algorithm is significantly better than the other two algorithms, and the average RPD value in the HV index is not lower than -23.41%; (2) The Wilcoxon test P values ​​of the TD3-IDDQN algorithm and the other two optimization algorithms in the HV index are 4.88E-04 and 1.61E-02, respectively, which indicates that there is a significant difference in performance between the algorithm in this paper and the other two optimization algorithms; (3) According to the rank test results (such as Figure 5 As shown in the figure, the average rank of the TD3-IDDQN algorithm is significantly higher than that of the other two optimization algorithms, indicating that the overall performance of the algorithm in this paper is the best; (4) Compared with Random, TD3-IDDQN can achieve better performance, which shows that the algorithm can learn effective action strategies and select action strategies that are adapted to the current environment.

[0157] Table 2. Calculation results of genetic algorithm and IMDEA / IOD algorithm

[0158] Although the preferred embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many specific modifications under the guidance of the present invention without departing from the spirit of the invention and the scope of protection of the claims, and these modifications all fall within the scope of protection of the present invention.

Claims

1. A method for integrated optimization of dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration, characterized in that, Includes the following steps: Step 1: Design the feeding agent and use the TD3 reinforcement learning algorithm to generate a feeding plan for the continuous action space, including the adjustment value of the feeding lot amount and the adjustment value of the feeding interval for each product. Step 2: Design a scheduling agent and use the IDDQN reinforcement learning algorithm to select composite scheduling rules for the discrete action space to realize workpiece allocation and machine scheduling decisions; Step 3: The feeding agent generates an initial feeding plan and transmits it to the scheduling agent; Step 4: The scheduling agent performs multiple rounds of scheduling iterations based on the current material feeding plan, selects the optimal scheduling scheme and the corresponding workshop status, and feeds it back to the material feeding agent. Step 5: The feeding intelligent agent corrects the feeding plan based on the feedback information, forming a closed-loop optimization of feeding and scheduling; Step 6: Use a parameter transfer mechanism between adjacent feeding plans to transfer the optimal weights of the trained agent to the initial network of the next training, thereby improving the learning efficiency of the agent and the quality of the scheduling solution.

2. The integrated optimization method for dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration as described in claim 1, characterized in that, The specific process of Step 1 is as follows: 1) Obtain state features Then, the state features are input into the TD3 online Actor network, and... Greedy strategy selects the optimal action ; 2) Rewards for interacting with the environment and new status and form an experience group ; 3) During the training phase of the TD3 algorithm, priority is calculated based on the temporal difference error of each sample in the empirical replay pool. And based on this, determine the probability of it being sampled. , specifically ,in To adjust the hyperparameters that influence the priority level, The total number of samples in the experience replay pool; 4) Sample a batch of samples from the empirical replay pool according to this probability and input them into the online system. In a network, calculate the target Values; two independent values ​​were introduced in TD3. network and And take the minimum of the two as the target. The value is calculated using the following formula: in As a discount factor, For the goal network; 5) The network updates using the minimum mean square error as the loss. And whenever Network Update After that, The network updates once; 6) A soft update strategy is adopted, and its calculation formula is as follows: in This is the weighting factor.

3. The integrated optimization method for dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration as described in claim 1, characterized in that, The specific process of Step 2 is as follows: 1) Obtain state features Then, the state features are input into the online network of IDDQN, which continuously learns and explores through the neural network to achieve... Greedy strategy selects the optimal action ; 2) Based on the optimal action Select the corresponding composite scheduling rule from the rule pool, and arrange the appropriate workpiece to the idle machine for processing according to the composite scheduling rule; 3) Receive instant rewards and new status and form an experience group Stored in the experience replay pool; 4) During the training phase of IDDQN, random samples are drawn from the experience replay pool. A sample is input into an online network to calculate the target. Value; the IDDQN algorithm uses a two-layer depth Network: A stable network generated from the target network Value estimation, while the online network is used for policy updates, thereby enabling the estimation of values. Dual constraints on values; Its goal The value is calculated as follows: in, and These represent the parameters of the online network and the target network, respectively. Discount factor; 5) The algorithm minimizes the prediction Values ​​and Objectives Mean square error of the value To update network parameters.

4. The integrated optimization method for dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration as described in claim 1, characterized in that, The specific process of Step 4 is as follows: 1) Define the state space of the scheduling agent, including the following two states: ① Machine status: Includes the average utilization rate and standard deviation of all machines, the average mold change utilization rate and standard deviation of all machines, and the average idle rate and standard deviation of all machines; ② Workpiece status: Includes the average completion rate and standard deviation of all workpieces, and the estimated and actual delay rates of all workpieces; 2) Define the state space of the scheduling agent, including the following six composite scheduling rules: First, the machining center with workpieces to be processed is identified. Then, a suitable machine is selected according to the machine rule MR. Next, the delay status of all workpieces in the buffer is calculated. If there are severely delayed workpieces, the corresponding workpieces are selected for processing according to the urgency level ED. Otherwise, the workpieces are selected for processing according to the mold change cost-effectiveness RH. Among them, the machine rule includes MR1 ​​and MR2, and the urgency level includes ED1, ED2 and ED3. This constitutes the action space of the scheduling agent, namely 6 composite scheduling rules: MR1+ED1+RH, MR1+ED2+RH, MR1+ED3+RH, MR2+ED1+RH, MR2+ED2+RH, MR2+ED3+RH. 3) Design the reward function for the scheduling agent, including step-by-step rewards and a final reward: ① The step-by-step reward is designed based on four dimensionless indicators: actual workpiece delay rate, workpiece assessment delay rate, machine idle rate, and machine mold change rate. Improvement of any one of these indicators will generate a positive reward, while deterioration of any one indicator will generate a negative reward. If there is no change in any indicator, the step-by-step reward will be zero. ② The final reward is designed based on the optimization objectives of maximum flow time, total mold change time, and total delay time. For multi-objective optimization problems, they are first decomposed into multiple sub-problems, and then aggregated using a weighted sum to form the final reward.

5. The integrated optimization method for dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration as described in claim 1, characterized in that, The specific process of Step 5 is as follows: 1) Define the state space of the feeding agent, including the following four states: ① Machine status: Includes the average utilization rate and standard deviation of all machines, the average mold change utilization rate and standard deviation of all machines, and the average idle rate and standard deviation of all machines; ② Workpiece status: Includes the average completion rate and standard deviation of all workpieces, and the estimated and actual delay rates of all workpieces; ③ Workshop congestion level: including the average waiting time for workpieces; ④ Overall scheduling performance: includes the maximum flow time under the current material feeding plan, the total mold change time under the current material feeding plan, and the total delay time under the current material feeding plan. 2) Define the state space of the feeding agent, which consists of the following two dimensions: ① Adjustment value for feed lot quantity; ② Feeding interval adjustment value. 3) Design the reward function for the feeding agent: Four sub-reward indicators are set for maximum flow time, total mold change time, total delay time, and material feeding timeout, and index normalization is used to ensure adaptability in different scenarios. Finally, the total reward is the equal-weighted average of the four indicators to comprehensively evaluate the overall quality of the material feeding plan.

6. The integrated optimization method for dynamic material feeding and production scheduling in a packaging and testing workshop based on heterogeneous dual-agent collaboration as described in claim 1, characterized in that, The specific process of Step 6 is as follows: In the The optimal weight parameters obtained during training under the next feeding plan Migrate to the Initial network parameters for the next feeding plan This improves the learning efficiency of the agent and the quality of the scheduling solution.

Images