Kit production workshop scheduling method and system based on hierarchical reinforcement learning

Abstract

The invention discloses a kit production workshop scheduling method and system based on hierarchical reinforcement learning. The method comprises the following steps: establishing an upper-layer optimization model including minimum completion time and minimum advanced delay total cost; the upper-layer optimization model selects an optimization target as a temporary optimization target of the lower-layer optimization model according to the current production progress and the equipment state; the lower-layer optimization model selects a kit and injection molding equipment which need to be subjected to injection molding in the next stage according to a temporary optimization target selected by the upper-layer optimization model; and solving the upper-layer optimization model and the lower-layer optimization model to realize the scheduling of the production workshop of the kit. According to the method, a layered structure is adopted, the upper layer and the lower layer pay attention to the scheduling environment from different perspectives, division of labor and cooperation of respective functions are facilitated, and the two sterile production facility scheduling targets of the cost brought by the completion time and the injection molding period can be optimized at the same time.

Description

Technical Field

[0001] The present invention relates to the field of workshop scheduling control, and in particular to a method and system for scheduling a test kit production workshop based on hierarchical reinforcement learning. Background Art

[0002] As an important part of the modern economy, manufacturing plays a key role globally. Production scheduling is a vital link in the manufacturing industry, which affects production efficiency, product quality and corporate competitiveness. With the changes in the demand market, consumers' pursuit of personalization and differentiation has become increasingly strong, making the production orders of manufacturing companies increasingly develop in the direction of small batches and multiple varieties. The integrated development of digital technology and manufacturing, digital transformation has become an important breakthrough in promoting the high-quality development of the manufacturing industry and accelerating the development of new industrialization. With the support of scheduling and related disciplines, the manufacturing industry has basically completed the modernization transformation and upgrading, and is moving towards the goal of digitalization, networking and intelligence.

[0003] Taking the automobile assembly line as an example, the factory's production efficiency can be greatly improved by rationally dividing the assembly process. The entire automobile production process has been transformed from the original traditional all-round employee process to an assembly line production mode with conveyor belt supply, which has shortened the assembly time of the car by 50% and created a new and efficient automobile manufacturing process. Therefore, workshop scheduling occupies an important position in intelligent manufacturing, has a wide range of application scenarios, and can effectively improve the production efficiency of enterprises and reduce costs.

[0004] Although the manufacturing industry has achieved modernization and digital transformation, existing technologies still face challenges. For example, in the process of producing medical test kits, injection molding equipment may suffer from problems such as wear of mechanical parts, aging of the heating system, or wear and deformation of the mold, which may lead to unstable injection molding quality and affect production efficiency. In addition, the order-driven production model requires companies to manage the production cycle while meeting the diverse needs of customers to avoid liquidated damages and additional warehousing costs. Excessive accumulation of deteriorated injection molding equipment will lead to longer production time and affect the realization of production scheduling goals. Traditional scheduling methods and metaheuristic methods have limitations in dealing with dynamic events and real-time scheduling, and cannot effectively respond to real-time changes and degradation problems in the production process. Therefore, more advanced scheduling methods are needed to optimize production processes, reduce costs, and improve production efficiency. Summary of the invention

[0005] The present invention proposes a method and system for scheduling a test kit production workshop based on hierarchical reinforcement learning, which solves the problem that the existing scheduling method for the test kit production workshop cannot respond to real-time changes and equipment degradation in the production process in a timely manner.

[0006] In order to solve the above technical problems, the present invention provides a method for scheduling a test kit production workshop based on hierarchical reinforcement learning, comprising the following steps: An upper-level optimization model is established that minimizes the completion time and minimizes the total cost of lead time and delay; the upper-level optimization model selects an optimization target as a temporary optimization target of the lower-level optimization model according to the current production progress and equipment status; the lower-level optimization model selects the test kit and injection molding equipment that need to be injected in the next stage according to the temporary optimization target selected by the upper-level optimization model; the upper-level optimization model and the lower-level optimization model are solved to realize the scheduling of the production workshop of the test kit.

[0007] Preferably, the objective function of the upper optimization model is expressed as: ; ; In the above formula, , are the total cost of advance and delay, and the maximum completion time of all bioassay kits, respectively; Biological detection kit The time when all parts are fully injection molded; Biological detection kit Deadline for Penalty factors for early and late injection completion for all bioassay kits.

[0008] Preferably, when solving the upper layer optimization model and the lower layer optimization model, constraints are also set, and the constraints include: (1) Processing order constraints between components: ; In the formula, For parts Time to start injection molding; For parts Actual injection time; (2) Constraints on the injection molding sequence of components of the same biological testing kit: ; In the formula, is the decision variable, when the component Injection molding equipment During injection molding , otherwise 0; (3) In each biological test kit, the start time constraint of the first component and the completion time constraint of the last component: ; ; In the formula, Biological detection kit The start time of the first component; Biological detection kit The time when all parts are fully injection molded; (4) Relationship constraints between biological testing kits and injection molding equipment: ; In the formula, For parts A collection of optional injection molding equipment; is the decision variable, when the component Can be injection molded During injection molding, , otherwise 0; is the decision variable, when the component Arranged in injection molding equipment During injection molding , otherwise 0; (5) Actual processing time constraints for components of biological test kits: ; (6) Injection sequence constraints between parts arranged on the same injection molding equipment: ; In the formula, , ,in For arrangement in injection molding equipment Injection molding o A combination of components of a biological detection kit.

[0009] Preferably, the upper layer optimization model and the lower layer optimization model are solved by a reinforcement learning method, comprising the following steps: Step S1: Initialize the experience replay pool, online network and target network of the upper optimization model and the lower optimization model; Step S2: Initialize the current instance, and initialize the states of the upper optimization model and the lower optimization model; Step S3: input the state of the upper optimization model into the upper neural network, and select a target as the temporary optimization target of the lower optimization model; input the state of the lower optimization model and the temporary optimization target into the lower neural network, and select the lower action; Step S4: Execute the selected lower-level action, update the upper-level state and the lower-level state, and obtain the global reward; Step S5: Update the experience replay pool of the upper optimization model and the lower optimization model, and determine whether the capacity of the experience replay pool exceeds the set capacity threshold. If so, execute step S6; if not, execute step S7; Step S6: determining whether the capacity of the upper experience replay pool and the lower experience replay pool exceeds a set maximum capacity threshold, and if so, updating the upper experience replay pool and the lower experience replay pool; Step S7: Determine whether the current number of iterations reaches the set maximum number of iterations. If not, return to step S2; otherwise, output the scheduling decision of the production workshop.

[0010] Preferably, for the upper experience replay pool and the lower experience replay pool, step S6 includes the following steps: Step S61: Calculate the probability of each experience in the experience replay pool being sampled, extract a subset of the size of the maximum capacity threshold from the experience replay pool through probability-based priority experience replay, and calculate the weight of the loss function of the subset: ; In the formula, is the weight of the loss function of the subset; is the maximum capacity threshold; represents the sampling weight coefficient; For the i The probability that an experience is sampled; Indicates taking the largest value; Step S62: Calculate the current target Q value as follows: ; In the formula, For the i Experience reward; is the discount factor; , They are online network and target network respectively; For the i +1 experience status; To maximize the Q value ; is the current solution target; , are the weights of the online network and the target network respectively; Step S63: Use the mean square loss function to calculate the loss of the optimization model, use the gradient descent and back propagation algorithm to update the network parameters, and calculate the mean square loss function The expression is: ; Step S64: Calculate the temporal difference error TD Error of all experiences, use the temporal difference error to update the probability of each sample being sampled, and calculate the temporal difference error The expression is: ; Update the probability of each sample being sampled as: ; Step S65: Repeat steps S61 to S64 until the set maximum number of iterations is reached.

[0011] Preferably, the upper state includes: the ratio between the current number of reagent kits and the number of injection molding equipment , Average injection molding equipment utilization rate , Variance of injection molding equipment utilization , the average value of the kit processing progress , Variance of the injection molding schedule of the test kit , time advance delay rate and the variance of the time advance delay rate ; Calculate the average injection molding equipment utilization The expression is: ; In the formula, is the decision variable, when the component Injection molding equipment During injection molding , otherwise 0; For parts The average of the initial injection time of all available injection molding equipment; The maximum completion time for all biological test kits; is the total number of injection molding equipment; Calculate the variance of the injection molding equipment utilization The expression is: ; ; In the formula, For injection molding equipment Total injection time; Calculate the average value of the processing progress of the kit The expression is: ; ; In the formula, Processing progress for the test kit; nis the total number of biological detection kits; Biological detection kit The serial number of the part currently being processed; Biological detection kit i Total number of parts; Calculate the variance of the injection molding schedule of the kit The expression is: ; Calculate the time advance delay rate The expression is: ; In the formula, Biological detection kit The moment when the last component of a component is completed; The average value of the time when all injection molding equipment completes the injection molding task of the biological detection kit; Biological detection kit Deadline for Biological detection kit Total completion time; Biological detection kit The sum of the average initial injection times of all parts.

[0012] Preferably, obtaining the state of the lower layer optimization model comprises the following steps: Step S21: obtaining status information of the biological detection kit and the injection molding device, and adjusting the status information of the biological detection kit and the injection molding device to a uniform size through an adaptive matrix filling mechanism; Step S22: using an attention mechanism to extract features of the filled biological detection kit and the injection molding equipment respectively; Step S23: Classify and filter the extracted features through the importance mechanism to generate two feature vectors, one representing the important features of the biological detection kit and the other representing the important features of the injection molding equipment. These two feature vectors are used as the state of the lower optimization model.

[0013] The present invention also provides a reagent kit production workshop scheduling system based on hierarchical reinforcement learning, which is implemented based on the above-mentioned reagent kit production workshop scheduling method based on hierarchical reinforcement learning, including a short-term optimization target decision module, a production workshop scheduling module and a real-time monitoring module; The short-term optimization target decision module: determines the short-term production optimization target according to the current production progress and equipment status; The production workshop scheduling module: performs production scheduling tasks according to the short-term production optimization target output by the temporary optimization target decision module; The real-time monitoring module collects real-time data on the production line, monitors the operating status of the production line, and ensures that the production process proceeds as planned.

[0014] Preferably, the production workshop scheduling module models the production scheduling task as a Markov decision process MDP.

[0015] Preferably, the production workshop scheduling module adopts a reinforcement learning method based on a dual value network to solve short-term production optimization objectives and perform production scheduling tasks.

[0016] The benefits of the present invention include at least: 1. It can consider multiple optimization objectives at the same time, such as completion time and advance delay cost, providing a more comprehensive and balanced production scheduling solution, and can dynamically adjust the optimization objectives according to the real-time production progress and equipment status, thus improving the adaptability and flexibility of the scheduling plan; 2. By optimizing the production process, waiting time and equipment idle time can be reduced, production efficiency can be improved, and by reducing orders for early delivery and delayed delivery, the penalty costs caused by violating delivery time can be reduced; this can more reasonably allocate injection molding tasks to different equipment, avoid equipment overload or idleness, and improve equipment utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention; Figure 2 A schematic diagram of a framework of a classification hierarchical reinforcement learning method according to an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a hierarchical deep reinforcement learning method according to an embodiment of the present invention; Figure 4 A schematic diagram of the state and environment details of the lower-level deep reinforcement learning according to an embodiment of the present invention; Figure 5 is a schematic diagram of an adaptive matrix filling mechanism according to an embodiment of the present invention; Figure 6 Schematic diagram of the convergence comparison of the classification hierarchical reinforcement learning method under different test cases in the embodiment of the present invention; Figure 7 Schematic diagram of the convergence comparison of the classification hierarchical reinforcement learning method under different learning rates in an embodiment of the present invention; Figure 8 A graph showing the variation trend of the average reward of the classification hierarchical reinforcement learning method according to an embodiment of the present invention with the number of iterations; Fig. 9 A schematic diagram showing a comparison of the effects of using a classification hierarchical reinforcement learning method and a deep reinforcement learning method to solve a double-layer optimization problem in an embodiment of the present invention; Fig.10A schematic diagram showing a comparison of scheduling results of the classification hierarchical reinforcement learning method according to an embodiment of the present invention and a deep Q network, a double deep Q network and a double-layer deep Q network at various scales; Fig.11 A schematic diagram showing a comparison of the generalization effects of the classification hierarchical reinforcement learning method of an embodiment of the present invention and a deep Q network, a double deep Q network, and a double-layer deep Q network under mixed example sets of various scales; Fig.12 A schematic diagram showing a comparison of scheduling results of the classification hierarchical reinforcement learning method according to an embodiment of the present invention and a deep Q network, a double deep Q network and a double-layer deep Q network under dynamic examples of different scales; Fig.13 A scheduling Gantt chart of the classification hierarchical reinforcement learning method in an embodiment of the present invention on a production instance; Fig.14 A Gantt chart showing the scheduling of a deep Q network in a production instance according to an embodiment of the present invention; Fig.15 A scheduling Gantt chart of a dual-depth Q network in a production instance according to an embodiment of the present invention; Fig.16 This is a scheduling Gantt chart of a double-layer deep Q network in a production instance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0019] The embodiment of the present invention is described with the immunology and molecular diagnosis production line of a medical enterprise as the object. The production of medical detection kits involves multiple steps, and the entire production process requires strict quality control and management to ensure the quality, stability and safety of the product to meet the needs of scientific research and clinical diagnosis. In addition to the reagents, most of the reagents are produced by injection molding, such as reaction pools and reaction tubes for carrying reaction liquids, filter elements and filters for filtration, and tips or dispensers for sucking or adding samples from the kit. The injection molding equipment is not static during the injection molding process. The wear of mechanical parts, such as screws, barrels, nozzles and other parts, will affect the plasticization performance and plastic fluidity due to long-term high-speed operation and high temperature. In addition, if the heating system of the injection molding equipment is affected by an adverse environment or long-term high-temperature operation, heating elements such as heating coils and thermocouples may age or be damaged, resulting in unstable temperature control, thereby affecting the quality and fluidity of the plastic melt. The mold used for injection molding may also wear and deform, affecting injection molding. Ordinary molds are generally replaced directly, but due to the high cost, multi-cavity molds used in injection molding equipment are generally not replaced as a whole. Instead, the worn cavities are sealed and then continued to be used. However, this method will reduce the available cavities and affect the injection molding efficiency. This phenomenon can be regarded as the deterioration of the injection molding equipment and thus be treated. The production model of the test kit is driven by customer orders, and the main customers include hospitals, testing institutions, etc. Because the company's products are diverse, involving multiple aspects such as immunodiagnosis and molecular diagnosis, so one order may contain multiple test kits. The production process of each test kit is different. At the same time, each test kit has a different injection molding period. If it is exceeded, a penalty will be incurred. If it is completed ahead of time, it requires sterile, refrigerated and other environments for warehousing, which incurs additional costs and brings economic losses to the company.

[0020] When degradation accumulates too much on the injection molding equipment, it will directly lead to a significant extension of the injection molding time of its subsequent injection molding, which will have an adverse impact on the scheduling objectives of the final sterile production facility. At the same time, the initial injection molding time of the component to the biological detection kit also corresponds to the actual injection molding time. In a flexible sterile production facility, not only is the real-time degradation of each injection molding equipment different, but the same component has a batch of available injection molding equipment with different injection molding times. Therefore, it is necessary to provide detailed status information for each injection molding equipment and biological detection kit at each rescheduling moment, so that not only can the appropriate biological detection kit and injection molding equipment be selected to complete the injection molding, but also the severely degraded injection molding equipment can be discovered in time for maintenance, reducing the impact of degradation on production. This poses a challenge to the existing scheduling methods for sterile production facilities.

[0021] The metaheuristic methods commonly used in the prior art cannot respond to dynamic events, and traditional reinforcement learning is limited to one-dimensional state representation, which can only generally display the state of the global environment and is powerless to small state changes caused by initial disturbances. When the impact of dynamic events is large enough to be represented by traditional one-dimensional state representation, the sterile production facility scheduling environment has been greatly disturbed, which is obviously not conducive to real-time sterile production facility scheduling. The two-dimensional state representation method can not only display the detailed status information of each component and injection molding equipment, but also timely and accurately reflect the deterioration location and degree of the injection molding equipment, as well as the injection molding information of each component required in the process of solving the problem. On the other hand, the actual scheduling of sterile production facilities cannot only focus on the completion time. The storage and overdue costs caused by the injection molding cycle arrangement are also factors that need to be considered in the scheduling of sterile production facilities. The hierarchy determines the goals through the cooperation of the upper and lower layers, and selects appropriate actions to achieve a balance between the goals.

[0022] In summary, the current solution to the aseptic production facility scheduling problem based on reinforcement learning still has the following shortcomings: 1. Only one-dimensional variables are used to represent the state, which cannot respond promptly to subtle changes caused by dynamic disturbances in the scheduling environment of sterile production facilities, thus having an adverse impact on the scheduling of sterile production facilities and the optimization of the ultimate goal.

[0023] 2. Using a single reward function for solving the problem cannot arrange appropriate scheduling goals based on the real-time scheduling environment, and cannot consider multiple scheduling optimization factors for aseptic production facilities. It is not suitable for the current complex production environment where both the injection completion time and the injection deadline are considered at the same time.

[0024] Based on the above problems, the embodiments of the present invention propose the following Figure 1 A method for scheduling a test kit production workshop based on hierarchical reinforcement learning is shown, comprising the following steps: An upper-level optimization model is established that minimizes the completion time and minimizes the total cost of lead delays. The upper-level optimization model selects an optimization target as the temporary optimization target of the lower-level optimization model based on the current production progress and equipment status. The lower-level optimization model selects the test kit and injection molding equipment that need to be injected in the next stage based on the temporary optimization target selected by the upper-level optimization model. The upper-level optimization model and the lower-level optimization model are solved to realize the scheduling of the production workshop of the test kit.

[0025] The scheduling problem set in the upper and lower layers is called the multi-objective shop dynamic scheduling problem considering degradation MODFJSP-MD. MODFJSP-MD includes n Biological detection kits ,in , each biological test kit i Both Parts ,in Need to be injection molded; Injection molding machines ,in Can be assigned to perform injection molding tasks. Biological Detection Kit The deadline is . Parts Have a set of injection molding equipment that can inject it , It is a spare part Arrange in injection molding equipment The initial injection time for injection molding is determined by the injection molding equipment. The influence of the degree of deterioration. The injection molding time of the injection molding equipment before maintenance reflects the degree of deterioration of the injection molding equipment. The actual injection molding time is equal to the initial injection molding time. Add the injection molding time from the last maintenance of the injection molding equipment to the injection molding equipment degradation rate R .

[0026] The assumptions of the MODJSP-MD problem are as follows: (1) Each component of a biological testing kit has a fixed injection molding order. The previous component must be processed first before the next component can be processed.

[0027] (2) Each injection molding machine is only allowed to mold one component of a biological detection kit at a time.

[0028] (3) The basic time required for the corresponding injection molding equipment to process each component of each biological testing kit is determined.

[0029] (4) For adjacent bioassay kits, including injection molding equipment maintenance and bioassay kit injection molding, ignoring the transportation time of the bioassay kits and the setting time of the injection molding equipment when performing these bioassay kits.

[0030] (5) Assume that the buffer zone of the injection molding equipment is infinite.

[0031] (6) Assuming that the injection molding quality of the biological test kit meets the standard, only one injection molding is required. In other words, no rework is required.

[0032] (7) Once the processing of the biological detection kit on the injection molding equipment has started, it cannot be interrupted.

[0033] (8) The injection molding equipment processes the parts in the order in which they are assigned.

[0034] (9) Each component of each biological detection kit can be selected for injection molding by multiple injection molding machines, but can only be arranged for injection molding on one injection molding machine.

[0035] In order to better describe the embodiment, the symbols and decision variables in the model are defined as follows: : No. i Biological detection kits; i :Serial number of the biological detection kit, ; : No. k Injection molding equipment; k :Serial number of injection molding equipment, ; :Biological detection kit No. j Parts; j :Parts serial number, ; :Biological detection kit The serial number of the part currently being processed; :Can be used for parts A collection of injection molding equipment for performing injection molding; :Arranged in injection molding equipment Injection molding o A combination of biological test kit components, ; :Planning on injection molding equipment The total number of biological test kit component combinations processed; :Can be used for parts The number of injection molding machines performing injection molding; : Parts Initial injection time; : Parts The average of the initial injection time of all available injection molding equipment; :Biological detection kit The sum of the average initial injection times of all parts; R : Deterioration coefficient, indicating the degree of loss of the injection molding equipment during the continuous injection molding process; DDT : Injection molding expectation coefficient, which reflects the customer's expectation of the injection molding completion time for this batch of biological detection kits; :Biological detection kit The deadline, ; : Penalty coefficients for early and delayed injection completion of all biological test kits; : Arranged in injection molding equipment Parts to be molded , in ; : Parts Time to start injection molding; :Biological detection kit The time when all parts are fully injection molded; :Injection molding equipment Cumulative injection molding time since the last maintenance; : Parts The time taken for actual injection molding; : ; : ; : ; : Discount factor.

[0036] Among them, the calculation components The average injection molding time The expression is: .

[0037] Computational Biology Assay Kits Average processing time The expression is: .

[0038] Calculation components Actual injection time The expression is: ; In the embodiment of the present invention, R is 0.2.

[0039] The objective function of the MODJSP-MD problem is: ; ; In the above formula, , are the total cost of advance and delay, and the maximum completion time of all bioassay kits, respectively.

[0040] The constraints are: (1) Processing order constraints between components: ; The above formula means that the processing of each component must be performed after the previous component is completed.

[0041] (2) Injection molding sequence constraints that need to be followed between the components of the same biological detection kit: .

[0042] (3) In each biological test kit, the start time constraint of the first component and the completion time constraint of the last component: ; ; In the formula, Biological detection kit The start time of the first component of .

[0043] (4) Relationship constraints between biological testing kits and injection molding equipment: ; The above formula indicates that each component of each biological detection kit can be selected for injection molding by multiple injection molding machines, but can only be arranged for injection molding on one injection molding machine.

[0044] (5) Actual processing time constraints for components of biological test kits: ; The above formula indicates that the actual processing time of each component of each biological detection kit must be greater than 0.

[0045] (6) Injection sequence constraints between parts arranged on the same injection molding equipment: ; The above formula means that the parts arranged on the same injection molding equipment must wait for the previously arranged parts to be completed before they can be injected.

[0046] The aseptic production facility scheduling problem in the embodiment of the present invention can be modeled as a Markov decision process MDP and can be represented as a five-tuple <S, A, P, R, >, ,in is the discount factor, is the state transfer equation.

[0047] In order to solve the problem of scheduling of sterile production facilities, the embodiment of the present invention constructs the following Figure 2 The classification hierarchical reinforcement learning method CAHRL shown includes a smart bioassay kit sterile production facility manufacturing environment, hierarchical agents, and reinforcement learning process.

[0048] Both the upper and lower optimization models use a two-layer deep reinforcement learning network DDQN, which can select more appropriate actions to reduce injection molding costs. The priority experience replay framework gives more weight to efficient experiences and improves the utilization of important experiences. The overall aseptic production facility scheduling process, including the reinforcement learning training process, includes the following steps: Step S1: Initialize the upper and lower experience replay pools and , the upper online network DDQN and the target network DDQN , the lower online network DDQN , and the target network DDQN ;according to n , m Load the training set with the same parameters.

[0049] Step S2: Initialize the current instance and the upper layer status , the state of the lower layer .

[0050] Step S3: Set the upper state Input to the upper neural network In the greedy strategy Select a higher level target , based on the target selected by the upper layer, determine the temporary optimization target of the current lower layer .

[0051] Step S4: Set the lower state and temporary optimization goals Input to the lower neural network together In, through Select the underlying action , execute the selected lower-level action , update the upper state and the lower state , get rewarded .

[0052] Step S5: Transition the upper experience Stored in the upper experience replay pool Middle and lower level experience Stored in the lower level experience replay pool middle.

[0053] Step S6: Determine whether the capacity of the upper experience playback pool exceeds the set capacity threshold If it exceeds, execute step S7, otherwise execute step S8.

[0054] Step S7: Updating the upper experience replay pool, including the following steps: Step S71: Calculate the probability of each experience in the upper experience replay pool being sampled : ; In the formula, For the upper experience replay pool i The priority of each sample; It is the priority weight of samples in the upper experience replay pool.

[0055] Step S72: Based on probability Priority experience playback from Pick one from Experience The subset consisting of , calculate the loss function weight of this subset

[0056] ; In the formula, It is the sampling weight coefficient of the upper experience replay pool.

[0057] Step S73: Calculate the current target Q value : ; in For the network Parameters.

[0058] Step S74: Use the mean square loss function as the loss function of the upper layer, then perform gradient descent and update the network through back propagation Parameters , the expression for calculating the mean square loss function is: .

[0059] Step S75: Calculate the time series difference error of all examples , update the probability of each sample being sampled , the expression for calculating the timing difference error is: .

[0060] Step S76: After each step C, reset , where C is the step size of the target network update interval.

[0061] Step S8: Determine whether the capacity of the lower experience playback pool exceeds the set capacity threshold If it exceeds, execute step S9, otherwise execute step S10.

[0062] Step S9: Updating the lower-level experience replay pool, including the following steps: Step S91: Calculate the probability of each experience being sampled in the lower experience replay pool: .

[0063] Step S92: Based on probability Priority experience playback from Pick one from Experience The subset consisting of , calculate the loss function weight of this subset

[0064] .

[0065] Step S93: Calculate the current target Q value : .

[0066] Step S94: Use the mean square loss function as the loss function of the lower layer, then perform gradient descent and update the network through back propagation Parameters , the expression for calculating the mean square loss function is: .

[0067] Step S95: Calculate the time series difference error of all examples , update the probability of each sample being sampled , the expression for calculating the timing difference error is: .

[0068] Step S96: After each step C, reset .

[0069] Step S10: Determine whether the current number of iterations reaches the set maximum number of iterations. If not, return to step S2; otherwise, output the scheduling decision of the production workshop.

[0070] The structure of the classification hierarchical reinforcement learning network constructed in the embodiment of the present invention is as follows: Figure 3 The upper layer observes the overall scheduling environment from a macro perspective, considers long-term information, and uses this to determine the temporary optimization target for each step. After extracting information from the input state of the lower layer through several convolutional layers, it is combined with the temporary target selected by the upper layer. The activation function of the classification layer reinforcement learning network is the ReLU layer set between layers, and the parameter optimization function is Adam. The final output is the q value of each action in the action space.

[0071] In the injection molding production process of an enterprise, the injection molding cost is one of the important factors that need to be considered. If the arrangement is reasonable, the storage cost and overdue penalty cost can be greatly reduced; on the contrary, if the production scheduling is improper, the injection molding cost will be greatly increased, affecting the final total cost, and then affecting the company's profit. Therefore, it is very important to control the injection molding cost through the scheduling of sterile production facilities. However, the traditional single-layer method cannot optimize the injection molding time and the injection molding advance delay cost at the same time, resulting in increased injection molding costs and increased total costs. The hierarchical structure of the embodiment of the present invention can observe the overall situation of the scheduling of sterile production facilities at the upper layer, such as the injection molding progress of the biological detection kit, the working time of the injection molding equipment, the time advance delay rate cost of the biological detection kit, etc., so as to select the target that best suits the current scheduling situation of the sterile production facility.

[0072] Defining Biological Assay Kits The completion time of the last component is , and define the injection molding equipment All parts previously scheduled for processing are completed at .set up It represents the average time when all injection molding equipments finish the biological test kits arranged on them. Biological detection kit The sum of the actual completion time. Since the upper layer needs to observe the overall situation at a macro level, rather than focusing on the injection molding process of a single injection molding machine or biological test kit, the following 7 parameters are used in the upper layer state space: (1) Ratio between the current number of test kits and the number of injection molding equipment : .

[0073] (2) Average injection molding equipment utilization rate : ; In the formula, The maximum completion time for all bioassay kits.

[0074] (3) Variance of injection molding equipment utilization : ; ; In the formula, It is injection molding equipment The total injection molding time comes from the injection molding of the arranged parts.

[0075] (4) Average value of kit processing progress : ; .

[0076] (5) Variance of the injection molding process of the test kit : .

[0077] (6) Time advance delay rate : .

[0078] (7) Variance of the time advance delay rate ; ; .

[0079] The problem studied in this paper has three high-level goals: completion time, total lead-delay cost, and a mixed reward of half of the two. After observing the environment, the upper layer selects the goal As the next temporary optimization target of the lower layer. When the completion time is selected as the target, the reward function The calculation formula is: .

[0080] When the total lead-delay cost is chosen as the objective, the reward function The calculation formula is: .

[0081] in The average bioassay kit injection progress is calculated by the following expression: ; .

[0082] When the completion time and total lead time cost are half each, the reward function The calculation formula is: .

[0083] The hierarchical structure means more complex environmental information, which poses a challenge to the traditional one-dimensional state representation. Therefore, the embodiment of the present invention proposes a hierarchical multi-channel matrix to represent environmental information, in which the state space of the biological detection kit and the state space of the injection molding equipment are modeled respectively. In this way, the complete state characteristics of the biological detection kit and the injection molding equipment can be extracted, and the information of the current aseptic production facility scheduling environment can be more comprehensively grasped, thereby reducing the injection molding cost.

[0084] Each cell in the state section of the biological test kit corresponds to an injection molding machine. Take the scheduling problem of creating a sterile production facility with six biological test kits and six injection molding machines, each with a maximum of six parts, as an example. The state and environment details are as follows: Figure 4 The second component of the second biological test kit is indicated by a dotted line. The corresponding position in the scheduling sequence of the sterile production facility is in the Gantt chart and the state. The vertical axis size of the matrix is ​​the number of biological test kits, and the horizontal axis size is the number of biological test kits with the largest number of parts among all biological test kits. The vertical axis size of the last three injection molding equipment is 1, and the horizontal axis size is the number of injection molding equipment. In this way, the biological test kit can be reasonably selected to reduce the injection molding cost. The contents and dimensions of the seven matrices that constitute the lower state space are as follows: (1) Average processing time: The average processing time of a component of a biological test kit on an available injection molding machine is stored.

[0085] (2) Processing time variance: The data source is the processing time of the biological test kit, but each cell contains the variance of the processing time of a certain component of a biological test kit by the available injection molding equipment.

[0086] (3) Component completion time: Before the start of aseptic production facility scheduling, the number in each cell is 0. When the component represented by the cell is completed, fill in the time when the component was completed.

[0087] (4) Utilization rate of the injection molding equipment selected for the component: Each cell is filled with a value, which is equal to the completion time of the component represented by the cell divided by the current completion time.

[0088] (5) Biological test kit early delay rate: The content of each cell is the completion time of the current component divided by the expiration date of the biological test kit to which the component belongs. , in order to fully understand the injection molding status of the biological detection kit and reduce the injection molding cost.

[0089] (6) Continuous injection time of injection molding equipment: The vertical axis size is 1, the horizontal axis size is the injection molding equipment number, and each cell is filled with the accumulated working time since the last maintenance of the injection molding equipment.

[0090] (7) Expected end time of injection molding equipment: The time when each cell filling is completed when the injection molding equipment completes processing of the currently scheduled biological test kit.

[0091] In the biological test kit status matrix, the first row represents the first biological test kit, the first column represents the first component, and so on. For rows where the number of components is less than the maximum number, the remaining cells are filled with zeros. The processing time mean and processing time variance are initially full, and the values ​​in the corresponding cells of each sterile production facility scheduling process are cleared to zero in the subsequent process. The initial values ​​of the values ​​of all cells in the biological test kit advance delay rate, injection equipment continuous injection time, and injection equipment expected end time are all zero, and the corresponding cells are filled with the corresponding values ​​of each sterile production facility scheduling component in the subsequent process.

[0092] In order to extract important features from the multi-channel state matrix, an embodiment of the present invention designs a method based on one-dimensional parameter input representation. However, the types and contents of the feature information contained in the biological detection kit state and the injection molding equipment state proposed in the multi-channel state matrix are not the same, and the scales of the two states are slightly different, so they cannot be directly entered into the network together. If they are all forcibly filled into the same size according to the conventional method, the amount of useless information will increase, affecting the judgment, so feature extraction must be performed separately for different categories.

[0093] Therefore, the embodiment of the present invention adopts a network for importance classification based on a plug-and-play lightweight attention mechanism ECA-net, including two parts that extract the state characteristics of the biological detection kit and the state characteristics of the injection molding equipment respectively. Then, each network is attached with an importance mechanism to filter the biological detection kit and injection molding equipment matrices containing different types of important information. This can avoid violent filling and reduce useless information. It is also possible to extract biological detection kit and injection molding equipment features according to local conditions to improve the efficiency of information acquisition. The one-dimensional kernel size used to complete cross-channel information exchange in ECA-net is adapted to the function change: ; Where c is the number of channels; , are two hyper parameters related to calculation, which are set to 2 and 1 in the embodiment of the present invention.

[0094] As the scheduling environment of sterile production facilities continues to expand, the scheduling problems of sterile production facilities of different scales in actual production have challenged the robustness of the framework. However, due to the characteristics of the neural network itself, its structure cannot be changed at will. Therefore, the embodiment of the present invention proposes an adaptive matrix filling mechanism, the network structure of which is as follows Figure 5 As shown in the figure. By filling the state matrices of the sterile production facility scheduling problems of various scales with a uniform size of 0, the calculation can proceed smoothly. The newly filled ones are surrounded by 0 and the old ones are in the center, retaining the initial feature information to the greatest extent. This matrix adaptive filling mechanism enables it to solve the sterile production facility scheduling problems of different scales and ensures the robustness of the method. After the matrix is ​​adaptively filled, the ReLU layer activation function and BatchNorm2d are input.

[0095] The operation of the biological test kit sterile production facility scheduling problem consists of two parts: selecting an unfinished biological test kit and then selecting an injection molding machine to process the next biological test kit. The expected rate of early or late arrival is , the calculation formula is In the embodiment of the present invention, four biological detection kit selection rules, three injection molding equipment selection rules and two maintenance selection rules are set. These rules constitute 4×3×2=24 combined aseptic production facility scheduling rules, which are used as the action space of the lower layer, and the action space of the upper layer is the three reward functions of the lower layer.

[0096] Through the designed four biological test kit scheduling rules, the optimal biological test kit is selected for injection molding by comprehensively considering the injection molding cost and injection molding time. When selecting the biological test kit, the two-dimensional state representation is used to have a deeper understanding of the overall injection molding, properly balance the injection molding cost, and reduce the total cost.

[0097] (1) Biological test kit scheduling rules 1 Select the bioassay kit that lags behind the injection molding progress on the part, and then select the bioassay kit with the smallest value among all the bioassay kits.

[0098] (2) Biological test kit scheduling rules 2 Calculating the expected lead delay rate for bioassay kits , if the value is greater than 0, the bioassay kit that lags behind in the injection molding progress in terms of parts is selected, and then the bioassay kit with the smallest value is selected among all the bioassay kits.

[0099] If less than 0, select the injection advance delay rate The largest bioassay kit, computational bioassay kit Injection advance delay rate The expression is: .

[0100] (3) Biological test kit scheduling rules 3 First calculate the expected lead delay rate of the bioassay kit If the value is greater than 0, the bioassay kit with a delayed injection progress in terms of time is selected so that its injection progress is equal to , and then select the smallest value among all biological assay kits.

[0101] If it is less than 0, the bioassay kit with the highest advance delay rate is selected.

[0102] (4) Biological test kit scheduling rules 4 Select a bioassay kit with a time lag in the injection schedule where the injection schedule is equal to , and then select the smallest value among all biological assay kits.

[0103] In order to improve the utilization rate of injection molding equipment and reasonably arrange the injection molding sequence of biological test kits while considering the injection molding cycle and injection molding cost, three injection molding equipment scheduling rules are adopted. The injection molding equipment is arranged to fully consider the delivery date of each biological test kit to avoid the storage cost caused by early injection molding and the penalty cost of delayed injection molding. At the same time, the actual injection molding completion time should also be considered when arranging the injection molding equipment. This is the parameter closely related to the injection molding cost, not the early or late arrangement. Of course, the cost increase or decrease involved in the injection molding constraint is also a factor that needs to be considered.

[0104] (1) Injection molding equipment scheduling rules 1 Select the injection molding equipment that can complete the next part to be injected as quickly as possible.

[0105] (2) Injection molding equipment scheduling rules 2 Select the molding machine that actually takes the shortest time to mold the next component of the biological test kit.

[0106] (3) Injection molding equipment scheduling rules 3 In all injection moldable parts In the injection molding equipment, select the waiting time based on the injection molding constraints The injection molding device with the shortest waiting time. If there are multiple candidate injection molding devices with zero waiting time, the injection molding device with the shortest initial processing time is selected.

[0107] After the biological test kit and injection molding equipment are determined, the following two injection molding equipment maintenance rules can help the injection molding equipment maintain good working condition and reduce the adverse effects of degradation.

[0108] (1) Injection molding equipment maintenance rules 1 Selecting Injection Molding Equipment After that, perform maintenance operations and then clear the injection time since the last maintenance time of the injection molding equipment. .

[0109] (2) Injection molding equipment maintenance rules 2 Selecting Injection Molding Equipment After that, no maintenance operation is performed, which will increase the cumulative processing time of the injection molding equipment since the last maintenance, that is, .

[0110] An embodiment of the present invention also provides a reagent kit production workshop scheduling system based on hierarchical reinforcement learning, which is implemented based on the above-mentioned reagent kit production workshop scheduling method based on hierarchical reinforcement learning, and includes a short-term optimization target decision module, a production workshop scheduling module and a real-time monitoring module.

[0111] Short-term optimization target decision module: Determine the short-term production optimization target based on the current production progress and equipment status.

[0112] Production workshop scheduling module: executes production scheduling tasks according to the short-term production optimization goals output by the temporary optimization goal decision module.

[0113] Real-time monitoring module: collects real-time data on the production line, monitors the operating status of the production line, and ensures that the production process proceeds as planned.

[0114] The hardware environment for the neural network training of the present invention is an Intercore i5 ninth-generation processor with 16G running memory and a GTX1650 graphics card, and the software is python3.6.

[0115] Considering that the constraint sets of some degradation problems in the present invention are unique compared with existing studies, and in order to expand the generalization of the method, randomly generated experimental examples are used. In order to train CAHRL to generalize the scheduling environment of sterile production facilities of different scales, the training set in the embodiment of the present invention consists of randomly generated samples, and the specific parameter settings are shown in Table 1.

[0116] Table 1 Parameter settings of CAHRL

[0117] Each training is performed on a new random sample. The () in the table indicates a random selection of integers, including the upper and lower limits in the brackets. It is designed to decrease linearly from 0.9 to 0.1 during training.

[0118] The embodiment of the present invention lists the approximately optimal hyperparameter configuration for the proposed problem. Test samples are generated according to the configuration in Table 2 to observe the performance of different hyperparameters during the training process.

[0119] Table 2 Optimal parameters of CAHRL

[0120] The capacity threshold minibatch and learning rate were tested on a test instance. Figure 6 The minibatch is tested on the test instance shown in Figure 1. The test results show that the larger the minibatch, the faster the convergence speed. When the minibatch is reduced, the convergence speed decreases, but the training performance improves. In order to find a higher quality solution, the minibatch with the best performance in the later stage, i.e. 64, is selected.

[0121] The training process trend diagram under different learning rates is as follows Figure 7 As shown in the figure, the smaller the learning rate, the better the training effect. However, it can be seen from the figure that if the learning rate is too small, the network will not be able to be updated effectively, resulting in unsatisfactory results. Taking various factors into consideration, the learning rate is set to 0.00001 in order to train the neural network more effectively and obtain better results for the scheduling of sterile production facilities.

[0122] Since the objectives and reward functions of the lower layer are the actions of the upper layer, changes in the lower layer behavior strategy will affect the determination of the upper layer objectives. When the lower layer network is updated, even the same upper layer objectives may cause the lower layer to make different judgments when choosing actions. Therefore, the capacity of the upper layer replay memory is designed to be consistent with the capacity of the minibatch to improve the timeliness of the experience in the upper layer replay memory, thereby ensuring the timely update of the upper layer neural network and preventing it from learning the later experience in the replay memory. Table 3 lists the hyperparameters of the neural network and method of the embodiment of the present invention.

[0123] Table 3 Hyperparameters of neural networks and methods

[0124] In order to verify the effectiveness and efficiency of CAHRL, the embodiment of the present invention designed a numerical comparison experiment, and used three methods as comparison methods, namely deep Q network DQN, dual deep Q network DDQN and double-layer deep Q network THDQN. In order to verify the advantages of CAHRL in multi-objective optimization problems, the state space, action space, and neural network settings of DDQN all adopt the same configuration as the bottom layer of CAHRL. Since a single-layer DDQN cannot use multiple reward functions, a weighted method is used to consider two objectives, and the weights of the two objectives are 0.5 respectively.

[0125] In reinforcement learning, the size of the average reward value for each training session represents whether more favorable actions can be selected during training, resulting in better scheduling results for sterile production facilities. Figure 8 The trend of the average reward value obtained in each training is shown. As can be seen from the figure, as the training progresses, a better aseptic production facility scheduling strategy is gradually obtained through interaction with the environment.

[0126] In multi-objective problems, it is very important to choose a strategy to balance the objectives. To demonstrate the importance of the hierarchical structure in solving MODJSP-MD, Fig. 9 The figure shows the changing trends of the two target values ​​of CAHRL and DDQN during the training process. It can be seen that compared with DDQN, CAHRL can select the most reasonable reward function at each step through the hierarchical structure, and can obtain a solution with both target values ​​close to the best more quickly.

[0127] In order to compare the versatility and generalization of the method in different production environments, some test samples were generated based on different scales and different parameter settings. The method of the embodiment of the present invention and three comparison methods were tested respectively, and the results of three performance indicators, generation distance (GD), inverse generation distance (IGD) and hyper volume (HV), were proposed for evaluation. True Pareto frontier It is composed of the non-dominated solutions of the method of the embodiment of the present invention and the sum of the solutions of all the comparative methods. The reference point used to calculate the hypervolume in the embodiment of the present invention is the maximum solution found for each objective in the problem. The optimal solution of each type of calculation instance is shown in bold.

[0128] The comparison results of GD of the four methods are shown in Table 4.

[0129] Table 4 Comparison results of GD

[0130] The comparison results of IGD are shown in Table 5.

[0131] Table 5 Comparison results of IGD

[0132] The comparison results of HV are shown in Table 6.

[0133] Table 6 Comparison results of HV

[0134] Through the above comparison, it can be found that the CAHRL designed by the present invention has better performance than other comparison methods in most cases. This proves that under different scales and different penalty coefficients for early and delayed injection completion, the upper layer of CAHRL can make a compromise between the two goals of minimizing the maximum completion time and minimizing the total early delay cost according to the current sterile production facility scheduling environment, the injection progress of the biological detection kit and the degree of degradation of the injection equipment, and select a temporary optimization goal, so that the lower layer can optimize the overall sterile production facility scheduling plan. The two-dimensional parameter input of the lower layer also expresses the current status of each biological detection kit and injection molding equipment in more detail than the ordinary one-dimensional parameter input, thereby selecting the action with the most rewards according to the temporary optimization goal.

[0135] From the examples tested above, five representative examples of different scales are selected to compare the two target values ​​of all methods in these examples. Fig.10 As shown. It is observed that in some examples, the result of one of the objectives of DDQN designed based on the lower layer of CAHRL is the same as that of CAHRL, but the other one is much worse, which shows that the CAHRL proposed in the present invention can achieve a good compromise between the objectives. It will not optimize only one objective and make the other objective worse. In addition, when the trained reinforcement learning method is applied to a new example, it can get a better solution in a few seconds under the same computer configuration. The metaheuristic method takes several minutes to calculate. This shows that the trained reinforcement learning method can achieve good generalization and quickly obtain excellent feasible solutions when facing different examples.

[0136] In order to further verify the generalization of CAHRL, a hybrid experiment was designed. The test dataset consists of 20 examples randomly generated according to the setting table in

[137] , including examples of different sizes. CAHRL and the three comparison methods were tested on the 20 examples respectively, and then the results were taken. and The two target values ​​of the four resulting solutions. It can be obtained that the advantage ratio of one of the target values ​​of the solution of each method on the instance is: ; in i Indicates the sequence number of the method.

[0137] Finally, the odds ratio box plots of the two targets for each method for each example are shown below. Fig.11 shown.

[0138] In the actual production environment, the degradation coefficient of the injection molding equipment is idealized, and each injection molding equipment tends to randomly degrade during the injection molding process. When the degradation coefficient changes, the above-mentioned aseptic production facility scheduling scheme is obviously no longer applicable. Therefore, the embodiment of the present invention conducts comparative experiments on four methods under the condition that the degradation rate before each injection molding randomly changes between 0.1 and 0.3. After each method is trained, it is independently run 20 times on the example and the results are recorded. The average Pareto front surface of the solution generated by each method is as follows: Fig.12 As shown in the figure, it can be seen that CAHRL can respond to unexpected disturbances in a timely manner in a dynamic aseptic production facility scheduling environment with random injection molding equipment degradation, and obtain a robust solution that balances the scheduling objectives of the two aseptic production facilities.

[0139] The scheduling method proposed in the present invention is applied to a test kit production example of a medical enterprise for analysis, and is compared with other comparative methods to verify the actual effect of the present invention in engineering.

[0140] The production process of the test kit mainly includes reagent preparation, production of reagent packaging boxes, reagent filling and assembly, sealing and packaging of various parts. First, select the mold according to the design requirements to ensure that the mold structure and size meet the product requirements; second, prepare high-quality plastic particles and put them into the hopper of the injection molding equipment; then, heat and melt the plastic particles through the injection molding equipment, and inject the molten plastic into the pre-prepared mold; then, wait for the plastic to cool and solidify in the mold to ensure the shape and size of the injection molded part are accurate; finally, open the mold, take out the molded injection molded part, and carry out subsequent removal of excess material, cleaning and quality inspection to ensure its appearance and function are intact. Then the prepared and configured reagents that have undergone strict quality control and performance verification are filled into the corresponding injection molding container according to requirements and packaged and disinfected.

[0141] As a medical product, the test kit needs to be safe and hygienic. As a container for carrying different biological agents, the test kit needs to ensure the hygiene and safety of the biological agents during the storage process to ensure the accuracy of the test results. Therefore, during the injection molding process of the test kit, the cleanliness of the injection mold and in some cases the sterility will be subject to stricter scrutiny. In addition, as a medical product, the production process of the test kit also has higher quality requirements. The inspection standards for some common quality problems of injection molded parts such as shrinkage, flash, black spots, flow marks, etc. are also more stringent. Therefore, the production of test kits is more sensitive to the deterioration of injection molding equipment.

[0142] Taking the production process of a virus / pathogen nucleic acid isolation kit as an example, some of its components are injection molded packaging box, injection molded sample collection funnel, injection molded sample collection tube, injection molded sample preservative cap, 550 mL binding solution injection molded reagent bottle, 1000 mL washing solution injection molded reagent bottle, 100 mL elution solution injection molded reagent bottle, 10 mL protease K Injection molding reagent bottle, 20mL DNA / RNA binding bead injection molding reagent bottle. These parts need to be molded by injection molding equipment before they can be loaded with the corresponding reagents. The molds of different injection molding equipment can be replaced with each other, which means that the same component can be molded by multiple injection molding equipment. Because medical test kits are designed to be hygienic and safe, the requirements for ambient temperature and sterile production facilities are high. Injection molding sterile production facilities must maintain a high degree of cleanliness and hygiene, control the temperature, humidity and cleanliness of the production environment, meet the requirements of medical device production, and regularly clean and disinfect injection molding equipment, molds and production areas. Table 7 shows the injection molding equipment that can be used for some components of two of the kits in production and the corresponding injection molding time.

[0143] Table 7 Optional injection molding equipment and corresponding injection molding time

[0144] The present invention tests CAHRL and three comparative methods on a kit production example, and presents the scheduling results in the form of a Gantt chart, as shown in FIG. Figures 13 to 16 As shown in the Gantt chart, it can be seen that the CAHRL method proposed in the present invention can reasonably select the maintenance time of the injection molding equipment, minimize the impact on the existing injection molding plan, and maintain the healthy state of the injection molding equipment.

[0145] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0146] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for scheduling a test kit production workshop based on hierarchical reinforcement learning, characterized in that: The following steps are involved: An upper-level optimization model is established that minimizes the completion time and minimizes the total cost of lead time and delay; the upper-level optimization model selects an optimization target as a temporary optimization target of the lower-level optimization model according to the current production progress and equipment status; the lower-level optimization model selects the test kit and injection molding equipment that need to be injected in the next stage according to the temporary optimization target selected by the upper-level optimization model; the upper-level optimization model and the lower-level optimization model are solved to realize the scheduling of the production workshop of the test kit.

2. The method for scheduling a test kit production workshop based on hierarchical reinforcement learning according to claim 1, characterized in that: The objective function of the upper optimization model is expressed as: ； ； In the above formula, , are the total cost of advance and delay, and the maximum completion time of all bioassay kits, respectively; Biological detection kit The time when all parts are fully injection molded; Biological detection kit Deadline for Penalty factors for early and late injection completion for all bioassay kits.

3. The method for scheduling a test kit production workshop based on hierarchical reinforcement learning according to claim 1, characterized in that: When solving the upper optimization model and the lower optimization model, constraints are also set, and the constraints include: (1) Processing order constraints between components: ； In the formula, For parts Time to start injection molding; For parts Actual injection time; (2) Constraints on the injection molding sequence of components of the same biological testing kit: ； In the formula, is the decision variable, when the component Injection molding equipment During injection molding , otherwise 0; (3) In each biological test kit, the start time constraint of the first component and the completion time constraint of the last component: ； ； In the formula, Biological detection kit The start time of the first component; Biological detection kit The time when all parts are fully injection molded; (4) Relationship constraints between biological testing kits and injection molding equipment: ； In the formula, For parts A collection of optional injection molding equipment; is the decision variable, when the component Can be injection molded During injection molding, , otherwise 0; is the decision variable, when the component Arranged in injection molding equipment During injection molding , otherwise 0; (5) Actual processing time constraints for components of biological test kits: ； (6) Injection sequence constraints between parts arranged on the same injection molding equipment: ； In the formula, , ,in For arrangement in injection molding equipment Injection molding o A combination of components of a biological detection kit.

4. The method for scheduling a test kit production workshop based on hierarchical reinforcement learning according to claim 1, characterized in that: The upper layer optimization model and the lower layer optimization model are solved by using a reinforcement learning method, comprising the following steps: Step S1: Initialize the experience replay pool, online network and target network of the upper optimization model and the lower optimization model; Step S2: Initialize the current instance, and initialize the states of the upper optimization model and the lower optimization model; Step S3: input the state of the upper optimization model into the upper neural network, and select a target as the temporary optimization target of the lower optimization model; input the state of the lower optimization model and the temporary optimization target into the lower neural network, and select the lower action; Step S4: Execute the selected lower-level action, update the upper-level state and the lower-level state, and obtain the global reward; Step S5: Update the experience replay pool of the upper optimization model and the lower optimization model, and determine whether the capacity of the experience replay pool exceeds the set capacity threshold. If so, execute step S6; if not, execute step S7; Step S6: determining whether the capacity of the upper experience replay pool and the lower experience replay pool exceeds a set maximum capacity threshold, and if so, updating the upper experience replay pool and the lower experience replay pool; Step S7: Determine whether the current number of iterations reaches the set maximum number of iterations. If not, return to step S2; otherwise, output the scheduling decision of the production workshop.

5. The method for scheduling a test kit production workshop based on hierarchical reinforcement learning according to claim 4, characterized in that: For the upper experience replay pool and the lower experience replay pool, step S6 includes the following steps: Step S61: Calculate the probability of each experience in the experience replay pool being sampled, extract a subset of the size of the maximum capacity threshold from the experience replay pool through probability-based priority experience replay, and calculate the weight of the loss function of the subset: ； In the formula, is the weight of the loss function of the subset; is the maximum capacity threshold; represents the sampling weight coefficient; For the i The probability that an experience is sampled; Indicates the largest value; Step S62: Calculate the current target Q value : ； In the formula, For the i Experience reward; is the discount factor; , They are online network and target network respectively; For the i +1 experience status; To maximize the Q value ; is the current solution target; , are the weights of the online network and the target network respectively; Step S63: Use the mean square loss function to calculate the loss of the optimization model, use the gradient descent and back propagation algorithm to update the network parameters, and calculate the mean square loss function The expression is: ； Step S64: Calculate the temporal difference error TD Error of all experiences, use the temporal difference error to update the probability of each sample being sampled, and calculate the temporal difference error The expression is: ； Update the probability of each sample being sampled as: ； Step S65: Repeat steps S61 to S64 until the set maximum number of iterations is reached.

6. The method for scheduling a test kit production workshop based on hierarchical reinforcement learning according to claim 4, characterized in that: The upper state includes: the ratio between the current number of test kits and the number of injection molding equipment , Average injection molding equipment utilization rate , Variance of injection molding equipment utilization , the average value of the kit processing progress , Variance of the injection molding schedule of the test kit , time advance delay rate and the variance of the time advance delay rate ; Calculate the average injection molding equipment utilization The expression is: ； In the formula, is the decision variable, when the component Injection molding equipment During injection molding , otherwise 0; For parts The average of the initial injection time of all available injection molding equipment; The maximum completion time for all biological test kits; is the total number of injection molding equipment; Calculate the variance of the injection molding equipment utilization The expression is: ； ； In the formula, For injection molding equipment Total injection time; Calculate the average value of the processing progress of the kit The expression is: ； ； In the formula, Processing progress for the test kit; n is the total number of biological detection kits; Biological detection kit The serial number of the part currently being processed; Biological detection kit i Total number of parts; Calculate the variance of the injection molding schedule of the kit The expression is: ； Calculate the time advance delay rate The expression is: ； In the formula, Biological detection kit The moment when the last component of a component is completed; The average value of the time when all injection molding equipment completes the injection molding task of the biological detection kit; Biological detection kit Deadline for Biological detection kit Total completion time; Biological detection kit The sum of the average initial injection times of all parts.

7. The method for scheduling a test kit production workshop based on hierarchical reinforcement learning according to claim 4, characterized in that: Obtaining the state of the lower optimization model comprises the following steps: Step S21: obtaining status information of the biological detection kit and the injection molding device, and adjusting the status information of the biological detection kit and the injection molding device to a uniform size through an adaptive matrix filling mechanism; Step S22: using an attention mechanism to extract features of the filled biological detection kit and the injection molding equipment respectively; Step S23: Classify and filter the extracted features through the importance mechanism to generate two feature vectors, one representing the important features of the biological detection kit and the other representing the important features of the injection molding equipment. These two feature vectors are used as the state of the lower optimization model.

8. A reagent kit production workshop scheduling system based on hierarchical reinforcement learning, implemented based on a reagent kit production workshop scheduling method based on hierarchical reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: It includes short-term optimization target decision-making module, production workshop scheduling module and real-time monitoring module; The short-term optimization target decision module: determines the short-term production optimization target according to the current production progress and equipment status; The production workshop scheduling module: performs production scheduling tasks according to the short-term production optimization target output by the temporary optimization target decision module; The real-time monitoring module collects real-time data on the production line, monitors the operating status of the production line, and ensures that the production process proceeds as planned.

9. The kit production workshop scheduling system based on hierarchical reinforcement learning according to claim 8, characterized in that: The production workshop scheduling module models the production scheduling task as a Markov decision process MDP.

10. The kit production workshop scheduling system based on hierarchical reinforcement learning according to claim 9, characterized in that: The production workshop scheduling module adopts a reinforcement learning method based on a dual value network to solve short-term production optimization goals and perform production scheduling tasks.

Images