Method for establishing flexible job-shop scheduling model and electronic device

Abstract

The application provides a flexible workshop scheduling model establishment method and an electronic device. The method comprises the following steps: creating a scheduling simulation environment model of a flexible workshop; creating an initial flexible workshop scheduling model, which comprises a generator network and a discriminator network; interacting with the scheduling simulation environment model through the generator network, generating a simulation scheme corresponding to a production task, and storing the simulation scheme in an experience pool; collecting expert schemes to establish an expert pool; training the discriminator network using a first training set, outputting a discrimination probability corresponding to the simulation scheme and the expert scheme, and optimizing the parameters of the discriminator network by taking the discrimination label as a target during the training; training the generator network using a second training set, and optimizing the parameters of the generator network by taking the discrimination probability in the second training set as a target during the training; recording the step number once for each optimization, and obtaining the flexible workshop scheduling model when the preset step number is reached. The learning process of the method is more stable, and the learning quality is higher.

Description

Technical Field

[0001] This invention relates to the field of flexible workshop scheduling technology, and in particular to a method for establishing a flexible workshop scheduling model and an electronic device. Background Technology

[0002] Intelligent manufacturing aims to address the need for more differentiated and customized services, enabling data-driven flexible production from large-scale manufacturing to multi-variety, small-batch production. Production scheduling is one of the most critical issues in flexible manufacturing systems. The Flexible Job-shop Scheduling Problem (FJSP) allocates production tasks to available production equipment while satisfying all resource constraints, determining the processing sequence of each operation on each machine to optimize shop floor production metrics. Flexible shop floor scheduling is an extension of the classic Job-shop scheduling problem (JSP). Unlike JSP, FJSP involves a set of selectable machines, and scheduling decisions include both process sequencing and machine selection, increasing the flexibility and complexity of scheduling. Therefore, FJSP is more complex than JSP.

[0003] Traditional methods for solving the FJSP problem include exact algorithms, heuristic methods, and metaheuristic methods. However, exact methods are only suitable for solving small-scale problems, heuristic methods have low accuracy, and metaheuristic methods struggle to achieve real-time responses.

[0004] With the emergence and development of deep learning and reinforcement learning technologies, researchers have begun to use deep reinforcement learning to solve the FJSP problem. However, reinforcement learning faces challenges such as difficulty in designing scheduling algorithms and setting reward functions, low data development and utilization efficiency, high dependence on simulation environment and sample data, and difficulty in obtaining high-quality solutions. Summary of the Invention

[0005] This invention provides a method and electronic device for establishing a flexible workshop scheduling model, which solves the defects of existing technologies such as difficulty in setting reward functions, high dependence on simulation environment and sample data, and low data development and utilization rate. It realizes a scheduling algorithm model with easy-to-obtain reward functions, low dependence on simulation environment and sample data, high data development and utilization rate, and higher stability and quality of learning process.

[0006] This invention provides a method for establishing a flexible workshop scheduling model, comprising:

[0007] Create a scheduling simulation environment model for the flexible workshop;

[0008] An initial flexible shop scheduling model is created, which includes a generator network and a discriminator network;

[0009] The generator network interacts with the scheduling simulation environment model to complete production tasks, generate simulation schemes corresponding to the production tasks, and store the simulation schemes in the experience pool.

[0010] Collect expert solutions to establish an expert pool, wherein the data format of the expert solutions is the same as the data format of the simulation solutions;

[0011] The discriminator network is trained using a first training set to output the discrimination probabilities corresponding to the simulated scheme and the expert scheme, respectively. The first training set includes the expert scheme and the simulated scheme. During the training process, the parameters of the discriminator network are optimized with the discrimination labels of the expert scheme and the simulated scheme as the target.

[0012] The generator network is trained using a second training set, which includes the simulation scheme and the corresponding discrimination probability. During the training process, the parameters of the generator network are optimized with the discrimination probability in the second training set as the target.

[0013] Each time the discriminator network and the generator network are optimized, the number of optimization steps is recorded. When the number of optimization steps reaches a preset number, the parameters of the generator network are saved to obtain the flexible workshop scheduling model.

[0014] According to the present invention, a method for establishing a flexible workshop scheduling model is provided, wherein the scheduling simulation environment model is established through an agent-based modeling method.

[0015] According to a method for establishing a flexible shop floor scheduling model provided by the present invention, the scheduling simulation environment model includes a flexible shop floor scheduling problem model and an optimization objective of the scheduling problem;

[0016] The flexible workshop scheduling problem model is a decentralized partially observable Markov decision model.

[0017] According to a method for establishing a flexible workshop scheduling model provided by the present invention, the scheduling simulation environment model includes a workpiece model, an equipment model, and a process model.

[0018] The states in the decentralized partially observable Markov decision model include the workpiece state, process state, and machine production state at each decision moment in the flexible workshop, and the actions are either the selected machine number or a preset special action.

[0019] According to a method for establishing a flexible workshop scheduling model provided by the present invention, the generator network interacts with the scheduling simulation environment model to complete production tasks and generate simulation schemes corresponding to the production tasks, including:

[0020] The initial production state is generated using the aforementioned scheduling simulation environment model;

[0021] The generator network outputs scheduling actions based on the initial production state.

[0022] The scheduling simulation environment model outputs the next production state based on the scheduling action;

[0023] Repeat the above steps until the production task is completed;

[0024] The simulation scheme is generated based on the production status and scheduling actions generated during the production task.

[0025] According to the present invention, a method for establishing a flexible workshop scheduling model is provided, wherein the generator network includes a distributed agent action value network and a centralized joint value network, the centralized joint value network includes a joint action value network and a joint state value network, and the agent includes the workpiece model.

[0026] According to the present invention, a method for establishing a flexible workshop scheduling model is provided, wherein the discriminator network comprises a multi-layer neural network.

[0027] According to the flexible workshop scheduling model establishment method provided by the present invention, the second training set further includes the expert scheme and the corresponding discrimination probability.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the flexible workshop scheduling model establishment method as described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the flexible workshop scheduling model establishment method as described above.

[0030] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the flexible workshop scheduling model establishment method as described above.

[0031] The flexible shop floor scheduling model establishment method provided by this invention, on the one hand, continuously generates simulated schemes by interacting with the scheduling simulation environment model through a generator network, and uses the generated simulated schemes as training data, with the discrimination probability of the discriminator network as the optimization objective, i.e., the reward function, making the data development and utilization for learning highly efficient and the reward function easy to obtain; on the other hand, by distinguishing between expert schemes and simulated schemes, the discriminator network enables the generator to learn scheduling strategies similar to expert schemes, making the generator network learning process more stable and the learning quality higher; furthermore, this flexible shop floor scheduling model has low dependence on the simulation environment and sample data. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the method for establishing a flexible workshop scheduling model provided by the present invention;

[0034] Figure 2 This is a diagram illustrating the scheduling strategy generated by the flexible workshop scheduling model provided by this invention.

[0035] Figure 3 This is a flowchart illustrating the process of generating a simulation scheme corresponding to the production task provided by the present invention;

[0036] Figure 4a This is a schematic diagram of the recurrent neural network (GQRNN) structure in the generator network provided by the present invention;

[0037] Figure 4b This is a schematic diagram of the Joint State Value Network (GVMLP) structure in the generator network provided by the present invention;

[0038] Figure 4c This is a schematic diagram of the Joint Action Value Network (GQMLP) structure in the generator network provided by the present invention;

[0039] Figure 5 This is a schematic diagram of the multilayer neural network (DMLP) structure provided by the present invention;

[0040] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0042] The following is combined Figures 1-5 The present invention describes a method for establishing a flexible workshop scheduling model.

[0043] like Figure 1 As shown, the first aspect of this invention provides a method for establishing a flexible workshop scheduling model, the method comprising:

[0044] S101: Create a scheduling simulation environment model for the flexible workshop.

[0045] Specifically, the scheduling simulation environment model is a general model that reflects the characteristics of the scheduling process of flexible job shops. The model can include a scheduling problem model and an optimization objective of the scheduling problem. Based on the needs of flexible job shop scheduling, the optimization objective of the scheduling problem can be to minimize the maximum actual completion time.

[0046] S102: Create an initial flexible shop floor scheduling model, which includes a generator network and a discriminator network.

[0047] Specifically, the initial flexible workshop scheduling model is a hybrid network model combining generative adversarial networks (GANs) and deep reinforcement learning. The structure of this hybrid network model is mainly a GAN structure, including a generator network and a discriminator network.

[0048] S103: The generator network interacts with the scheduling simulation environment model to complete the production task, generate a simulation scheme corresponding to the production task, and store the simulation scheme in the experience pool.

[0049] Specifically, the generator network interacts with the scheduling simulation environment model. During the interaction, production states and scheduling actions are generated. The interaction stops when the production task is completed, thus obtaining a decision sequence consisting of the production states and scheduling actions of the completed production task. This decision sequence is a simulation scheme, and the obtained simulation scheme is put into the experience pool.

[0050] S104: Collect expert solutions to establish an expert pool, wherein the data format of the expert solutions is the same as the data format of the simulation solutions.

[0051] Specifically, expert solutions can be solutions collected from the actual production line in the workshop. An expert solution is a decision sequence consisting of production states and scheduling actions from the start to the end of a production task. Before adding expert solutions to the expert pool, the data format in the expert solutions needs to be processed according to the data input and output formats in the initial flexible workshop scheduling model mentioned above.

[0052] S105: The discriminator network is trained using the first training set, and the discrimination probabilities corresponding to the simulation scheme and the expert scheme are output respectively. The first training set includes the expert scheme and the simulation scheme. During the training process, the parameters of the discriminator network are optimized with the discrimination labels of the expert scheme and the simulation scheme as the target.

[0053] Specifically, when the number of expert solutions in the experience pool and the expert pool reaches a preset number, the discriminator network and the generator network are trained.

[0054] When training the discriminator network, a first training set is formed by collecting a certain number of expert schemes and simulated schemes. For example, the number of expert schemes and simulated schemes in the first training set can be the same or different. The discriminator network determines whether the input scheme is an expert scheme or a simulated scheme and outputs the discrimination probability.

[0055] The discrimination labels are pre-defined based on expert and simulated schemes. For example, the discrimination label for expert schemes is preset to 1, and the discrimination label for simulated schemes is preset to 0. Expert and simulated schemes are collected from the experience pool and expert pool to form the first training set for training the discriminator network. During training, the discrimination labels are the target, that is, the parameters of the discriminator network are optimized so that the discriminator network can accurately distinguish whether a scheme is an expert scheme or a simulated scheme.

[0056] S106: The generator network is trained using a second training set, which includes the simulation scheme and the corresponding discrimination probability. During the training process, the parameters of the generator network are optimized with the discrimination probability in the second training set as the target.

[0057] Specifically, the second training set includes a batch of simulation schemes obtained from the experience pool and the corresponding discrimination probabilities output by the discriminator network after the simulation schemes are judged. When training the generator network, the production state in the simulation scheme is used as the input data of the generator network, the scheduling action in the simulation scheme is used as the output, and the discrimination probability is used as the target to optimize the parameters of the generator network.

[0058] S107: Record the number of optimization steps for each optimization of the discriminator network and the generator network. When the number of optimization steps reaches a preset number, save the parameters of the generator network to obtain the flexible workshop scheduling model.

[0059] Specifically, each optimization step of the discriminator and generator networks is recorded. For example, optimizing the discriminator and generator networks once is recorded as 1 step, and optimizing them twice is recorded as 2 steps. When the recorded number of optimization steps is less than the preset number, the first and second training sets are built to train and optimize the discriminator and generator networks respectively. When the recorded number of optimization steps equals the preset number, optimization stops, and the parameters of the generator network are saved, thus obtaining the flexible workshop scheduling model.

[0060] The resulting flexible workshop scheduling model can then be used to generate corresponding production strategies based on the actual workshop production situation. Figure 2 As shown, Figure 2 This diagram illustrates the scheduling strategy generated by the flexible workshop scheduling model based on actual production conditions. In the diagram, J represents the workpiece, M represents the machine, the horizontal axis represents time, and the bar graph represents the time taken for one process of the workpiece.

[0061] The flexible workshop scheduling model establishment method provided in this embodiment has several advantages. First, it continuously generates simulated schemes by interacting with the scheduling simulation environment model through a generator network. The generated simulated schemes are used as training data, and the discrimination probability of the discriminator network is used as the optimization objective, i.e., the reward function. This makes the data development and utilization efficiency for learning high and the reward function easy to obtain. Second, by distinguishing between expert schemes and simulated schemes, the discriminator network enables the generator to learn scheduling strategies similar to expert schemes, making the generator network learning process more stable and the learning quality higher. Furthermore, this flexible workshop scheduling model has low dependence on the simulation environment and sample data.

[0062] In one embodiment, the scheduling simulation environment model is established using an agent-based modeling method.

[0063] Specifically, the main goal of agent-based behavior modeling is to accurately model the behavior of real objects (including reactive and intelligent behaviors) so that they can be simulated on a computer.

[0064] In one embodiment, the scheduling simulation environment model includes a flexible shop floor scheduling problem model and an optimization objective for the scheduling problem;

[0065] The flexible workshop scheduling problem model is a decentralized partially observable Markov decision model.

[0066] Specifically, the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) ​​is a model for handling multi-agent sequential decision-making problems. For the flexible shop floor scheduling problem, the optimization objective can be to minimize the maximum actual completion time.

[0067] minC max

[0068] Among them, C max This indicates the actual maximum completion time for all workpieces.

[0069] In one embodiment, the scheduling simulation environment model includes a workpiece model, an equipment model, and a process model;

[0070] The states in the decentralized partially observable Markov decision model include the workpiece state, process state, and machine production state at each decision moment in the flexible workshop, and the actions are either the selected machine number or a preset special action.

[0071] For example, the flexible shop floor scheduling problem is modeled as a decentralized, partially observable Markov decision model, represented by a single tuple, as follows: <J,G,{L i}, {A i}, R, γ>. Where J represents the set of workpieces, G represents the set of global states; the global state at a certain moment is represented as g∈G; L i This represents the set of observed states for the i-th workpiece, i.e., workpiece i; l i A represents the observed state of the i-th workpiece at a certain moment; i Let a represent the set of actions for the i-th workpiece; let a represent the action of workpiece i performing a certain process at a certain moment. i ∈A i A represents the set of joint action vectors for all workpieces. The joint action of all workpieces performing a certain process at a given moment is represented by a joint action vector a = (a... 1 , ..., a i , ..., a n ), a∈A; R represents the global reward function, which represents the reward value obtained by the system when all workpieces perform the joint action a in the global state g. In this embodiment, the global reward function is the discrimination probability; γ is the reward discount factor, with a value range of 0<γ<1.

[0072] Specifically, the global state at time t refers to the global information at time t, including the state of the production machine and the state of the workpiece, such as the quantity completed and the remaining workload, expressed as: in, This indicates whether the production machine is busy at time t and the remaining operating time. This includes the completion rate of tasks waiting to be done at time t and the proportion of remaining total task time. This representation of the global state reflects the crucial global information needed for each decision by the agent, without increasing the dimensionality of the global state due to an increase in the number of tasks.

[0073] Each workpiece is assigned an observation state, including its operational information, the status of the next process, and the status of the target machine. The observation state of the j-th workpiece at time t can be specifically represented as:

[0074]

[0075] in, This reflects the operational status of the j-th workpiece at time t, including whether it is busy, the number of incomplete processes, and their duration. This reflects the next operation and duration for the j-th workpiece. Describe the operational status of the target machine required for the next process of the j-th workpiece, such as whether it is busy, the number of pending processes, and their duration. The agent's action space consists of a set of selectable machines and preset special actions. The preset special actions include three special actions: executing a task, waiting for a task, and ending a task. i This represents the action performed by the i-th workpiece at a certain moment, such as its assigned machine number or one of its preset special actions. The set of actions for the i-th workpiece can be specifically represented as:

[0076]

[0077] in, This indicates that the i-th agent has been assigned a work machine and is ready to go to the target work machine to perform the next process. This indicates that the agent is performing a task. At this time, the agent cannot be interrupted and transferred to perform the next task, nor can it wait and do nothing. However, it is allowed to perform other tasks in parallel while performing the current task. This indicates that the agent is performing a waiting action. At this time, the agent is not in a work process and cannot move to the next process; it can only wait. This indicates that the agent has finished its work. At this point, the agent has completed all procedures and there is no need to continue scheduling.

[0078] In one embodiment, the method for the generator network to interact with the scheduling simulation environment model to complete production tasks and generate simulation schemes corresponding to the production tasks includes:

[0079] S301: Generate the initial production state through the scheduling simulation environment model.

[0080] For example, the initial production state is generated and provided by the scheduling simulation environment model, and the production state includes workpiece state, machine state, and process state.

[0081] S302: Output scheduling actions based on the initial production state through the generator network.

[0082] For example, the generator network takes the initial production state as input and outputs a scheduling action, which is either the selected machine number or a preset special action.

[0083] S303: The next production state is output based on the scheduling action through the scheduling simulation environment model.

[0084] Specifically, based on the scheduling actions output by the generator network, the scheduling simulation environment model generates the next production state input to the generator network.

[0085] S304: Repeat the above steps until the production task is completed.

[0086] Specifically, the above steps are repeated in a loop to output the next scheduling action and production status until a production task is completely completed.

[0087] S305: Generate the simulation scheme based on the production status and scheduling actions generated during the production task.

[0088] Specifically, based on a production task, the above steps generate a series of production states and scheduling actions. The sequence of all production states and scheduling actions from the start to the completion of a production task constitutes the simulation scheme.

[0089] In one embodiment, reference Figure 4a , 4b As shown in 4c, the generator network includes a distributed agent action value network and a centralized joint value network. The centralized joint value network includes a joint action value network and a joint state value network. The agent includes the workpiece model.

[0090] For example, the intelligent agent includes the workpiece model, that is, the workpiece model can be defined as an intelligent agent. The action value network of the distributed intelligent agent is a recurrent neural network (GQRNN), and each intelligent agent, i.e., each workpiece model, has a recurrent neural network (GQRNN), and the recurrent neural networks (GQRNNs) of all intelligent agents share network parameters. Figure 4a The diagram shows the structure of a Generic QR Neural Network (GQRNN). The input to this network can be the observation state l of the i-th agent at a certain moment. i and hidden state h i Output qi Let be the action value of the i-th agent at a certain moment. Here, the hidden state is an abstract concept, referring to the interaction states recorded over multiple historical interactions during the training process, represented as a continuous data vector.

[0091] Centralized joint value networks include joint action value networks and joint state value networks.

[0092] Figure 4b The diagram illustrates the structure of a Joint State Value Network (GVMLP). As shown, the GVMLP is a multi-layer fully connected neural network that outputs global state values. The network's inputs are the global states g and H of the agents at a given time, where H represents the hidden states h of all agents at that time. i The network is composed of multiple parts, and the output v is the joint state value at a certain time. The global state g of the agent at a certain time serves as the input to the joint state value network, which helps the recurrent neural network (GQRNN) learn global decision-making knowledge.

[0093] Figure 4c The diagram illustrates the structure of the Joint Action Value Network. As shown in the figure, the Joint Action Value Network is a multi-layer fully connected neural network (GQMLP) that outputs the joint action value. The inputs to this network are the global state g at a certain moment, the set of joint action vectors A and H for all workpieces, and H being the hidden states h of all workpieces at a certain moment. i The components are assembled, and the output Q represents the value of the combined action at a certain moment.

[0094] To reduce bias in the generator network, the distributed agent's action value network and joint action value network of the generator network are configured with an evaluation network and a target network, respectively. The evaluation network is used to estimate the action value of the current state, and the target network is used to estimate the action value of the next state. The target network and the evaluation network have identical structures, but the evaluation network updates its parameters after training and learning optimization, while the target network does not participate in the parameter updates during training and learning optimization; it simply replicates the parameters of the evaluation network at preset intervals.

[0095] For the action value network (GQRNN) of the distributed agent in the generator network, an evaluation network (GQRNN) is set up. eval and the target network GQRNN target Evaluation networks GQMLP are set up separately for the Joint Action Value Network (GQMLP). eval and target network GQMLP target Then the value function of the mixed actions of each workpiece, i.e., the intelligent agent, is represented as GHQ:

[0096]

[0097] Where: g represents the global state at a certain moment, l represents the observation state of each agent at a certain moment, a represents the joint action vector of all workpieces executing a certain process at a certain moment, and l i Let a represent the observed state of agent i at a certain moment. i h represents the action that agent i performs at a certain moment in a certain process. i GQRNN represents the hidden state of agent i at a certain moment. i (l i ,a i ,h i ) represents the action value output of the recurrent neural network (GQRNN) of agent i at a certain time, and GVMLP(g) represents the output value of the global state value network at a certain time.

[0098] The sub-loss function of the generator network is:

[0099]

[0100]

[0101] L nopt (g,l,a)=(GHQ(g,l,a)-GQMLP eval (g,a)) 2

[0102] Where: g represents the global state at a certain moment, l represents the observation state of each agent at a certain moment, and a represents the joint action vector of all workpieces performing a certain process at a certain moment. GQMLP represents the joint action vector with the highest value for an agent at a given moment. eval (g, a) represents the output value of the evaluation network function of the joint action value network at a certain moment, and g′ represents the global state at the next moment. This represents the joint action vector with the highest value for the agent at a certain moment in the next moment. This represents the output value of the target network function of the joint action value network at a certain moment, and the value at the next moment.

[0103] The objective function of the generator network is:

[0104]

[0105] Joint action value network parameters θ of generator network GQMLP The update formula is:

[0106]

[0107] in, L represents the objective optimization function G For parameter θGQMLP The gradient of α is given by α, which represents the hyperparameter. The parameters are updated using gradient descent.

[0108] Joint state value network parameters θ of generator network GVMLP The update formula is:

[0109]

[0110] in, L represents the objective optimization function G For parameter θ GVMLP The gradient of β is given by β, which represents the hyperparameter. The parameters are updated using gradient descent.

[0111] The action value network parameters θ of the distributed agent in the generator network GQRNN The update formula is:

[0112]

[0113] in, L represents the objective optimization function G For parameter θ GQRNN The gradient of α, β, and δ represents the hyperparameter. The parameters are updated using gradient descent. The hyperparameters α, β, and δ can be the same or different.

[0114] In one embodiment, reference Figure 5 As shown, the discriminator network comprises a multi-layer neural network.

[0115] Specifically, the discriminator network is a multilayer neural network (DMLP). The network's inputs are the global state g at a given time, the joint action vector set A, and noise. This network is responsible for distinguishing between the expert scheme and the simulated scheme, outputting the discrimination probability of either the expert scheme or the simulated scheme. For example, the noise can be Gaussian noise, and the discrimination probability is converted into a numerical value using the sigmoid function. For example, when the preset discrimination label for the expert scheme is 1 and the discrimination label for the simulated scheme is 0, the numerical value converted by the sigmoid function ∈ [0, 1]. The activation function of this multilayer neural network (DMLP) can be the linearly rectified ReLU or a leaky linearly rectified ReLU.

[0116] The objective function for optimizing the discriminator network is:

[0117]

[0118] Where DMLP() represents the discriminant probability output, e_plan i Represents the expert plan, g_plan iLet denot , where 'a' represents the number of expert schemes and 'b' represents the number of simulation schemes. When the number of expert schemes and simulation schemes are both 'b', the objective function of the discriminator network is:

[0119]

[0120] Discriminator network parameters θ D The updated formula is: in, Describing the optimization objective function L D For θ D The gradient of , where τ is a hyperparameter, and the hyperparameter τ can be the same as or different from the hyperparameters α, β and δ. Under normal circumstances, the hyperparameters α, β, δ and τ are all equal.

[0121] In one embodiment, the second training set further includes the expert scheme and the corresponding discrimination probability.

[0122] Specifically, in order to improve the convergence of the generator network, at least one expert scheme and its corresponding discrimination probability are randomly sampled from the expert pool and put into the second training set each time the generator network is trained. During training, the expert scheme and the simulated scheme are used together as the input of the generator network to guide the generator network to update the gradient in a better direction, which to a certain extent prevents the generator network from getting lost in the process of learning when it cannot obtain high-quality experience.

[0123] For example, in order to make the data format of the expert scheme the same as that of the simulation scheme, the data of the expert scheme can be processed according to the definitions of global state, observation state and action in the above embodiments.

[0124] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a flexible shop floor scheduling model establishment method. This method includes: creating a flexible shop floor scheduling simulation environment model; creating an initial flexible shop floor scheduling model, which includes a generator network and a discriminator network; interacting with the scheduling simulation environment model through the generator network to complete production tasks, generating simulation schemes corresponding to the production tasks, and storing the simulation schemes in an experience pool; collecting expert schemes to establish an expert pool, wherein the data format of the expert schemes is the same as the data format of the simulation schemes; training the discriminator network using a first training set, and outputting the simulation schemes and the expert schemes. The first training set includes the expert scheme and the simulation scheme, respectively, and during training, the parameters of the discriminator network are optimized with the discriminant labels of the expert scheme and the simulation scheme as the target. The generator network is trained using a second training set, which includes the simulation scheme and the corresponding discriminant probabilities. During training, the parameters of the generator network are optimized with the discriminant probabilities in the second training set as the target. Each optimization step is recorded for each optimization of the discriminator network and the generator network. When the optimization step count reaches a preset number of steps, the parameters of the generator network are saved to obtain the flexible workshop scheduling model.

[0125] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the flexible workshop scheduling model establishment method provided by the above methods, the method comprising:

[0127] A flexible workshop scheduling simulation environment model is created; an initial flexible workshop scheduling model is created, which includes a generator network and a discriminator network; the generator network interacts with the scheduling simulation environment model to complete production tasks, generate simulation schemes corresponding to the production tasks, and store the simulation schemes in an experience pool; expert schemes are collected to establish an expert pool, the data format of the expert schemes being the same as the data format of the simulation schemes; the discriminator network is trained using a first training set, and the discrimination probabilities corresponding to the simulation schemes and the expert schemes are output, respectively. The first training set includes the expert schemes. The scheme and the simulation scheme are described, wherein during training, the parameters of the discriminator network are optimized with the discrimination labels of the expert scheme and the simulation scheme as the target; the generator network is trained using a second training set, the second training set including the simulation scheme and the corresponding discrimination probabilities, wherein during training, the parameters of the generator network are optimized with the discrimination probabilities in the second training set as the target; each optimization step is recorded after optimizing the discriminator network and the generator network, and when the optimization step reaches a preset number of steps, the parameters of the generator network are saved to obtain the flexible workshop scheduling model.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for establishing a flexible shop floor scheduling model provided by the methods described above. This method includes: creating a scheduling simulation environment model of a flexible shop floor; creating an initial flexible shop floor scheduling model, the initial flexible shop floor scheduling model including a generator network and a discriminator network; interacting with the scheduling simulation environment model through the generator network to complete production tasks, generating simulation schemes corresponding to the production tasks, and storing the simulation schemes in an experience pool; collecting expert schemes to establish an expert pool, the data format of the expert schemes being the same as the data format of the simulation schemes; and using a first training set to train the discriminator network... The training process outputs the discrimination probabilities corresponding to the simulated scheme and the expert scheme, respectively. The first training set includes the expert scheme and the simulated scheme. During training, the parameters of the discriminator network are optimized with the discrimination labels of the expert scheme and the simulated scheme as the target. The generator network is trained using a second training set, which includes the simulated scheme and the corresponding discrimination probabilities. During training, the parameters of the generator network are optimized with the discrimination probabilities in the second training set as the target. Each optimization step is recorded for each optimization of the discriminator network and the generator network. When the optimization step reaches a preset number of steps, the parameters of the generator network are saved to obtain the flexible workshop scheduling model.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for establishing a flexible workshop scheduling model, characterized in that, include: Create a scheduling simulation environment model for the flexible workshop; An initial flexible shop scheduling model is created, which includes a generator network and a discriminator network; The generator network interacts with the scheduling simulation environment model to complete production tasks, generate simulation schemes corresponding to the production tasks, and store the simulation schemes in the experience pool. Collect expert solutions to establish an expert pool, wherein the data format of the expert solutions is the same as the data format of the simulation solutions; The discriminator network is trained using a first training set to output the discrimination probabilities corresponding to the simulated scheme and the expert scheme, respectively. The first training set includes the expert scheme and the simulated scheme. During the training process, the parameters of the discriminator network are optimized with the discrimination labels of the expert scheme and the simulated scheme as the target. The generator network is trained using a second training set, which includes the simulation scheme and the corresponding discrimination probability. During the training process, the production state in the simulation scheme is used as the input data of the generator network, the scheduling action in the simulation scheme is used as the output, and the discrimination probability is used as the target to optimize the parameters of the generator network. Each time the discriminator network and the generator network are optimized, the number of optimization steps is recorded. When the number of optimization steps reaches a preset number, the parameters of the generator network are saved to obtain the flexible workshop scheduling model.

2. The method for establishing a flexible workshop scheduling model according to claim 1, characterized in that, The scheduling simulation environment model was established using an agent-based modeling method.

3. The method for establishing a flexible workshop scheduling model according to claim 2, characterized in that, The scheduling simulation environment model includes a flexible shop scheduling problem model and an optimization objective for the scheduling problem; The flexible workshop scheduling problem model is a decentralized partially observable Markov decision model.

4. The method for establishing a flexible workshop scheduling model according to claim 3, characterized in that, The scheduling simulation environment model includes a workpiece model, an equipment model, and a process model; The states in the decentralized partially observable Markov decision model include the workpiece state, process state, and machine production state at each decision moment in the flexible workshop, and the actions are either the selected machine number or a preset special action.

5. The method for establishing a flexible workshop scheduling model according to claim 4, characterized in that, The generator network interacts with the scheduling simulation environment model to complete production tasks and generate simulation schemes corresponding to the production tasks, including: The initial production state is generated using the aforementioned scheduling simulation environment model; The generator network outputs scheduling actions based on the initial production state. The scheduling simulation environment model outputs the next production state based on the scheduling action; Repeat the above steps until the production task is completed; The simulation scheme is generated based on the production status and scheduling actions generated during the production task.

6. The method for establishing a flexible workshop scheduling model according to claim 4 or 5, characterized in that, The generator network includes a distributed agent action value network and a centralized joint value network. The centralized joint value network includes a joint action value network and a joint state value network. The agent includes the workpiece model.

7. The method for establishing a flexible workshop scheduling model according to any one of claims 1 to 5, characterized in that, The discriminator network comprises a multi-layer neural network.

8. The method for establishing a flexible workshop scheduling model according to any one of claims 1 to 5, characterized in that, The second training set also includes the expert scheme and the corresponding discrimination probability.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the flexible workshop scheduling model establishment method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the flexible workshop scheduling model establishment method as described in any one of claims 1 to 8.

Images