Pulse production line scheduling method and device, equipment, storage medium and product

Abstract

The invention provides a pulsating production line scheduling method and device, equipment, a storage medium and a product, and the method comprises the steps: inputting the static information of a current scheduling problem into a resource prediction agent, and obtaining a prediction peak value of each type of current production resources outputted by the resource prediction agent; inputting the static information of the current scheduling problem, the predicted peak value of each type of current production resources and the current scheduling condition into a process scheduling agent, and obtaining the probability value of each current process output by the process scheduling agent as next scheduling; determining a next scheduling scheme according to the probability value that each current working procedure is used as next scheduling; according to the scheduling scheme of the next step, scheduling processing is carried out on the pulsation production line; if the pulse production line still has the working procedures which are not started, returning to execute the steps; otherwise, ending the pulsation production line scheduling. According to the scheme, load balance of the pulsation production line is achieved, and the scheduling accuracy of the pulsation production line is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of production line scheduling, and in particular to a pulsating production line scheduling method, device, equipment, storage medium and product. Background Art

[0002] The pulsating production line is an advanced assembly production line, mainly used in the aviation field. As an example, the aircraft assembly process is implemented using the pulsating production line. Aircraft assembly is a key link in aircraft manufacturing. How to reasonably schedule the aircraft assembly pulsating production line to achieve cost reduction and efficiency improvement is an important scientific issue in the field of intelligent manufacturing. However, the aircraft assembly pulsating production line has a complex scenario, and assembling a single aircraft involves tens of thousands of processes, which brings new challenges to the efficient solution of aircraft assembly scheduling problems.

[0003] In the prior art, the Pointer Network with Actor-Critic (PN-AC) model based on reinforcement learning is used to schedule pulsating pipelines. Currently, the Markov Decision Processes (MDPs) for pulsating pipeline scheduling problems are often based on a greedy framework. In practical applications, the reinforcement learning agent dynamically selects the task with the highest priority, and the initial execution time of the task is determined by the greedy framework.

[0004] However, the goal of the greedy algorithm is to advance the start time of the task as much as possible within the local optimal period. Therefore, the current pulsating production line scheduling method has an unbalanced load and poor scheduling accuracy for aircraft assembly production lines. Summary of the invention

[0005] The present invention provides a pulsating production line scheduling method, device, equipment, storage medium and product, which are used to solve the defects of the current pulsating production line scheduling method in the prior art, such as load imbalance and poor scheduling accuracy of the pulsating production line, realize the load balance of the pulsating production line, and improve the scheduling accuracy of the pulsating production line.

[0006] The present invention provides a pulsating production line scheduling method, comprising: obtaining static information of a current scheduling problem of a pulsating production line; inputting the static information of the current scheduling problem into a resource prediction agent, and obtaining the predicted peak value of each type of current production resource output by the resource prediction agent; wherein the resource prediction agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a resource prediction training set; the resource prediction training set includes static information of historical scheduling problems and peak values ​​of each type of historical production resources; inputting the static information of the current scheduling problem, the predicted peak value of each type of current production resources, and the current shift situation into a process scheduling agent, and obtaining the current peak value of each process output by the process scheduling agent. as the probability value of the next scheduling; wherein the process scheduling agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a process scheduling training set; the process scheduling training set includes the static information of the historical scheduling problem, the historical peak value of each type of production resource, the historical scheduling situation, and the probability value of each process in history as the next scheduling; according to the probability value of each current process as the next scheduling, the next scheduling plan is determined; according to the next scheduling plan, the pulsating production line is scheduled; if there are still processes that have not been started in the pulsating production line, return to execute the step of obtaining the static information of the current scheduling problem of the pulsating production line; otherwise, the pulsating production line scheduling is terminated.

[0007] According to a pulsating production line scheduling method provided by the present invention, before obtaining the static information of the current scheduling problem of the pulsating production line, the method also includes: constructing the resource prediction training set and the process scheduling training set based on the historical pulsating production line assembly data; constructing the initial resource prediction intelligent agent and the initial process scheduling intelligent agent based on the reinforcement learning algorithm; based on the resource prediction training set and the process scheduling training set, alternately training the initial resource prediction intelligent agent and the initial process scheduling intelligent agent until the resource prediction intelligent agent and the process scheduling intelligent agent are obtained.

[0008] According to a pulsating production line scheduling method provided by the present invention, the initial resource forecasting agent and the initial process scheduling agent are alternately trained, specifically comprising: gradually increasing the training scale through a course learning training method, and alternately training the initial resource forecasting agent and the initial process scheduling agent.

[0009] According to a pulsating production line scheduling method provided by the present invention, the method also includes: normalizing the reward functions under different training scales; during the training process, after the currently trained intelligent agent performs an action, obtaining the current reward function of the external environment feedback, and inputting the current reward function into the currently trained intelligent agent to optimize the currently trained intelligent agent.

[0010] According to a pulsating production line scheduling method provided by the present invention, the next scheduling plan is determined according to the probability value of each current process as the next scheduling, including: if the variance of the probability value of each current process as the next scheduling is greater than a first threshold, then based on a multiple reasoning method of search, the next scheduling plan is determined; if the variance of the probability value of each current process as the next scheduling is not greater than the first threshold, then based on a data augmentation method, multiple sub-problems corresponding to the current scheduling problem are constructed; the solution of each sub-problem is calculated to obtain multiple equivalent solutions corresponding to the current scheduling problem; multiple equivalent solutions corresponding to the current scheduling problem are evaluated by an evaluation function, and the optimal solution is used as the scheduling plan for the next step.

[0011] According to a pulsating production line scheduling method provided by the present invention, after obtaining the static information of the current scheduling problem of the pulsating production line, the method also includes: based on the static information of the current scheduling problem, constructing the current scheduling problem input graph through the graph attention network; inputting the static information of the current scheduling problem into the resource prediction intelligent agent, specifically includes: inputting the current scheduling problem input graph into the resource prediction intelligent agent.

[0012] The present invention also provides a pulsating production line scheduling device, comprising the following modules: an acquisition module, used to acquire static information of the current scheduling problem of the pulsating production line; a first input module, used to input the static information of the current scheduling problem into a resource prediction agent, and obtain the predicted peak value of each type of production resource output by the resource prediction agent; wherein the resource prediction agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a resource prediction training set; the resource prediction training set includes static information of historical scheduling problems and peak values ​​of each type of production resources in history; a second input module, used to input the static information of the current scheduling problem, the predicted peak value of each type of production resources and the current shift situation into a process scheduling agent, and obtain the current peak value of each type of production resource output by the process scheduling agent. The process scheduling agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a process scheduling training set; the process scheduling training set includes static information of the historical scheduling problem, the peak value of each type of production resource in the history, the historical scheduling situation and the probability value of each process in the history as the next scheduling; a determination module is used to determine the next scheduling plan according to the probability value of each current process as the next scheduling; an execution module is used to perform scheduling processing on the pulsating production line according to the next scheduling plan; a processing module is used to return to the step of obtaining the static information of the current scheduling problem of the pulsating production line if there are still processes that have not been started in the pulsating production line; otherwise, the pulsating production line scheduling is terminated.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the pulsating production line scheduling methods described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the pulsating production line scheduling method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the pulsating production line scheduling method described above is implemented.

[0016] In the pulsating production line scheduling method, device, equipment, storage medium and product provided by the present invention, the static information of the current scheduling problem is input into the resource prediction intelligent body, and the predicted peak value of each type of production resource output by the resource prediction intelligent body is obtained; the static information of the current scheduling problem, the predicted peak value of each type of production resource and the current scheduling situation are input into the process scheduling intelligent body, and the probability value of each current process as the next step of scheduling output by the process scheduling intelligent body is obtained; according to the probability value of each current process as the next step of scheduling, the next scheduling plan is determined; according to the next scheduling plan, the pulsating production line is scheduled; if there are still processes that have not started in the pulsating production line, return to execute the above steps; otherwise, the pulsating production line scheduling is ended. The solution of the present invention adopts a two-layer reinforcement learning model to decompose the scheduling problem of the pulsating production line into two sub-problems: resource prediction and process scheduling. The peak value of each type of production resource is predicted by resource prediction intelligence. Then, based on the predicted peak value of each type of production resource, the process scheduling intelligent body outputs the probability value of each current process as the next step of scheduling. Then, according to the probability value of each current process as the next step of scheduling, the next step of scheduling is determined. Further, the next step of scheduling is executed, and after the execution is completed, it is determined whether there are still unstarted processes in the pulsating production line. If there are still unstarted processes, the above steps are returned until there are no unstarted processes in the pulsating production line, thereby achieving load balancing of the pulsating production line and improving the scheduling accuracy of the pulsating production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the pulsating production line scheduling method provided by the present invention.

[0019] Figure 2 This is the second flow chart of the pulsating production line scheduling method provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the pulsating production line scheduling device provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Figure 1 and Figure 2 The pulsating production line scheduling method of the present invention is described.

[0024] In practical applications, the execution subject of the pulsating production line scheduling method can be a pulsating production line scheduling device, which can be implemented in many ways, for example, it can be implemented through a computer program, such as application software, etc.; or, for example, a chip, etc. It can also be implemented as a medium storing relevant computer programs, such as a U disk, a cloud disk, etc.; or, it can also be implemented through a physical device integrated or installed with relevant computer programs, such as a server, etc.

[0025] Figure 1 It is one of the flow charts of the pulse production line scheduling method provided by the present invention, such as Figure 1 As shown, the method includes steps 101 to 107.

[0026] Step 101: Obtain static information of the current scheduling problem of the pulse production line.

[0027] The static information of the current scheduling problem includes the topological relationship between all processes in the pulsating production line, the number of workers required for each process, and the processing time of each process. In this embodiment, the static information of the current scheduling problem can be pre-set according to the current production needs. As an example, the pulsating production line can be applied to the production process of aircraft assembly, and the aircraft assembly pulsating production line sets the static information of the current scheduling problem according to the aircraft assembly requirements.

[0028] In practical applications, the pulsating production line includes multiple processes, which requires the reasonable allocation of various types of production resources in the production system. Among them, the scheduling problem refers to the scheduling problem of various types of production resources. The goal of the scheduling problem is generally to minimize the completion time, the most balanced production line load, or the least consumption of production resources.

[0029] Step 102: Input the static information of the current scheduling problem into the resource forecasting agent, and obtain the predicted peak value of each type of current production resources output by the resource forecasting agent.

[0030] Among them, the resource prediction agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the resource prediction training set; the resource prediction training set includes static information of historical scheduling problems and historical peak values ​​of each type of production resources.

[0031] In practical applications, the production system includes various types of production resources, such as human resources, production equipment resources, material resources, etc. In this embodiment, the resource prediction agent receives static information of the current scheduling problem to predict the peak value of each type of production resource.

[0032] Specifically, the resource prediction agent can be expressed as , the number of types of production resources is recorded as K, and the resource prediction agent Receive static information about the current scheduling problem, resource prediction agent The output of the current predicted peak value of each type of production resource is recorded as .

[0033] In addition, in order to obtain a more accurate predicted peak value for each type of production resource, the static information of the current scheduling problem can be feature extracted, and then the feature extracted information can be input into the resource prediction agent. In this embodiment, there is no specific limitation on the feature extraction method. In one example, the static information of the current scheduling problem is feature extracted by the attention mechanism (AM). In another example, the static information of the current scheduling problem can be feature extracted by the graph attention network (GAT). In one possible implementation, after the above step 101, the method further includes: based on the static information of the current scheduling problem, constructing the input graph of the current scheduling problem through the graph attention network; the above step 102 specifically includes: inputting the current scheduling problem input graph into the resource prediction agent, and obtaining the predicted peak value of each type of production resource output by the resource prediction agent.

[0034] In this implementation, the current scheduling problem input graph G is a topological graph constructed by the graph attention network (GAT) based on the static information of the current scheduling problem. In the current scheduling problem input graph G, each task process is a node in the graph, and the constraint relationship between the nodes is a directed edge. The structure of the topological graph and the node information change with the solution process. For the i-th layer network, the graph attention network representation calculation formula of the j-th node is as follows:

[0035] in express The node's successor node, Indicates The value matrix in the layer network, is the attention mechanism, and its calculation formula is

[0036] in, and Respectively represent The query and key matrices in the layer network.

[0037] Specifically, the resource prediction agent The input is the current scheduling problem input graph G, and the resource prediction agent The output of the two-layer perceptron network (Multilayer Perceptron, MLP) is a Gaussian distribution of dimension K, with a variance of 1 and a mean of ,in, Resource prediction agent The probability output is is the predicted peak value of each type of current production resources.

[0038] Step 103: Input the static information of the current scheduling problem, the predicted peak value of each type of production resource, and the current scheduling situation into the process scheduling agent, and obtain the probability value of each current process output by the process scheduling agent as the next scheduling step.

[0039] Among them, the process scheduling agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the process scheduling training set; the process scheduling training set includes static information of historical scheduling problems, historical peak values ​​of each type of production resources, historical scheduling situations, and the probability value of each historical process as the next step of scheduling.

[0040] Specifically, the process scheduling agent can be expressed as , process scheduling agent It receives the static information of the current scheduling problem, the predicted peak value of each type of production resource, and the current scheduling situation, and outputs the probability value of each current process as the next scheduling through the Pointer Network (PN).

[0041] In practical applications, process scheduling agents Output distribution ,distributed The dimension is consistent with the number of processes in the pulsating production line. The probability value of each current process as the next step of scheduling can be determined.

[0042] Step 104: Determine the next scheduling plan based on the probability value of each current process being used as the next scheduling.

[0043] Step 105: According to the next step scheduling plan, the pulsating production line is scheduled.

[0044] In practical applications, according to the probability value of each current process as the next step of scheduling, an unstarted process is selected from all processes and assigned to the earliest time allowed by resources to start execution. Output distribution The situation, from the distribution In the sampling, a process that has not yet started is selected and assigned to start execution at the earliest time allowed by resources.

[0045] Step 106, determine whether there are any unstarted processes in the pulsation production line; if there are no unstarted processes in the pulsation production line, end.

[0046] Step 107: If there are still unstarted processes in the pulse production line, return to step 101.

[0047] Furthermore, after the scheduling plan is executed, it is determined whether there are any unstarted processes in the pulsating production line. If there are any unstarted processes, return to execute the above steps until there are no unstarted processes in the pulsating production line.

[0048] In the prior art, in the reinforcement learning scenario for solving combinatorial optimization problems, the typical strategy for constructing a Markov decision process (MDP) is based on the traditional greedy framework. In this method, the reinforcement learning agent dynamically selects the task with the highest priority, and the initial execution time of the task is determined by the greedy framework. The goal of the greedy algorithm is to achieve a local optimal schedule, so the start time of the task will be advanced as much as possible. Therefore, the current pulsating production line scheduling method is load-unbalanced and the scheduling accuracy of the aircraft assembly line is poor.

[0049] In response to the above technical problems, the present invention decomposes the scheduling problem of the pulsating production line into two mutually coordinated sub-problems, namely the resource prediction problem and the process scheduling problem, by constructing a two-layer reinforcement learning framework. Under the two-layer reinforcement learning framework, the resource prediction agent predicts the peak value of each type of current production resources; further, under the restriction of the predicted peak value of each type of current production resources, the process scheduling agent outputs the probability value of each current process as the next step of scheduling; then, according to the probability value of each current process as the next step of scheduling, the next step of scheduling is determined. Furthermore, the next step of scheduling is executed, and after the execution is completed, it is determined whether there are any unstarted processes in the pulsating production line. If there are any unstarted processes, return to the above steps until there are no unstarted processes in the pulsating production line, thereby achieving load balancing of the pulsating production line and improving the scheduling accuracy of the pulsating production line.

[0050] Combined with the above description, the resource forecasting agent and the process scheduling agent are both constructed based on the reinforcement learning algorithm and are pre-trained agents. Specifically, for the training process of the resource forecasting agent and the process scheduling agent, as an example, in a possible implementation, Figure 2 This is the second flow chart of the pulsating production line scheduling method provided by the present invention, such as Figure 2 As shown, before the above step 101, the method further includes: steps 201 to 203.

[0051] Step 201: construct a resource prediction training set and a process scheduling training set based on historical pulse production line assembly data.

[0052] In combination with the above description, the resource prediction training set includes the static information of the historical scheduling problem and the historical peak value of each type of production resource. The process scheduling training set includes the static information of the historical scheduling problem, the historical peak value of each type of production resource, the historical scheduling situation, and the probability value of each process as the next step of scheduling.

[0053] In practical applications, the historical pulsating production line assembly data is the actual pulsating production line assembly data in the historical stage. The historical pulsating production line assembly data at least includes: static information of the historical scheduling problem, the historical peak value of each type of production resources, and the probability value of each process in history as the next step of scheduling. As an example, during the operation of the pulsating production line, the pulsating production line assembly data is recorded in real time in the log. Correspondingly, the historical pulsating production line assembly data can be obtained from the log, and based on the historical pulsating production line assembly data, a resource prediction training set and a process scheduling training set are constructed.

[0054] Step 202: Based on the reinforcement learning algorithm, construct an initial resource forecasting agent and an initial process scheduling agent.

[0055] As an example, the initial resource prediction agent can be based on a multilayer perceptron (MLP) model. The initial process scheduling agent can be based on a reinforcement learning pointer network model (Pointer Network with Actor-Critic, PN-AC).

[0056] Step 203: Based on the resource prediction training set and the process scheduling training set, the initial resource prediction agent and the initial process scheduling agent are alternately trained until a resource prediction agent and a process scheduling agent are obtained.

[0057] In practical applications, when training the resource forecasting agent and the process scheduling agent, an alternating training mode imitating coordinate descent is adopted. Specifically, when training the resource forecasting agent, the process scheduling agent is placed in the evaluation (eval) mode, that is, the process scheduling agent is regarded as the external environment of the resource forecasting agent. The static information of the historical scheduling problem is input into the current resource forecasting agent to obtain the predicted peak value of each type of production resource output by the current resource forecasting agent. The current resource forecasting agent receives the current reward function fed back by the external environment and optimizes the current resource forecasting agent according to the current reward function. By comparing the predicted peak value of each type of production resource output by the current resource forecasting agent with the historical peak value of each type of production resource, the performance of the current resource forecasting agent can be evaluated.

[0058] Specifically, the resource prediction agent The reward function is expressed as minimizing the total number of production resources, that is, In practical applications, in order to prevent resource prediction agents Output is too low for total production resources. If the process scheduling agent Unable to predict the agent in the resource Given the production resources, the resource forecasting agent builds a scheduling solution that meets the requirements. A significant penalty term is added to the reward function.

[0059] Correspondingly, when training the process scheduling agent, the resource prediction agent is placed in the evaluation (eval) mode, that is, the resource prediction agent is used as the external environment of the process scheduling agent. The static information of the historical scheduling problem, the historical peak value of each type of production resource, and the historical scheduling situation are input into the current process scheduling agent to obtain the probability value of each current process as the next step of scheduling output by the current process scheduling agent. The current process scheduling agent receives the corresponding reward function and optimizes the current process scheduling agent according to the reward function. By comparing the probability value of each current process as the next step of scheduling with the probability value of each historical process as the next step of scheduling, the performance of the current process scheduling agent can be evaluated.

[0060] Specifically, the process scheduling agent After a schedule is fully generated, the external environment will evaluate and return the reward signal for the entire decision trajectory. The reward function is the inverse of the total time to complete all processes. If the final generated schedule fails to meet all constraints, a significant penalty term will also be added to the reward function.

[0061] As an example, when training resource forecasting agents and process scheduling agents, you can use algorithms such as reinforcement learning (REINFORCE), proximal policy optimization (PPO), and advantage actor-critic (A2C) for optimization. As an example, the REINFORCE algorithm formula is as follows:

[0062] Among them, represents a set of generated heuristic solutions, represents the reward evaluation for this set of solutions, Indicates in the problem The baseline under (baseline), using the baseline REINFORCE algorithm to train can reduce the variance.

[0063] Combined with the above description, the resource prediction agent The input can be the current scheduling problem input graph G, and the resource prediction agent The output is a Gaussian distribution with dimension K. In practical applications, when training resource prediction agents When The actions can be sampled from the Gaussian distribution, and the mean of the Gaussian distribution will be used in the evaluation. The actions of the resource prediction agent are well explainable. In order to achieve more stable and efficient exploration, For example, the sampling result must be an integer and can only be within the Gaussian distribution mean. Within the range of , the value of a certain production resource is increased or decreased respectively. On this basis, the resource prediction agent Based on the reward function finally returned, we can further optimize the production resources that can be reduced, or jump out of the illegal solution area.

[0064] In this embodiment, based on the resource prediction training set and the process scheduling training set, the resource prediction agent and the process scheduling agent are pre-trained to obtain a two-layer reinforcement learning model. On this basis, the scheduling problem of the pulsating production line is decomposed into two sub-problems: resource prediction and process scheduling. The peak value of each type of production resource is predicted by the resource prediction intelligence. Then, based on the predicted peak value of each type of production resource, the process scheduling agent outputs the probability value of each current process as the next step of scheduling. Then, according to the probability value of each current process as the next step of scheduling, the next step of scheduling is determined. Further, the next step of scheduling is executed, and after the execution is completed, whether there are still unstarted processes in the pulsating production line, if there are still unstarted processes, return to the above steps until there are no unstarted processes in the pulsating production line, so as to achieve the load balance of the pulsating production line and improve the scheduling accuracy of the pulsating production line.

[0065] In practical applications, a pulsating production line involves multiple processes. For example, a complex aircraft assembly pulsating production line involves dozens of workstations and tens of thousands of processes. The training scale of the reinforcement learning agent is large, and it is difficult for the agent to obtain meaningful strategies in the exploration space of this scale at the beginning of training. The training is time-consuming and difficult. In order to improve the efficiency of training, as an example, in a possible implementation, the initial resource prediction agent and the initial process scheduling agent are alternately trained in the above step 203, specifically including: through the training method of course learning, gradually increasing the training scale, and alternately training the initial resource prediction agent and the initial process scheduling agent.

[0066] In this embodiment, the agent is trained by gradually increasing the training scale, that is, the agent learns from easy to difficult. For example, first let the agent be trained on the assembly problem of LB=50 processes to evaluate the learning of the agent. Then, according to the function value of the reward function obtained by the agent, the upper limit UB of the scale of solving the problem is gradually increased until the assembly problem of a given size can be solved. The scale of each training is sampled from the uniform distribution U(LB, UB). It can be understood that the method in this embodiment can effectively reduce the difficulty of the agent to explore difficult problems, can effectively reduce the training time, and improve the efficiency of training.

[0067] Specifically, for the aircraft assembly problem, the solution process is to decompose the original problem into smaller sub-problems. For example, an aircraft assembly problem with 100 processes can be regarded as an aircraft assembly problem with 97 processes after completing 3 processes. Therefore, training on smaller-scale problems has a positive benefit for learning large-scale problems, which is why the scale of course learning training is not directly increased, but sampling is used. However, during the agent training process, larger-scale problems can obtain potentially higher rewards, while small-scale problems have lower rewards, which will lead to unstable optimization. In order to improve the stability and reliability of agent training, as an example, in a possible implementation, the above-mentioned pulsating production line scheduling method also includes: normalizing the reward function under different training scales; during the training process, after the currently trained agent performs an action, the current reward function of the external environment feedback is obtained, and the current reward function is input into the currently trained agent to optimize the currently trained agent.

[0068] In this implementation, the reward functions under different training scales are normalized to ensure that the reward functions under different training scales are on the same basis, thereby improving the stability and reliability of the agent training.

[0069] Combined with the above description, after obtaining the probability value of each current process as the next scheduling output by the process scheduling agent, the next scheduling plan is determined according to the probability value of each current process as the next scheduling. In one example, the next scheduling plan is determined by a single reasoning method. Specifically, based on the greedy algorithm, the process with the largest probability value is directly used as the next process, and it is assigned to the earliest time allowed by resources for execution.

[0070] In another example, the next scheduling plan is determined by multiple reasoning. As an example, in one possible implementation, the above step 104 includes: if the variance of the probability value of each current process as the next scheduling is greater than a first threshold, then based on the search multiple reasoning method, the next scheduling plan is determined; if the variance of the probability value of each current process as the next scheduling is not greater than the first threshold, then based on the data augmentation method, multiple sub-problems corresponding to the current scheduling problem are constructed; the solution of each sub-problem is calculated to obtain multiple equivalent solutions corresponding to the current scheduling problem; multiple equivalent solutions corresponding to the current scheduling problem are evaluated by an evaluation function, and the optimal solution is used as the next scheduling plan.

[0071] During the reasoning process, when the training of the reinforcement learning agent is close to convergence, the variance of the probability value of each current process as the next step of scheduling is very small, and it is impossible to use the search technology to sample and obtain solutions with large differences, which makes it easier to fall into the local extreme value of the problem. In this implementation, based on the data augmentation method, multiple sub-problems corresponding to the current scheduling problem are constructed; the solution of each sub-problem is calculated to obtain multiple equivalent solutions corresponding to the current scheduling problem; the multiple equivalent solutions corresponding to the current scheduling problem are evaluated by the evaluation function, and the optimal solution is used as the next scheduling solution, thereby improving the accuracy of the pulsating production line scheduling. Correspondingly, when the variance of the probability value of each current process as the next step of scheduling is greater than the first threshold, the multiple inference method of search is used to determine the next scheduling solution.

[0072] Specifically, the data augmentation method can be a directed acyclic graph (DAG) modeling method. As an example, in view of the characteristics of multiple parallel processes in the aircraft assembly problem and the use of directed acyclic graph (DAG) to model dependencies, the present invention proposes to solve several sub-problems of the original problem in addition to solving the original problem, and the solution space of these sub-problems is also a subset of the original problem. Specifically, on the one hand, randomly deleting several nodes with a degree of 0 and their corresponding edges in the DAG and adding them to the scheduling plan is equivalent to directly starting some unconstrained processes. On the other hand, randomly adding several edges to the DAG is equivalent to adding some constraints on the processing procedures. The DAGs of these sub-problems are very different from the original problem, which will lead to different strategies of the intelligent agent. In this way, some equivalent solutions to the original problem can be obtained without reducing the quality of the solution, and then the optimal solution is found through the evaluation function as the final output scheduling plan.

[0073] In the pulsating production line scheduling method provided in this embodiment, by inputting the static information of the current scheduling problem into the resource prediction agent, the predicted peak value of each type of production resource output by the resource prediction agent is obtained; the static information of the current scheduling problem, the predicted peak value of each type of production resource and the current scheduling situation are input into the process scheduling agent, and the probability value of each current process as the next scheduling is obtained; according to the probability value of each current process as the next scheduling, the next scheduling plan is determined; according to the next scheduling plan, the pulsating production line is scheduled; if there are still unstarted processes in the pulsating production line, the above steps are returned to execute, otherwise, the pulsating production line scheduling is terminated. The solution of this embodiment adopts a two-layer reinforcement learning model to decompose the scheduling problem of the pulsating production line into two sub-problems of resource prediction and process scheduling. The peak value of each type of production resource is predicted by the resource prediction intelligence. Then, based on the predicted peak value of each type of production resource, the process scheduling agent outputs the probability value of each current process as the next scheduling, and then, according to the probability value of each current process as the next scheduling, the next scheduling plan is determined. Furthermore, the next step of scheduling plan is executed, and after the execution is completed, it is determined whether there are any unstarted processes in the pulsating production line. If there are any unstarted processes in the pulsating production line, the above steps are returned; otherwise, the scheduling of the pulsating production line is terminated, the load balance of the pulsating production line is achieved, and the scheduling accuracy of the pulsating production line is improved.

[0074] The pulsating production line scheduling device provided by the present invention is described below. The pulsating production line scheduling device described below and the pulsating production line scheduling method described above can be referred to each other.

[0075] Figure 3 It is a structural schematic diagram of the pulsating production line scheduling device provided by the present invention, such as Figure 3 As shown, the pulse production line scheduling device includes: an acquisition module 31, a first input module 32, a second input module 33, a determination module 34, an execution module 35 and a processing module 36.

[0076] The acquisition module 31 is used to acquire the static information of the current scheduling problem of the pulsating production line; wherein the static information of the current scheduling problem includes the topological relationship between all processes in the pulsating production line, the number of workers required for each process, and the processing time of each process.

[0077] In this embodiment, the static information of the current scheduling problem can be pre-set according to the current production needs. As an example, the pulsating production line can be applied to the production process of aircraft assembly, and the aircraft assembly pulsating production line sets the static information of the current scheduling problem according to the aircraft assembly needs.

[0078] In practical applications, the pulsating production line includes multiple processes, which requires the reasonable allocation of various types of production resources in the production system. Among them, the scheduling problem refers to the scheduling problem of various types of production resources. The goal of the scheduling problem is generally to minimize the completion time, the most balanced production line load, or the least consumption of production resources.

[0079] The above-mentioned first input module 32 is used to input the static information of the current scheduling problem into the resource prediction agent, and obtain the predicted peak value of each type of production resource output by the resource prediction agent; wherein, the resource prediction agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the resource prediction training set; the resource prediction training set includes the static information of historical scheduling problems and the historical peak value of each type of production resources.

[0080] In practical applications, the production system includes various types of production resources, such as human resources, production equipment resources, material resources, etc. In this embodiment, the resource prediction agent receives static information of the current scheduling problem to predict the peak value of each type of production resource.

[0081] Specifically, the resource prediction agent can be expressed as , the number of types of production resources is recorded as K, and the resource prediction agent Receive static information about the current scheduling problem, resource prediction agent The output of the current predicted peak value of each type of production resource is recorded as .

[0082] In addition, in order to obtain a more accurate predicted peak value for each type of production resource, the static information of the current scheduling problem can be feature extracted, and then the feature-extracted information can be input into the resource prediction agent. In this embodiment, there is no specific limitation on the feature extraction method. In one example, the static information of the current scheduling problem is feature extracted through the attention mechanism (AM). In another example, the static information of the current scheduling problem can be feature extracted through the graph attention network (GAT). In a possible implementation, the pulsating production line scheduling device further includes: a feature extraction module, which is used to construct a current scheduling problem input graph through a graph attention network based on the static information of the current scheduling problem; the first input module 32 is specifically used to: input the current scheduling problem input graph into the resource prediction agent to obtain the current predicted peak value of each type of production resource output by the resource prediction agent.

[0083] In this implementation, the current scheduling problem input graph G is a topological graph constructed by the graph attention network (GAT) based on the static information of the current scheduling problem. In the current scheduling problem input graph G, each task process is a node in the graph, and the constraint relationship between the nodes is a directed edge. The structure of the topological graph and the node information change with the solution process. For the i-th layer network, the graph attention network representation calculation formula of the j-th node is as follows:

[0084] in express The node's successor node, Indicates The value matrix in the layer network, is the attention mechanism, and its calculation formula is

[0085] in, and Respectively represent The query and key matrices in the layer network.

[0086] Specifically, the resource prediction agent The input is the current scheduling problem input graph G, and the resource prediction agent The output of the two-layer perceptron network (Multilayer Perceptron, MLP) is a Gaussian distribution of dimension K, with a variance of 1 and a mean of ,in, Resource prediction agent The probability output is is the predicted peak value of each type of current production resources.

[0087] The above-mentioned second input module 33 is used to input the static information of the current scheduling problem, the current predicted peak value of each type of production resources and the current scheduling situation into the process scheduling agent, and obtain the probability value of each current process output by the process scheduling agent as the next step of scheduling; wherein, the process scheduling agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the process scheduling training set; the process scheduling training set includes the static information of historical scheduling problems, the historical peak value of each type of production resources, the historical scheduling situation and the probability value of each historical process as the next step of scheduling.

[0088] Specifically, the process scheduling agent can be expressed as , process scheduling agent Receive the static information of the current scheduling problem, the predicted peak value of each type of production resource, and the current scheduling situation, and output the probability value of each current process as the next scheduling through the Pointer Network (PN).

[0089] In practical applications, process scheduling agents Output distribution ,distributed The dimension is consistent with the number of processes in the pulsating production line. The probability value of each current process as the next step of scheduling can be determined.

[0090] The determination module 34 is used to determine the next scheduling plan according to the probability value of each current process being used as the next scheduling.

[0091] The execution module 35 is used to execute the scheduling process for the pulse production line according to the next scheduling plan.

[0092] In actual application, the execution module 35 selects an unstarted process from all processes according to the probability value of each current process as the next scheduling, and assigns it to the earliest time allowed by resources for execution. Output distribution In the case of In the sampling, a process that has not yet started is selected and assigned to start execution at the earliest time allowed by resources.

[0093] The processing module 36 is used to return to the step of obtaining static information of the current scheduling problem of the pulsating production line if there are still unstarted processes in the pulsating production line; otherwise, the scheduling of the pulsating production line is terminated.

[0094] Furthermore, after the scheduling plan is executed, the processing module 36 determines whether there are any unstarted processes in the pulsating production line. If there are any unstarted processes, the above steps are returned to execute until there are no unstarted processes in the pulsating production line.

[0095] In this embodiment, by constructing a two-layer reinforcement learning framework, the scheduling problem of the pulsating production line is decomposed into two mutually coordinated sub-problems, namely the resource prediction problem and the process scheduling problem. Under the two-layer reinforcement learning framework, the resource prediction agent predicts the peak value of each type of current production resources; further, under the restriction of the predicted peak value of each type of current production resources, the process scheduling agent outputs the probability value of each current process as the next step of scheduling; then, according to the probability value of each current process as the next step of scheduling, the next step of scheduling is determined. Further, the next step of scheduling is executed, and after the execution is completed, it is determined whether there are still unstarted processes in the pulsating production line. If there are still unstarted processes, return to the above steps until there are no unstarted processes in the pulsating production line, thereby achieving load balancing of the pulsating production line and improving the scheduling accuracy of the pulsating production line.

[0096] In a possible implementation, the pulse production line scheduling device further includes: a first construction module, a second construction module and a training module.

[0097] The first building module is used to build a resource prediction training set and a process scheduling training set based on historical pulse production line assembly data.

[0098] In combination with the above description, the resource prediction training set includes the static information of the historical scheduling problem and the historical peak value of each type of production resource. The process scheduling training set includes the static information of the historical scheduling problem, the historical peak value of each type of production resource, the historical scheduling situation, and the probability value of each process as the next step of scheduling.

[0099] In practical applications, the historical pulsating production line assembly data is the actual pulsating production line assembly data in the historical stage. The historical pulsating production line assembly data at least includes: static information of historical scheduling problems, historical peak values ​​of each type of production resources, and the probability value of each process in history as the next step of scheduling. As an example, during the operation of the pulsating production line, the pulsating production line assembly data is recorded in real time in the log. Correspondingly, the first construction module can obtain the historical pulsating production line assembly data from the log, and construct a resource prediction training set and a process scheduling training set based on the historical pulsating production line assembly data.

[0100] The second building module is used to build an initial resource forecasting agent and an initial process scheduling agent based on a reinforcement learning algorithm.

[0101] As an example, the initial resource forecasting agent can be based on a multi-layer perceptron network (MLP) model. The initial process scheduling agent can be based on a pointer network model (PN-AC) based on reinforcement learning.

[0102] The above training module is used to alternately train the initial resource forecasting agent and the initial process scheduling agent based on the resource forecasting training set and the process scheduling training set until the resource forecasting agent and the process scheduling agent are obtained.

[0103] In practical applications, when training the resource forecasting agent and the process scheduling agent, the training module adopts an alternating training mode that imitates coordinate descent. Specifically, when training the resource forecasting agent, the training module puts the process scheduling agent in the evaluation (eval) mode, that is, the process scheduling agent is regarded as the external environment of the resource forecasting agent. The static information of the historical scheduling problem is input into the current resource forecasting agent to obtain the predicted peak value of each type of production resource output by the current resource forecasting agent. The current resource forecasting agent receives the current reward function fed back by the external environment and optimizes the current resource forecasting agent according to the current reward function. By comparing the predicted peak value of each type of production resource output by the current resource forecasting agent with the historical peak value of each type of production resource, the performance of the current resource forecasting agent can be evaluated.

[0104] Specifically, the resource prediction agent The reward function is expressed as minimizing the total number of production resources, that is, In practical applications, in order to prevent resource prediction agents Output is too low for total production resources. If the process scheduling agent Unable to predict the agent in the resource Given the production resources, the resource forecasting agent builds a scheduling solution that meets the requirements. A significant penalty term is added to the reward function.

[0105] Correspondingly, when training the process scheduling agent, the training module puts the resource prediction agent into the evaluation (eval) mode, that is, the resource prediction agent is used as the external environment of the process scheduling agent. The static information of the historical scheduling problem, the historical peak value of each type of production resource, and the historical scheduling situation are input into the current process scheduling agent to obtain the probability value of each current process as the next step of scheduling output by the current process scheduling agent. The current process scheduling agent receives the corresponding reward function and optimizes the current process scheduling agent according to the reward function. By comparing the probability value of each current process as the next step of scheduling with the probability value of each historical process as the next step of scheduling, the performance of the current process scheduling agent can be evaluated.

[0106] Specifically, the process scheduling agent After a schedule is fully generated, the external environment will evaluate and return the reward signal for the entire decision trajectory. The reward function is the inverse of the total time to complete all processes. If the final generated schedule fails to meet all constraints, a significant penalty term will also be added to the reward function.

[0107] As an example, when training resource forecasting agents and process scheduling agents, algorithms such as reinforcement learning (REINFORCE), proximal policy optimization (PPO), and advantage actor-critic (A2C) can be used for optimization.

[0108] Combined with the above description, the resource prediction agent The input can be the current scheduling problem input graph G, and the resource prediction agent The output is a Gaussian distribution with dimension K. In practical applications, when training resource prediction agents When The actions can be sampled from the Gaussian distribution, and the mean of the Gaussian distribution will be used in the evaluation. The actions of the resource prediction agent are well explainable. In order to achieve more stable and efficient exploration, On this basis, the resource prediction agent Based on the reward function finally returned, we can further optimize the production resources that can be reduced, or jump out of the illegal solution area.

[0109] In this embodiment, based on the resource prediction training set and the process scheduling training set, the resource prediction agent and the process scheduling agent are pre-trained to obtain a two-layer reinforcement learning model. On this basis, the scheduling problem of the pulsating production line is decomposed into two sub-problems: resource prediction and process scheduling. The peak value of each type of production resource is predicted by the resource prediction intelligence. Then, based on the predicted peak value of each type of production resource, the process scheduling agent outputs the probability value of each current process as the next step of scheduling. Then, according to the probability value of each current process as the next step of scheduling, the next step of scheduling is determined. Further, the next step of scheduling is executed, and after the execution is completed, it is determined whether there are still unstarted processes in the pulsating production line. If there are still unstarted processes, return to the above steps until there are no unstarted processes in the pulsating production line, thereby achieving load balancing of the pulsating production line and improving the scheduling accuracy of the pulsating production line.

[0110] In order to improve the efficiency of training, as an example, in one possible implementation, when the above-mentioned training module is used to alternately train the initial resource forecasting intelligent agent and the initial process scheduling intelligent agent, it is specifically used to: gradually increase the training scale through the course learning training method, and alternately train the initial resource forecasting intelligent agent and the initial process scheduling intelligent agent.

[0111] In this embodiment, the intelligent agent is trained by gradually increasing the training scale, that is, the intelligent agent learns from easy to difficult, which can effectively reduce the difficulty of the intelligent agent in exploring difficult problems, effectively reduce the training time, and improve the efficiency of training.

[0112] In order to improve the stability and reliability of agent training, as an example, in one possible implementation, the above-mentioned pulsating production line scheduling device also includes: a normalization module, used to normalize the reward function under different training scales; a third input module, used in the training process, after the currently trained agent performs an action, to obtain the current reward function of the external environment feedback, and input the current reward function into the currently trained agent to optimize the currently trained agent.

[0113] In this implementation, the reward functions under different training scales are normalized to ensure that the reward functions under different training scales are on the same basis, thereby improving the stability and reliability of the agent training.

[0114] Combined with the above description, after obtaining the probability value of each current process as the next scheduling output by the process scheduling agent, the next scheduling plan is determined according to the probability value of each current process as the next scheduling. In one example, the next scheduling plan is determined by a single reasoning method. Specifically, based on the greedy algorithm, the process with the largest probability value is directly used as the next process, and it is assigned to the earliest time allowed by resources for execution.

[0115] In another example, the next scheduling plan is determined by multiple reasoning. As an example, in one possible implementation, the above-mentioned determination module 34 is specifically used to: if the variance of the probability value of each current process as the next step of scheduling is greater than a first threshold, then based on the search multiple reasoning method, determine the next scheduling plan; if the variance of the probability value of each current process as the next step of scheduling is not greater than the first threshold, then based on the data augmentation method, construct multiple sub-problems corresponding to the current scheduling problem; calculate the solution of each sub-problem to obtain multiple equivalent solutions corresponding to the current scheduling problem; evaluate multiple equivalent solutions corresponding to the current scheduling problem through an evaluation function, and use the optimal solution as the next scheduling plan.

[0116] During the reasoning process, when the training of the reinforcement learning agent is close to convergence, the variance of the probability value of each current process as the next step of scheduling is very small, and it is impossible to use the search technology to sample and obtain solutions with large differences, which makes it easier to fall into the local extreme value of the problem. In this implementation, based on the data augmentation method, multiple sub-problems corresponding to the current scheduling problem are constructed; the solution of each sub-problem is calculated to obtain multiple equivalent solutions corresponding to the current scheduling problem; the multiple equivalent solutions corresponding to the current scheduling problem are evaluated by the evaluation function, and the optimal solution is used as the next scheduling solution, thereby improving the accuracy of the pulsating production line scheduling. Correspondingly, when the variance of the probability value of each current process as the next step of scheduling is greater than the first threshold, the multiple inference method of search is used to determine the next scheduling solution.

[0117] In the pulsating production line scheduling device provided by the present invention, the acquisition module acquires the static information of the current scheduling problem of the pulsating production line; the first input module obtains the predicted peak value of each type of production resource output by the resource prediction agent by inputting the static information of the current scheduling problem into the resource prediction agent; the second input module inputs the static information of the current scheduling problem, the predicted peak value of each type of production resource and the current scheduling situation into the process scheduling agent, and obtains the probability value of each current process as the next step of scheduling output by the process scheduling agent; the determination module determines the next scheduling plan according to the probability value of each current process as the next step of scheduling; the execution module performs scheduling processing on the pulsating production line according to the next step of scheduling; if there are still processes that have not been started in the pulsating production line, the processing module returns to execute the above steps; otherwise, the processing module ends the pulsating production line scheduling. The solution of this embodiment adopts a two-layer reinforcement learning model to decompose the scheduling problem of the pulsating production line into two sub-problems: resource prediction and process scheduling. The peak value of each type of production resource is predicted through resource prediction intelligence. Then, based on the predicted peak value of each type of production resource, the process scheduling agent outputs the probability value of each current process as the next step of scheduling. Then, according to the probability value of each current process as the next step of scheduling, the next step of scheduling is determined. Further, the next step of scheduling is executed, and after the execution is completed, it is determined whether there are still unstarted processes. If there are still unstarted processes in the pulsating production line, the above steps are returned to execute. Otherwise, the scheduling of the pulsating production line is terminated, the load balance of the pulsating production line is achieved, and the scheduling accuracy of the pulsating production line is improved.

[0118] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communications bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other via the communications bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the pulsating production line scheduling method, which includes: obtaining static information of the current scheduling problem of the pulsating production line; inputting the static information of the current scheduling problem into the resource prediction agent, and obtaining the predicted peak value of each type of production resource output by the resource prediction agent; wherein the resource prediction agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a resource prediction training set; the resource prediction training set includes the static information of historical scheduling problems and the peak value of each type of production resources in history; the static information of the current scheduling problem, the predicted peak value of each type of production resources and the current shift situation are input into the process scheduling agent, and the process scheduling is obtained. The process scheduling agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the process scheduling training set; the process scheduling training set includes the static information of historical scheduling problems, the peak values ​​of each type of production resources in history, the historical scheduling conditions and the probability value of each process in history as the next scheduling; according to the probability value of each current process as the next scheduling, the next scheduling plan is determined; according to the next scheduling plan, the pulsating production line is scheduled; if there are still processes that have not been started in the pulsating production line, return to the step of obtaining the static information of the current scheduling problem of the pulsating production line; otherwise, the pulsating production line scheduling is terminated.

[0119] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pulsating production line scheduling method provided by the above methods, the method comprising: obtaining static information of the current scheduling problem of the pulsating production line; inputting the static information of the current scheduling problem into a resource prediction intelligent agent, and obtaining the predicted peak value of each type of current production resource output by the resource prediction intelligent agent; wherein the resource prediction intelligent agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a resource prediction training set; the resource prediction training set includes static information of historical scheduling problems and historical peak values ​​of each type of production resources; the static information of the current scheduling problem and the current peak values ​​of each type of production resources are input into a resource prediction intelligent agent, and the resource prediction intelligent agent outputs the predicted peak value of each type of current production resource ... pre-trained based on a resource prediction training set; the resource prediction training set includes static information of historical scheduling problems and historical peak values ​​of each type of production resources; the static information of the current scheduling problem and the current peak values ​​of each type of current production resources are input into a resource prediction intelligent agent, and the resource prediction intelligent agent outputs the predicted peak value of each type of current production resource; wherein the resource prediction intelligent agent is pre-trained based on a resource prediction training set; the resource prediction training set includes static information of historical scheduling problems and historical peak values ​​of each type of production resources; the resource prediction intelligent agent outputs the predicted peak value of each type of current production resource; the resource prediction intelligent agent outputs the predicted peak value of each type of current production resource; the resource prediction intelligent agent outputs the predicted peak value of each type The predicted peak value of resources and the current scheduling situation are input into the process scheduling agent, and the probability value of each current process as the next scheduling is obtained from the output of the process scheduling agent; wherein, the process scheduling agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the process scheduling training set; the process scheduling training set includes static information of historical scheduling problems, historical peak values ​​of each type of production resources, historical scheduling situations and the probability value of each historical process as the next scheduling; according to the probability value of each current process as the next scheduling, the next scheduling plan is determined; according to the next scheduling plan, the pulsating production line is scheduled; if there are still processes that have not been started in the pulsating production line, return to the step of obtaining the static information of the current scheduling problem of the pulsating production line; otherwise, the pulsating production line scheduling is terminated.

[0121] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the pulsating production line scheduling method provided by the above methods, the method comprising: obtaining static information of the current scheduling problem of the pulsating production line; inputting the static information of the current scheduling problem into a resource prediction agent, and obtaining the predicted peak value of each type of production resource output by the resource prediction agent; wherein the resource prediction agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a resource prediction training set; the resource prediction training set includes the static information of historical scheduling problems and the peak values ​​of each type of production resources in history; the static information of the current scheduling problem, the predicted peak value of each type of production resources and the current scheduling situation are input into a resource prediction agent; The situation is input into the process scheduling agent, and the probability value of each current process as the next step of scheduling is obtained from the output of the process scheduling agent; wherein, the process scheduling agent is constructed based on the reinforcement learning algorithm and is pre-trained based on the process scheduling training set; the process scheduling training set includes the static information of the historical scheduling problem, the peak value of each type of production resources in history, the historical scheduling situation and the probability value of each historical process as the next step of scheduling; according to the probability value of each current process as the next step of scheduling, the next scheduling plan is determined; according to the next scheduling plan, the pulsating production line is scheduled; if there are still processes that have not been started in the pulsating production line, return to the step of obtaining the static information of the current scheduling problem of the pulsating production line; otherwise, the pulsating production line scheduling is terminated.

[0122] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pulse production line scheduling method, characterized in that: include: Obtain static information about the current scheduling problem of the pulsating production line; Input the static information of the current scheduling problem into the resource prediction agent, and obtain the predicted peak value of each type of production resource output by the resource prediction agent; wherein the resource prediction agent is constructed based on a reinforcement learning algorithm and pre-trained based on a resource prediction training set; the resource prediction training set includes the static information of historical scheduling problems and the peak value of each type of production resource in history; The static information of the current scheduling problem, the predicted peak value of each type of production resource and the current shift scheduling are input into the process scheduling agent, and the probability value of each current process output by the process scheduling agent as the next step of scheduling is obtained; wherein the process scheduling agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a process scheduling training set; the process scheduling training set includes the static information of the historical scheduling problem, the peak value of each type of production resource, the historical shift scheduling and the probability value of each historical process as the next step of scheduling; Determine the next scheduling plan according to the probability value of each current process as the next scheduling plan; perform scheduling processing on the pulsating production line according to the next scheduling plan; If there are still unstarted processes in the pulsating production line, the process returns to the step of obtaining static information of the current scheduling problem of the pulsating production line; otherwise, the scheduling of the pulsating production line is terminated.

2. The pulsating production line scheduling method according to claim 1, characterized in that: Before obtaining the static information of the current scheduling problem of the pulse production line, the method further includes: Based on the historical pulse production line assembly data, construct the resource prediction training set and the process scheduling training set; Based on the reinforcement learning algorithm, construct the initial resource forecasting agent and the initial process scheduling agent; Based on the resource forecasting training set and the process scheduling training set, the initial resource forecasting agent and the initial process scheduling agent are alternately trained until the resource forecasting agent and the process scheduling agent are obtained.

3. The pulsating production line scheduling method according to claim 2, characterized in that: The alternate training of the initial resource prediction agent and the initial process scheduling agent specifically includes: By using a course learning training method, the training scale is gradually increased, and the initial resource forecasting agent and the initial process scheduling agent are trained alternately.

4. The pulsating production line scheduling method according to claim 3 is characterized in that: The method further comprises: Normalize the reward function at different training scales; During the training process, after the currently trained agent performs an action, a current reward function of the external environment feedback is obtained, and the current reward function is input into the currently trained agent to optimize the currently trained agent.

5. The pulsating production line scheduling method according to claim 1, characterized in that: The step of determining the next scheduling plan according to the probability value of each current process being used as the next scheduling includes: If the variance of the probability value of each current process as the next scheduling is greater than a first threshold, then based on the multiple reasoning method of the search, determine the next scheduling plan; If the variance of the probability value of each current process as the next step of scheduling is not greater than the first threshold, then based on the data augmentation method, multiple sub-problems corresponding to the current scheduling problem are constructed; the solution of each sub-problem is calculated to obtain multiple equivalent solutions corresponding to the current scheduling problem; the multiple equivalent solutions corresponding to the current scheduling problem are evaluated by an evaluation function, and the optimal solution is used as the scheduling plan for the next step.

6. The pulsating production line scheduling method according to any one of claims 1 to 5, characterized in that: After obtaining the static information of the current scheduling problem of the pulse production line, the method further includes: Based on the static information of the current scheduling problem, construct an input graph of the current scheduling problem through a graph attention network; The step of inputting the static information of the current scheduling problem into the resource prediction agent specifically includes: The current scheduling problem input graph is input into the resource prediction agent.

7. A pulsating production line scheduling device, characterized in that: include: The acquisition module is used to obtain static information of the current scheduling problem of the pulsating production line; A first input module is used to input the static information of the current scheduling problem into the resource prediction agent, and obtain the predicted peak value of each type of production resource output by the resource prediction agent; wherein the resource prediction agent is constructed based on a reinforcement learning algorithm and pre-trained based on a resource prediction training set; the resource prediction training set includes the static information of the historical scheduling problem and the peak value of each type of production resource in history; The second input module is used to input the static information of the current scheduling problem, the predicted peak value of each type of production resources and the current scheduling situation into the process scheduling agent, and obtain the probability value of each current process output by the process scheduling agent as the next step of scheduling; wherein the process scheduling agent is constructed based on a reinforcement learning algorithm and is pre-trained based on a process scheduling training set; the process scheduling training set includes the static information of the historical scheduling problem, the peak value of each type of production resources in the history, the historical scheduling situation and the probability value of each process in the history as the next step of scheduling; A determination module, used to determine the next scheduling plan according to the probability value of each current process being used as the next scheduling; An execution module, used for executing a scheduling process on the pulse production line according to the scheduling plan for the next step; The processing module is used to return to the step of obtaining static information of the current scheduling problem of the pulsating production line if there are still unstarted processes in the pulsating production line; otherwise, the scheduling of the pulsating production line is terminated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the pulsating production line scheduling method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the pulsating production line scheduling method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the pulsating production line scheduling method according to any one of claims 1 to 6 is implemented.

Images