Production scheduling plan generation method and device, equipment and medium

Abstract

The embodiment of the invention provides a production scheduling plan generation method and device, equipment and a medium, which are used for solving the problems of difficulty in coping with a complicated and changeable production environment, low production efficiency and the like in related technologies. The electronic equipment inputs the randomly determined identifier of the target machine for executing each process and the sequence of the corresponding process executed by the target machine into the pre-trained reinforcement learning model, and obtains the identifier of the standard machine for executing each process output by the reinforcement learning model and the standard sequence of the corresponding process executed by the standard machine; through the reinforcement learning model, the optimal standard execution machine and order of each process can be determined, production tasks can be distributed to each machine more reasonably, resources are ensured to be utilized optimally, and waiting time and production bottlenecks are reduced. According to the method, the production scheduling plan can be accurately and effectively determined, so that a complicated and changeable production environment can be effectively handled, and the production efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of genetics and reinforcement learning technology, and in particular to a method, device, equipment and medium for generating a production schedule. Background Art

[0002] With the rapid development of information technology, manufacturing companies are increasingly inclined to adopt advanced production scheduling technologies to improve production efficiency, reduce costs and improve product quality. Production scheduling, that is, production planning and scheduling, is a key link in the manufacturing industry, which is directly related to production efficiency, cost control and product quality. The traditional scheduling method mainly relies on the experience and intuition of production managers to determine the scheduling plan and determines the scheduling plan through preset rules and algorithms. However, the two methods in the existing technology can no longer meet the needs of the more complex production process of modern manufacturing industry, and there are problems such as low production efficiency. Summary of the invention

[0003] The embodiments of the present application provide a production scheduling generation method, device, equipment and medium, which are used to solve the problems in the related technology that it is difficult to cope with complex and changeable production environments and there is low production efficiency.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a production schedule, the method comprising:

[0005] Obtain each task carried in the received production scheduling request; obtain each process included in each task;

[0006] For each of the processes, randomly determine a target machine among each machine executing the process, and randomly determine the order in which the target machine executes the process in each process executed by the target machine;

[0007] The identification of the target machine that executes each process and the order in which the target machine executes the corresponding process are input into a pre-trained reinforcement learning model, and the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process are obtained based on the reinforcement learning model.

[0008] In a second aspect, an embodiment of the present application further provides a production scheduling plan generating device, the device comprising:

[0009] An acquisition module, used to acquire each task carried in the received production scheduling request; and acquire each process included in each task;

[0010] A determination module, configured to randomly determine a target machine among each machine executing the process for each process, and randomly determine an order in which the target machine executes the process in each process executed by the target machine;

[0011] The processing module is used to input the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process into a pre-trained reinforcement learning model, and obtain the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process based on the reinforcement learning model.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0013] A memory for storing program instructions;

[0014] The processor is used to call the program instructions stored in the memory and execute the steps included in any one of the above-mentioned methods for generating a production schedule according to the obtained program instructions.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for generating a production schedule as described in any one of the above items.

[0016] Since in the embodiment of the present application, the electronic device randomly determines the target machine for each process in each machine executing the process, and randomly determines the order in which the target machine executes the process, the identification of the target machine executing each process and the order in which the target machine executes the corresponding process are input into the pre-trained reinforcement learning model, and the identification of the standard machine executing each process output by the reinforcement learning model and the standard order in which the standard machine executes the corresponding process are obtained, the optimal standard execution machine and order for each process can be determined through the reinforcement learning model, and the production tasks can be more reasonably allocated to each machine, ensuring that resources are optimally utilized, reducing waiting time and production bottlenecks. The production schedule can be determined accurately and effectively, so that the complex and changing production environment can be effectively responded to and production efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A process diagram of a method for generating a production schedule provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a problem space provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a production scheduling plan provided in an embodiment of the present application;

[0021] Figure 4 A structural diagram of a production scheduling plan generating device provided in an embodiment of the present application;

[0022] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0024] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0025] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0026] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0027] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0028] In order to cope with the complex and ever-changing production environment, accurately and effectively determine the production schedule, and improve production efficiency, the embodiments of the present application provide a production schedule generation method, device, equipment and medium.

[0029] The production scheduling plan generation method includes: the electronic device obtains each task carried in the received production scheduling request; obtains each process included in each task; for each process, randomly determines a target machine in each machine that executes the process, and randomly determines the order in which the target machine executes the process in each task; inputs the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process in the task into a pre-trained reinforcement learning model, and based on the reinforcement learning model, obtains the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process in each task.

[0030] Embodiment 1:

[0031] Figure 1 A process diagram of a production scheduling method provided in an embodiment of the present application, the process includes the following steps:

[0032] S101: Obtain each task carried in the received production scheduling request; obtain each process included in each task.

[0033] The production scheduling plan generation method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0034] In order to generate production schedules accurately and efficiently, when the user needs to determine the production plan for the business, the user can select each task to be performed on the preset page of the device he is using. Specifically, the selection can be the name or other identification of each task. It should be noted that different tasks have different identifications. After selecting the task to be performed, click the preset button, for example, the "OK" button, and the device used by the user can send the production schedule request to the electronic device.

[0035] After receiving the production scheduling request, the electronic device can obtain each task carried in the production scheduling request. In order to accurately determine the production scheduling plan, the electronic device can obtain each process included in each task. In a possible implementation, the electronic device stores the corresponding relationship between the business and the process, and the electronic device determines each process stored for each acquired task.

[0036] For example, each task carried in the production scheduling request is a drug task J1, a drug task J2, and a drug task J3, and the process corresponding to the drug task J1 is process O. 11 And process O 12 ; The process corresponding to drug task J2 is process O 21 And process O 22 ; The process corresponding to drug task J3 is process O 31 And process O 32 .

[0037] S102: For each process, randomly determine a target machine in each machine executing the process, and randomly determine the order in which the target machine executes the process in each process executed by the target machine.

[0038] In order to accurately and effectively determine the production schedule, the electronic device can obtain each machine that can execute the process and randomly determine the target machine among each machine that can execute the process for each process, and the electronic device can also randomly determine the order of executing the process according to each process executed by the target machine. Specifically, the electronic device can determine the number of each process executed by the target machine, and randomly determine a value based on the number, which is the order in which the target machine executes the process.

[0039] For example, the determined execution process O 11 The target machine is machine M1, and the determined process performed by M1 includes process O 11 And process O 21 The electronic device randomly determines that the target machine executes the process O in each process executed by the target machine. 11 The order is 1, and the target machine is randomly determined to execute the process O in each process executed by the target machine. 21The order is 2, which means that the target machine executes the process O first. 21 , and then execute the process O 11 .

[0040] Table 1 is a schematic diagram of the machine for performing each process provided in the embodiment of the present application:

[0041]

[0042] Table 1

[0043] The first column in Table 1 is the task identifier, the second column in Table 1 is the process identifier, and the third column in Table 1 is the selectable machine and the time it takes for the machine to execute the process. As can be seen from Table 1, the process included in Task J1 is Process O 11 and process O 12 ; The process included in task J2 is process O 21 and process O 22 ; The process included in task J3 is process O 31 and process O 32 ; Machine M1, machine M2 and machine M3 can all perform process O 11 , and execute process O 11 The durations are 4 hours, 8 hours and 4 hours respectively; Machine M1, Machine M2 and Machine M3 can all execute process O 12 , and execute process O 12 The durations are 5 hours, 6 hours and 4 hours respectively; both machine M2 and machine M3 can perform process O 21 , and execute process O 21 The durations are 5 hours and 8 hours respectively; Machine M1, Machine M2 and Machine M3 can all perform process O 22 , and execute process O 22 The durations are 5 hours, 9 hours and 6 hours respectively; Machine M1, Machine M2 and Machine M3 can all perform process O 31 , and execute process O 31 The durations are 5 hours, 4 hours and 5 hours respectively; both machine M1 and machine M3 can perform process O 32 , and execute process O 32 The durations are 6 hours and 4 hours respectively.

[0044] S103: Input the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process into a pre-trained reinforcement learning model, and obtain the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process based on the reinforcement learning model.

[0045] The electronic device can obtain the identification of the target machine that executes each process. In a possible implementation, for each process, the electronic device can obtain each machine that can execute the process, and sort each of the acquired machines according to the pre-saved sorting order of each machine, and randomly determine a numerical value in the sorting value of each machine. The machine with the sorting value of this numerical value is the randomly determined target machine, and the numerical value is determined as the identification of the randomly determined target machine that executes the process. For example, the pre-saved sorting order of each machine is machine A, machine B, machine C, and machine D, and each machine that can execute the process is machine D and machine A, then the sorting result is machine A and machine D; if the randomly determined numerical value is 1, the target machine is machine A, and the identification of the target machine is 1.

[0046] The electronic device can input the obtained identification of the target machine that executes each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model to obtain the output of the reinforcement learning model, which is the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process. Through the trained reinforcement learning model, it is possible to obtain a relatively accurate identification of the standard machine that executes the process and the standard order in which the standard machine executes the corresponding process, thereby enabling accurate and effective production scheduling.

[0047] The relevant technologies may determine the production schedule based on intelligent optimization algorithms, simulation technologies, and integrated production scheduling systems. Among them, intelligent optimization algorithms refer to genetic algorithms, ant colony algorithms, etc., which can perform optimization calculations based on actual production data to generate efficient production plans. Simulation technology is based on simulating the actual production process, predicting production results, and optimizing production plans. The integrated production scheduling system integrates production plans, material requirement plans, workshop scheduling, etc. to achieve coordination and optimization of the entire production process. However, although the application of intelligent optimization algorithms, simulation technologies, and integrated production scheduling systems has significantly improved the scientificity and accuracy of production plans, it still faces challenges in data accuracy, algorithm complexity, and system integration.

[0048] After determining the identifier of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process, the electronic device can determine the order of the processes contained in each task, where the next process can be executed only after the previous process is completed, and then determine when which machine will execute which process. It should be noted that when executing the process, the problem is described as follows: there are n work tasks and m machines; each task contains several processes in sequence (1 to k); each process can be processed by one machine; each machine can only be engaged in one process at the same time.

[0049] Figure 2 A schematic diagram of a problem space provided in an embodiment of the present application.

[0050] Depend on Figure 2 It can be seen that Task 1-Task n include Process 1-Process k respectively. In order to determine the production schedule, the electronic device determines the machine that executes each process respectively.

[0051] Taking the machines that execute each process and the time for each machine to execute the corresponding process as Table 1 as an example, the generated production schedule can be Figure 3 Schematic diagram shown.

[0052] Figure 3 A schematic diagram of a production scheduling plan provided in an embodiment of the present application.

[0053] Depend on Figure 3 It can be seen that machine M1 first performs process O 11 , and then execute process O 22 ; Machine M2 first executes process O 21 , and then execute process O 31 ; Machine M3 first executes process O 12 , and then execute process O 32 ; Among them, process O 11 And process O 12 These are two processes that are performed in sequence for the same task. Therefore, machine M3 is used after machine M1 completes process O. 11 Then execute process O 12 ; Process O 21 And process O 22 These are two processes that are performed in sequence for the same task. Therefore, machine M1 is used after machine M2 completes process O. 21 Then execute process O 22 ; Process O 31 And process O 32 These are two processes that are performed in sequence for the same task. Therefore, machine M3 is used after machine M2 completes process O. 31 Then execute process O 32 Among them, machine M1 executes O in the 0-4 hours of the start of execution 11 , perform step O in 5-9 hours 22 ; Machine M2 executes O in the 0-5 hours after the start of execution 21 , perform step O in 5-8 hours 31 ; Machine M3 executes O in the first 4-8 hours of execution 12 , perform step O in 4-9 hours 32 .

[0054] The embodiment of the present application is equivalent to providing a reinforcement learning production scheduling method based on heuristic exploration.

[0055] Since in the embodiment of the present application, the electronic device randomly determines the target machine for each process in each machine executing the process, and randomly determines the order in which the target machine executes the process, the identification of the target machine executing each process and the order in which the target machine executes the corresponding process are input into the pre-trained reinforcement learning model, and the identification of the standard machine executing each process output by the reinforcement learning model and the standard order in which the standard machine executes the corresponding process are obtained, the optimal standard execution machine and order for each process can be determined through the reinforcement learning model, and the production tasks can be more reasonably allocated to each machine, ensuring that resources are optimally utilized, reducing waiting time and production bottlenecks. The production schedule can be determined accurately and effectively, so that the complex and changing production environment can be effectively responded to and production efficiency can be improved.

[0056] Embodiment 2:

[0057] In order to improve production efficiency accurately and effectively, based on the above embodiments, in the embodiment of the present application, after the target machine is randomly determined to execute the process, and the order of each process executed by the target machine, before the identifier of the target machine executing each process and the order of the target machine executing the corresponding process are input into the pre-trained reinforcement learning model, the method further includes:

[0058] Determine in sequence the identifier of the target machine that executes each of the processes as the elements in the first row of the matrix, and use the order of the corresponding target machine executing the corresponding process in each process executed by the corresponding target machine as the elements in the second row of the matrix;

[0059] The step of inputting the identifier of the target machine that executes each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model includes:

[0060] The matrix is ​​input into a pre-trained reinforcement learning model.

[0061] In order to improve the recognition capability of the reinforcement learning model, the electronic device may generate a matrix including the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process, and the reinforcement learning model performs recognition based on the matrix.

[0062] Specifically, the electronic device may sequentially determine the identifier of the target machine executing each process as the element in the first row of the matrix, and determine the order in which the corresponding target machine executes the corresponding process as the element in the second row of the matrix. It should be noted that the identifier of the target machine executing the same process and the order in which the target machine executes the process are elements in the same column of the matrix.

[0063] After the matrix is ​​determined, it can be input into a pre-trained reinforcement learning model.

[0064] In the embodiment of the present application, the production schedule is represented by the operation sequence (OS) gene and the machine selection (MS) gene. The OS gene represents the number of processing of the corresponding process on the corresponding machine, and the MS gene represents the number of optional processing of the corresponding process on the machine. This process is equivalent to mapping the problem space to the gene space, and using genes to represent the solutions of the two sub-problems. The embodiment of the present application is equivalent to randomly generating a set of initial solutions, called a population. The solution corresponding to each element is called an individual.

[0065] Table 2 shows the identifiers of the target machines that execute each process, and the order in which the corresponding target machines execute the corresponding processes, determined in the embodiment of the present application:

[0066]

[0067]

[0068] Table 2

[0069] The first line of Table 2 is the identification of each task, the second line of Table 2 is the identification of each process included in each task, the third line of Table 2 is the identification of the machine that executes the corresponding process, and the fourth line of Table 2 is the order in which the corresponding machine executes the process. It can be seen from Table 2 that process O can be executed. 11 The first machine in each machine is the one that performs process O 11 The machine first performs process O 11 ; Process O can be executed 12 The third machine in each machine is for performing process O 12 The machine that first performs process O 12 ; Process O can be executed 21 The first machine in each machine is the one that performs process O 21 The machine first performs process O 21 ; Process O can be executed 22 The first machine in each machine is the one that performs process O 22 The machine is the second to perform process O 22 , that is, the machine executes this process after completing the previous process O 22 ; Process O can be executed 31 The second machine in each machine is for performing operation O 22 The machine is the second to perform process O 31, that is, the machine executes this process after completing the previous process O 31 ; Process O can be executed 32 The second machine in each machine is for performing operation O 32 The machine is the second to perform process O 32 , that is, the machine executes this process after completing the previous process O 32 .

[0070] Taking the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process as Table 1 as an example, the generated matrix is: 1 3 1 1 2 2 1 1 1 2 2 2

[0073] Embodiment 3:

[0074] In order to accurately and effectively determine the production scheduling plan, based on the above embodiments, in the embodiment of the present application, the identification of the standard machine that executes each process and the standard order of the standard machine executing the corresponding process in each task based on the reinforcement learning model include:

[0075] The first sub-model of the reinforcement learning model determines an intermediate matrix based on a cross product of a pre-trained parameter matrix and the matrix; wherein the dimension of the intermediate matrix is ​​the same as the dimension of the matrix;

[0076] A second sub-model of the reinforcement learning model determines a sum matrix of the matrix and the intermediate matrix;

[0077] The elements in the first row of the sum matrix are sequentially determined as the identifiers of the standard machines that perform each process, and the elements in the second row of the sum matrix are sequentially determined as the standard order in which the corresponding standard machines perform the corresponding processes in each process performed.

[0078] In order to accurately and efficiently formulate production scheduling, the reinforcement learning model contains two sub-models. The first sub-model uses the pre-trained parameter matrix and the matrix input to the reinforcement learning model to perform a cross multiplication operation to generate an intermediate matrix. The dimension of this intermediate matrix is ​​consistent with the input matrix. Then, the second sub-model calculates the sum of the input matrix and the intermediate matrix to obtain the sum matrix and uses it as the output.

[0079] When the electronic device receives the sum value matrix output by the second sub-model, the electronic device will sequentially determine the elements in the first row of the sum value matrix as the identifiers of the standard machines that perform each process. At the same time, the elements in the second row of the sum value matrix will sequentially determine the standard order of these standard machines when performing the corresponding processes, wherein the standard order of a certain standard machine performing a certain process refers to the execution order of the standard machine when it is its turn to perform the process in the process of performing all processes.

[0080] The method provided in the embodiment of the present application is equivalent to using a deep neural network as a policy network and initializing its weight parameters. The first sub-model of the deep neural network takes the parental individual gene as input and outputs a difference equal to the parental individual gene, and the second sub-model of the deep neural network adds the parental individual gene to the difference to obtain the offspring individual gene. The parental individual gene is the parental gene described in the embodiment of the present application, and the offspring individual gene is the sum matrix described in the embodiment of the present application.

[0081] Embodiment 4:

[0082] In order to accurately and effectively determine the production schedule, based on the above embodiments, in the embodiment of the present application, the reinforcement learning model is trained in the following manner:

[0083] Obtain any sample group in a pre-saved sample set, wherein the sample group includes a first preset number of sample pairs, and each sample pair includes a sample parent matrix and a sample child matrix; wherein the elements in the first row of the sample parent matrix and the sample child matrix are the identifiers of the machines that execute each sample process, and the elements in the second row are the order in which the corresponding machines execute the corresponding sample processes; for each sample pair, the total time for the machine corresponding to the identifier contained in the sample parent matrix in the sample pair to execute each sample task in the corresponding order is greater than the total time for the machine corresponding to the identifier contained in the sample child matrix in the sample pair to execute each sample task in the corresponding order;

[0084] For each sample pair, the sample parent matrix in the sample pair is input into the original reinforcement learning model to obtain the output child matrix output by the original reinforcement learning model; and the reward value is determined according to the similarity between the output child matrix and the sample child matrix in the sample pair;

[0085] According to the determined reward value of each sample pair, a target reward value of the sample group is determined; and the original reinforcement learning model is trained based on the target reward value.

[0086] In order to accurately and effectively determine the production schedule, the electronic device can train the reinforcement learning model, and, in order to realize the training of the reinforcement learning model, the electronic device locally pre-stores a sample set, which contains multiple sample groups. The electronic device can first obtain any sample group from the pre-stored sample set, and the sample group contains a first preset number of sample pairs, and each sample pair is composed of a sample parent matrix and a sample child matrix. In these two matrices, the first row records the identification of the machine that executes each sample process, and the second row lists in detail the order in which these machines execute the corresponding processes in each executed process. It is worth noting that for each sample pair, the total time required to execute all sample tasks according to the machine identification and order in the sample parent matrix is ​​greater than the total time required to execute all sample tasks according to the machine identification and order in the sample child matrix, which reflects the optimization of the efficiency of the child matrix.

[0087] For each sample pair in the sample group, the electronic device can input the sample parent matrix into the original reinforcement learning model to obtain the output of the reinforcement learning model, and the output of the reinforcement learning model is the output offspring matrix. In order to evaluate the quality of the output of the reinforcement learning model, the electronic device calculates the similarity between the output offspring matrix and the sample offspring matrix, and determines a reward value based on this similarity. In a possible implementation, the similarity can be determined as the reward value. The level of the reward value reflects the closeness of the model output to the ideal optimization state (i.e., the sample offspring matrix).

[0088] After obtaining the reward value of each sample pair in the sample group, the electronic device determines the sum of each reward value as the target reward value of the sample group, which is a comprehensive consideration of the sample pair. The electronic device uses this target reward value to train the original reinforcement learning model, aiming to improve its output quality by adjusting the parameters of the reinforcement learning model, so that the model can more accurately generate an output offspring matrix that is close to or even better than the sample offspring matrix.

[0089] Taking a sample group containing three sample pairs as an example, the sample group includes sample pair A, sample pair B and sample pair C. The situation of the sample pairs in the sample group may be that the sample parent matrix in sample pair A is the sample child matrix in sample pair B, and the sample parent matrix in sample pair B is the sample child matrix in sample pair C.

[0090] It should be noted that the electronic device can also determine a difference matrix between the sample offspring matrix and the sample parent matrix, and train only the first sub-model of the reinforcement learning model based on the difference matrix and the sample parent matrix.

[0091] In the embodiment of the present application, the goal of optimization is to maximize the target reward value of each sample group, and the parameter update can be achieved by deriving the parameters of the original reinforcement learning model. In the embodiment of the present application, the above steps are repeated from selecting and executing actions to updating the policy network until the set total number of fragments M is reached or other stopping conditions are met, such as strategy convergence, stable performance, etc. In practical applications, convergence can be judged by observing the performance of the agent, such as cumulative rewards, winning rate and other indicators. If the performance of the agent has not been significantly improved over a period of time, it may be considered that the algorithm has converged.

[0092] Embodiment 5:

[0093] In order to accurately and effectively train the original reinforcement learning model, based on the above embodiments, in the embodiment of the present application, the target reward value of the sample group is determined according to the determined reward value of each sample pair, including:

[0094] According to whether the sample offspring matrix in each sample pair is the sample parent matrix in other sample pairs in the sample group, each sample pair is sorted; wherein, if the sample parent matrix in the first sample pair is the sample offspring matrix in the second sample pair, the first sample pair is sorted after the second sample pair; the sample parent matrix in any sample pair in the sample group is the sample offspring matrix in any other sample pair, or the sample offspring matrix in any sample pair in the sample group is the sample parent matrix in any other sample pair;

[0095] The target reward value of the sample group is determined according to the determined reward value of each sample pair and the weight saved for the sorted value of each sample pair; wherein the weight of the sample pair sorted first is greater than the weight of the sample pair sorted later.

[0096] After obtaining the reward value of each sample pair in the sample group, if the target reward value of the sample group is determined directly based on the sum of each reward value obtained, the calculation result of the reward value may not be accurate and reasonable because the differences and importance between the sample pairs are not fully considered. In order to improve the accuracy and effectiveness of the reward value determination, the electronic device can sort each sample pair, and determine the target reward value of the sample group based on the weight corresponding to the sort value of each sample pair and the reward value of each sample pair.

[0097] Specifically, the electronic device may sort each sample pair according to whether the sample child matrix in each sample pair is the sample parent matrix in other sample pairs in the sample group. If the sample parent matrix in the first sample pair is the sample child matrix in the second sample pair, the first sample pair is sorted after the second sample pair. The sample parent matrix in any sample pair in the sample group is the sample child matrix in any other sample pair, or the sample child matrix in any sample pair in the sample group is the sample parent matrix in any other sample pair.

[0098] After sorting each sample pair, the electronic device may determine the target reward value of the sample group according to the reward value of each sample pair and the weight stored for the sorted ranking value of each sample pair. The weight of the sample pair in the front ranking is greater than the weight of the sample pair in the back ranking. Specifically, the electronic device may obtain the ranking value of the sample pair after sorting, determine the weight stored for the ranking value, and determine the product of the weight and the reward value of the determined sample pair, and determine the sum of the products of each determined sample pair as the target reward value of the sample group.

[0099] For each pair of samples, the electronic device can calculate a ranking value. Next, the electronic device uses a preset value as the base, which is less than 1 and is called the discount factor γ. Then, the difference between the ranking value and 1 is calculated, and this difference is used as a power. By using the preset value (i.e., the discount factor γ) as the base and the difference as a power for calculation, the result is used as the weight of the sample pair. In short, the embodiment of the present application uses the discount factor γ to attenuate the weight of each sample pair.

[0100] The electronic device may determine the target reward value using the following formula:

[0101]

[0102] Among them, R(τ) is the target reward value, γ is the preset weight discount factor, t is the ranking value of the sample pair, r t is the reward value of the sample pair with ranking value t.

[0103] For example, the sample parent matrix in sample pair A is the sample child matrix in sample pair B, and the sample parent matrix in sample pair B is the sample child matrix in sample pair C. The generated sorting results can be sample pair C, sample pair B, and sample pair A.

[0104] In the embodiment of the present application, all individuals that have evolved from the first generation to the end are called a track, where the first generation refers to the sample parent matrix included in the sample group is not the sample child matrix in other sample pairs, and the sample child matrix included is the sample parent matrix in other sample pairs. For each track, calculate its reward sum R(τ), which is the weighted sum of all rewards in the track.

[0105] Embodiment 6:

[0106] In order to accurately and effectively train the original reinforcement learning model, based on the above embodiments, in the embodiment of the present application, the reward value is determined according to the similarity between the output offspring matrix and the sample offspring matrix in the sample pair, including:

[0107] If the similarity between the output offspring matrix and the sample offspring matrix in the sample pair is greater than a threshold, a first preset reward value is determined as the reward value of the sample pair; otherwise, a second preset value is determined as the reward value of the sample pair; wherein the second preset value is less than the first preset reward value; or,

[0108] The reward value of the sample pair is determined as the product of the similarity between the output offspring matrix and the sample offspring matrix in the sample pair and the first preset reward value.

[0109] In order to determine the reward value of each sample pair, the electronic device can determine whether the similarity is greater than a threshold after determining the similarity between the output offspring matrix and the sample offspring matrix in the sample pair. If the similarity is greater than the threshold, the first preset reward value can be determined as the reward value of the sample pair; otherwise, the second preset reward value can be determined as the reward value of the sample pair, wherein the second preset value is less than the first preset value, and the second preset value can be 0.

[0110] The electronic device may also determine the similarity between the output offspring matrix and the sample offspring matrix in the sample pair, and determine the product of the similarity and the first preset reward value as the reward value of the sample pair.

[0111] In order to accurately and effectively train the original reinforcement learning model, based on the above embodiments, in the embodiment of the present application, the sample group in the sample set is obtained by the following method:

[0112] Randomly obtain a second preset number of sample tasks, and obtain each sample process included in each sample task; for each sample process, randomly determine a sample machine in each machine that executes the sample process, and randomly determine the sample order of the sample process executed by the sample machine in each executed process; sequentially determine the identifier of the machine that executes each sample process as the element in the first row of the sample parent matrix, and sequentially determine the order in which the corresponding machine executes the corresponding sample process as the element in the second row of the sample parent matrix;

[0113] Repeat the following steps until the number of sample pairs determined reaches the first preset number:

[0114] Based on the crossover and / or mutation method in genetics, a third preset number of candidate offspring matrices corresponding to the sample parent matrix are generated; and based on the total time for the machine corresponding to the identifier contained in each candidate offspring matrix to perform each sample task in the corresponding order, the candidate offspring matrix with the shortest total time is determined as the sample offspring matrix in the sample pair containing the sample parent matrix; if the number of determined sample pairs is not the first preset number, the sample offspring matrix is ​​determined as the sample parent matrix in another sample pair.

[0115] In order to train the original reinforcement learning model, the electronic device constructs a sample group in the sample set. Specifically, the electronic device randomly selects a certain number of sample tasks, where the certain number can be a second preset number, and for each sample process in each task, randomly selects one machine from the machines that can execute the process as a sample machine, and randomly determines the sample order of the machine executing the process. Among them, the process executed by each sample machine refers to all the processes that the machine needs to complete in the randomly selected sample task. The electronic device first organizes this information into a sample parent matrix, where the first row records the machine identifier that executes each sample process, and the second row records the corresponding execution order.

[0116] After determining the sample parent matrix, the electronic device may repeatedly perform the following steps until the number of determined sample pairs reaches a first preset number:

[0117] Subsequently, the electronic device derives the candidate offspring matrix from the sample parent matrix through crossover and / or mutation operations under the framework of the genetic algorithm. This process simulates the genetic variation in biological evolution and aims to explore the potential better solution space. The crossover operation generates a new matrix that combines the characteristics of both parties by exchanging some elements of the parent matrix; the mutation operation randomly adjusts some elements of a single parent matrix to increase the diversity of the population.

[0118] After obtaining the third preset number of candidate offspring matrices, the electronic device will evaluate the performance of each candidate offspring matrix to obtain the candidate offspring matrix with the highest fitness. Specifically, the electronic device will calculate the total time required to execute each sample task based on the machine identification and order in the candidate offspring matrix, and compare it with the total time corresponding to other candidate offspring matrices. If a candidate offspring matrix has the shortest total time, it is considered to have improved efficiency and is determined as the sample offspring matrix in the sample pair. This step reflects the selection principle in the genetic algorithm, that is, retaining excellent individuals and eliminating poor individuals.

[0119] At the same time, the electronic device will also store information such as the genes of the parent individuals, the individuals with the highest fitness in the offspring, and the difference between them in the experience playback buffer D. During the training phase, this buffer will be used to randomly extract experience playback to train the neural network. This can break the correlation between data, improve data usage efficiency, and stabilize the training process.

[0120] If the number of currently determined sample pairs does not reach the first preset number, the electronic device will use the most recently determined sample child matrix as a new starting point, that is, determine it as the sample parent matrix of another sample pair, and continue to iterate this process until the number of sample pairs reaches the first preset number.

[0121] The embodiment of the present application combines reinforcement learning with genetic algorithms, and uses genetic algorithms to replace the initial exploration stage of reinforcement learning, thereby solving the problem of slow and inefficient initial exploration. At the same time, it also defines a production scheduling problem model and uses reinforcement learning to solve the problem.

[0122] In an embodiment of the present application, reinforcement learning is combined with a genetic algorithm, and a genetic algorithm is used to replace the initial exploration stage of reinforcement learning to solve the problem of slow and inefficient initial exploration. In addition, a production scheduling problem model is defined, and reinforcement learning is used to solve the production scheduling problem.

[0123] Embodiment 7:

[0124] Based on the same inventive concept, the present application embodiment provides a production scheduling plan generation device, please refer to Figure 4 , the device comprises:

[0125] The acquisition module 401 is used to acquire each task carried in the received production scheduling request; and acquire each process included in each task;

[0126] A determination module 402 is used to randomly determine a target machine among each machine executing the process for each process, and randomly determine an order in which the target machine executes the process in each process executed by the target machine;

[0127] Processing module 403 is used to input the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process into a pre-trained reinforcement learning model, and obtain the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process based on the reinforcement learning model.

[0128] In a possible implementation, the processing module 403 is further configured to sequentially determine the identifiers of the target machines that execute each process as the elements in the first row of the matrix, and use the order of the corresponding target machines executing the corresponding processes in each process executed by the corresponding target machines as the elements in the second row of the matrix;

[0129] The processing module 403 is specifically used to input the matrix into a pre-trained reinforcement learning model.

[0130] In one possible implementation, the processing module 403 is specifically used for the first sub-model of the reinforcement learning model to determine an intermediate matrix based on the cross product of the pre-trained parameter matrix and the matrix; wherein the dimension of the intermediate matrix is ​​the same as the dimension of the matrix; the second sub-model of the reinforcement learning model determines a sum matrix of the matrix and the intermediate matrix; the elements in the first row of the sum matrix are sequentially determined as the identifiers of the standard machines that perform each of the processes, and the elements in the second row of the sum matrix are sequentially determined as the standard order of the corresponding standard machines performing the corresponding processes in each of the processes performed.

[0131] In a possible implementation, the processing module 403 is further configured to train the reinforcement learning model in the following manner:

[0132] Obtain any sample group in a pre-saved sample set, wherein the sample group contains a first preset number of sample pairs, and each sample pair contains a sample parent matrix and a sample child matrix; wherein the elements in the first row of the sample parent matrix and the sample child matrix are the identifiers of the machines that execute each sample process, and the elements in the second row are the order in which the corresponding machines execute the corresponding sample processes; for each sample pair, the total time for the machine corresponding to the identifier contained in the sample parent matrix in the sample pair to execute each sample task in the corresponding order is greater than the total time for the machine corresponding to the identifier contained in the sample child matrix in the sample pair to execute each sample task in the corresponding order; for each sample pair, input the sample parent matrix in the sample pair into the original reinforcement learning model to obtain the output child matrix output by the original reinforcement learning model; determine the reward value according to the similarity between the output child matrix and the sample child matrix in the sample pair; determine the target reward value of the sample group according to the determined reward value of each sample pair; and train the original reinforcement learning model based on the target reward value.

[0133] In a possible implementation, the processing module 403 is specifically configured to sort each sample pair according to whether the sample offspring matrix in each sample pair is the sample parent matrix in other sample pairs in the sample group; wherein, if the sample parent matrix in the first sample pair is the sample offspring matrix in the second sample pair, the first sample pair is sorted after the second sample pair; the sample parent matrix in any sample pair in the sample group is the sample offspring matrix in any other sample pair, or the sample offspring matrix in any sample pair in the sample group is the sample parent matrix in any other sample pair; and determine the target reward value of the sample group according to the determined reward value of each sample pair and the weight saved for the sorted value of each sample pair after sorting; wherein, the weight of the sample pair sorted first is greater than the weight of the sample pair sorted later.

[0134] In a possible implementation, the processing module 403 is specifically configured to determine a first preset reward value as the reward value of the sample pair if the similarity between the output offspring matrix and the sample offspring matrix in the sample pair is greater than a threshold value; otherwise, determine a second preset value as the reward value of the sample pair; wherein the second preset value is less than the first preset reward value; or, determine the reward value of the sample pair by multiplying the similarity between the output offspring matrix and the sample offspring matrix in the sample pair by the first preset reward value.

[0135] In a possible implementation, the processing module 403 is specifically used to obtain the sample group in the sample set in the following manner: randomly obtain a second preset number of sample tasks, and obtain each sample process included in each sample task; for each sample process, randomly determine a sample machine in each machine that executes the sample process, and randomly determine the sample order of the sample process executed by the sample machine in each executed process; sequentially determine the identifier of the machine that executes each sample process as the element in the first row of the sample parent matrix, and sequentially determine the order in which the corresponding machine executes the corresponding sample process as the sample parent matrix. elements in the second row of the sample matrix; repeatedly performing the following steps until the number of determined sample pairs is the first preset number: based on the crossover and / or mutation method in genetics, generating a third preset number of candidate offspring matrices corresponding to the sample parent matrix; and based on the total time for the machine corresponding to the identifier contained in each candidate offspring matrix to perform each sample task in the corresponding order, determining the candidate offspring matrix with the shortest total time as the sample offspring matrix in the sample pair containing the sample parent matrix; if the number of determined sample pairs is not the first preset number, determining the sample offspring matrix as the sample parent matrix in another sample pair.

[0136] Embodiment 8:

[0137] Based on the same inventive concept, the embodiment of the present application provides an electronic device, which can realize the function of generating the production schedule mentioned above, please refer to Figure 5 The device includes a processor 501, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0138] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps:

[0139] Obtain each task carried in the received production scheduling request; obtain each process included in each task;

[0140] For each of the processes, randomly determine a target machine among each machine executing the process, and randomly determine the order in which the target machine executes the process in each process executed by the target machine;

[0141] The identification of the target machine that executes each process and the order in which the target machine executes the corresponding process are input into a pre-trained reinforcement learning model, and the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process are obtained based on the reinforcement learning model.

[0142] In a possible implementation, after randomly determining that the target machine executes the process and the order of each process executed by the target machine, and before inputting the identifier of the target machine that will execute each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model, the method further includes:

[0143] Determine in sequence the identifier of the target machine that executes each of the processes as the elements in the first row of the matrix, and use the order of the corresponding target machine executing the corresponding process in each process executed by the corresponding target machine as the elements in the second row of the matrix;

[0144] The step of inputting the identifier of the target machine that executes each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model includes:

[0145] The matrix is ​​input into a pre-trained reinforcement learning model.

[0146] In a possible implementation, the acquiring, based on the reinforcement learning model, an identifier of a standard machine that executes each process and a standard order in which the standard machine executes the corresponding process in each task includes:

[0147] The first sub-model of the reinforcement learning model determines an intermediate matrix based on a cross product of a pre-trained parameter matrix and the matrix; wherein the dimension of the intermediate matrix is ​​the same as the dimension of the matrix;

[0148] A second sub-model of the reinforcement learning model determines a sum matrix of the matrix and the intermediate matrix;

[0149] The elements in the first row of the sum matrix are sequentially determined as the identifiers of the standard machines that perform each process, and the elements in the second row of the sum matrix are sequentially determined as the standard order in which the corresponding standard machines perform the corresponding processes in each process performed.

[0150] In one possible implementation, the reinforcement learning model is trained in the following manner:

[0151] Obtain any sample group in a pre-saved sample set, wherein the sample group includes a first preset number of sample pairs, and each sample pair includes a sample parent matrix and a sample child matrix; wherein the elements in the first row of the sample parent matrix and the sample child matrix are the identifiers of the machines that execute each sample process, and the elements in the second row are the order in which the corresponding machines execute the corresponding sample processes; for each sample pair, the total time for the machine corresponding to the identifier contained in the sample parent matrix in the sample pair to execute each sample task in the corresponding order is greater than the total time for the machine corresponding to the identifier contained in the sample child matrix in the sample pair to execute each sample task in the corresponding order;

[0152] For each sample pair, the sample parent matrix in the sample pair is input into the original reinforcement learning model to obtain the output child matrix output by the original reinforcement learning model; and the reward value is determined according to the similarity between the output child matrix and the sample child matrix in the sample pair;

[0153] According to the determined reward value of each sample pair, a target reward value of the sample group is determined; and the original reinforcement learning model is trained based on the target reward value.

[0154] In a possible implementation, determining the target reward value of the sample group according to the determined reward value of each sample pair includes:

[0155] According to whether the sample offspring matrix in each sample pair is the sample parent matrix in other sample pairs in the sample group, each sample pair is sorted; wherein, if the sample parent matrix in the first sample pair is the sample offspring matrix in the second sample pair, the first sample pair is sorted after the second sample pair; the sample parent matrix in any sample pair in the sample group is the sample offspring matrix in any other sample pair, or the sample offspring matrix in any sample pair in the sample group is the sample parent matrix in any other sample pair;

[0156] The target reward value of the sample group is determined according to the determined reward value of each sample pair and the weight saved for the sorted value of each sample pair; wherein the weight of the sample pair sorted first is greater than the weight of the sample pair sorted later.

[0157] In a possible implementation, determining the reward value according to the similarity between the output offspring matrix and the sample offspring matrix in the sample pair includes:

[0158] If the similarity between the output offspring matrix and the sample offspring matrix in the sample pair is greater than a threshold, a first preset reward value is determined as the reward value of the sample pair; otherwise, a second preset value is determined as the reward value of the sample pair; wherein the second preset value is less than the first preset reward value; or,

[0159] The reward value of the sample pair is determined as the product of the similarity between the output offspring matrix and the sample offspring matrix in the sample pair and the first preset reward value.

[0160] In a possible implementation manner, the sample group in the sample set is obtained in the following manner:

[0161] Randomly obtain a second preset number of sample tasks, and obtain each sample process included in each sample task; for each sample process, randomly determine a sample machine in each machine that executes the sample process, and randomly determine the sample order of the sample process executed by the sample machine in each executed process; sequentially determine the identifier of the machine that executes each sample process as the element in the first row of the sample parent matrix, and sequentially determine the order in which the corresponding machine executes the corresponding sample process as the element in the second row of the sample parent matrix;

[0162] Repeat the following steps until the number of sample pairs determined reaches the first preset number:

[0163] Based on the crossover and / or mutation method in genetics, a third preset number of candidate offspring matrices corresponding to the sample parent matrix are generated; and based on the total time for the machine corresponding to the identifier contained in each candidate offspring matrix to perform each sample task in the corresponding order, the candidate offspring matrix with the shortest total time is determined to be the sample offspring matrix in the sample pair containing the sample parent matrix; if the number of determined sample pairs is not the first preset number, the sample offspring matrix is ​​determined to be the sample parent matrix in another sample pair.

[0164] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0165] The communication interface 502 is used for communication between the electronic device and other devices.

[0166] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0167] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0168] Embodiment 9:

[0169] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0170] Obtain each task carried in the received production scheduling request; obtain each process included in each task;

[0171] For each of the processes, randomly determine a target machine among each machine executing the process, and randomly determine the order in which the target machine executes the process in each process executed by the target machine;

[0172] The identification of the target machine that executes each process and the order in which the target machine executes the corresponding process are input into a pre-trained reinforcement learning model, and the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process are obtained based on the reinforcement learning model.

[0173] In a possible implementation, after randomly determining that the target machine executes the process and the order of each process executed by the target machine, and before inputting the identifier of the target machine that will execute each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model, the method further includes:

[0174] Determine in sequence the identifier of the target machine that executes each of the processes as the elements in the first row of the matrix, and use the order of the corresponding target machine executing the corresponding process in each process executed by the corresponding target machine as the elements in the second row of the matrix;

[0175] The step of inputting the identifier of the target machine that executes each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model includes:

[0176] The matrix is ​​input into a pre-trained reinforcement learning model.

[0177] In a possible implementation, the acquiring, based on the reinforcement learning model, an identifier of a standard machine that executes each process and a standard order in which the standard machine executes the corresponding process in each task includes:

[0178] The first sub-model of the reinforcement learning model determines an intermediate matrix based on a cross product of a pre-trained parameter matrix and the matrix; wherein the dimension of the intermediate matrix is ​​the same as the dimension of the matrix;

[0179] A second sub-model of the reinforcement learning model determines a sum matrix of the matrix and the intermediate matrix;

[0180] The elements in the first row of the sum matrix are sequentially determined as the identifiers of the standard machines that perform each process, and the elements in the second row of the sum matrix are sequentially determined as the standard order in which the corresponding standard machines perform the corresponding processes in each process performed.

[0181] In one possible implementation, the reinforcement learning model is trained in the following manner:

[0182] Obtain any sample group in a pre-saved sample set, wherein the sample group includes a first preset number of sample pairs, and each sample pair includes a sample parent matrix and a sample child matrix; wherein the elements in the first row of the sample parent matrix and the sample child matrix are the identifiers of the machines that execute each sample process, and the elements in the second row are the order in which the corresponding machines execute the corresponding sample processes; for each sample pair, the total time for the machine corresponding to the identifier contained in the sample parent matrix in the sample pair to execute each sample task in the corresponding order is greater than the total time for the machine corresponding to the identifier contained in the sample child matrix in the sample pair to execute each sample task in the corresponding order;

[0183] For each sample pair, the sample parent matrix in the sample pair is input into the original reinforcement learning model to obtain the output child matrix output by the original reinforcement learning model; and the reward value is determined according to the similarity between the output child matrix and the sample child matrix in the sample pair;

[0184] According to the determined reward value of each sample pair, a target reward value of the sample group is determined; and the original reinforcement learning model is trained based on the target reward value.

[0185] In a possible implementation, determining the target reward value of the sample group according to the determined reward value of each sample pair includes:

[0186] According to whether the sample offspring matrix in each sample pair is the sample parent matrix in other sample pairs in the sample group, each sample pair is sorted; wherein, if the sample parent matrix in the first sample pair is the sample offspring matrix in the second sample pair, the first sample pair is sorted after the second sample pair; the sample parent matrix in any sample pair in the sample group is the sample offspring matrix in any other sample pair, or the sample offspring matrix in any sample pair in the sample group is the sample parent matrix in any other sample pair;

[0187] The target reward value of the sample group is determined according to the determined reward value of each sample pair and the weight saved for the sorted value of each sample pair; wherein the weight of the sample pair sorted first is greater than the weight of the sample pair sorted later.

[0188] In a possible implementation, determining the reward value according to the similarity between the output offspring matrix and the sample offspring matrix in the sample pair includes:

[0189] If the similarity between the output offspring matrix and the sample offspring matrix in the sample pair is greater than a threshold, a first preset reward value is determined as the reward value of the sample pair; otherwise, a second preset value is determined as the reward value of the sample pair; wherein the second preset value is less than the first preset reward value; or,

[0190] The reward value of the sample pair is determined as the product of the similarity between the output offspring matrix and the sample offspring matrix in the sample pair and the first preset reward value.

[0191] In a possible implementation manner, the sample group in the sample set is obtained in the following manner:

[0192] Randomly obtain a second preset number of sample tasks, and obtain each sample process included in each sample task; for each sample process, randomly determine a sample machine in each machine that executes the sample process, and randomly determine the sample order of the sample process executed by the sample machine in each executed process; sequentially determine the identifier of the machine that executes each sample process as the element in the first row of the sample parent matrix, and sequentially determine the order in which the corresponding machine executes the corresponding sample process as the element in the second row of the sample parent matrix;

[0193] Repeat the following steps until the number of sample pairs determined reaches the first preset number:

[0194] Based on the crossover and / or mutation method in genetics, a third preset number of candidate offspring matrices corresponding to the sample parent matrix are generated; and based on the total time for the machine corresponding to the identifier contained in each candidate offspring matrix to perform each sample task in the corresponding order, the candidate offspring matrix with the shortest total time is determined to be the sample offspring matrix in the sample pair containing the sample parent matrix; if the number of determined sample pairs is not the first preset number, the sample offspring matrix is ​​determined to be the sample parent matrix in another sample pair.

[0195] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0196] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating a production schedule, characterized in that: The method comprises: Obtain each task carried in the received production scheduling request; obtain each process included in each task; For each of the processes, randomly determine a target machine among each machine executing the process, and randomly determine the order in which the target machine executes the process in each process executed by the target machine; The identification of the target machine that executes each process and the order in which the target machine executes the corresponding process are input into a pre-trained reinforcement learning model, and the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process are obtained based on the reinforcement learning model.

2. The method according to claim 1, characterized in that After randomly determining the target machine to execute the process and the order of each process executed by the target machine, before inputting the identifier of the target machine that will execute each process and the order of the target machine to execute the corresponding process into the pre-trained reinforcement learning model, the method further includes: Determine in sequence the identifier of the target machine that executes each of the processes as the elements in the first row of the matrix, and use the order of the corresponding target machine executing the corresponding process in each process executed by the corresponding target machine as the elements in the second row of the matrix; The step of inputting the identifier of the target machine that executes each process and the order in which the target machine executes the corresponding process into the pre-trained reinforcement learning model includes: The matrix is ​​input into a pre-trained reinforcement learning model.

3. The method according to claim 2, characterized in that The step of obtaining the identifier of a standard machine that executes each process and the standard order in which the standard machine executes the corresponding process in each task based on the reinforcement learning model includes: The first sub-model of the reinforcement learning model determines an intermediate matrix based on a cross product of a pre-trained parameter matrix and the matrix; wherein the dimension of the intermediate matrix is ​​the same as the dimension of the matrix; A second sub-model of the reinforcement learning model determines a sum matrix of the matrix and the intermediate matrix; The elements in the first row of the sum matrix are sequentially determined as the identifiers of the standard machines that perform each process, and the elements in the second row of the sum matrix are sequentially determined as the standard order in which the corresponding standard machines perform the corresponding processes in each process performed.

4. The method according to claim 1, characterized in that: The reinforcement learning model is trained in the following way: Obtain any sample group in a pre-saved sample set, wherein the sample group includes a first preset number of sample pairs, and each sample pair includes a sample parent matrix and a sample child matrix; wherein the elements in the first row of the sample parent matrix and the sample child matrix are the identifiers of the machines that execute each sample process, and the elements in the second row are the order in which the corresponding machines execute the corresponding sample processes; for each sample pair, the total time for the machine corresponding to the identifier contained in the sample parent matrix in the sample pair to execute each sample task in the corresponding order is greater than the total time for the machine corresponding to the identifier contained in the sample child matrix in the sample pair to execute each sample task in the corresponding order; For each sample pair, the sample parent matrix in the sample pair is input into the original reinforcement learning model to obtain the output child matrix output by the original reinforcement learning model; and the reward value is determined according to the similarity between the output child matrix and the sample child matrix in the sample pair; According to the determined reward value of each sample pair, a target reward value of the sample group is determined; and the original reinforcement learning model is trained based on the target reward value.

5. The method according to claim 4, characterized in that Determining a target reward value for the sample group according to the determined reward value for each sample pair includes: According to whether the sample offspring matrix in each sample pair is the sample parent matrix in other sample pairs in the sample group, each sample pair is sorted; wherein, if the sample parent matrix in the first sample pair is the sample offspring matrix in the second sample pair, the first sample pair is sorted after the second sample pair; the sample parent matrix in any sample pair in the sample group is the sample offspring matrix in any other sample pair, or the sample offspring matrix in any sample pair in the sample group is the sample parent matrix in any other sample pair; The target reward value of the sample group is determined according to the determined reward value of each sample pair and the weight saved for the sorted value of each sample pair; wherein the weight of the sample pair sorted first is greater than the weight of the sample pair sorted later.

6. The method according to claim 4, characterized in that The step of determining a reward value according to the similarity between the output offspring matrix and the sample offspring matrix in the sample pair comprises: If the similarity between the output offspring matrix and the sample offspring matrix in the sample pair is greater than a threshold, a first preset reward value is determined as the reward value of the sample pair; otherwise, a second preset value is determined as the reward value of the sample pair; wherein the second preset value is less than the first preset reward value; or, The reward value of the sample pair is determined as the product of the similarity between the output offspring matrix and the sample offspring matrix in the sample pair and the first preset reward value.

7. The method according to claim 4, characterized in that The sample groups in the sample set are obtained in the following way: Randomly obtain a second preset number of sample tasks, and obtain each sample process included in each sample task; for each sample process, randomly determine a sample machine in each machine that executes the sample process, and randomly determine the sample order of the sample process executed by the sample machine in each executed process; sequentially determine the identifier of the machine that executes each sample process as the element in the first row of the sample parent matrix, and sequentially determine the order in which the corresponding machine executes the corresponding sample process as the element in the second row of the sample parent matrix; Repeat the following steps until the number of sample pairs determined reaches the first preset number: Based on the crossover and / or mutation method in genetics, a third preset number of candidate offspring matrices corresponding to the sample parent matrix are generated; and based on the total time for the machine corresponding to the identifier contained in each candidate offspring matrix to perform each sample task in the corresponding order, the candidate offspring matrix with the shortest total time is determined to be the sample offspring matrix in the sample pair containing the sample parent matrix; if the number of determined sample pairs is not the first preset number, the sample offspring matrix is ​​determined to be the sample parent matrix in another sample pair.

8. A production scheduling plan generating device, characterized in that: The device comprises: An acquisition module, used to acquire each task carried in the received production scheduling request; and acquire each process included in each task; A determination module, configured to randomly determine a target machine among each machine executing the process for each process, and randomly determine an order in which the target machine executes the process in each process executed by the target machine; The processing module is used to input the identification of the target machine that executes each process and the order in which the target machine executes the corresponding process into a pre-trained reinforcement learning model, and obtain the identification of the standard machine that executes each process and the standard order in which the standard machine executes the corresponding process based on the reinforcement learning model.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method for generating a production schedule according to any one of claims 1-7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer executes the steps included in the method for generating a production schedule as described in any one of claims 1 to 7.

Images