Real-time scheduling method for intelligent workshop with workpiece flow based on D3QN reinforcement learning

Abstract

The present invention discloses a real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning. The method first collects the workshop operation data and processes it into Markov decision chain data, trains an online Q network of D3QN and a target Q network, and then trains the target Q network to realize the real-time scheduling of intelligent workshops with workpiece flow. ‑ The network is trained online and uses the trained Q network as a knowledge network to match the real-time status of the workshop, select the best production schedule, and repeat this process until the production is completed. This invention achieves real-time scheduling of intelligent manufacturing workshops by collecting multi-channel resource view status in real time and matching the reasoning of the knowledge network model. It can efficiently and quickly guide workshop operation scheduling and improve enterprise production efficiency.

Description

Technical Field

[0001] The present invention relates to the field of workshop job scheduling and artificial intelligence algorithm technology, and more specifically, to a real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning. Background Art

[0002] Real-time job scheduling in the shop floor is a key component of enterprise intelligence and automation, and has practical significance for improving production efficiency and economic returns. Therefore, job scheduling has become a current research hotspot and a difficult problem in the scheduling field. However, modern manufacturing workshops are plagued by large production scales and numerous resource constraints, which urgently require real-time scheduling algorithms to possess high-quality solution performance and rapid response capabilities.

[0003] The discrete workshop scheduling problem is a typical non-deterministic polynomial problem (NP-Hard). Existing workshop scheduling methods mostly use deterministic algorithms or meta-heuristic algorithms to solve it. Although these algorithms have significant optimization capabilities for small-scale discrete manufacturing problems, in actual large-scale production scenarios, such algorithms inevitably face a combinatorial explosion in the search for scheduling objectives. They may fall into local optimal solutions or have long solution times. Therefore, such algorithms are difficult to adapt to real-time scheduling in actual workshops.

[0004] The prior art discloses a flow shop scheduling method based on deep reinforcement learning. First, the workshop resources required for scheduling decisions are classified and a multi-dimensional, flexibly scalable resource information matrix is constructed, which is aggregated to obtain the corresponding workshop status view. Secondly, the workshop status is used as the input of a multi-channel convolutional neural network model, and the degree of fit between the maximum completion time and the expected production target is used as a reward. Finally, the network model is trained and the parameters are saved so that the network fits the optimal policy function. In practical problems, the workshop status can be constructed by selecting the processing time of each workpiece in its process as resource node information, and this is used as the input of the optimal policy network. The workpiece scheduling tasks with the best policy value are selected in sequence until all workpieces are selected and a complete scheduling sequence is obtained. The flow shop scheduling method described in the present invention can obtain near-optimal solutions for small-scale problems and better solutions than heuristic algorithms and genetic algorithms for large-scale problems. However, a simple deep learning DQN algorithm may lead to overestimation, resulting in large deviations in the scheduling results. Summary of the Invention

[0005] The present invention provides a real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning, which overcomes the problems of traditional algorithms in solving problems in today's large-scale manufacturing workshops, such as being prone to falling into local optimal solutions and taking a long time to solve.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] A real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning includes the following steps:

[0008] S1: Collect and record all operation data of the production process from the time an order is placed in the smart workshop to the completion of finished product processing;

[0009] S2: Classify and process the operation data collected in step S1 into Markov decision chain data;

[0010] S3: Load the Markov decision chain data processed in step S2 into a sample data pool with a capacity of C;

[0011] S4: Determine whether the sample data pool is full. If not, repeat steps S1 to S3. If full, execute S5.

[0012] S5: Construct two convolutional neural networks with the same structure as the online Q network and target Q of D3QN (Dueling Double DQN, two-layer dueling network algorithm) - network;

[0013] S6: Randomly extract several Markov decision chain data from the sample data pool, train the online Q network, and update the target Q at a fixed training frequency. - Network parameters;

[0014] S7: Determine whether the current online Q network has converged. If not, return to step S6. If converged, execute S8.

[0015] S8: Call the converged online Q network as the knowledge network, match the real-time status of the workshop, select the best production schedule, and repeat the process until the production is completed.

[0016] Preferably, all the operation data of the production process in step S1 are specifically:

[0017] All the operation data of the production process are various resource type data collected through various methods during the production process. The production process includes the placement of workshop material orders, workshop workpiece process scheduling and workshop completion results. The operation data covers the changes in various resource information of "people, machines, materials, methods and environment" in the workshop, including the working status data of production equipment collected in real time by the Internet of Things, information data of workpieces to be processed and workpiece scheduling transfer data, as well as production order data, material data and processing technology data imported into the workshop by other production systems.

[0018] Preferably, in step S2, the operation data collected in step S1 is classified and processed into Markov decision chain data, specifically:

[0019] Classify by workshop resource category and build multi-channel resource view as status s i , a state s i Contains all resource dimension status information taken into consideration. Each resource dimension status information is represented by a channel status view in the form of a matrix, expressed as follows:

[0020] s i = <P m×c ,M m×c ,W m×c ,F m×c ,……,E m×c >,i∈(0,1,2…,n-1)

[0021] Where, P m×c is the workshop personnel information matrix, M m×c is the machine equipment information matrix, W m×c is the workpiece material information matrix, F m×c is the process information matrix, E m×c is the environmental information matrix, m×c represents the matrix size, m represents the total number of workpieces in the workshop, and c represents the total number of columns. This modeling approach transforms complex, high-dimensional workshop state information into multiple low-dimensional state matrices, reducing workshop modeling complexity. The state matrices for each resource dimension can be flexibly expanded to include other resource types used for decision-making, based on production logic needs.

[0022] Build actions based on the flow of workshop workpieces between equipment action For a specific workpiece to be transferred from a certain process equipment to the next process equipment, the workpiece θ H Directed flow Tr between processing equipment K To represent, the expression is as follows:

[0023]

[0024] Where θ H For H-type workpiece, Tr(O,Z) K Indicates transfer from O-type equipment to Z-type equipment. G is the action code, H is the workpiece category code, and K is the flow code between equipment. The coding rules for each element are as follows:

[0025] G=(H×sum(Tr))+K

[0026] Where sum(Tr) represents the total number of directional flows between similar equipment in the workshop;

[0027] In the workshop status i The transferable action set of the next job is Job(s i ), with a finite number of actions To represent, the expression is as follows:

[0028]

[0029] Job is an intelligent function that can effectively express the workshop status s i The workpiece can choose the direction of transfer. The actual production transfer will be executed by selecting one direction from many directions. Every time the smart workshop status is refreshed, i ,Job(s i ) will also be updated accordingly;

[0030] According to the degree of fit between the scheduling target tar and the completion result T, the workshop is assigned to the state s i Next action Entering a new state i+1 The corresponding reward R;

[0031] The workshop starts from the order release state s0 to the completion state s n End, form Markov decision chain data one by one according to the order of n scheduling

[0032] Preferably, the reward function R is:

[0033]

[0034] Where r + and r - They represent positive and negative reward values respectively, and tar is the scheduling target.

[0035] Preferably, the online Q network and the target Q in step S5 - The network structure of the network is consistent, specifically:

[0036] The online Q network and the target Q - The network structure of the network is multi-input channel and dual-output subnet;

[0037] Among them, the number of network input channels is the same as the state s in step S2 i The total number of resource status view channels contained is equal. One network input channel matches the input of a type of workshop resource information. The dual output subnet includes a front-end and an output subnet. The input of the network enters the output subnet after passing through the front-end. The output subnet includes a price function network Vn and an advantage function network An. The price function network Vn is responsible for evaluating the state s i The value of the advantage function network An is responsible for each action in state s The relative advantages and disadvantages of the network, the final network value output Q (s, a; ω, α, β) is obtained by the linear combination of the two:

[0038]

[0039] Where ω represents the network parameters before the dual-output subnet, a and β represent the parameters of the price function network Vn and the advantage function network An respectively, A represents the set of all actions, and a′∈A.

[0040] Preferably, the front end is composed of multiple layers of network units, each layer of network units is composed of a convolutional layer, a normalization layer and an activation function, and the output subnetworks are all constructed by fully connected layers.

[0041] Preferably, the front end is connected to the output subnet via a flattening layer, which unfolds the features.

[0042] Preferably, in step S6, a number of Markov decision chain data are randomly extracted from the sample data pool to train the online Q network and update the target Q at a fixed training frequency. - Network parameters, specifically:

[0043] The state s i Input the online Q network to estimate its Q value and convert the state s i+1 Input target Q - The network combines the reward value r to obtain the target value Y, and uses stochastic gradient descent to the loss function L between the Q value and the Y value. The training updates the online Q network parameters, and the online Q network parameters are copied at a fixed training frequency f to update the target Q. - Network parameters.

[0044] Preferably, the mathematical expression of the loss function L between the Q value and the Y value is as follows:

[0045]

[0046]

[0047] Where subscript j is the training step size, represents the mean square error calculation, Represents the training sample data extracted from the sample data pool, γ is the discount factor, and the larger the discount factor, the more the network training focuses on future state changes. Indicates that the D3QN online Q network evaluates the next state s i+1 The most valuable action is a m .

[0048] Preferably, in step S8, the converged online Q network is called as the knowledge network, the real-time status of the workshop is matched, the optimal production is scheduled, and the process is repeated until the production is completed. Specifically,

[0049] A workshop online knowledge matching mechanism is constructed. Within this mechanism, the converged online Q network is called as the knowledge network to determine the real-time status s of each resource information in the manufacturing workshop. The task to be scheduled Job(s) under the real-time status is determined. The real-time status s is input into the knowledge network to obtain the value of the action to be scheduled. The optimal scheduling action is selected and executed in the workshop. The real-time status s′ and the task Job(s′) are refreshed and input into the knowledge network again for optimal scheduling. This process is repeated until the end of production.

[0050] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0051] This paper proposes a multi-channel resource view modeling method tailored to varying resource requirements. This method uses the D3QN algorithm, based on multi-channel resource view input, to construct and train a knowledge network model. This method enables real-time scheduling of intelligent manufacturing workshops by capturing the state of the multi-channel resource view in real time and then reasoning with the knowledge network model. This method can self-learn without human supervision. Once the D3QN value network has fitted the optimal policy function, it can efficiently and quickly guide workshop job scheduling, improving enterprise production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the method of the present invention.

[0053] Figure 2 Schematic diagram of a mixed-flow manufacturing workshop with 26 equipment resource nodes in three stages provided in the embodiment.

[0054] Figure 3 The manufacturing workshop status s provided for the embodiment i view.

[0055] Figure 4 A schematic diagram of the convolutional neural network structure provided in the embodiment.

[0056] Figure 5 A schematic diagram of the workshop knowledge matching mechanism provided in the embodiment. DETAILED DESCRIPTION

[0057] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0058] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0059] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0060] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0061] Example 1

[0062] This embodiment provides a real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning. Figure 1 As shown, the following steps are included:

[0063] S1: Collect and record all operation data of the production process from the time an order is placed in the smart workshop to the completion of finished product processing;

[0064] S2: Classify and process the operation data collected in step S1 into Markov decision chain data;

[0065] S3: Load the Markov decision chain data processed in step S2 into a sample data pool with a capacity of C;

[0066] S4: Determine whether the sample data pool is full. If not, repeat steps S1 to S3. If full, execute S5.

[0067] S5: Construct two convolutional neural networks with the same structure as the online Q network and target Q of D3QN - network;

[0068] S6: Randomly extract several Markov decision chain data from the sample data pool, train the online Q network, and update the target Q at a fixed training frequency. - Network parameters;

[0069] S7: Determine whether the current online Q network has converged. If not, return to step S6. If converged, execute S8.

[0070] S8: Call the converged online Q network as the knowledge network, match the real-time status of the workshop, select the best production schedule, and repeat the process until the production is completed.

[0071] Example 2

[0072] This embodiment, based on the first embodiment, further discloses the following contents:

[0073] All operation data of the production process in step S1 are specifically:

[0074] All the operation data of the production process are various resource type data collected through various methods during the production process. The production process includes the placement of workshop material orders, workshop workpiece process scheduling and workshop completion results. The operation data covers the changes in various resource information of "people, machines, materials, methods and environment" in the workshop, including the working status data of production equipment collected in real time by the Internet of Things, information data of workpieces to be processed and workpiece scheduling transfer data, as well as production order data, material data and processing technology data imported into the workshop by other production systems.

[0075] In this embodiment, a copper clad laminate manufacturing workshop case is provided, which has a scale of three stages and a total of 26 equipment nodes, and the order resources include 10 types of processing workpieces. Figure 2 As shown in Figure 2, this example belongs to the hybrid flowshop (HFS) scheduling problem, which is a typical scheduling problem of a flowshop combined with similar parallel equipment.

[0076] In step S2, the operation data collected in step S1 is classified and processed into Markov decision chain data, specifically:

[0077] Classify by workshop resource category and build multi-channel resource view as status s i , a state s i Contains all resource dimension status information taken into consideration. Each resource dimension status information is represented by a channel status view in the form of a matrix, expressed as follows:

[0078] s i = <P m×c ,M m×c ,W m×c ,F m×c ,……,E m×c >,i∈(0,1,2…,n-1)

[0079] Where, P m×c is the workshop personnel information matrix, M m×c is the machine equipment information matrix, W m×c is the workpiece material information matrix, F m×c is the process information matrix, E m×c is the environmental information matrix, m×c represents the matrix size, the total number of workpieces in the workshop m is the number of matrix rows, and the total number of processing equipment c is the number of matrix columns;

[0080] Build actions based on the flow of workshop workpieces between equipment action For a specific workpiece to be transferred from a certain process equipment to the next process equipment, the workpiece θ H Directed flow Tr between processing equipment K To represent, the expression is as follows:

[0081]

[0082] Where θ H For H-type workpiece, Tr(O,Z) k Indicates transfer from O-type equipment to Z-type equipment. G is the action code, H is the workpiece category code, and K is the flow code between equipment. The coding rules for each element are as follows:

[0083] G=(H×sum(Tr))+K

[0084] In the formula, sum(Tr) represents the total number of directed flows between similar equipment in the workshop. In this embodiment, there are 22 types of directed flows. Through the above encoding, one action can be uniquely associated with one type of workpiece transfer event. In addition, if an action code G is known, the corresponding workpiece and transfer information can be obtained through inverse encoding. For example:

[0085] Indicates: the action coded 28 represents the transfer of a type 1 workpiece from a type A device to a type F device (following the above coding rule: 1×22+6=28);

[0086] In the workshop status i The transferable action set of the next job is Job(s i ), with a finite number of actions To represent, the expression is as follows:

[0087]

[0088] Job is an intelligent function that can effectively express the workshop status s i The workpiece can choose the direction of transfer. The actual production transfer will be executed by selecting one direction from many directions. Every time the smart workshop status is refreshed, i ,Job(s i ) will also be updated accordingly;

[0089] According to the degree of fit between the scheduling target tar and the completion result T, the workshop is assigned to the state s i Next action Entering a new state i+1 The corresponding reward R;

[0090] The workshop starts from the order release state s0 to the completion state s n End, form Markov decision chain data one by one according to the order of n scheduling

[0091] The reward function R is:

[0092]

[0093] Where r + and r - Represents positive and negative reward values, tar is the scheduling target, in this example, r + ,r - Set them to 15, -1, and tar to 150 respectively.

[0094] In this embodiment, the data pool capacity C is 100,000 Markov decision chain data

[0095] In this embodiment, based on the workshop workpiece resource information, equipment resource information, and production scheduling information, the workpiece location identification matrix (MM), workpiece production scheduling matrix (ST), and workpiece processing time matrix (MT) are constructed respectively. These three types of channel resource view matrices constitute the state s i , the data in the matrix is filled with the information data collected by S1, as shown in the example Figure 3 , state s i The expression is as follows:

[0096] s i = <MM m×c ,ST m×c ,MT m×c >,i∈(0,1,2…,n-1)

[0097] The symbol m×c represents the matrix size, with the number of workpieces in the workshop m as the number of matrix rows, and the number of similar parallel devices c as the number of matrix columns. In this example, m and c are 10 and 14 respectively, and the number of similar parallel devices is as shown in the example Figure 2 Class C1 and C2 equipment.

[0098] Example 3

[0099] This embodiment, based on Embodiment 1 and Embodiment 2, further discloses the following contents:

[0100] The online Q network and target Q in step S5 - The network structure of the network is consistent, specifically:

[0101] The online Q network and the target Q - The network structure of the network is multi-input channel and dual-output subnet;

[0102] Among them, the number of network input channels is the same as the state s in step S2 i The total number of resource status view channels is equal. One network input channel matches and inputs one type of workshop resource information. The number of network input channels in this embodiment is 3. The network structure is as follows: Figure 4 As shown; the dual output subnet includes a front-end and an output subnet. The input of the network enters the output subnet after passing through the front-end. The output subnet includes a price function network Vn and an advantage function network An. The price function network Vn is responsible for evaluating the state s i The value of the advantage function network An is responsible for each action in state s The relative advantages and disadvantages of the network, the final network value output Q (s, a; ω, α, β) is obtained by the linear combination of the two:

[0103]

[0104] Where ω represents the network parameters before the dual-output subnet, a and β represent the parameters of the price function network Vn and the advantage function network An respectively, A represents the set of all actions, and a′∈A.

[0105] The front end is composed of multiple layers of network units, each layer of network units is composed of a convolutional layer, a normalization layer and an activation function, and the output subnets are all built by fully connected layers.

[0106] The front end is connected to the output subnet through a flattening layer, which expands the features.

[0107] In step S6, a number of Markov decision chain data are randomly extracted from the sample data pool to train the online Q network and update the target Q at a fixed training frequency. - Network parameters, specifically:

[0108] In this embodiment, k=240 Markov decision chain data are randomly extracted from the sample data pool, and the state s i Input the online Q network to estimate its Q value and convert the state s i+1 Input target Q - The network combines the reward value r to obtain the target value Y, and uses stochastic gradient descent to the loss function L between the Q value and the Y value. The training updates the online Q network parameters, and the online Q network parameters are copied at a fixed training frequency f to update the target Q. - Network parameters.

[0109] The mathematical expression of the loss function L between the Q value and the Y value is as follows:

[0110]

[0111]

[0112] Where subscript j is the training step size, represents the mean square error calculation, Represents the training sample data extracted from the sample data pool, γ is the discount factor, and the larger the discount factor, the more the network training focuses on future state changes. Indicates that the D3QN online Q network evaluates the next state s i+1 The most valuable action is a m .

[0113] In step S8, the converged online Q network is called as the knowledge network to match the real-time status of the workshop, select the best production schedule, and repeat the process until the production is completed. Specifically,

[0114] Construct an online knowledge matching mechanism for the workshop, in which the converged online Q network is called as the knowledge network, and the three types of resource information in the manufacturing workshop, namely the workpiece location identification (MM), workpiece scheduling time (ST), and workpiece processing time (MT), are collected to form the real-time state s. The task Job(s) to be scheduled in the real-time state is determined, and the real-time state s is input into the knowledge network to obtain the value of the action to be scheduled. The optimal value scheduling action is selected and executed in the workshop, and the real-time state s′ and the task Job(s′) are refreshed. The knowledge network is input again for optimal scheduling, and the execution is repeated until the end of production. The online knowledge matching mechanism is as follows: Figure 5 As shown in the figure, the workpiece location information (MM), production scheduling information (ST), and processing time information (MT) of the manufacturing workshop at time τ are extracted through real-time IoT technology to construct the real-time workshop state s. Input s into the online knowledge network to obtain the value of each action in the workshop's scheduled task Job(s) at the same time, as shown in the figure Q(s,a 9 ),Q(s,a 10 ),Q(s,a 31 ),Q(s,a 32 ). The greater the action value, the higher the execution priority, so the most valuable production scheduling action a in the workshop at this moment is directly obtained through the argmax function 9 , that is, workpiece θ0 is transferred to type H equipment first. The above scheduling reasoning and production scheduling method is repeatedly executed until the workshop production is completed and the final scheduling goal is achieved.

[0115] The same or similar reference numerals correspond to the same or similar components;

[0116] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0117] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning, characterized by: The following steps are involved: S1: Collect and record all operation data of the production process from the time an order is placed in the smart workshop to the completion of finished product processing; S2: Classify and process the operation data collected in step S1 into Markov decision chain data; S3: Load the Markov decision chain data processed in step S2 into a sample data pool with a capacity of C; S4: Determine whether the sample data pool is full. If not, repeat steps S1 to S3. If full, execute S5. S5: Construct two convolutional neural networks with the same structure as the online Q network and target Q of D3QN - network; S6: Randomly extract several Markov decision chain data from the sample data pool, train the online Q network, and update the target Q at a fixed training frequency. - Network parameters, specifically: The state s i Input the online Q network to estimate its Q value and convert the state s i+1 Input target Q - The network combines the reward value r to obtain the target value Y, and uses stochastic gradient descent to the loss function L between the Q value and the Y value. The training updates the online Q network parameters, and the online Q network parameters are copied at a fixed training frequency f to update the target Q. - Network parameters; The mathematical expression of the loss function L between the Q value and the Y value is as follows: Where ω represents the network parameters before the dual-output subnet. The dual-output subnet includes the front-end and the output subnet. The input of the network enters the output subnet after passing through the front-end. The output subnet includes the price function network Vn and the advantage function network An. The price function network Vn is responsible for evaluating the state s. i The value of the advantage function network An is responsible for each action in state s The relative advantages and disadvantages of α and β are the parameters of the price function network Vn and the advantage function network An, respectively. The subscript j is the training step size. represents the mean square error calculation, represents the training sample data extracted from the sample data pool, γ is the discount factor, the larger the discount factor, the more the network training focuses on future state changes; Indicates that the D3QN online Q network evaluates the next state s i+1 The most valuable action is a m ; S7: Determine whether the current online Q network has converged. If not, return to step S6. If converged, execute S8. S8: Call the converged online Q network as the knowledge network, match the real-time status of the workshop, select the best production schedule, and repeat the process until the production is completed. Specifically: A workshop online knowledge matching mechanism is constructed. Within this mechanism, the converged online Q network is called as the knowledge network to determine the real-time status s of each resource information in the manufacturing workshop. The task to be scheduled Job(s) under the real-time status is determined. The real-time status s is input into the knowledge network to obtain the value of the action to be scheduled. The optimal scheduling action is selected and executed in the workshop. The real-time status s′ and the task Job(s′) are refreshed and input into the knowledge network again for optimal scheduling. This process is repeated until the end of production.

2. The real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning according to claim 1 is characterized in that: All operation data of the production process in step S1 are specifically: All the operation data of the production process are various resource type data collected through various methods during the production process. The production process includes the placement of workshop material orders, workshop workpiece process scheduling and workshop completion results. The operation data covers the changes in various resource information of "people, machines, materials, methods and environment" in the workshop, including the working status data of production equipment collected in real time by the Internet of Things, information data of workpieces to be processed and workpiece scheduling and transfer data, as well as production order data, material data and processing technology data imported into the workshop by other production systems.

3. The real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning according to claim 2 is characterized in that: In step S2, the operation data collected in step S1 is classified and processed into Markov decision chain data, specifically: Classify by workshop resource category and build multi-channel resource view as status s i , a state s i Contains all resource dimension status information taken into consideration. Each resource dimension status information is represented by a channel status view in the form of a matrix, expressed as follows: s i =<P m×c ,M m×c ,W m×c ,F m×c ,……,E m×c >,i∈(0,1,2…,n-1) Where, P m×c is the workshop personnel information matrix, M m×c is the machine equipment information matrix, W m×c is the workpiece material information matrix, F m×c is the process information matrix, E m×c is the environmental information matrix, m×c represents the matrix size, the total number of workpieces in the workshop m is the number of matrix rows, and the total number of processing equipment c is the number of matrix columns. Among them, the state matrix of each resource dimension can be flexibly expanded to other resource types used for decision-making according to the needs of production logic; Build actions based on the flow of workshop workpieces between equipment action For a specific workpiece to be transferred from a certain process equipment to the next process equipment, the workpiece θ H Directed flow Tr between processing equipment K To represent, the expression is as follows: Where θ H For H-type workpiece, Tr(O,Z) K Indicates transfer from O-type equipment to Z-type equipment. G is the action code, H is the workpiece category code, and K is the flow code between equipment. The coding rules for each element are as follows: G=(H×sum(Tr))+K Where sum(Tr) represents the total number of directional flows between similar equipment in the workshop; In the workshop status i The transferable action set of the next job is Job(s i ), with a finite number of actions To represent, the expression is as follows: Job is an intelligent function that can effectively express the workshop status s i The workpiece can choose the direction of transfer. The actual production transfer will be executed by selecting one direction from many directions. Every time the smart workshop status is refreshed, i ,Job(s i ) will also be updated accordingly; According to the degree of fit between the scheduling target tar and the completion result T, the workshop is assigned to the state s i Next action Entering a new state i+1 The corresponding reward r; The workshop starts from the order release state s0 to the completion state s n End, form Markov decision chain data one by one according to the order of n scheduling 4. The real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning according to claim 3 is characterized in that: The reward function R is: Where r + and r - They represent positive and negative reward values respectively, and tar is the scheduling target.

5. The real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning according to claim 4 is characterized in that: The online Q network and target Q in step S5 - The network structure of the network is consistent, specifically: The online Q network and the target Q - The network structure of the network is multi-input channel and dual-output subnet; Among them, the number of network input channels is the same as the state s in step S2 i The total number of resource status view channels contained is equal. One network input channel matches the input of a type of workshop resource information. The dual output subnet includes a front-end and an output subnet. The input of the network enters the output subnet after passing through the front-end. The output subnet includes a price function network Vn and an advantage function network An. The price function network Vn is responsible for evaluating the state s i The value of the advantage function network An is responsible for each action in state s The relative advantages and disadvantages of the network, the final network value output Q (s, a; ω, α, β) is obtained by the linear combination of the two: Where ω represents the network parameters before the dual-output subnet, α and β represent the parameters of the price function network Vn and the advantage function network An, respectively, A represents the set of all actions, a′∈A.

6. The real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning according to claim 5 is characterized in that: The front end is composed of multiple layers of network units, each layer of network units is composed of a convolutional layer, a normalization layer and an activation function, and the output subnets are all built by fully connected layers.

7. The real-time scheduling method for intelligent workshops with workpiece flow based on D3QN reinforcement learning according to claim 6 is characterized in that: The front end is connected to the output subnet through a flattening layer, which unfolds the features.

Images