Tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning
Through the combination of graph convolution network and reinforcement learning, different process path scheduling in tin smelting production is optimized, and the problem of low efficiency and low resolution quality in the existing technology is solved, and the efficiency improvement of tin smelting production and equipment continuity are achieved. optimization.
Patent Information
- Application Number
- CN202410871482.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to effectively optimize the scheduling of different process paths in tin smelting production, resulting in low production efficiency and difficult to obtain feasible solutions that meet actual production requirements in a short period of time.
Using a combination of graph convolutional networks and reinforcement learning, the tin smelting process path is represented as an excised graph through the deep deterministic strategy gradient algorithm (DDPG), and the node feature information is extracted using graph neural network and input into the DDPG model. Generate the optimal scheduling strategy and optimize the process flow path.
The process flow path optimization under the conditions of output and quality constraints is achieved, the efficiency of tin smelting production and the quality of scheduling strategies are improved, and the continuity of equipment and operation optimization is ensured.
Smart Images

Figure CN118917571A_ABST
Abstract
Description
Technical Field
[0001] The present invention proposes a scheduling optimization method based on graph convolutional network and reinforcement learning, and relates to the technical field of tin smelting production scheduling optimization. Background Art
[0002] Tin smelting production has many links, multiple processes are coupled and related, and the process flow changes caused by different processes of multi-grade tin. At present, tin smelting plants mainly rely on personal experience to determine production scheduling plans, resulting in low tin ingot production efficiency. In order to reduce enterprise resource consumption, improve production efficiency, and enhance enterprise competitiveness, a more reasonable production scheduling strategy is urgently needed. Therefore, the scheduling of tin smelting plants under different process paths has become an important research content of industrial scheduling. Tin smelting production scheduling is to determine the corresponding processing equipment and the corresponding processing start time of each process under the premise of ensuring that the equipment does not produce operation conflicts and interruptions during the smelting process, with the uniqueness of the operating equipment of each process in any processing time and the time continuity of any processing equipment as constraints, so as to form a production operation schedule and achieve the goal of minimum total completion time.
[0003] At present, industrial scheduling technology mainly includes precise methods, heuristic methods and intelligent methods. Traditional precise methods search for the optimal solution in the entire solution space, but their calculation time is usually unacceptable, and the algorithm is usually only suitable for solving small-scale problems. Heuristic methods can solve scheduling problems in a reasonable time, but these methods are usually based on specific heuristic rules, which can quickly obtain solutions to the problem, but it is difficult to guarantee the quality of the solution. Reinforcement learning aims to continuously adjust strategies to maximize cumulative rewards through the interaction between the intelligent agent and the environment. Although the method combining heuristic algorithms with reinforcement learning can overcome the local optimal problem to a certain extent, it is still possible to fall into the local optimal in high-dimensional complex scheduling problems.
[0004] The above existing methods are difficult to guarantee the solution quality of the scheduling optimization problem under different process paths of the tin smelting plant, and it is difficult to obtain a feasible solution that meets the actual production requirements in a short time. In order to solve the above problems, the present invention proposes a scheduling method under different process paths of tin smelting based on graph convolutional network and reinforcement learning. The deep deterministic policy gradient algorithm is used to optimize the process flow path of different grades of tin under the constraints of output and quality, so as to realize the control of tin smelting production process. Summary of the invention
[0005] The present invention provides a scheduling optimization method based on graph convolutional network and reinforcement learning for tin smelting production. A graph structure is used to express the complex relationship in the scheduling problem of different process paths of tin smelting. Then, the node feature information is extracted using the graph structure through the graph neural network to generate a feature vector. Finally, the feature vector is input into the DDPG model, and the optimal scheduling strategy is obtained through DDPG.
[0006] The specific steps are:
[0007] Step 1: Represent the process path of tin smelting as a disjunctive graph G = (O, C, D), describe the relationship and constraints of the production process, and establish the time series of each equipment operating each process;
[0008] Step 1.1: Represent the process as a node (O);
[0009] Step 1.2: The fixed sequence relationship between the processes of the directed edge finished products is represented as a connected arc (C);
[0010] Step 1.3: Represent the relationship between processes that can be operated on the same equipment as disjunctive arcs (D);
[0011] Step 1.4: Establish the equipment operation time series and represent the processing time of process O on machine M as T.
[0012] Step 2: Use graph neural network to encode the input disjunctive graph, use Transformer Encoder to process the device status information, and use the encoded feature vector as the input of the reinforcement model;
[0013] Step 2.1: Extract the global state information of the disjunctive graph through the graph neural network;
[0014] First, for the nodes in the disjunctive graph, the graph neural network calculates the attention coefficient through the learned weights and features. For each node O ij and its neighbor node O kl , the attention coefficient is calculated according to the following formula:
[0015] e ij,kl =LeakyReLU(a T ·[W·h ij ||W·h kl ])
[0016] Among them, W is the weight matrix for linear transformation, a T is the parameter of the attention mechanism, || represents the vector connection operation, and LeakyReLU is the ReLU activation function with leakage;
[0017] Next, the attention coefficient is normalized using the softmax function:
[0018]
[0019] in, Represents the set of neighbor nodes of a node;
[0020] According to the calculated attention coefficient, the features of neighboring nodes are weighted and aggregated to the current node, and the feature vector of each node is updated:
[0021]
[0022] Where: h′ ij is the updated node O ij The feature vector of , σ is the activation function;
[0023] After neighborhood aggregation, the feature vector of each node is the final feature representation of the node.
[0024] Step 2.2: The device status is feature extracted through Transformer Encoder;
[0025] Step 2.3: Use the extracted feature vector as the input of the policy network in subsequent reinforcement learning.
[0026] Step 3: Based on the obtained node features, the features are input into the DDPG policy network. The policy network selects actions based on the input feature vector to generate scheduling strategies, including job operation strategies and machine selection strategies.
[0027] Step 3.1: Input the node feature vector generated by the graph neural network into the job operation strategy encoder, and the encoder calculates the priority of each operation based on the input process information;
[0028] Step 3.2: Based on the input operation priority and device status, the machine selection strategy encoder calculates the priority score of each device in combination with the operation priority score and the current device status information;
[0029] Step 3.3: At each time step, the agent selects the highest priority operation from all pending operations based on the priority scores generated in Step 3.1, and selects the most appropriate device for the selected operation based on the device priority calculated in Step 3.2.
[0030] Step 4: The agent executes the operation selected by the DDPG policy network and updates the environment state and reward based on the results of the operation;
[0031] Step 4.1: The agent will execute the job operation and machine selection selected by the policy network in Step 3, update the disjunctive graph and equipment information status, and generate a new state;
[0032] Step 4.2: Store the current experience quadruple (state S, action A, reward R, next state S′) into the experience pool;
[0033] Step 4.3: The reward function is expressed as r(s, a) = -d t , where d t =C(s t+1 )-C(s t ), represents the difference in scheduling strategy between any two consecutive time steps t and t+1, C(s t ) is the maximum completion time of all operations at time step t.
[0034] Step 5: DDPG’s evaluation network evaluates the effect of the action, and the policy network continuously optimizes its strategy based on the evaluation results until the iteration ends.
[0035] Step 5.1: Evaluation network evaluates the value of the action selected by the strategy network, that is, the value function Q(s, a), which represents the future cumulative reward that may be obtained by taking a specific action in a given state;
[0036]
[0037] Where α is the learning rate, γ is the discount factor, and r is the current reward;
[0038] Step 5.2: The policy network optimizes the scheduling strategy based on the feedback from the evaluation network. The parameters of the policy network are updated in the following ways:
[0039]
[0040] where θπ is the policy network, Q π (s, a) is the output of the value network.
[0041] Beneficial effects: The present invention first represents the process path of tin smelting production as a disjunctive graph, and uses nodes and arcs to represent the various steps of the production process and their sequential relationship. Secondly, the disjunctive graph is encoded using a graph convolutional network, and the state of the disjunctive graph is extracted through a graph attention network to generate a feature vector. And the feature vector of the equipment state is further processed using a Transformer Encoder to extract the complex relationship of the equipment state. Subsequently, these feature vectors are input into the policy network of the deep deterministic policy gradient algorithm (DDPG), which contains a process operation strategy and a machine selection strategy, and generates a scheduling strategy based on the feature vector. The evaluation network of DDPG evaluates the value of the action selected by the policy network, and the policy network continuously optimizes its strategy according to the evaluation results until the iteration ends. Finally, the interaction between the policy network and the evaluation network is continuously adjusted and optimized to ultimately achieve optimal scheduling. The present invention effectively optimizes the scheduling strategy of tin smelting production by combining a graph convolutional network with reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 is a flow chart of the steps of the present invention;
[0043] Figure 2 is a schematic diagram of the disjunctive graph of the example in Table 1. DETAILED DESCRIPTION
[0045] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.
[0046] like Figure 1 As shown in the figure, a scheduling method for a tin smelting plant based on graph convolutional network and reinforcement learning is shown in the figure. The specific steps are as follows:
[0047] Step 1: In the scheduling of the tin smelting plant, the process paths of each grade as shown in Table 1 are represented by a disjunctive graph G = (O, C, D) to represent the production relationships and constraints (such as Figure 2 As shown in Figure 1), the disjunctive graph can represent the various steps of the production process and their sequence relationships in the form of nodes and arcs. is the set of all operations, O ij is the jth process of the i-th workpiece, Start and End are the start and end operations. C is a set of arcs, representing the precedence constraints between consecutive operations from the same process. D is a set of undirected arcs, each connecting operations that can be processed by the same equipment, represented by different colors. In addition, the equipment state will be the operation O ij On machine M k The processing time on sequence.
[0048] Table 1 Examples of tin smelting process paths
[0049]
[0050]
[0051] Step 2: After converting the process path into a disjunctive graph, the next step is to use a graph neural network to encode the input disjunctive graph and use the encoded feature vector as the input of the reinforcement learning model. First, for the nodes in the graph, GAT calculates the attention coefficient using the learned weights and features. For each node O ij and its neighbor node O kl , the attention coefficient is calculated according to the following formula:
[0052] e ij,kl =LeakyReLU(a T ·[W·h ij ||W·h kl ])
[0053] Where W is the weight matrix used for linear transformation. T is the parameter of the attention mechanism. || represents the vector connection operation, and LeakyReLU is the ReLU activation function with leakage.
[0054] Next, GAT normalizes the attention coefficients using a softmax function.
[0055]
[0056] in, Represents the set of neighbor nodes of a node.
[0057] According to the calculated attention coefficient, the features of neighboring nodes are weighted and aggregated to the current node, and the feature vector of each node is updated.
[0058]
[0059] Where: h′ ij is the updated node O ij σ is the activation function.
[0060] After neighborhood aggregation, the feature vector of each node is the final feature representation of the node.
[0061] The device state is extracted through the Transformer Encoder to extract feature information. First, the self-attention mechanism is used to process all machine features and calculate the mutual influence between different machine features. Then, in order to better capture information from different angles, the transformer adopts a multi-head attention mechanism to split the feature space into multiple heads, each head processing a subset of the input features. Finally, the output of the attention layer is further processed by a feedforward neural network. Through these steps, the transformer encoder is able to analyze the machine state.
[0062] The processed feature vector will be used as the input of the subsequent scheduling strategy network.
[0063] Step 3: After the graph neural network encodes the disjunctive graph and the Transformer Encoder encodes the equipment status, the policy network of deep deterministic policy gradient is used to generate the scheduling strategy. The policy network of DDPG includes process operation strategy and machine selection strategy, which can select actions according to the input feature vector, including job operation and machine selection. At each time step, the agent will select the highest priority operation from all pending operations based on the priority score generated by the job operation policy encoder. In addition, according to the equipment priority calculated by the machine selection policy encoder, the highest priority equipment is selected for the selected operation.
[0064] Step 4: After the agent completes the specified operation, it needs to update the state information of the environment, including updating the node attributes and edge attributes in the disjunctive graph, as well as the device status. At the same time, the current experience quadruple (state S, action A, reward R, next state S′) is stored in the experience pool. In the subsequent training process, the agent will randomly sample a batch of experience from the experience pool to update the policy network and value network. The reward function is expressed as r(s, a) = -d according to the scheduling target. t . Where d t =C(s t+1 )-C(s t ), represents the difference in scheduling strategy between any two consecutive time steps t and t+1. t ) is the maximum completion time of all operations at time step t.
[0065] Step 5: The agent optimizes the scheduling strategy through the interaction between the evaluation network and the policy network. First, the evaluation network evaluates the value of the action selected by the policy network, that is, the value function Q(s, a), which represents the future cumulative reward that may be obtained by taking a specific action in a given state. By calculating the value function Q(s, a), the evaluation network quantifies the pros and cons of the current scheduling strategy and provides a basis for strategy improvement. The evaluation network updates the Q value through the Bellman equation:
[0066]
[0067] Where α is the learning rate, γ is the discount factor, and r is the current reward.
[0068] Secondly, the policy network optimizes the scheduling strategy based on the feedback from the evaluation network and selects the action that can maximize the future cumulative reward under the current state. The parameters of the policy network are updated in the following way:
[0069]
[0070] where θπ is the policy network, Q π (s, a) is the output of the value network.
[0071] The intelligent agent continuously adjusts and optimizes through the interaction between the evaluation network and the policy network, and ultimately achieves the optimal scheduling.
[0072] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning, characterized in that: Step 1: Represent the process path of tin smelting as a disjunctive graph G = (O, C, D), describe the relationship and constraints of the production process, and establish the time series of each equipment operating each process; Step 2: Use graph neural network to encode the input disjunctive graph, use Transformer Encoder to process the device status information, and use the encoded feature vector as the input of the reinforcement model; Step 3: Based on the obtained node features, the features are input into the DDPG policy network. The policy network selects actions based on the input feature vector to generate scheduling strategies, including job operation strategies and machine selection strategies. Step 4: The agent executes the operation selected by the DDPG policy network and updates the environment state and reward based on the results of the operation; Step 5: DDPG’s evaluation network evaluates the effect of the action, and the policy network continuously optimizes its strategy based on the evaluation results until the iteration ends.
2. The tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning according to claim 1 is characterized in that: The Step 1 is specifically as follows: Step 1.1: Represent the process as a node (O); Step 1.2: The fixed sequence relationship between the processes of the directed edge finished products is represented as a connected arc (C); Step 1.3: Represent the relationship between processes that can be operated on the same equipment as disjunctive arcs (D); Step 1.4: Establish the equipment operation time series and represent the processing time of process O on machine M as T.
3. The tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning according to claim 1 is characterized in that: The Step 2 is specifically as follows: Step 2.1: Extract the global state information of the disjunctive graph through the graph neural network; Step 2.2: The device status is feature extracted through Transformer Encoder; Step 2.3: Use the extracted feature vector as the input of the policy network in subsequent reinforcement learning.
4. The tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning according to claim 3 is characterized in that: The Step 2.1 is specifically as follows: First, for the nodes in the disjunctive graph, the graph neural network calculates the attention coefficient through the learned weights and features. For each node O ij and its neighbor node O kl , the attention coefficient is calculated according to the following formula: and ij,kl =LeakyReLU(a T ·[W·h ij ||W h kl ]) Among them, W is the weight matrix for linear transformation, a T is the parameter of the attention mechanism, || represents the vector connection operation, and LeakyReLU is the ReLU activation function with leakage; Next, the attention coefficient is normalized using the softmax function: in, Represents the set of neighbor nodes of a node; According to the calculated attention coefficient, the features of neighboring nodes are weighted and aggregated to the current node, and the feature vector of each node is updated: Where: h' ij is the updated node O ij The feature vector of , σ is the activation function; After neighborhood aggregation, the feature vector of each node is the final feature representation of the node.
5. The tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning according to claim 1 is characterized in that: The Step 3 is specifically as follows: Step 3.1: Input the node feature vector generated by the graph neural network into the job operation strategy encoder, and the encoder calculates the priority of each operation based on the input process information; Step 3.2: Based on the input operation priority and device status, the machine selection strategy encoder calculates the priority score of each device in combination with the operation priority score and the current device status information; Step 3.3: At each time step, the agent selects the highest priority operation from all pending operations based on the priority scores generated in Step 3.1, and selects the most appropriate device for the selected operation based on the device priority calculated in Step 3.
2.
6. The tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning according to claim 1 is characterized in that: The Step 4 is specifically as follows: Step 4.1: The agent will execute the job operation and machine selection selected by the policy network in Step 3, update the disjunctive graph and equipment information status, and generate a new state; Step 4.2: Store the current experience quadruple (state S, action A, reward R, next state S') into the experience pool; Step 4.3: The reward function is expressed as r(s,a)=-d t , where d t =C(s t+1 )-C(s t ), represents the gap in scheduling strategies between any two consecutive time steps t and t+1, C(s t ) is the maximum completion time of all operations at time step t.
7. The tin smelting production scheduling optimization method based on graph convolutional network and reinforcement learning according to claim 1 is characterized in that: The Step 5 is specifically as follows: Step 5.1: Evaluation network evaluates the value of the action selected by the policy network, i.e., the value function Q(s,a), which represents the future cumulative reward that may be obtained by taking a specific action in a given state; Where α is the learning rate, γ is the discount factor, and r is the current reward; Step 5.2: The policy network optimizes the scheduling strategy based on the feedback from the evaluation network. The parameters of the policy network are updated in the following ways: where θ is π Policy network, Q π (s,a) is the output of the value network.
Citation Information
Cited By
Retired battery disassembly scheduling method based on reinforcement learning
CN119849887A
A Retired Battery Disassembly Scheduling Method Based on Reinforcement Learning
CN119849887B
Roadway local cooling control method and system based on deep reinforcement learning
CN119882879A
Deep reinforcement learning flexible manufacturing workshop job scheduling method based on double-path graph attention
CN119960401A
Tin smelting chemical reaction optimization control method based on reinforcement learning
CN120178693A
