Production line layout optimization method based on multi-agent reinforcement learning

Abstract

The invention relates to the technical field of production line layout, in particular to a production line layout optimization method based on multi-agent reinforcement learning, and the method comprises the steps: constructing a topological structure of a production line through employing a directed weighted network based on a complex network theory; carrying out community division on nodes in the topological structure based on a modularity optimized condensed hierarchical clustering algorithm to obtain a plurality of communities; and carrying out optimized layout on the equipment in each community based on a multi-agent layout algorithm of MAPPO to complete layout of all the equipment in each community, and then carrying out manual layout on all the communities to obtain an optimized production line layout. According to the invention, on the basis of production line equipment community classification of a complex network, modular recombination of equipment clusters is realized through community division, and the cross-community logistics complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of production line layout, and in particular to a production line layout optimization method based on multi-agent reinforcement learning. Background Art

[0002] With the rapid development of Industry 4.0 and intelligent manufacturing, China's manufacturing industry is currently at a critical stage of transformation and upgrading. It is undergoing a profound transformation from a traditional model to one that is digital, networked, and intelligent, demonstrating a trend toward intelligent, green, and high-end manufacturing. Market demand is also gradually shifting toward high-variety, low-batch production. High-variety, low-batch production is a manufacturing model driven by diverse market demand. It involves the simultaneous production of multiple specifications and models within a specific cycle, with relatively low yields per product. However, with the growing demand for product customization, the increasing heterogeneity of production equipment, and the frequent occurrence of dynamic disruptions such as urgent order insertions and equipment downtime, traditional layout optimization methods face significant challenges. This is particularly true in the manufacturing of complex products such as aerospace and new energy vehicles, where production lines comprise hundreds of pieces of equipment, process flows are complex, and logistical relationships between equipment exhibit highly nonlinear coupling. Against this backdrop, achieving rapid response and global optimization in dynamic environments has become a key issue in the design of intelligent manufacturing systems.

[0003] Layout problems often involve multi-dimensional decision-making (such as equipment selection, location allocation, and path planning). Traditional single-agent approaches struggle to efficiently solve these problems due to the exploding dimensionality of the action space. In recent years, reinforcement learning has evolved from single-agent to multi-agent approaches. The distributed collaboration and dynamic adaptability of multi-agent reinforcement learning can effectively address the shortcomings of traditional approaches.

[0004] Deep reinforcement learning (DRL) has achieved breakthroughs in areas such as robot path planning and resource scheduling, but its application to production line layout remains exploratory. Multi-agent reinforcement learning (MARL) further expands the scope of RL, effectively solving multi-objective optimization problems in complex workshops through the collaborative decision-making of distributed agents. However, existing MARL research has primarily focused on gaming or traffic control scenarios, and targeted algorithm design and engineering validation for the manufacturing industry remain underdeveloped.

[0005] The prior art discloses an adaptive layout optimization system and planning method for digital workshops. This method achieves improved logistics efficiency, efficient use of space resources, optimized algorithm convergence (obtaining a stable optimal solution within 200 generations), and enhanced engineering applicability in complex workshop scenarios such as shipbuilding through equipment partition modeling, multi-objective optimization constraints, and improved adaptive genetic algorithms. It supports customization of equipment size and workshop boundary parameters and is suitable for discrete manufacturing and process industry scenarios. Although an improved adaptive genetic algorithm is used (such as dynamic adjustment of crossover rate and mutation rate), the core mechanism of the genetic algorithm still relies on population iteration and random search. However, there is a risk of local optimal solution, high computational complexity, dependence of parameter adjustment on experience, and model simplification that limits application scenarios.

[0006] Prior art also discloses a multi-objective optimization layout method for workshop equipment based on an adaptive genetic algorithm. This method primarily provides a compact arrangement for workshop equipment layout, balances algorithm diversity and convergence through adaptive parameter adjustment, and improves area utilization while reducing logistics costs. However, the multi-objective optimization weight setting is highly subjective, the adaptive genetic algorithm parameters lack theoretical support, and the solution adjustment process relies heavily on manual intervention, resulting in insufficient automation.

[0007] The prior art also discloses a workshop facility layout optimization method based on an improved artificial bee colony algorithm. This method optimizes the layout of workshop facilities by improving the discretization processing, local search strategy, and crossover operation of the artificial bee colony algorithm, reducing logistics and handling costs between processing units and improving layout efficiency. However, the local search strategy suffers from insufficient efficiency and diversity, the crossover operation has adaptability issues, the population diversity maintenance mechanism is weak, and there are bottlenecks in computational complexity and scalability.

[0008] Metaheuristic algorithms are currently the most widely used algorithms in the field of shop floor layout. While they can obtain acceptable solutions within a reasonable timeframe for large-scale combinatorial optimization problems, they are prone to becoming trapped in local optima within complex search spaces. This problem is particularly pronounced when production line layout involves multiple constraints. Algorithm performance is highly dependent on parameter settings (such as the crossover rate and mutation rate in genetic algorithms), and the optimal parameter combinations for different production line layout problems vary significantly, requiring extensive experimental adjustments and increasing the difficulty of practical application. Furthermore, due to the influence of the initial solution and random perturbations, the algorithm's convergence speed and solution quality can fluctuate significantly, making it difficult to meet the real-time and stability requirements of shop floor scheduling.

[0009] Reinforcement learning shows promise in dynamic optimization, but single-agent RL models equipment layout as a single decision sequence, causing the action space to explode exponentially with the number of devices. The distributed collaboration and dynamic adaptability of multi-agent reinforcement learning can effectively address the shortcomings of traditional methods, but few studies have yet applied multi-agent reinforcement learning to production line layout. Summary of the Invention

[0010] In order to solve the above technical problems, the present invention mainly aims at the fact that existing MARL research mostly adopts homogeneous agent design, which is difficult to adapt to the heterogeneous decision-making requirements of equipment selection and location allocation. The present invention provides a production line layout optimization method based on multi-agent reinforcement learning. The method applies the multi-agent reinforcement learning algorithm to the production line layout problem, in order to improve the efficiency of solving the layout optimization problem through algorithm innovation and to open up new solution ideas. The present invention proposes a DA-LA heterogeneous agent architecture: the equipment selection agent (DA) makes discrete decisions based on logistics correlation, and the location allocation agent (LA) allocates equipment coordinates. The two achieve collaboration through shared layout matrix observation.

[0011] The first object of the present invention is to provide a production line layout optimization method based on multi-agent reinforcement learning, which is used for the layout of multiple devices in a production line, including:

[0012] Based on complex network theory, a directed weighted network is used to construct the topology of the production line;

[0013] Based on modularity optimization, an agglomerative hierarchical clustering algorithm is used to divide the nodes in the topology structure into communities to obtain multiple communities. Each community includes multiple nodes, and each node is a device in the production line.

[0014] The MAPPO-based multi-agent layout algorithm optimizes the layout of equipment in each community, completes the layout of all equipment in each community, and then manually layouts all communities to obtain the optimized production line layout.

[0015] Preferably, the MAPPO-based multi-agent layout algorithm optimizes the layout of devices in each community, including:

[0016] Obtain status information in any community; the status information includes location status and device status;

[0017] Configuring a corresponding selection agent and location agent for each device in the community; wherein the selection agent and location agent contain their respective corresponding policy networks;

[0018] Select the agent observation space, location agent observation space and shared observation space according to the state information setting;

[0019] The device is selected by selecting the agent's corresponding policy network based on the agent's observation space and the shared observation space. Then, the location agent selects the device selected by the selection agent according to the corresponding policy network based on the location agent's observation space until the layout of all devices in the community is completed.

[0020] Determine a reward value using a target reward mechanism according to the layout of all devices in the community, and optimize the selection agent and the position agent according to the reward value;

[0021] Among them, the target reward mechanism is determined based on minimizing the total logistics volume.

[0022] Preferably, the equipment layout process includes:

[0023] Obtain the state space corresponding to each device based on the location state and device state;

[0024] Setting an action space for a selection agent to execute based on the device state, and setting an action space for a location agent to execute based on the location state;

[0025] The policy network corresponding to the selection agent and the policy network corresponding to the position agent are both trained by deep reinforcement algorithms.

[0026] Preferably, the action space of the policy network corresponding to the selection agent is the number of optional devices; the action space of the policy network corresponding to the location agent is the size of the area where the community is located.

[0027] Preferably, the target reward mechanism includes an immediate reward for the selection agent, an immediate reward for the placement agent, and a delayed reward shared by the selection agent and the placement agent when the layout is fully completed;

[0028] The instant reward of the selected agent is:

[0029] The immediate reward of the position agent is:

[0030] The delayed reward is:

[0031] Where, Perform actions for optional devices; is the action space for the number of optional devices;

[0032] w ia is the logistics volume between equipment i and equipment a;

[0033] d ia is the Manhattan distance between device i and device a;

[0034] M(Vi ,S t ) is a binary character, where the node set V = {v i ,i=1,2,...,m},V i Indicates that the i-th device is in state S t Next, when V i 1 if it has been scheduled, otherwise 0;

[0035] w ij is the logistics volume between equipment i and equipment j;

[0036] d ij is the Manhattan distance between device i and device j.

[0037] Preferably, the agent observation space is selected as the logistics relationship between the equipment that has not been laid out and the equipment that has been laid out; the location agent observation space is a matrix with the same dimension and size as the layout matrix, and the value of the matrix element is the sum of the Manhattan distance between the location and the laid out equipment; the shared observation space is the current layout matrix.

[0038] Preferably, a directed weighted network is used to construct the topology of the production line, including:

[0039] The equipment in the production line is defined as nodes in the topology;

[0040] Each edge in the topological structure is a directed edge, representing the processing route from one node to another;

[0041] The weight of each directed edge represents the number of times the processing route is reused, that is, the logistics volume between one node and another.

[0042] Preferably, an agglomerative hierarchical clustering algorithm based on modularity optimization is used to divide the nodes in the topological structure into communities, including: first initializing each node as a community, then traversing all possible community mergers, selecting the merger method with the largest modularity gain, and repeatedly traversing the communities until the modularity cannot be improved.

[0043] Preferably, the modularity is the ratio of the total number of edges within the community to the total number of edges in the network minus an expected value, where the expected value is the ratio of the total number of edges within the community to the total number of edges in the network when the network is set as a random network and the same community distribution is formed.

[0044] A second object of the present invention is to provide a production line layout optimization system based on multi-agent reinforcement learning, comprising:

[0045] Topology construction module, which is used to construct the topology structure of the production line using a directed weighted network based on complex network theory;

[0046] The community partitioning module is used to divide the nodes in the topology structure into communities based on the agglomerative hierarchical clustering algorithm optimized by modularity, and obtain multiple communities; each community includes multiple nodes, and each node is a device in the production line;

[0047] The layout module is used to optimize the layout of equipment in each community based on MAPPO's multi-agent layout algorithm, complete the layout of all equipment in each community, and then manually layout all communities to obtain the optimized production line layout.

[0048] The present invention has at least the following beneficial effects:

[0049] The present invention provides a production line layout optimization method based on multi-agent reinforcement learning. The method divides the production line logistics relationship network into communities through the FN algorithm, and the modularity Q reaches 0.34, indicating that the equipment in the production line system obviously has a community structure, and the classification algorithm based on the complex network has better classification effect than other algorithms. The algorithm design for reinforcement learning is based on the classification results, which significantly reduces the strategy search space of the layout problem and provides a structural basis for multi-agent collaborative optimization. The present invention decomposes the equipment layout problem through the division of labor and cooperation between the equipment selection agent (DA) and the location agent (LA), and reduces the action space dimension from the product of the total number of equipment and the total number of positions to the sum of the total number of equipment and the total number of positions, solving the problem of high difficulty in solving the problem caused by the excessively large action space of the traditional single-agent method. Experimental results show that the optimal solution given by multi-agent reinforcement learning is better than the traditional genetic algorithm.

[0050] This paper proposes a DA-LA dual-agent collaboration mechanism. Through the division of labor and collaboration between a device selection agent (DA) and a location allocation agent (LA), the layout problem is decomposed into two subtasks: device selection and location allocation. This compresses the action space and solves the layout of communities classified by the FN algorithm. DA selects devices based on logistics relationships, while LA allocates device locations based on the current layout state. Dynamic interaction between the agents and the environment, coupled with collaborative decision-making through a shared observation space (global layout matrix), enables collaborative decision-making. This method reduces the dimensionality of the action space from the product of the total number of devices and the total number of locations to the sum of the total number of devices and the total number of locations, effectively alleviating the curse of dimensionality.

[0051] This paper designs a hybrid reward function and shared observation mechanism, combining immediate rewards (logistics connections between devices) with delayed rewards (global logistics costs), guiding agents to balance local optimization with global goals, enhancing optimization capabilities. Based on complex network-based production line equipment community classification, this paper achieves modular reorganization of equipment clusters through community division, reducing cross-community logistics complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1It is a complex network diagram of the production line system;

[0053] Figure 2 A community classification diagram for complex networks;

[0054] Figure 3 This is the flow chart of the multi-agent layout algorithm based on MAPPO;

[0055] Figure 4 Layout results for the MAPPO algorithm;

[0056] Figure 5 It is the minimum iteration process of MAPPO algorithm;

[0057] Figure 6 Comparison chart of the optimal solutions of MAPPO and GA. DETAILED DESCRIPTION

[0058] In order to illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description with reference to the embodiments.

[0059] The present invention aims at the distributed collaboration and dynamic adaptability of existing multi-agent reinforcement learning, which can effectively make up for the shortcomings of traditional methods. However, there are few articles that apply multi-agent reinforcement learning to production line layout problems. Therefore, the present invention proposes to apply multi-agent reinforcement learning algorithms to production line layout problems, in order to improve the efficiency of solving layout optimization problems through algorithm innovation and to open up new solution ideas. Existing MARL research mostly adopts homogeneous agent design, which is difficult to adapt to the heterogeneous decision-making requirements of equipment selection and location allocation. The present invention proposes a DA-LA heterogeneous agent architecture: the equipment selection agent (DA) makes discrete decisions based on logistics correlation, and the location agent (LA) allocates equipment coordinates. The two achieve collaboration through shared layout matrix observation.

[0060] To achieve the above objectives, a production line layout optimization method based on multi-agent reinforcement learning is proposed for the layout of multiple devices in a production line, including:

[0061] S1. Based on complex network theory, a directed weighted network is used to construct the topology of the production line;

[0062] A directed weighted network is used to construct the topology of the production line, including:

[0063] The equipment in the production line is defined as nodes in the topology;

[0064] Each edge in the topological structure is a directed edge, representing the processing route from one node to another;

[0065] The weight of each directed edge represents the number of times the processing route is reused, that is, the logistics volume between one node and another.

[0066] S2. An agglomerative hierarchical clustering algorithm based on modularity optimization is used to divide the nodes in the topology structure into communities to obtain multiple communities; each community includes multiple nodes, and each node is a device in the production line; an agglomerative hierarchical clustering algorithm based on modularity optimization is used to divide the nodes in the topology structure into communities, including: first initializing each node as a community, then traversing all possible community mergers, selecting the merger method with the largest modularity gain, and repeatedly traversing the communities until the modularity cannot be improved.

[0067] The modularity is the ratio of the total number of edges within a community to the total number of edges in the network minus an expected value, which is the ratio of the total number of edges within a community to the total number of edges in the network when the network is set as a random network with the same community distribution.

[0068] S3. The multi-agent layout algorithm based on MAPPO optimizes the layout of the equipment in each community, completes the layout of all equipment in each community, and then manually layouts all communities to obtain the optimized production line layout.

[0069] The MAPPO-based multi-agent layout algorithm optimizes the layout of devices in each community, including:

[0070] Obtain status information in any community; the status information includes location status and device status;

[0071] Configuring a corresponding selection agent and location agent for each device in the community; wherein the selection agent and location agent contain their respective corresponding policy networks;

[0072] Select the agent observation space, location agent observation space and shared observation space according to the state information setting;

[0073] The device is selected by selecting the agent's corresponding policy network based on the agent's observation space and the shared observation space. Then, the location agent selects the device selected by the selection agent according to the corresponding policy network based on the location agent's observation space until the layout of all devices in the community is completed.

[0074] Among them, the selection agent observation space is the logistics relationship between the equipment that has not been laid out and the equipment that has been laid out; the location agent observation space is a matrix with the same dimension and size as the layout matrix, and the value of the matrix element is the sum of the Manhattan distance between the location and the laid out equipment; the shared observation space is the current layout matrix.

[0075] The location status is the layout matrix at the current moment; the device status is whether the current device is laid out or not, that is, whether the device is not laid out or has been laid out.

[0076] In this embodiment, a policy network corresponding to an agent is selected, whose input includes local observations and shared global observations;

[0077] Local observation: Obs1 is a one-dimensional array of dimension M×1, where M is the total number of devices and each element represents the sum of the logistics correlation between unplaced devices and placed devices:

[0078]

[0079] where w ij is the logistics volume from device i to device j, and δ(·) is the indicator function.

[0080] Shared global observation: The global layout matrix S, assuming its size is 10×10, represents the plane layout status of the workshop. The matrix element S[x][y] represents the equipment number placed at the coordinate (x, y) (0 when not placed).

[0081] The output is: action probability distribution, the output dimension is M×1, which represents the probability of selecting each device from M devices, specifically the device selection action space A DA The discrete probability distribution of ,guides DA to select devices that are not currently deployed.

[0082] In this embodiment, the policy network corresponding to the location agent has an input including a location heat map encoding;

[0083] Location heat map encoding: The 10×10 matrix Obs2 represents the "layout feasibility" score of each location in the workshop. The value of the matrix element (x, y) is the sum of the Manhattan distances between the location and the already laid out equipment. It is used to guide LA to select locations with a high degree of logistics correlation with the already laid out equipment.

[0084] The specific calculation formula is:

[0085]

[0086] Among them, d i,(x,y) is the Manhattan distance between the current position of device i and the coordinate (x, y)

[0087] The output is the action probability distribution: Assuming the workshop is a 10×10 grid, the output dimension is 100×1, which represents the probability of selecting each position from 100 discrete positions, specifically the position allocation action space A LA The discrete probability distribution of , guides LA to select the optimal location in the layout area.

[0088] Determine a reward value using a target reward mechanism according to the layout of all devices in the community, and optimize the selection agent and the position agent according to the reward value;

[0089] Among them, the target reward mechanism is determined based on minimizing the total logistics volume.

[0090] The equipment layout process includes:

[0091] Obtain the state space corresponding to each device based on the location state and device state;

[0092] Setting an action space for a selection agent to execute based on the device state, and setting an action space for a location agent to execute based on the location state;

[0093] The policy network corresponding to the selection agent and the policy network corresponding to the position agent are both trained by deep reinforcement algorithms.

[0094] The action space of the policy network corresponding to the selection agent is the number of optional devices; the action space of the location agent for the policy network is the size of the community area.

[0095] The target reward mechanism includes an immediate reward for the selection agent, an immediate reward for the placement agent, and a delayed reward shared by the selection agent and the placement agent when the layout is fully completed;

[0096] The instant reward of the selected agent is:

[0097] The immediate reward of the position agent is:

[0098] The delayed reward is:

[0099] Where, Perform actions for optional devices; is the action space of the number of optional devices;

[0100] w ia is the logistics volume between equipment i and equipment a;

[0101] d ia is the Manhattan distance between device i and device a;

[0102] M(V i ,S t ) is a binary character, where the node set V = {v i ,i=1,2,...,m},V i Indicates that the i-th device is in state S t Next, when V i 1 if it has been scheduled, otherwise 0;

[0103] w ijis the logistics volume between equipment i and equipment j;

[0104] d ij is the Manhattan distance between device i and device j.

[0105] In order to illustrate the production line layout optimization method based on multi-agent reinforcement learning provided by the present invention, it is described with reference to the accompanying drawings.

[0106] Step 1: Community classification based on complex networks

[0107] Based on complex network theory, the topological structure of the product production line is represented by a directed weighted network G(V, E, W). The equipment in the production line is defined as a node, and the node set is V = {v i ,i=1,2,...,n}, where v i Represents the i-th device; the directed edge set E={e ij ,i,j=1,2,...,n}, directed edge e ij Indicates that the product is from node v i To node v j The processing route is described by a binary relationship to describe the connectivity of the directed edge, that is, when two nodes are connected, e ij =1, when disconnected, e ij =0. In addition, due to the directed nature of the edge, e ij ≠e ji ; directed edge E ij The weight set W={w ij ,i,j=1,2,...,n}, which indicates the number of times the processing route is reused, that is, node v i With node v j The amount of logistics between them.

[0108] For example, consider the production line of an aircraft engine parts workshop. This production line includes 15 types of parts and 41 pieces of equipment. Its process route can be represented by the equipment's sequence number in the node set, as shown in Table 1.

[0109] Table 1 Process route

[0110]

[0111]

[0112] Use Spring Layout to draw complex network diagrams. For complex network diagrams, see Figure 1 shown.

[0113] The adjacency matrix is ​​one of the core mathematical tools in complex network research, which is used to quantitatively describe the topological connection relationship between nodes in the network. For a directed weighted network containing N nodes, its adjacency matrix A is an N×N square matrix. In the adjacency matrix of a directed weighted network, the matrix element a ij is a non-negative real number w in the complex network ij , representing the slave node v i Points to node v j The weight of the edge. In the complex network model of the production line, the element a of its adjacency matrix ij For node v i and v j At the same time, materials will not flow to a node after completing their tasks at a node, that is, the node will not form a self-loop. Therefore, the value of the diagonal of the adjacency matrix is ​​always zero. In addition, due to the directed nature of the edge, E ij ≠E ji , that is, the adjacency matrix is ​​an asymmetric matrix. The adjacency matrix is ​​shown in A1 below.

[0114]

[0115] Modularity Q (Modularity Q) is the most widely used indicator for judging the strength of community characteristics. Modularity optimization has an accuracy rate of over 85% in detecting communities in networks, significantly higher than spectral clustering and random walk algorithms.

[0116] Modularity Q is defined as the ratio of the total number of edges within a community to the total number of edges in the network minus an expected value. The expected value is the ratio of the total number of edges within a community formed by the same community distribution when the network is set as a random network to the total number of edges in the network. Its physical meaning is the difference between the number of edges connecting nodes within the community and the number of edges in a random situation. Its value range is [-1 / 2, 1), reflecting the difference between the internal connection density of the module and the random network. Q = 0.3 is generally used as the lower bound of the network with a clear community structure. When the Q value is between [0.3, 0.7), it indicates that the clustering effect is good. A high modularity Q indicates that there are dense internal connections and sparse cross-community connections in the network.

[0117] The calculation formula of Q value is as follows:

[0118]

[0119] m is the total number of edges in the network, which is calculated as follows:

[0120]

[0121] Among them, A ij For node v i With node v jThe edge weight between them (1 for unweighted graphs). For node v i With node v j The degree of a node refers to the number of other nodes connected to the node, which represents the importance of the node in the network. The node degree in a directed network is divided into out-degree and in-degree. Refers to v i The number of edges pointing to other nodes, indegree It refers to the direction from other nodes to v i The number of edges. is the sum of each row of the adjacency matrix; in-degree is the sum of each column of the adjacency matrix. i ,c j ) is the indicator function, when the node v i With node v j It is 1 if they belong to the same community, and 0 otherwise.

[0122] The FN algorithm is an agglomerative hierarchical clustering algorithm based on modularity optimization. It maximizes modularity by gradually merging nodes or communities. The steps are: first initialize each node as a community, then traverse all possible community mergers, select the merger method with the largest modularity gain, and repeat the traversal of the community until the modularity cannot be improved. Its advantage is high computational efficiency and a time complexity of O(n 2 ), suitable for medium-sized networks; the results are stable, a deterministic process without random interference; and the hierarchical output can provide multi-scale community delineation results. However, its disadvantage is its high computational cost, making it unsuitable for ultra-large networks. In this example, the modularity Q value of the FN algorithm classification result is 0.34. The community classification results are shown in Table 2.

[0123] Table 2 Classification results of FN algorithm

[0124]

[0125] By comparison, the FN algorithm has the highest modularity, which is greater than 0.3, proving that the production line system has a distinct community structure. The FN algorithm, which has a high modularity and relatively even classification results, is selected to classify the data into complex network communities, and the next step of layout is carried out based on the classification results of the FN algorithm.

[0126] The complex network diagram of the production line process route after classification, see Figure 2 Show. Figure 2 The same color markers are used to illustrate the distribution of device nodes within the same community. Nodes between communities are arranged using force-directed layout. The modularity Q after classification also indicates significant community differentiation among production equipment. Compared to traditional classification methods, the division strategy based on weighted network community structure is more accurate.

[0127] Step 2: Multi-agent layout algorithm based on MAPPO

[0128] 2.1 Markov parameter design

[0129] (1) State space

[0130] Assume a 10×10 matrix S, where each matrix element represents a 1×1 area, and the number represents the equipment number placed at that location. As shown below, S1 indicates placing equipment No. 2, 3, and 4 in a 2×2 workshop, where equipment 2 is 1×2 in size, and equipment 3 and equipment 4 are 1×1 in size.

[0131]

[0132] According to the characteristics of the layout problem, the system state S at time t t By location status and device status Among them, the position state The device status is represented by the current layout matrix S. Represents the set of devices that are not currently laid out, consisting of an optional node set M(V i ,S t ), where the node set V = {V i ,i=1,2,...,n},v i Indicates the i-th device, M(V i ,S t ) is a binary character, in state S t Next, when v i 1 if it has been scheduled, 0 otherwise.

[0133] (2) Action Design

[0134] At time t, the joint action a t Depend on and The two actions correspond to the selection of equipment by agent DA and the selection of location by agent LA. By device status The action space that the agent DA can perform is the set of all unconfigured devices in the current state, that is, M(V i ,S t ) t The device v corresponding to 0 i A collection of By location status The action space that the agent LA can perform is determined by the time matrix S at time t. t The set of positions (x,y) that are 0.

[0135] right The action space is the number of optional devices. The action space is the workshop length × workshop width. To simplify the calculation, the two-dimensional position (x, y) is encoded into a one-dimensional discrete action (0 to 99), which is decoded by rounding / / and taking the remainder %. The values ​​of x and y are as follows:

[0136]

[0137] (3) State Transfer

[0138] At time t, the agent DA performs actions based on the available equipment Agent LA takes actions based on DA Execute an action Select the location. Then update the layout matrix S and the optional device set M (V i ,S t ).

[0139] (4) Rewards

[0140] In this paper, the goal of the layout problem is to minimize the total logistics volume, with immediate feedback on the volume during the layout process. The reward is divided into two components: an immediate reward (RI), which represents the logistical correlation between the new equipment selected by the DA and the already placed equipment; and a delayed reward (RC), which is given after all equipment has been placed. Furthermore, the DA and LA agents solve different subproblems, requiring the design of corresponding reward signals for each.

[0141] For the agent DA, when it is performing equipment selection tasks, it will be penalized if the equipment it selects has already been laid out; for the agent LA, when it is performing location selection tasks, it will be penalized if the location it selects has already been placed or is beyond the scope of the workshop.

[0142] For the agent DA, the immediate reward RI1 is as follows:

[0143]

[0144] For agent LA, the immediate reward RI2 is as follows:

[0145]

[0146] Among them, a is the action of the intelligent agent DA, m is the total number of devices, M(V i ,S t ) is a binary character, where the node set V = {v i ,i=1,2,...,m},V iIndicates that the i-th device is in state S t Next, when V i 1 if it has been scheduled, otherwise 0, w ia is the logistics volume between device i and device a, d ia is the Manhattan distance between device i and device a.

[0147] Agents DA and LA share a delayed reward RC as shown below, which is given when the layout is completed:

[0148]

[0149] Among them, w ij is the logistics volume between equipment i and equipment j, d ij is the Manhattan distance between device i and device j.

[0150] 2.2 Observation space setting

[0151] (1) DA observation space

[0152] In this paper, the intelligent agent DA needs to select a suitable device from the optional devices. The intelligent agent DA needs to make a decision based on the current layout matrix S and the information related to the device. Therefore, the local observation space of the intelligent agent DA is the logistics relationship between the devices that have not been laid out and the devices that have been laid out. Therefore, the observation space Obs1 = [o i ,i=1,2,…,m] is designed as a one-dimensional array with a dimension of M×1. If the device V i If the device has already been laid out, its corresponding element is 0. Otherwise, it is the sum of the logistics relationships between the device and the already laid out devices. This encourages placing highly correlated devices in adjacent locations to reduce logistics costs. The element values ​​are as follows.

[0153]

[0154] Among them, m is the total amount of equipment, M(V j ,S t ) is a binary character, where the node set V = {v i ,i=1,2,...,m},V i Indicates that the i-th device is in state S t Next, when V j 1 if it has been scheduled, otherwise 0, w ij is the logistics volume between device i and device j.

[0155] (2) LA observation space

[0156] The intelligent agent LA needs to select a location in the layout area to place the device selected by DA. The intelligent agent LA needs to obtain the location information of the current layout matrix S. Therefore, the local observation space Obs2 of LA is designed to be a matrix with the same dimension and size as the layout matrix, that is, the value at (x, y) represents the sum of the distances from the point to the already laid out devices.

[0157] (3) Shared observation space

[0158] To ensure that the agents DA and LA can make optimal decisions based on the global state, they need to be able to observe the current device layout. Therefore, the current layout matrix S is set as the shared observation space of the agents DA and LA.

[0159] 2.3 MAPPO-based multi-agent layout algorithm process

[0160] During the production line layout process, the equipment selection agent (DA) and the location allocation agent (LA) need to make collaborative decisions: after the DA selects the equipment, the LA places the equipment according to the workshop space, calculates the logistics cost and provides rewards. Based on the above process, see Figure 3 As shown, the entire Markov decision process can be standardized as the following process:

[0161] (1) Environment initialization

[0162] A 10×10 layout matrix is ​​constructed, and the equipment size and logistics association matrix is ​​loaded from a CSV file. A dual-agent system is defined: the action space of the equipment selection agent DA is the set of selectable equipment, and the action space of the location allocation agent LA is the index of valid locations. The action spaces of both agents are discrete, with the action space size of the equipment selection agent DA being the total number of equipment, and the action space size of the location allocation agent LA being the total number of locations, i.e., the workshop length × the workshop width. The shared observation space is the global layout matrix, and the independent observation spaces are the equipment association vector (DA) and the location heat map encoding (LA). The environment initialization process is shown in Table 3.

[0163] Table 3 Production line layout problem environment design

[0164]

[0165] (2) Centralized training architecture

[0166] The CTDE (Centralized Training with Decentralized Execution) framework is adopted, in which the global layout matrix and joint actions are input, and the critic network is used to determine the output state value estimate; each intelligent agent has an independent policy network, and the local observations (DA is the device association and LA is the location heat map) are input and evaluated by the actor network.

[0167] The advantage function is calculated by dynamically adjusting the policy gradient through GAE (Generalized Advantage Estimation, GAE) combined with TD error:

[0168] A(s,a)=Q(s,a)-V(s)

[0169] Among them, Q(s,a) is the action value; V(s) is the state value; through the advantage function, the algorithm can more accurately identify high-yield actions and reduce the variance of strategy updates.

[0170] (3) Strategy Optimization Process

[0171] Trajectory sampling: The parallel environment generates multiple sets of trajectory data, recording the state, action, and reward sequences.

[0172] Value estimation: Critic network computing state value V(S t ), and update the target value through the Bellman equation.

[0173] Policy update: calculate the probability ratio of the new and old policies, trim the objective function, and add an entropy regularization term.

[0174] Parameter synchronization: Update the Actor-Critic network parameters every K steps, and keep the old strategy for importance sampling.

[0175] (5) Termination and Evaluation

[0176] When the termination condition is met, that is, when all devices have completed their layout or the maximum step size (step size = total number of devices) is reached, the calculation terminates and the evaluation metrics are recorded: the value function loss (critic prediction error), policy entropy (exploration level), and global logistics cost. The optimal policy is saved. The MAPPO pseudocode is shown in Table 4 below.

[0177] Table 4 MAPPO algorithm process

[0178]

[0179]

[0180] See also Figure 4As shown in the figure, the blue box is the device, and the red dotted box is the device layout box. The size is the device size plus the device spacing and rounded up. That is, the blank space between the blue box and the red box is the device gap.

[0181] The device spacing is 20% of the device's length and width. The sum of the horizontal and vertical spacing is calculated and rounded up to construct the device layout box. The length and width are calculated as follows:

[0182]

[0183] Where, L i With H i are the length and width of device i, Δx and Δy are the horizontal and vertical intervals between devices, respectively. Δx and Δy are 20% of the length and width of the device, indicating rounding up.

[0184] See also Figure 5 As shown in the figure, it intuitively reflects the effectiveness of the algorithm in multi-agent logistics optimization tasks: through the combination of centralized training and distributed execution, and the stable guidance of the policy gradient by GAE, the system gradually learns the optimal collaborative strategy to reduce logistics volume. The smooth convergence trend of the curve also verifies the successful design of the algorithm in balancing exploration and utilization and controlling variance.

[0185] In order to verify the optimization performance of the multi-agent proximal policy optimization algorithm (MAPPO), 10 independent experiments were conducted to compare it with the traditional genetic algorithm (GA), and the optimal solution was recorded. The comparison of the optimized solutions is shown in Figure 2. Figure 6 As shown in the figure, the red dotted line is the genetic algorithm and the blue solid line is the multi-agent proximal strategy optimization algorithm.

[0186] See also Figure 6 As shown in Figure 2, the MAPPO algorithm exhibits significant advantages in the following aspects:

[0187] (1) Strong stability: The standard deviation of MAPPO’s 10-run results is only 1.22, which is better than 5.97 of GA, showing better robustness.

[0188] (2) Good convergence: 80% of the running results are stable in the range of 79.5-81, reflecting the convergence ability of the algorithm. At the same time, the average value of the optimized solution is 80.35, and the average value of the optimized solution of the traditional genetic algorithm is 88.70. The average value of the optimized solution of the algorithm is significantly better than the average value of the optimized solution of the traditional genetic algorithm.

[0189] (3) Strong anti-fluctuation: Even in the worst case (83.5), MAPPO still outperforms the average performance of GA algorithm (88.7).

[0190] The present invention provides a production line layout optimization system based on multi-agent reinforcement learning, comprising:

[0191] Topology construction module, which is used to construct the topology structure of the production line using a directed weighted network based on complex network theory;

[0192] The community partitioning module is used to divide the nodes in the topology structure into communities based on the agglomerative hierarchical clustering algorithm optimized by modularity, and obtain multiple communities; each community includes multiple nodes, and each node is a device in the production line;

[0193] The layout module is used to optimize the layout of equipment in each community based on MAPPO's multi-agent layout algorithm, complete the layout of all equipment in each community, and then manually layout all communities to obtain the optimized production line layout.

[0194] In summary, the present invention provides a multi-agent reinforcement learning framework based on a multi-agent proximal policy optimization algorithm, which decomposes the layout problem into two subtasks: device selection and location allocation, and realizes collaborative decision-making through a shared observation space.

[0195] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A production line layout optimization method based on multi-agent reinforcement learning, characterized in that: Used for the layout of multiple equipment in a production line, including: Based on complex network theory, a directed weighted network is used to construct the topology of the production line; Based on modularity optimization, an agglomerative hierarchical clustering algorithm is used to divide the nodes in the topology structure into communities to obtain multiple communities. Each community includes multiple nodes, and each node is a device in the production line. The MAPPO-based multi-agent layout algorithm optimizes the layout of equipment in each community, completes the layout of all equipment in each community, and then manually layouts all communities to obtain the optimized production line layout.

2. The production line layout optimization method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The MAPPO-based multi-agent layout algorithm optimizes the layout of devices in each community, including: Obtain status information in any community; the status information includes location status and device status; Configuring a corresponding selection agent and location agent for each device in the community; wherein the selection agent and location agent contain their respective corresponding policy networks; Select the agent observation space, location agent observation space and shared observation space according to the state information setting; The device is selected by selecting the agent's corresponding policy network based on the agent's observation space and the shared observation space. Then, the location agent selects the device selected by the selection agent according to the corresponding policy network based on the location agent's observation space until the layout of all devices in the community is completed. Determine a reward value using a target reward mechanism according to the layout of all devices in the community, and optimize the selection agent and the position agent according to the reward value; Among them, the target reward mechanism is determined based on minimizing the total logistics volume.

3. The production line layout optimization method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The equipment layout process includes: Obtain the state space corresponding to each device based on the location state and device state; Setting an action space for a selection agent to execute based on the device state, and setting an action space for a location agent to execute based on the location state; The policy network corresponding to the selection agent and the policy network corresponding to the position agent are both trained by deep reinforcement algorithms.

4. The production line layout optimization method based on multi-agent reinforcement learning according to claim 3 is characterized in that: The action space of the policy network corresponding to the selection agent is the number of optional devices; the action space of the location agent for the policy network is the size of the community area.

5. The production line layout optimization method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The target reward mechanism includes an immediate reward for the selection agent, an immediate reward for the placement agent, and a delayed reward shared by the selection agent and the placement agent when the layout is fully completed; The instant reward of the selected agent is: The immediate reward of the position agent is: The delayed reward is: Where, Perform actions for optional devices; is the action space for the number of optional devices; w ia is the logistics volume between equipment i and equipment a; d ia is the Manhattan distance between device i and device a; M(V i ,S t ) is a binary character, where the node set V = {v i ,i=1,2,...,m},V i Indicates that the i-th device is in state S t Next, when V i 1 if it has been scheduled, otherwise 0; w ij is the logistics volume between equipment i and equipment j; d ij is the Manhattan distance between device i and device j.

6. The production line layout optimization method based on multi-agent reinforcement learning according to claim 2, characterized in that: The agent observation space is selected as the logistics relationship between the equipment that has not been laid out and the equipment that has been laid out. The location agent observation space is a matrix with the same dimension as the layout matrix, and the value of the matrix element is the sum of the Manhattan distance between the location and the laid out equipment. The shared observation space is the current layout matrix.

7. The production line layout optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: A directed weighted network is used to construct the topology of the production line, including: The equipment in the production line is defined as nodes in the topology; Each edge in the topological structure is a directed edge, representing the processing route from one node to another; The weight of each directed edge represents the number of times the processing route is reused, that is, the logistics volume between one node and another.

8. The production line layout optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: An agglomerative hierarchical clustering algorithm based on modularity optimization divides nodes in the topological structure into communities, including: first initializing each node as a community, then traversing all possible community mergers, selecting the merger method with the largest modularity gain, and repeatedly traversing the communities until the modularity cannot be improved.

9. The production line layout optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: The modularity is the ratio of the total number of edges within a community to the total number of edges in the network minus an expected value, which is the ratio of the total number of edges within a community to the total number of edges in the network when the network is set as a random network with the same community distribution.

10. A production line layout optimization system based on multi-agent reinforcement learning, characterized in that: include: Topology construction module, which is used to construct the topology structure of the production line using a directed weighted network based on complex network theory; The community partitioning module is used to divide the nodes in the topology structure into communities based on the agglomerative hierarchical clustering algorithm optimized by modularity, and obtain multiple communities; each community includes multiple nodes, and each node is a device in the production line; The layout module is used to optimize the layout of equipment in each community based on MAPPO's multi-agent layout algorithm, complete the layout of all equipment in each community, and then manually layout all communities to obtain the optimized production line layout.

Images