Multi-agent deep reinforcement learning-based double-gantry chip mounter taking and pasting sequence optimization method

Abstract

The invention relates to a double-gantry chip mounter taking and mounting sequence optimization method based on multi-agent deep reinforcement learning, and belongs to the field of PCB mounting. The method comprises the following steps: defining a multi-agent deep reinforcement learning basic component under the optimization of a double-gantry chip mounter taking and pasting sequence; building an encoder network model; building a decoder network model embedded with an attention mechanism; building a commentator network model under an MAC-AC algorithm framework; introducing a mask mechanism to accelerate training; and performing neural network model training through an MAC-AC algorithm to obtain a pasting sequence optimization result. From the overall perspective, the problem of taking and pasting sequence optimization of the double-gantry chip mounter is solved end to end, and the problem solving quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of PCB mounting, and in particular relates to a method for optimizing the mounting sequence of a double gantry mounting machine based on multi-agent deep reinforcement learning. Background Art

[0002] With the increasingly widespread application of surface mount technology (SMT), the assembly of electronic products has made a qualitative leap compared to manual assembly, and the key equipment in the SMT assembly line, the SMT machine, has received widespread attention. In order to further improve the performance of the SMT machine, one of the key factors is to improve the efficiency of the SMT. For a single SMT machine, there are two key issues in improving the efficiency of the SMT: the position allocation problem of the SMT machine feeder and the optimization problem of the mounting sequence of components. However, for the mounting sequence of LED products, due to the limited types of LED chips to be mounted, but the large number of LED chips of the same type to be mounted, the traditional mounting settings no longer meet the requirements, and special automated equipment and processes are required to complete this process.

[0003] The LED mounting sequence problem requires that the equipment mounting speed is fast enough, with a minimum mounting speed of more than 18,000 points / h. In order to achieve accurate and efficient LED mounting, the LED chip is accurately placed in the target position. Therefore, the mounting sequence needs to be carefully planned and optimized to ensure high-speed operation without reducing the mounting quality. For LED mounting technology, optimizing the mounting sequence becomes particularly important. By shortening the mounting time, companies can meet customer needs faster, increase profits, and reduce idle time and equipment maintenance costs.

[0004] A good workload balance of a double gantry placement machine is the key to the optimization of the placement sequence. Some people solve it as an optimization problem. Some studies use workload balance as an objective measure, while others apply heuristic strategies and apply hill climbing heuristics to find a good solution for the placement sequence. Kulak et al. solved the placement sequence problem of the double gantry placement machine by improving GA, integrating a clustering algorithm called density search construction method (DSCM) into GA. Since there are many operation decision problems in the optimization of double gantry placement sequence, a hybrid genetic algorithm is usually proposed to solve this problem. Sund et al. developed a genetic algorithm for solving the component allocation problem, combined with a greedy heuristic algorithm for workload balancing and placement sequence. However, it has been noted that the performance of GA decreases significantly when the number of placement operations increases, because the efficiency of GA operators deteriorates in a wider search space.

[0005] In summary, using deep reinforcement learning to solve the problem of optimizing the pick-up and placement sequence of a dual-gantry SMT machine requires overcoming many difficulties: 1. The dual-gantry LED SMT machine has a much larger solution set space, and it is more difficult to find a reliable solution. 2. The optimization of the pick-up and placement sequence of a dual-gantry LED SMT machine is difficult to model as a single-agent reinforcement learning problem. 3. Using a multi-agent reinforcement learning algorithm to solve the problem of optimizing the pick-up and placement sequence of a dual-gantry LED SMT machine requires a reasonable design to balance the abstract game relationship between the agents. Summary of the invention

[0006] The purpose of the present invention is to overcome the problems existing in the prior art and provide a method for optimizing the pick-up and sticking sequence of a double gantry placement machine based on multi-agent deep reinforcement learning, which solves the pick-up and sticking sequence optimization problem of the double gantry placement machine end-to-end from an overall perspective and reduces operating costs.

[0007] To achieve the above object, the technical solution of the present invention is: a method for optimizing the pick-up order of a double gantry placement machine based on multi-agent deep reinforcement learning, comprising:

[0008] Step S1, defining the basic components of multi-agent deep reinforcement learning under the optimization of the pick-up sequence of the double gantry placement machine;

[0009] Step S2, building an encoder network model;

[0010] Step S3, building a decoder network model;

[0011] Step S4, building a critic network model under the MAC-AC algorithm framework;

[0012] Step S5, introducing a mask mechanism to accelerate training;

[0013] Step S6: determine the action selection mechanism;

[0014] Step S7: Perform neural network model training using the MAC-AC algorithm to obtain posting order optimization results.

[0015] In one embodiment of the present invention, the pick-up sequence model of the double gantry placement machine is described as follows:

[0016] Premise: The time for each material picking, placement and visual inspection by the placement head is a constant value; the acceleration and deceleration time of the placement head movement is ignored, and the movement is assumed to be uniform.

[0017] The optimization goal established is to minimize the total path of the placement head during the placement process (including the two processes of picking up materials and placing them), and the optimization objective function is:

[0018]

[0019] Where T represents the time step of one round of placement action performed by the placement machine, i is the number of the placement head, It represents the distance required for the Tth round of component picking performed by the i-th placement head. It represents the distance required for the Tth round of component placement performed by the i-th placement head; Compare the travel distances of the two gantries to determine the longest distance.

[0020] (1) Distance required for material collection:

[0021]

[0022] in It is represented by the coordinates of the camera corresponding to the placement head i station of the placement machine, The coordinates of the feeding group are the average values ​​of the set of corresponding feeding trough coordinates.

[0023] (2) Update the real-time position coordinates of patch head i:

[0024]

[0025] It represents the real-time position of the current patch head i. After the patch head i visits any placement point k, its position coordinate Li is updated: Where X k The coordinates corresponding to the visited mounting point.

[0026] (3) Distance required for mounting process:

[0027]

[0028] Among them, t represents the time step of the placement machine to perform material picking or placement, n represents the maximum time step in a round of placement cycle T, and this formula represents the total distance that the placement head i starts from feeding group i, passes through placement points 1 to n in sequence, and finally returns to the camera to wait. According to the process flow of the double gantry placement machine, the following constraints need to be introduced:

[0029] st

[0030] 3≤n≤H+2I

[0031]

[0032] Among them, I represents the number of placement heads involved in the work, and the double gantry here should be 2; H represents the total number of nozzles contained in all placement heads. The above formula contains the total component constraints. Once the placement head participates in the placement cycle, it must at least perform three steps: picking up materials, placing, and returning to the camera. At most, it can complete the placement of the number of nozzles of the placement head and return to the camera.

[0033] In one embodiment of the present invention, in step S1, the basic components of multi-agent deep reinforcement learning are defined, including:

[0034] (1) Define the patch heads on each gantry as independent intelligent entities;

[0035] (2) The behavior of each patch head in deciding the next target node is defined as an action;

[0036] (3) The state consists of two parts: the environment state and the agent state.

[0037] 1) Environmental status includes:

[0038] a. All feeding group coordinates include and For the convenience of training, it is directly defined as: X0, X1, where the coordinates of the feeding group are the average values ​​of the set of coordinates corresponding to the feeding trough and the camera;

[0039] b. The position coordinates X of all points to be mounted on the PCB k ;

[0040] 2) Agent states include:

[0041] The current coordinates of each placement head and the number of remaining unplaced components currently contained;

[0042] (4) Define the state transfer function:

[0043] After the patch head i visits any placement point k, its position coordinates L i renew: Where X k is the static coordinate corresponding to the visited mounting point; d k The number of placement points required and the number of components remaining on the placement head i l i renew: Among them, 1 here represents the component demand of the placement point, and l represents the number of components currently remaining in the placement head; after the placement head accesses the corresponding feeding group node, it re-absorbs the full components: Where C represents the number of nozzles of the patch head;

[0044] (5) Define the reward function:

[0045] For the placement sequence problem of the double-gantry multi-head placement machine, considering that if the two gantries move at the same time, only the distance of the gantry with the longest moving distance is calculated, the particularity of the type of components to be placed and the type of nozzle in the placement sequence problem of the placement machine shows that the time from the camera to the feeding group should be a constant, that is, the material picking time should be a constant. It can be known that for the distance cost of the total process, the distance cost generated by the material picking process can be directly ignored. Therefore, the total distance moved by all placement heads within a time step minus all material picking distances is used as the reward function. Considering the definition of the feeding group coordinates, the final reward function R(t) is the target optimization function:

[0046] In one embodiment of the present invention, in step S2, the encoder network model is specifically:

[0047] The encoder network consists of a single-layer linear network and a gated recurrent neural network, namely a GRU network. The encoder network receives the local observation of patch head i at time t. The initial embedding is calculated by linear projection; the initial embedding is combined with the GRU network hidden layer state h t , calculate the feature vector e of each agent's local observation through the GRU network i .

[0048] In one embodiment of the present invention, in step S3, the decoder network model embeds an attention mechanism to avoid unnecessary communication interfering with the agent's decision-making, thereby helping the agents to autonomously learn the real-time communication needs between each other.

[0049] In one embodiment of the present invention, in step S3, the decoder network model is specifically:

[0050] The decoder network consists of an attention mechanism module: the attention mechanism aggregates the observation information e of all patch heads i i Combined with the hard attention weight W h , for each patch header i, calculate the correlation weight δ between it and other patch headers i , δ i The weighted sum of the values ​​of other patch headers yields: i =∑αV i , where the value V i Through the patch head i corresponding to the feature vector e i After the matrix W V The linear transformation results in that the attention weight α compares the feature vector e using the query key system i With e j , W q E i Convert to query, W k E jConvert to a key and input the matching value into the SoftMax function; to prevent the gradient from disappearing, according to W q With W k The matching value is scaled by the dimension of , and combined with the hard attention weight W h , and get the attention weight α:

[0051] Finally, merge the feature vectors e corresponding to each patch header i i With the correlation weight δ i , calculate the action value function Q of each patch head i i :Q i =f(g(e i ,δ i )), where f is a multi-layer linear network and g is a single linear layer.

[0052] In one embodiment of the present invention, in step S4, the MAC-AC algorithm is a multi-agent actor-critic algorithm under a fully cooperative relationship setting.

[0053] In one embodiment of the present invention, in step S4, a critic network model under the MAC-AC algorithm framework is constructed, specifically as follows:

[0054] The critic network model consists of two parts: the evaluation network and the target critic network. They are multi-layer linear networks with the same network dimension but different network parameters. The evaluation network receives the environment state S at time t. t Estimated state value The target critic network receives the environment state at time t+1 and estimates the value of the environment state at time t+1.

[0055] In one embodiment of the present invention, in step S5, the mask mechanism sets the logarithmic probability of all nodes that should not be visited by the tile heads to -∞ during the training process to mask infeasible solutions and force a solution when specific conditions are met.

[0056] In one embodiment of the present invention, the mask mechanism is specifically as follows:

[0057] 1) The placement head is not allowed to move to a placement point where placement has been completed; 2) When the number of remaining components on the placement head is 0, it is forced to return to the corresponding feeding group to absorb components; 3) When the number of remaining components on the placement head is not 0 and there are points to be placed, it is forced to move to the points to be placed for placement; 4) When there are no points to be placed, it is forced to return to its corresponding feeding group and the process ends.

[0058] In one embodiment of the present invention, in step S6, an action selection mechanism is determined:

[0059] The Greedy method is used as the action selection strategy. The intelligent agent selects the action value function and selects the action corresponding to the maximum Q value.

[0060] In one embodiment of the present invention, in step S7, the neural network model is trained by the MAC-AC algorithm to obtain the posting order optimization result:

[0061] Using the parameters θ, W and W - Parameterize the encoder and decoder, evaluate all trainable variables in the network and the target critic network respectively; Advantage function, composed of The calculation results show that the joint action value function Q tot Depend on Approximately, where γ is the reward discount rate, and The evaluation network and the target critic network are approximated respectively, and the parameter θ i The update method is as follows: The evaluation network parameter W is updated using the TD difference algorithm: In order to make the training more stable, the evaluation network parameters W are copied every T times to update the network parameters W of the target critic network - After the training reaches the set maximum number of times, the solution with the largest reward obtained during the training process is regarded as the solution to the problem.

[0062] The present invention also provides a double gantry placement machine pick-up and placement sequence optimization system based on multi-agent deep reinforcement learning, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0063] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0064] Compared with the prior art, the present invention has the following beneficial effects: Different from the traditional heuristic algorithm's solution approach of "grouping first and then planning", the method of the present invention uses high-level feature information, learns to cooperate with each other through communication, plans from the perspective of the problem as a whole, and improves the quality of the solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the process of the present invention.

[0066] Figure 2 This is a time sequence diagram of the present invention.

[0067] Figure 3 It is a schematic diagram of the operation of the encoder and decoder of the present invention.

[0068] Figure 4 Schematic diagram of the interaction between the environment and the agent in a single training of the present invention.

[0069] Figure 5 Schematic diagram of the network model training process of the present invention. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0071] The present invention provides a method for optimizing the pick-and-place sequence of a double-gantry placement machine based on multi-agent deep reinforcement learning, which specifically comprises the following steps:

[0072] Step S1, establishing a pick-and-place sequence model of a double gantry placement machine;

[0073] Step S2, defining the basic components of multi-agent deep reinforcement learning under the optimization of the pick-up sequence of the double gantry placement machine;

[0074] Step S3, building an encoder network model;

[0075] Step S4: Build a decoder network model. To avoid unnecessary communication interfering with the decision-making of the intelligent agent, an attention mechanism is embedded in the model to help the intelligent agents autonomously learn the real-time communication needs between each other.

[0076] Step S5, building a critic network model under the framework of MAC-AC algorithm (Multi-Agent Cooperative Actor-Critic, a multi-agent actor-critic algorithm under a fully cooperative relationship setting);

[0077] Step S6, introducing a mask mechanism to accelerate training;

[0078] Step S7, determining an action selection mechanism;

[0079] Step S8: Perform neural network model training using the MAC-AC algorithm to obtain posting order optimization results.

[0080] The following is a specific implementation process of the present invention. The method of the present invention specifically includes the following eight steps:

[0081] 1. Establishment of the pick-and-place sequence model of the double gantry placement machine:

[0082] Premise: The time for each material picking, placement and visual inspection by the placement head is a constant value; the acceleration and deceleration time of the placement head movement is ignored, and the movement is assumed to be uniform.

[0083] The optimization goal is to minimize the total path of the placement head during the placement process (including the two processes of picking up materials and placing them). The optimization objective function is as follows: Figure 1 As shown:

[0084]

[0085] Where T represents the time step of one round of placement action performed by the placement machine, i is the number of the placement head, It represents the distance required for the Tth round of component picking performed by the i-th placement head. It represents the distance required for the Tth round of component placement performed by the i-th placement head; Compare the travel distances of the two gantries to determine the longest distance, e.g. Figure 2 shown.

[0086] (1) Distance required for material collection:

[0087]

[0088] in It is represented by the coordinates of the camera corresponding to the placement head i station of the placement machine, The coordinates of the feeding group are the average values ​​of the set of corresponding feeding trough coordinates.

[0089] (2) Update the real-time position coordinates of patch head i:

[0090]

[0091] It represents the real-time position of the current patch head i. After the patch head i visits any placement point k, its position coordinate Li is updated: Where X k The coordinates corresponding to the visited mounting point.

[0092] (3) Distance required for mounting process:

[0093]

[0094] Among them, t represents the time step of the placement machine to perform material picking or placement, n represents the maximum time step in a round of placement cycle T, and this formula represents the total distance that the placement head i starts from feeding group i, passes through placement points 1 to n in sequence, and finally returns to the camera to wait. According to the process flow of the double gantry placement machine, the following constraints need to be introduced:

[0095] st

[0096] 3≤n≤H+2I

[0097]

[0098] Among them, I represents the number of patch heads involved in the work, and the double gantry here should be 2; H represents the total number of nozzles contained in all patch heads.

[0099] 2. Define the basic components of multi-agent deep reinforcement learning for the optimization of the pick-up order of the double gantry placement machine:

[0100] (1) Define the patch heads on each gantry as independent intelligent entities;

[0101] (2) The behavior of each patch head in deciding the next target node is defined as an action;

[0102] (3) The state consists of two parts: the environment state and the agent state.

[0103] 1) Environmental status includes:

[0104] a. All feeding group coordinates include and For the convenience of training, it is directly defined as: X0, X1, where the coordinates of the feeding group are the average values ​​of the set of coordinates corresponding to the feeding trough and the camera;

[0105] b. The position coordinates X of all points to be mounted on the PCB k .

[0106] 2) Agent states include:

[0107] The current coordinates of each placement head and the number of remaining unplaced components currently contained.

[0108] (4) Define the state transfer function:

[0109] After the patch head i visits any placement point k, its position coordinates L i renew: Where X k is the static coordinate corresponding to the visited mounting point; d k The number of placement points required and the number of components remaining on the placement head i l i renew: Among them, 1 here represents the component demand of the placement point, and l represents the number of components currently remaining in the placement head; after the placement head accesses the corresponding feeding group node, it re-absorbs the full components: Wherein, C represents the number of nozzles of the patch head.

[0110] (5) Define the reward function:

[0111] The total distance moved by all patch heads in one time step minus all material fetching distances is used as the reward function. Considering the definition of the feeding group coordinates, the final reward function is the target optimization function:

[0112] 3. Build the encoder network model:

[0113] The encoder network consists of a single-layer linear network and a GRU network (Gated Recurrent Neural Network). The GRU network enables the agent to memorize and record its recent visit trajectory. The encoder network receives the local observation of gantry i at time t at each time step. The initial embedding is calculated by linear projection, and then the initial embedding is combined with the GRU hidden layer state h t , the feature vector e of each agent's local observation is calculated through the GRU network i .

[0114] 4. Build the decoder network model:

[0115] The decoder is composed of an attention mechanism module: the attention mechanism aggregates the observation information e of all patch heads i i Combined with the hard attention weight W h , for each patch header i, calculate the correlation weight δ between it and other patch headers i , δ i The weighted sum of the values ​​of other patch headers yields: i =∑αV i , where the value V i Through the patch head i corresponding to the feature vector e i After the matrix W V The linear transformation results in that the attention weight α compares the feature vector e using the query key system i With e j , W q E i Convert to query, W k E j Convert to a key and input the matching value into the SoftMax function. To prevent the gradient from disappearing, according to W q With W k The matching value is scaled by the dimension of , and combined with the hard attention weight W h , and get the attention weight α:

[0116] Finally, merge the feature vectors e corresponding to each patch header i i With the correlation weight δ i , calculate the action value function Q of each patch head i i :Q i =f(g(e i , δ i )), where f is a multi-layer linear network and g is a single linear layer.

[0117] Figure 3 It is a schematic diagram of the operation of the encoder and decoder of the present invention.

[0118] 5. Build a critic network model under the MAC-AC algorithm framework:

[0119] The critic network model consists of two parts: the evaluation network and the target critic network. They are multi-layer linear networks with the same network dimension but different network parameters. The evaluation network receives the environment state S at time t. t Estimated state value The target critic network receives the environment state at time t+1 and estimates the value of the environment state at time t+1.

[0120] 6. Introduce mask mechanism to accelerate training:

[0121] During the training process, the mask sets the corresponding logarithmic probabilities of all nodes that the patch head should not visit to -∞ to cover infeasible solutions and force solutions when specific conditions are met. The working mechanism of the mask is as follows: 1) The patch head is not allowed to move to the placement point that has been completed; 2) When the number of remaining components of the patch head is 0, it is forced to return to the corresponding feeding group to absorb components; 3) When the number of remaining components of the patch head is not 0 and there are points to be placed, it is forced to move to the points to be placed for placement; 4) When there are no points to be placed, it is forced to return to its corresponding feeding group.

[0122] 7. Determine the action selection mechanism:

[0123] The Greedy method is used as the action selection strategy. The intelligent agent selects the action value function and selects the action corresponding to the maximum Q value.

[0124] 8. Use the MAC-AC algorithm to train the neural network model and obtain the posting order optimization results:

[0125] Using the parameters θ, W and W - Parameterize the encoder and decoder, evaluate all trainable variables in the network and the target critic network respectively; Advantage function, composed of The calculation results show that the joint action value function Q tot Depend on Approximately, where γ is the reward discount rate, and The evaluation network and the target critic network are approximated respectively, and the parameter θ i The update method is as follows: The evaluation network parameter W is updated using the TD difference algorithm: In order to make the training more stable, the evaluation network parameters W are copied every T times to update the network parameters W of the target critic network - After the training reaches the set maximum number of times, the solution with the largest reward obtained during the training process is regarded as the solution to the problem.

[0126] 8. Description of the interaction between the environment and the agent during training:

[0127] like Figure 4 As shown in:

[0128] At time step t, all patch heads obtain their own local observations O from the environment i And communicate with each other. Under the constraint of the mask matrix, the patch head decides the action at the next moment through the policy network: select the next target node. The environment updates the environment information, each patch head information and the mask matrix according to the actions taken by each patch head. This process is repeated until all the mounting points are mounted. A cycle ends and a public global reward is given to improve the critic network. The policy network improves the distribution strategy through the feedback evaluation of the critic network, thus completing a single training.

[0129] Figure 5 Schematic diagram of the network model training process of the present invention.

[0130] Preferred parameter settings and optimizer selection of the present invention:

[0131] The RMS optimizer is used with learning rates α and β of 5×10 -4 With 5×10 -4 The encoder, decoder, and critic network parameters are optimized. The hidden dimension of the GRU in the encoder network is 128, and the linear hidden dimension of the critic network is 256. Different parameter settings are used for small-scale instances with less than or equal to 50 nodes, medium-scale instances with 50-100 nodes, and large-scale instances with more than 100 nodes. For small-scale instances, 1×10 4 For medium-sized instances, 4×10 4 For large-scale examples, 6×10 4 Round training. Target critic network parameters W - , the training cycles are 25, 50 and 50 respectively for small-scale instances, medium-scale instances and large-scale instances.

[0132] The present invention also provides a double gantry placement machine pick-up and placement sequence optimization system based on multi-agent deep reinforcement learning, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0133] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for optimizing the pick-and-place sequence of a double-gantry placement machine based on multi-agent deep reinforcement learning, characterized in that: include: Step S1, defining the basic components of multi-agent deep reinforcement learning under the optimization of the pick-up sequence of the double gantry placement machine; Step S2, building an encoder network model; Step S3, building a decoder network model; Step S4, building a critic network model under the MAC-AC algorithm framework; Step S5, introducing a mask mechanism to accelerate training; Step S6: determine the action selection mechanism; Step S7: Perform neural network model training using the MAC-AC algorithm to obtain posting order optimization results.

2. According to the method for optimizing the pick-up and placement sequence of a double gantry placement machine based on multi-agent deep reinforcement learning according to claim 1, it is characterized in that: In step S1, the basic components of multi-agent deep reinforcement learning are defined, including: (1) Define the patch heads on each gantry as independent intelligent entities; (2) The behavior of each patch head in deciding the next target node is defined as an action; (3) The state consists of two parts: the environment state and the agent state. 1) Environmental status includes: a. Coordinates of all feeding groups; b. The position coordinates of all points to be mounted on the PCB; 2) Agent states include: The current coordinates of each placement head and the number of remaining unplaced components currently contained; (4) Define the state transfer function: After the patch head i visits any placement point k, its position coordinates L i renew: Where X k is the static coordinate corresponding to the visited mounting point; d k The number of placement points required and the number of components remaining on the placement head i l i renew: Among them, 1 here represents the component demand of the placement point, and l represents the number of components currently remaining in the placement head; after the placement head accesses the corresponding feeding group node, it re-absorbs the full components: Where C represents the number of nozzles of the patch head; (5) Define the reward function.

3. According to claim 2, a method for optimizing the pick-up and placement sequence of a double gantry placement machine based on multi-agent deep reinforcement learning is characterized in that: In step S2, the encoder network model is specifically: The encoder network consists of a single-layer linear network and a gated recurrent neural network, namely a GRU network. The encoder network receives the local observation of patch head i at time t. The initial embedding is calculated by linear projection; the initial embedding is combined with the GRU network hidden layer state h t , calculate the feature vector e of each agent's local observation through the GRU network i .

4. According to the method for optimizing the pick-up and placement sequence of a double gantry placement machine based on multi-agent deep reinforcement learning in claim 1, it is characterized in that: In step S3, the decoder network model embeds an attention mechanism to avoid unnecessary communication interfering with the agent's decision-making, thereby helping the agents to autonomously learn the real-time communication needs between each other.

5. The method for optimizing the pick-and-place sequence of a double gantry placement machine based on multi-agent deep reinforcement learning according to claim 3, characterized in that: In step S3, the decoder network model is specifically: The decoder network consists of an attention mechanism module: the attention mechanism aggregates the observation information e of all patch heads i i Combined with the hard attention weight W h , for each patch header i, calculate the correlation weight δ between it and other patch headers i , δ i The weighted sum of the values ​​of other patch headers yields: δ i =∑αV i , where the value V i Through the patch head i corresponding to the feature vector e i After the matrix W V The linear transformation results in that the attention weight α compares the feature vector e using the query key system i With e j , W q E i Convert to query, W k E j Convert to a key and input the matching value into the SoftMax function; to prevent the gradient from disappearing, according to W q With W k The matching value is scaled by the dimension of , and combined with the hard attention weight W h , and get the attention weight α: Finally, merge the feature vectors e corresponding to each patch header i i With the correlation weight δ i , calculate the action value function Q of each patch head i i :Q i =f(g(e i ,δ i )), where f is a multi-layer linear network and g is a single linear layer.

6. The method for optimizing the pick-and-place sequence of a double gantry placement machine based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In step S4, the MAC-AC algorithm is a multi-agent actor-critic algorithm under a fully cooperative relationship setting.

7. The method for optimizing the pick-and-place sequence of a double gantry placement machine based on multi-agent deep reinforcement learning according to claim 5, characterized in that: In step S5, the mask mechanism sets the logarithmic probability of all nodes that should not be visited by the tile heads to -∞ during the training process to mask infeasible solutions and force a solution when specific conditions are met.

8. The method for optimizing the pick-and-place sequence of a double gantry placement machine based on multi-agent deep reinforcement learning according to claim 7, characterized in that: The masking mechanism is as follows: 1) The placement head is not allowed to move to a placement point where placement has been completed; 2) When the number of remaining components on the placement head is 0, it is forced to return to the corresponding feeding group to absorb components; 3) When the number of remaining components on the placement head is not 0 and there are points to be placed, it is forced to move to the points to be placed for placement; 4) When there are no points to be placed, it is forced to return to its corresponding feeding group and the process ends.

9. A dual-gantry placement machine pick-and-place sequence optimization system based on multi-agent deep reinforcement learning, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium storing computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images