Catheter assembly sequence optimization method based on multi-agent reinforcement learning

Abstract

This invention relates to a method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning, belonging to the field of quality and reliability engineering. It includes the following steps: S1, introducing a multi-agent reinforcement learning framework based on conduit assembly, defining agents and their state spaces; S2, defining the action space of the agents within the state space; S3, defining the reward function corresponding to the agent taking a specified action in each state, and a discount factor reflecting the degree of influence of future rewards on the current decision; S4, defining the constraints on each agent when making action decisions; S5, proposing a deep multi-agent hierarchical Q-network algorithm to optimize the conduit assembly sequence; S6, validating the conduit assembly position optimization method. This invention proposes a hierarchical deep multi-agent Q-network algorithm, which optimizes the assembly sequence, improves the consistency and stability of product manufacturing process capabilities and quality, and provides a basis for quantitative technical support and decision-making.

Description

Technical Field

[0001] This invention belongs to the field of quality and reliability engineering, and specifically relates to a method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning. Background Technology

[0002] Metal conduits, as key aircraft components, are widely used in critical parts of aircraft hydraulic, environmental control, and fuel systems, playing a vital role in transmitting energy and power. When conduits experience significant assembly stress, it can lead to leaks of air or oil, causing malfunctions such as depressurization, energy leakage, and even fires in various aircraft systems, severely impacting aircraft performance and safety. During conduit assembly, the magnitude of assembly stress is influenced by multiple factors, including the conduit material, the preload during installation, and the conduit's design parameters. In addition to these factors, the assembly sequence also affects assembly stress. Due to assembly errors, an improper assembly sequence can lead to a significant accumulation of errors. If some conduits are installed too early, it may affect the positioning and alignment of subsequent conduits, causing a gradual increase in system deviations, resulting in higher assembly stress and reduced sealing performance of the entire system. Therefore, a proper assembly sequence is crucial during conduit assembly.

[0003] In the process of duct assembly, if we consider the position of the duct's end at a certain moment as the state of the entire system at that moment, then the entire duct assembly process can be viewed as a dynamic transition process. Each time a new duct is assembled, its end position changes. At each stage, we need to decide which duct to assemble and the specific assembly angle; the entire process is a typical multi-stage sequential decision-making process. Since the problem is based on three-dimensional space, the solution space is enormous, and it contains both discrete and continuous variables. Therefore, ordinary dynamic programming and Monte Carlo tree search algorithms are difficult to solve and are not suitable for this method. Dynamic programming requires a precise understanding of the entire environment and some prior knowledge, but in practical applications, reasonable prior knowledge is often difficult to obtain. Monte Carlo trees are mostly used when the solution space consists of discrete variables, which is clearly not suitable for this problem.

[0004] Therefore, there is an urgent need to provide a new method to solve the problem of optimizing the catheter assembly sequence. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a catheter assembly sequence optimization method based on multi-agent reinforcement learning. This method addresses the problem of increased assembly stress and decreased sealing performance in pipe fittings due to error accumulation caused by improper assembly sequence during catheter assembly. A multi-agent reinforcement learning-based catheter assembly sequence optimization algorithm is constructed to optimize the catheter assembly sequence, thereby solving the aforementioned problems. The method first establishes an optimization model for the catheter assembly sequence. Then, considering the characteristic that its decision space contains both discrete and continuous variables, a hierarchical multi-agent reinforcement learning algorithm is designed to solve the problem, calculating the catheter assembly sequence with the minimum error accumulation under given constraints, as well as the specific assembly position coordinates of each catheter.

[0006] The method of this invention introduces a multi-agent reinforcement learning (MARL) framework, designing three agents to make decisions on discrete and continuous variables respectively. Then, a reinforcement learning model for optimizing the assembly position of the catheter is constructed. Subsequently, a hierarchical deep multi-agent Q-network algorithm is proposed and numerical simulations are performed to calculate the optimal assembly sequence and assembly position, thereby improving the consistency and stability of product manufacturing process capabilities and product quality, and providing a basis for data-driven quantitative technology support and decision-making.

[0007] To achieve the above objectives, the present invention discloses the following technical solution:

[0008] Specifically, this invention provides a method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning, the specific steps of which are as follows:

[0009] S1. Introduce a multi-agent reinforcement learning framework and define the agents and their state spaces;

[0010] S2. Based on the defined agent and its state space, define the agent's action space within the state space;

[0011] S3. Based on the defined agent, state space, and action space, define the reward function corresponding to the agent taking a specified action in each state, and the discount factor reflecting the degree of influence of future rewards on the current decision.

[0012] S4. Define the constraints for each agent when making action decisions. The constraints include assembly angle deviation constraints, assembly end position constraints, and discrete action space constraints, forming a reinforcement learning model for optimizing the assembly position of the conduit.

[0013] S5. Based on a reinforcement learning model for optimizing catheter assembly position, a deep multi-agent hierarchical Q-network algorithm is proposed.

[0014] S6. Numerical simulation analysis was conducted based on the deep multi-agent hierarchical Q-network algorithm to verify the catheter assembly position optimization method.

[0015] Preferably, step S1 specifically includes the following sub-steps:

[0016] S11. Define the overall intelligent agent and the sub-intelligent agents respectively, where the overall intelligent agent is Agent. tot The agent is responsible for selecting the sequence number of the assembly catheter and the starting position for assembly; sub-agents Agent1 and Agent2 are each responsible for assembling the catheter from one end.

[0017] S12. For each agent, define its state space in stage t. The state space of a sub-agent includes: the position coordinates of the end of the conduit after assembly in stage t when assembly starts from a certain end, and the length of the conduit selected in the current stage. The state space of the total agent includes: the position coordinates of the end of the conduit after assembly in stage t for each sub-agent, and the angle between the straight line of the conduit and the positive directions of the x, y, and z axes after assembly in stage t for each sub-agent.

[0018] The state space s1(t) of the sub-agent Agent1 is:

[0019]

[0020] The state space s2(t) of the sub-agent Agent2 is:

[0021]

[0022] in, This indicates the position coordinates of the guide tube end after assembly is completed in stage t, starting from the first endpoint. This indicates the length of the catheter selected at the current stage; This indicates the position coordinates of the guide tube end after assembly is completed in stage t, starting from the second endpoint.

[0023] Total Intelligent Agent tot The state space is:

[0024]

[0025] in,

[0026]

[0027] in, These represent the actions of the two sub-agents in the previous stage; α, β, and Let α and β represent the angles between the line containing the conduit and the positive directions of the x, y, and z axes, respectively; in the initial stage, α at both ends is 0, and β and β are... Both are π / 2; in subsequent stages, the decision updates are performed by two sub-agents, Agent1 and Agent2.

[0028] Preferably, step S2 specifically includes the following sub-steps:

[0029] S21. For each agent, define its action in stage t; where the action information of the overall agent includes: the length of the i-th duct to be assembled in stage t, and information for deciding whether the selected duct should be assembled from the left or the right; the action information of the sub-agents includes the angle between the straight lines of the two end ducts and the positive directions of the x, y, and z axes.

[0030] S22. Update the agent's state in stage t+1 based on the agent's state and actions in stage t.

[0031] Preferably, in step S2, the total intelligent agent (Agent) tot action a tot (t) is:

[0032]

[0033] in, This indicates that the i-th conduit is selected for assembly in stage t. Let I(t) be the length of the i-th catheter, and I(t) be the value used to decide whether the catheter should be assembled starting from the first or second end.

[0034]

[0035] Execute a in the current stage tot After (t), s tot (t) transforms to s tot (t+1):

[0036]

[0037] Where, if I(t) = 1, then only update and If I(t) = 0, then only update. and Remain unchanged;

[0038] and All updates are performed by the policy network of the sub-agents;

[0039] The actions a1(t) and a2(t) of the two sub-agents are defined as follows:

[0040]

[0041] a1(t) and a2(t) represent the angles between the two end guide tubes and the positive x, y, and z axes, respectively; in the initial stage,

[0042]

[0043]

[0044] At the same time, a1(t) and a2(t) satisfy:

[0045]

[0046] After executing a1(t) in the current stage, s1(t) is transformed into s1(t+1):

[0047]

[0048] After executing a2(t) in the current stage, s2(t) is transformed into s2(t+1):

[0049]

[0050] in, The length of the conduit selected for the overall agent.

[0051] Preferably, step S3 specifically includes the following sub-steps:

[0052] S31. When assembling from both ends, establish an optimization function that minimizes the distance between the end points of the two middle conduits:

[0053]

[0054] Where d is the distance between the two end points of the middle two conduits;

[0055] And the maximum angular deviation Δ of all conduits during assembly. A Minimal optimization function:

[0056]

[0057] S32. Construct the global reward function for the overall intelligent agent and the local reward functions for the sub-intelligent agents:

[0058] The global reward function is:

[0059] r tot (t) = r(t) - r(t+1);

[0060] The local reward function is:

[0061]

[0062] in, and These are the local reward functions for the two sub-agents, x. m y m , z m Let ω be the coordinates of the midpoint of the line connecting the two fixed endpoints, and ω be the weighting coefficient for positional and angular deviations.

[0063] S33. Based on the degree of influence of future rewards on current decisions, define a discount factor and combine it with the reward function to construct the agent's total reward in stage t.

[0064] The discount factor γ is a number between 0 and 1, and the agent's total reward G in stage t is... t Defined as:

[0065] G(r tot (t))=r tot (t+1)+γr tot (t+2)+γr tot (t+3)+...

[0066]

[0067] Among them, G(r) tot (t)), and These represent the total rewards for the main agent and the two sub-agents, respectively.

[0068] Preferably, step S4 specifically includes the following sub-steps:

[0069] S41. Define the upper limit of the angle deviation between adjacent guide tubes and establish the expression of the included angle of the straight lines of the two guide tubes to form an assembly angle deviation constraint.

[0070] The angle θ between the lines containing the two ducts and the angles α and β between each duct and the positive directions of the x, y, and z axes, respectively. It has the following relationship:

[0071]

[0072] The assembly angle deviation constraints for Agent1 and Agent2 are:

[0073]

[0074] S42. Define the upper limits of radial and length deviations between the end guides assembled at both ends, and establish expressions for the distance from the end point of one guide to the axis of the other guide and the length deviation, thus forming assembly end position constraints.

[0075] S43. Analyze the discrete action space constraints and other constraints, and establish the corresponding expressions;

[0076] The discrete action space constraints are:

[0077]

[0078] Other constraints are:

[0079]

[0080] That is, in the initial stage t=0, α at both ends is 0, and β and Both are π / 2;

[0081] S44. Based on the defined reward function, discount factor, action space, and state space, a reinforcement learning model for optimizing catheter assembly position is formed.

[0082] Preferably, in step S42, the terminal catheters assembled at both ends are constrained by radial and length deviations. The radial deviation is the distance d from the end point of one catheter to the axis of the other catheter. r1 and d r2 These are the radial deviations of the two conduits, and the radial deviation constraints are as follows:

[0083]

[0084] in, and These represent the lengths of the last catheter assembled from both ends, and τ is the radial deviation per unit length specified in the relevant catheter standards and specifications.

[0085] d l1 and d l2 d represents the length deviation of the two catheters respectively. l1 d l2 and d r1 d r2 The following relationship exists between them:

[0086]

[0087] Preferably, step S5 specifically includes the following sub-steps:

[0088] S51. Design a distributed strategy with a two-layer network structure. In each round, the decision of the main agent affects the decision of the sub-agents, and the decision of the sub-agents is also fed back to the main agent to guide the decision of the main agent in the next round.

[0089] S52. For the upper-layer network structure, the DQN structure is used to select the discrete actions of the overall agent and perform centralized training; for the lower-layer network structure, the corresponding continuous action parameters of the sub-agents are updated according to the discrete actions selected by the overall agent, forming a deep multi-agent hierarchical Q-network algorithm.

[0090] Preferably, in step S51, in Q tot Noise was introduced into the network parameters, and the final TD target and TD error are as follows:

[0091]

[0092] Where, ω tot It is the upper-level value network The parameter, ω tot- ξ and ξ' are the target network parameters, and ξ and ξ' are random noise generated according to the standard normal distribution. And I is the Agent tot Discrete actions, s tot and s tot ′ respectively correspond to Agent tot The current state and the state in the next stage. After training is complete, maximize the upper-layer Q. tot Network output is used to update the upper-layer agent. tot Discrete action strategy:

[0093]

[0094] In step S52, an improved deep deterministic policy gradient algorithm is used to update the corresponding continuous action parameters based on the Actor-Critical framework.

[0095] Preferably, step S6 specifically includes the following sub-steps:

[0096] S61. Use random numbers to generate sample data for the simulation experiment, simulate the catheter length parameter, and set the hyperparameters in the simulation model.

[0097] S62. Conduct simulation experiments based on sample data.

[0098] Compared with the prior art, the present invention has the following beneficial effects:

[0099] (1) The duct assembly sequence optimization method based on multi-agent reinforcement learning provided by this invention can adapt to complex environments. By introducing a multi-agent reinforcement learning system, it can handle dynamically changing complex assembly environments. In the whole system, the total agent and sub-agents can adapt well to complex and dynamic assembly processes through their independent policy learning and the fusion of global policies, thereby optimizing duct assembly and meeting the requirements of duct assembly work.

[0100] (2) The method of this invention can handle non-convex and complex objective functions, and multi-agent reinforcement learning can handle non-convex optimization problems in catheter assembly. Since catheter assembly sequence problems usually have multiple local optima, traditional optimization methods may get stuck in local minima, while the multi-agent reinforcement learning algorithm MARL can explore a wider search space and avoid getting stuck in local optima. Through iterative optimization and self-adjustment of the policy, MARL is able to approach the global optimum.

[0101] (3) The method of this invention can cope with complex constraints and physical limitations. The multi-agent reinforcement learning algorithm MARL can handle various constraints in duct assembly, such as pipe length, assembly angle, collision detection, etc., through a flexible reward mechanism. Through a carefully designed reward function, sub-agents can learn how to optimize their actions and complete the assembly task under these complex constraints.

[0102] (4) The method of this invention can be continuously improved through the experience pool mechanism, and the multi-agent reinforcement learning algorithm (MARL) can learn and improve from historical experience. During the execution of a task, each agent stores its actions, states, and rewards in the experience pool and continuously optimizes its behavior by updating the policy network. In this way, the system can gradually optimize the agent's action selection and assembly sequence for the conduit, ensuring that the next action is the optimal conduit assembly sequence, thereby improving the overall assembly efficiency and accuracy. Attached Figure Description

[0103] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0104] Figure 2 This is a schematic diagram of the catheter assembly of the present invention;

[0105] Figure 3 This is a schematic diagram of the lower-level network algorithm structure of the present invention;

[0106] Figure 4 This is a schematic diagram of the Deep MAHQN algorithm structure of the present invention;

[0107] Figure 5 This is a schematic diagram of the training iteration curve of the present invention. Detailed Implementation

[0108] Exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0109] In this embodiment, as Figure 2As shown, there are n conduits, where L is the length vector, starting from two fixed endpoints. Begin assembly. Each time assembly begins, select one conduit, then decide whether to begin assembling from the left or right end, and so on, until all conduits are assembled. The goal is to minimize the distance between the end of the conduit assembled from the first end (left end) and the end of the conduit assembled from the second end (right end) after all conduits have been assembled. Example follows. Figure 2 As shown, the arrows indicate the assembly direction of the catheter. In this embodiment, the left end point is the first end, and the right end point is the second end.

[0110]

[0111] The optimization goal of catheter sequencing is to select the optimal catheter assembly sequence and assembly angle, so that... Minimum.

[0112] Specifically, embodiments of the present invention provide a method for optimizing catheter assembly sequence based on multi-agent reinforcement learning, such as... Figure 1 As shown, the specific steps of this method are as follows:

[0113] S1. Based on conduit assembly, a multi-agent reinforcement learning framework is introduced, defining the agents and their state spaces.

[0114] Step S1 specifically includes the following sub-steps:

[0115] S11. Define the overall intelligent agent and the sub-intelligent agents respectively, where the overall intelligent agent is Agent. tot The agent is responsible for selecting the sequence number of the assembly conduit, i.e., choosing which conduit to assemble, and determining the starting point for assembly, i.e., which end to begin. Sub-agents Agent1 and Agent2 are each responsible for assembling the conduit from one end. Sub-agent Agent1 is responsible for assembling the conduit from the left end, and sub-agent Agent2 is responsible for assembling the conduit from the right end.

[0116] S12. For each agent, define its state space in stage t. The state space of a sub-agent includes: the position coordinates of the end of the conduit after assembly in stage t when assembly starts from a certain end, and the length of the conduit selected in the current stage. The state space of the total agent includes: the position coordinates of the end of the conduit after assembly in stage t for each sub-agent, and the angle between the straight line of the conduit and the positive directions of the x, y, and z axes after assembly in stage t for each sub-agent.

[0117] The state space s1(t) of the sub-agent Agent1 is as follows:

[0118]

[0119] The state space s2(t) of the sub-agent Agent2 is:

[0120]

[0121] in, This indicates the position coordinates of the guide tube end after assembly is completed in stage t, starting from the left endpoint (i.e., the first end), during the initial stage. This indicates the length of the catheter selected at the current stage; This indicates the position coordinates of the guide tube end after assembly is completed in stage t, starting from the right endpoint (the second end), during the initial stage.

[0122] Total Intelligent Agent tot The state space is:

[0123]

[0124] in,

[0125]

[0126] in, These represent the actions of the two sub-agents in the previous stage; α, β, and These represent the angles between the straight line containing the conduit and the positive directions of the x, y, and z axes, respectively; superscript 1 represents sub-agent Agent1, and superscript 2 represents sub-agent Agent2; in the initial stage, α at both ends is 0, and β and Both are π / 2; in subsequent stages, the decision updates are performed by two sub-agents, Agent1 and Agent2.

[0127] S2. Based on the defined agent and its state space, define the agent's action space within the state space.

[0128] Step S2 specifically includes the following sub-steps:

[0129] S21. For each agent, define its action in stage t; wherein, the action information of the overall agent includes: the length of the i-th conduit to be assembled in stage t and information for deciding whether the selected conduit should be assembled from the first end or the second end; the action information of the sub-agents includes the angle between the straight line containing the two conduits and the positive directions of the x, y, and z axes.

[0130] S22. Update the agent's state in stage t+1 based on the agent's state and actions in stage t.

[0131] Total Intelligent Agent tot action a tot (t) is:

[0132]

[0133] in, This indicates that the i-th conduit is selected for assembly in stage t. Let I(t) be the length of the i-th catheter, and I(t) be the value used to decide whether the catheter should be assembled starting from the first or second end.

[0134]

[0135] Execute a in the current stage tot After (t), s tot (t) transforms to s tot (t+1):

[0136]

[0137] Where, if I(t) = 1, then only update and If I(t) = 0, then only update. and It remains unchanged.

[0138] and All updates are performed by the policy network of the sub-agents.

[0139] The actions a1(t) and a2(t) of the two sub-agents are defined as follows:

[0140]

[0141] a1(t) and a2(t) represent the angles between the two end guide tubes and the positive x, y, and z axes, respectively; in the initial stage,

[0142]

[0143]

[0144] At the same time, a1(t) and a2(t) satisfy:

[0145]

[0146] After executing a1(t) in the current stage, s1(t) is transformed into s1(t+1):

[0147]

[0148] After executing a2(t) in the current stage, s2(t) is transformed into s2(t+1):

[0149]

[0150] in, The length of the conduit selected for the overall agent.

[0151] S3. Based on the defined agent, state space, and action space, define the reward function corresponding to the agent taking a specified action in each state, as well as the discount factor reflecting the degree of influence of future rewards on the current decision.

[0152] Step S3 specifically includes the following sub-steps:

[0153] S31. When assembling from both ends, establish an optimization function that minimizes the distance between the end points of the two middle conduits:

[0154]

[0155] Where d is the distance between the two end points of the middle two conduits. Let be the coordinates of the distal end of the left duct in stage t. The coordinates are the position of the distal end of the right duct in stage t.

[0156] And the maximum angular deviation Δ of all conduits during assembly. A Minimal optimization function:

[0157]

[0158] S32. Construct the global reward function for the overall intelligent agent and the local reward functions for the sub-intelligent agents:

[0159] The global reward function is:

[0160] r tot (t)=r(t)-r(t+1)

[0161]

[0162] The local reward function is:

[0163]

[0164] in, and These are the local reward functions for the two sub-agents, (x) m y m , z m ) represents the coordinates of the midpoint of the line connecting the two fixed endpoints, and ω is the weighting coefficient for positional deviation and angular deviation.

[0165] S33. Based on the degree of influence of future rewards on current decisions, define a discount factor and combine it with the reward function to construct the agent's total reward in stage t.

[0166] The discount factor γ is a number between 0 and 1, and the agent's total reward G in stage t is... t Defined as:

[0167] G(r tot (t))=r tot (t+1)+γr tot (t+2)+γr tot (t+3)+...

[0168]

[0169]

[0170] Among them, G(r) tot (t)), and These represent the total rewards for the main agent and the two sub-agents, respectively.

[0171] S4. Define the constraints for each agent when making action decisions. The constraints include assembly angle deviation constraints, assembly end position constraints, and discrete action space constraints, forming a reinforcement learning model for optimizing the assembly position of the conduit.

[0172] Step S4 specifically includes the following sub-steps:

[0173] S41. Define the upper limit of the angle deviation between adjacent guide tubes and establish the expression of the included angle between the two guide tubes to form an assembly angle deviation constraint.

[0174] The angle θ between the lines containing the two ducts and the angles α and β between each duct and the positive directions of the x, y, and z axes, respectively. It has the following relationship:

[0175]

[0176] The assembly angle deviation constraints for Agent1 and Agent2 are:

[0177]

[0178] S42. Define the upper limits of radial and length deviations between the end guides assembled at both ends, and establish expressions for the distance from the end point of one guide to the axis of the other guide and the length deviation, thus forming the position constraints of the assembly end.

[0179] In step S42, the end catheters assembled at both ends are subject to constraints of radial and length deviations. The radial deviation is the distance from the end point of one catheter to the axis of the other catheter. Figure 2 As shown, d r1 and dr2 These are the radial deviations of the two catheters. According to relevant standards, the radial deviation per unit length of catheter cannot exceed τ units of length. The radial deviation constraints are as follows:

[0180]

[0181] in, and These represent the lengths of the last conduit assembled from both ends.

[0182] d l1 and d l2 d represents the length deviation of the two catheters respectively. l1 d l2 and d r1 d r2 The following relationship exists between them:

[0183]

[0184]

[0185] S43. Analyze the discrete action space constraints and other constraints, and establish the corresponding expressions.

[0186] The discrete action space constraints are:

[0187]

[0188] Other constraints are:

[0189]

[0190] That is, in the initial stage t=0, α at both ends is 0, and β and Both are π / 2.

[0191] S44. Based on the defined reward function, discount factor, action space, and state space, a reinforcement learning model for optimizing catheter assembly position is formed.

[0192] S5. Based on a reinforcement learning model for optimizing catheter assembly position, a deep multi-agent hierarchical Q-network algorithm is proposed to optimize the catheter assembly sequence.

[0193] Step S5 specifically includes the following sub-steps:

[0194] S51. Design a distributed strategy with a two-layer network structure. In each round, the decision of the main agent affects the decision of the sub-agents, and the decisions of the sub-agents are also fed back to the main agent to guide the decision of the main agent in the next round.

[0195] In step S51, in Qtot Noise was introduced into the network parameters, and the final TD target and TD error are as follows:

[0196]

[0197] Where, ω tot It is the upper-level value network The parameter, ω tot- ξ and ξ' are the target network parameters, and ξ and ξ' are random noise generated according to the standard normal distribution. And I is the Agent tot Discrete actions, s tot and s tot 'Corresponding to Agent respectively tot The current state and the state in the next stage. After training is complete, maximize the upper-layer Q. tot Network output is used to update the upper-layer agent. tot Discrete action strategy:

[0198]

[0199] S52. For the upper-layer network structure, the DQN structure is used to select the discrete actions of the overall agent and perform centralized training; for the lower-layer network structure, the corresponding continuous action parameters of the sub-agents are updated according to the discrete actions selected by the overall agent, forming a deep multi-agent hierarchical Q-network algorithm.

[0200] In step S52, an improved deep deterministic policy gradient algorithm, hereinafter referred to as the TD3 algorithm, is used to update the corresponding continuous action parameters based on the Actor-Critic framework. After the discrete action is selected, the sub-agent updates the parameters according to the selected discrete action k. i Generate new state observations s i Among them, when I * When I = 1, Agent 1 is selected for decision-making; when I = 1, Agent 1 is selected for decision-making * When the value is 0, Agent2 is selected for decision-making. The corresponding sub-agent will update its continuous action parameters based on the lower-layer network. The training process of the TD3 algorithm is as follows:

[0201] For each sub-agent, truncated double-Q learning is used to train two value networks and one policy network:

[0202]

[0203] in, and It refers to the parameters of the individual value network of each agent in the lower-level network, φ. i These are the parameters of the continuous action policy network for each agent.

[0204] There are three other target networks that correspond one-to-one with it:

[0205]

[0206] and These are the parameters corresponding to the target network.

[0207] First, the parameters of the continuous actions are calculated using the target policy network:

[0208]

[0209] Where, ξ i It is random noise drawn from a truncated normal distribution. The truncated normal distribution is denoted as […]. This represents a normal distribution with a mean of 0 and a standard deviation of σ, but the probability that the variable falls outside the range [-c, c] is 0. This setting helps to avoid noise ξ. i Too large an amount can negatively impact training effectiveness.

[0210] Calculate the TD target y using two target value networks respectively. i :

[0211]

[0212] Update the lower-level value network:

[0213]

[0214] Every K rounds, the low-level policy network μ is updated using the policy gradient method. i And three target networks:

[0215]

[0216]

[0217] Since the Actor-Critic framework uses a value network to guide the policy network update, if the value network itself is unreliable, it will not guide the policy network update. Therefore, we need to reduce the frequency of updating the policy and target networks, making it slower than the value network update. We set the value network to be updated once per round, but the policy and target networks to be updated every K rounds. This ensures that when the value network evaluates the policy's quality, it is sufficiently stable and has been trained for a certain period. The algorithmic structure of the entire lower-layer network is as follows. Figure 3 As shown.

[0218] At the start of training, the lower-level network is trained independently for a period to stabilize its policy output before the upper and lower-level networks are trained together. During joint training, the output of the lower-level network serves as the input to the higher-level network, enabling the higher-level network to learn better discrete action selection policies. The overall algorithm structure is as follows: Figure 4 As shown in Table 1, the overall update process of the algorithm is as follows.

[0219] Table 1. Overall update process of the algorithm

[0220]

[0221]

[0222]

[0223] S6. Numerical simulation analysis was conducted based on the deep multi-agent hierarchical Q-network algorithm to verify the catheter assembly position optimization method.

[0224] Step S6 specifically includes the following sub-steps:

[0225] S61. Use random numbers to generate sample data for the simulation experiment, simulate the catheter length parameter, and set the hyperparameters in the simulation model.

[0226] S62. Conduct simulation experiments based on sample data.

[0227] In this embodiment, during the numerical simulation, 10 samples are randomly selected from the N~(5, 0.05) distribution as the length of the catheter to be assembled, and the right end point is taken. The training was performed using the Deep MAHQN algorithm. The hyperparameter settings used during training are shown in Table 2 below.

[0228] Table 2 Hyperparameter Settings

[0229]

[0230] The training iteration curve is as follows Figure 5 As shown in the figure, the algorithm converges well. To test the performance of the model, two sets of duct data were randomly generated in the same way for testing. The final calculation results are shown in Table 3 below.

[0231] Table 3 Simulation Results

[0232]

[0233] As can be seen from Table 3 above, the algorithm exhibits good performance on both test sets, with minimal error accumulation in the calculation results, while also meeting the standard requirements for radial and length deviations.

[0234] In summary, the duct assembly sequence optimization method based on multi-agent reinforcement learning provided by this invention can adapt to complex environments. By introducing a multi-agent reinforcement learning system, it can handle dynamically changing and complex assembly environments. In the entire system, the overall agent and sub-agents learn their own independent policies and fuse global policies, enabling them to adapt well to complex and dynamic assembly processes, thereby optimizing duct assembly and meeting the requirements of duct assembly work.

[0235] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning, characterized in that: The specific steps of this method are as follows: S1. Based on conduit assembly, a multi-agent reinforcement learning framework is introduced, defining the agents and their state spaces; Step S1 specifically includes the following sub-steps: S11. Define the overall intelligent agent and the sub-intelligent agents respectively, where the overall intelligent agent... Responsible for selecting the sequence number of the assembly conduit and the starting position for assembly; sub-agent Harmony agent Each person is responsible for assembling the tubing starting from one end; S12. For each agent, define its state space in stage t. The state space of a sub-agent includes: the position coordinates of the end of the conduit after assembly in stage t when assembly starts from a certain end, and the length of the conduit selected in the current stage. The state space of the total agent includes: the position coordinates of the end of the conduit after assembly in stage t for each sub-agent, and the angle between the straight line of the conduit and the positive directions of the x, y, and z axes after assembly in stage t for each sub-agent. Sub-agent state space for: ； Sub-agent state space for: ； in, This indicates that the assembly begins from the first endpoint. The position coordinates of the conduit end after the stage assembly is completed, in the initial stage. ; This indicates the length of the catheter selected at the current stage; This indicates that assembly begins from the second endpoint. The position coordinates of the conduit end after the stage assembly is completed, in the initial stage. ; Total intelligent agent state space for: ； in, ； ； in, These represent the actions of the two sub-agents in the previous stage; , and Representing the straight line containing the catheter and The included angle along the positive axis; the subscript 'last' represents the previous stage, and the superscript '1' represents the sub-agent. The superscript 2 represents a sub-agent. In the initial stage, both ends =0, and All In the subsequent stage, two sub-agents... , Update decisions; S2. Based on the defined agent and its state space, define the agent's action space within the state space; S3. Based on the defined agent, state space, and action space, define the reward function corresponding to the agent taking a specified action in each state, and the discount factor reflecting the degree of influence of future rewards on the current decision. S4. Define the constraints for each agent when making action decisions. The constraints include assembly angle deviation constraints, assembly end position constraints, and discrete action space constraints, forming a reinforcement learning model for optimizing the assembly position of the conduit. S5. Based on the reinforcement learning model for optimizing catheter assembly position, a deep multi-agent hierarchical Q-network algorithm is obtained to optimize the catheter assembly sequence. S6. Numerical simulation analysis was conducted based on the deep multi-agent hierarchical Q-network algorithm to verify the catheter assembly position optimization method.

2. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S2 specifically includes the following sub-steps: S21. For each agent, define its action in stage t; wherein, the action information of the overall agent includes the length of the i-th conduit to be assembled in stage t and the information used to decide whether the selected conduit should be assembled from the first end or the second end; the action information of the sub-agent includes the angle between the straight line containing the two conduits and the positive directions of the x, y, and z axes. S22. Update the agent's state in stage t+1 based on the agent's state and actions in stage t.

3. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 2, characterized in that: In step S2, the total intelligent agent action for: ； in, , indicating in Phase selection assembly Root vessels, Let be the length of the i-th duct. The conduit used for decision-making should be assembled starting from either the first or second end: ； Execution at the current stage back, Transform to : ； Among them, if Then only update ,and Remain unchanged; if Then only update ,and Remain unchanged; and All updates are performed by the policy network of the sub-agents; Define the actions of the two sub-agents respectively. , as follows: ； ； , These represent the two ends of the catheter and... The angle between the positive axes; in the initial stage, ； ； at the same time and satisfy: ； ； Execution at the current stage back, Transform to : ； Execution at the current stage back, Transform to : ； in, The length of the conduit selected for the overall agent.

4. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 3, characterized in that: Step S3 specifically includes the following sub-steps: S31. When assembling from both ends, establish an optimization function that minimizes the distance between the end points of the two middle conduits: ； in, This is the distance between the two end points of the middle two conduits; and the maximum angular deviation of all conduits during assembly. Minimal optimization function: ； S32. Construct the global reward function for the overall intelligent agent and the local reward functions for the sub-intelligent agents: Global reward function for: ； ； The local reward function is: ； ； in, Let be the reward functions for stage t and stage t+1, respectively. and These are the local reward functions for the two sub-agents. The coordinates of the midpoint of the line connecting the two fixed endpoints. These are the weighting coefficients for positional and angular deviations; These are the reward functions for the two sub-agents at stage t and stage t+1, respectively. S33. Based on the degree of influence of future rewards on current decisions, define a discount factor and combine it with the reward function to construct the agent's total reward in stage t. Discount factor The total reward of the agent in stage t is a number between 0 and 1. Defined as: ； ； ； in, , and These represent the total rewards for the main agent and the two sub-agents, respectively.

5. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 4, characterized in that: Step S4 specifically includes the following sub-steps: S41. Define the upper limit of the angle deviation between two adjacent guide tubes and establish the expression of the included angle of the straight lines containing the two guide tubes to form an assembly angle deviation constraint. The angle between the lines containing the two catheters Rather than each with Angle along the positive axis It has the following relationship: ； and The assembly angle deviation constraint is: ； S42. Define the upper limits of radial and length deviations between the end guides assembled at both ends, and establish expressions for the distance from the end point of one guide to the axis of the other guide and the length deviation, thus forming assembly end position constraints. S43. Analyze the discrete action space constraints and other constraints, and establish the corresponding expressions; The discrete action space constraints are: ； Other constraints are: ； That is, in the initial stage t=0, both ends =0, and All ; S44. Based on the defined reward function, discount factor, action space, and state space, a reinforcement learning model for optimizing catheter assembly position is formed by combining the neural network model.

6. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 5, characterized in that: In step S42, the end catheters assembled at both ends are subject to constraints of radial deviation and length deviation. The radial deviation is the distance from the end point of one catheter to the axis of the other catheter. and These represent the radial deviations of the two conduits, with the following radial deviation constraints: ； in, and These represent the lengths of the last conduit assembled from both ends. The radial deviation per unit length as specified in the catheter standard specification; and These represent the length deviations of the two catheters, and The following relationship exists between them: ； but .

7. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 6, characterized in that: Step S5 specifically includes the following sub-steps: S51. Design a distributed strategy with a two-layer network structure. In each round, the decision of the main agent affects the decision of the sub-agents, and the decision of the sub-agents is fed back to the main agent to guide the decision of the main agent in the next round. S52. For the upper-layer network structure, the DQN structure is used to select the discrete actions of the overall agent and perform centralized training; for the lower-layer network structure, the corresponding continuous action parameters of the sub-agents are updated according to the discrete actions selected by the overall agent, forming a deep multi-agent hierarchical Q-network algorithm.

8. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 7, characterized in that: In step S51, in the upper-layer network structure Noise is introduced into the network parameters, ultimately affecting the TD target. and TD error as follows: ； in, It is the upper-layer network structure Network parameters, These are the target network parameters. and It is random noise generated according to the standard normal distribution; and It is the overall intelligent agent discrete actions, and Corresponding to the total intelligent agent The current and next phase states; after training, maximizing the upper network structure... The network output is used to update the overall intelligent agent. Discrete action strategy: ； In step S52, the lower-layer network structure uses an improved deep deterministic policy gradient algorithm based on the Actor-Critic framework to update the corresponding continuous action parameters.

9. The method for optimizing the assembly sequence of conduits based on multi-agent reinforcement learning according to claim 8, characterized in that: Step S6 specifically includes the following sub-steps: S61. Use random numbers to generate sample data for the simulation experiment, simulate the catheter length parameter, and set the hyperparameters in the simulation model. S62. Conduct simulation experiments based on sample data.

Images