Multi-agent reinforcement learning-based conduit assembly sequence optimization method

Abstract

The invention relates to a conduit assembly sequence optimization method based on multi-agent reinforcement learning, and belongs to the field of quality and reliability engineering, and the method comprises the following steps: S1, introducing a multi-agent reinforcement learning framework based on conduit assembly, and defining agents and state spaces thereof; s2, defining an action space of the intelligent agent in the state space; s3, defining a reward function corresponding to the appointed action adopted by the agent in each state and a discount factor reflecting the influence degree of future return on the current decision; s4, defining constraint conditions of the intelligent agents during action decision making; s5, proposing a deep multi-agent layered Q network algorithm to optimize the assembly sequence of the conduits; and S6, verifying the conduit assembly position optimization method. The invention provides a layered deep multi-agent Q network algorithm, the optimization of the assembly sequence is realized, the consistency and the stability of the product manufacturing process capability and the quality can be improved, and a basis is provided for quantitative technical support and decision making.

Description

Technical Field

[0001] The invention belongs to the field of quality and reliability engineering, and in particular relates to a catheter assembly sequence optimization method based on multi-agent reinforcement learning. Background Art

[0002] As the main parts of aircraft, metal conduits are widely used in key parts such as aircraft hydraulic, environmental control, and fuel systems, playing the role of transmitting energy and power. When there is a large assembly stress in the conduit, it will cause air leakage, oil leakage, and other phenomena in the pipeline, which will cause various systems in the aircraft to lose pressure, leak energy, catch fire, and other failures, seriously affecting the performance and safety of the aircraft system. During the assembly process of the conduit, the size of the assembly stress is affected by multiple factors, including the material of the conduit, the preload during installation, and the design parameters of the conduit itself. In addition to these factors, the assembly order of the conduit will also affect the assembly stress. Due to the existence of assembly errors, improper assembly order will lead to a large accumulation of assembly errors. If some conduits are installed too early, it may affect the positioning and alignment of subsequent conduits, resulting in a gradual increase in deviations in the system, thereby increasing the assembly stress and reducing the sealing performance of the entire system. Therefore, a reasonable assembly order is very important during the assembly of the conduit.

[0003] In the process of catheter assembly, if we regard the end position of the catheter assembled at a certain moment as the state of the entire system at the current moment, then the entire catheter assembly process can be regarded as a dynamic transfer process. Every time a new catheter is assembled, its end position will change. At each stage, we need to decide which catheter to assemble and the specific assembly angle. The whole process is a typical multi-stage sequential decision-making process. Since the whole problem is based on three-dimensional space, the whole solution space is very large, and there are both discrete variables and continuous variables. Therefore, ordinary dynamic programming and Monte Carlo tree search algorithms are difficult to solve and are not suitable for this method. For dynamic programming methods, it is necessary to have an accurate understanding of the entire environment in advance and require certain prior knowledge, but in practical applications, it is often difficult to obtain reasonable prior knowledge; for Monte Carlo trees, they are mostly used in situations where the solution space is composed of discrete variables, which is obviously not applicable to this problem.

[0004] Therefore, there is an urgent need to provide a new method to solve the problem of optimizing the catheter assembly sequence. Summary of the invention

[0005] In order to solve the deficiencies of the above-mentioned prior art, the present invention provides a catheter assembly sequence optimization method based on multi-agent reinforcement learning, which can solve the problem that due to improper assembly sequence during the catheter assembly process, errors accumulate, resulting in increased assembly stress of pipeline connectors and decreased sealing performance. A catheter assembly sequence optimization algorithm based on multi-agent reinforcement learning is constructed to optimize the catheter assembly sequence, thereby solving the above-mentioned problem. The method of the present invention first establishes an optimization model for the catheter assembly sequence, and then designs a hierarchical multi-agent reinforcement learning algorithm based on the characteristics that its decision space contains both discrete variables and continuous variables, which is used to solve the problem, so as to calculate the catheter assembly sequence with the minimum error accumulation under given constraints and the specific assembly position coordinates of each catheter.

[0006] The method of the present invention introduces a multi-agent reinforcement learning (MARL) framework, designs three agents to decide discrete variables and continuous variables respectively, and then constructs a reinforcement learning model for optimizing the catheter assembly position. Then, a hierarchical deep multi-agent Q-network algorithm is proposed, and numerical simulation is performed to realize the calculation of the optimal assembly sequence and assembly position, thereby improving the consistency and stability of the product manufacturing process capability and product quality, and providing a basis for data-driven quantitative technical support and decision-making.

[0007] To achieve the above object, the present invention discloses the following technical solution:

[0008] Specifically, the present invention provides a catheter assembly sequence optimization method based on multi-agent reinforcement learning, and the specific steps of the method are as follows:

[0009] S1. Introduce the multi-agent reinforcement learning framework and define the agents and their state spaces;

[0010] S2. Based on the defined agent and its state space, define the action space of the agent in the state space;

[0011] S3. Based on the defined agent, state space and action space, define the reward function corresponding to the agent taking a specified action in each state, as well as the discount factor that reflects the impact of future returns on the current decision;

[0012] S4. Define the constraints of each agent when making action decisions, including assembly angle deviation constraints, assembly end position constraints, and discrete action space constraints, to form a reinforcement learning model for optimizing the catheter assembly position;

[0013] S5. Based on the reinforcement learning model for optimizing the catheter assembly position, a deep multi-agent hierarchical Q network algorithm is proposed;

[0014] S6. Based on the deep multi-agent hierarchical Q network algorithm, numerical simulation analysis is performed to verify the catheter assembly position optimization method.

[0015] Preferably, step S1 specifically includes the following sub-steps:

[0016] S11. Define the overall agent and sub-agents respectively, where the overall agent Agent tot Responsible for selecting the serial number of the catheter and the position to start assembling the catheter; Agent 1 and Agent 2 are responsible for assembling the catheter from one end respectively;

[0017] S12. For each agent, define its state space at stage t, wherein the state space of the sub-agent includes: the position coordinates of the end of the catheter after the assembly is completed at stage t when assembling starts from a certain end, and the length of the catheter selected at the current stage; the state space of the overall agent includes: the position coordinates of the end of the catheter of each sub-agent after the assembly is completed at stage t, and the angle between the straight line where the catheter is located and the positive direction of the x, y, and z axes after the assembly of each sub-agent is completed at stage t;

[0018] The state space s1(t) of the sub-agent Agent1 is:

[0019]

[0020] The state space s2(t) of the sub-agent Agent2 is:

[0021]

[0022] in, Indicates the position coordinates of the catheter end after the assembly is completed at stage t when the assembly starts from the first end point. Indicates the length of the catheter selected at the current stage; Indicates the position coordinates of the catheter end after the assembly is completed at stage t when the assembly starts from the second end point.

[0023] Total Agent tot The state space is:

[0024]

[0025] in,

[0026]

[0027] in, Respectively represent the actions of the two sub-agents in the previous stage; α, β and They represent the angles between the straight line where the catheter is located and the positive directions of the x, y, and z axes respectively; in the initial stage, α at both ends is 0, β and Both are π / 2; in the subsequent stage, the two sub-agents Agent1 and Agent2 make decision updates.

[0028] Preferably, step S2 specifically includes the following sub-steps:

[0029] S21. For each agent, define its action at stage t; wherein the action information of the master agent includes: the length of the i-th catheter to be assembled at stage t, and the information used to decide whether the selected catheter should be assembled from the left or the right; the action information of the sub-agent includes the angle between the straight line where the two end catheters are located and the positive direction of the x, y, and z axes;

[0030] S22. Update the state of the agent at stage t+1 based on the state and action of the agent at stage t.

[0031] Preferably, in step S2, the general agent Agent tot Action tot (t) is:

[0032]

[0033] in, Indicates that the i-th catheter is selected for assembly at stage t. is the length of the i-th tube, I(t) is used to decide whether the selected tube should be assembled from the first end or the second end:

[0034]

[0035] Execute a in the current stage tot (t) after, s tot (t) converted to s tot (t+1):

[0036]

[0037] If I(t) = 1, then only update and Remain unchanged; if I(t)=0, only update and remain unchanged;

[0038] and All are updated by the sub-agent’s policy network;

[0039] The actions a1(t) and a2(t) of the two sub-agents are defined as follows:

[0040]

[0041] a1(t) and a2(t) represent the angles between the two end tubes and the positive directions of the x, y, and z axes, respectively. In the initial stage,

[0042]

[0043]

[0044] At the same time, a1(t) and a2(t) satisfy:

[0045]

[0046] After executing a1(t) in the current stage, s1(t) is transformed to s1(t+1):

[0047]

[0048] After executing a2(t) in the current stage, s2(t) is transformed to s2(t+1):

[0049]

[0050] in, The length of the catheter selected for the overall agent.

[0051] Preferably, step S3 specifically includes the following sub-steps:

[0052] S31. When assembling from both ends, establish an optimization function to minimize the distance between the end points of the two middle catheters:

[0053]

[0054] Where d is the distance between the end points of the two middle catheters;

[0055] and the maximum angular deviation of all conduits during assembly Δ A The smallest optimized function:

[0056]

[0057] S32. Construct the global reward function of the overall agent and the local reward function of the sub-agent:

[0058] The global reward function is:

[0059] r tot (t) = r(t) - r(t+1);

[0060] The local reward function is:

[0061]

[0062] in, and are the local reward functions of the two sub-agents, x m ,y m , z m is the midpoint coordinate of the line connecting the two fixed endpoints, ω is the weight coefficient of position deviation and angle deviation;

[0063] S33. Define the discount factor based on the impact of future returns on current decisions, and combine it with the reward function to construct the total return of the agent in stage t;

[0064] The discount factor γ is a number between 0 and 1, and the total reward G of the agent in stage t is t Defined as:

[0065] G(r tot (t)) = r tot (t+1)+γr tot (t+2)+γr tot (t+3)+...

[0066]

[0067] Among them, G(r tot (t))、 and Represent the total rewards of the total agent and the two sub-agents respectively.

[0068] Preferably, step S4 specifically includes the following sub-steps:

[0069] S41, clarifying the upper limit of the angle deviation between adjacent conduits, and establishing an angle expression of the straight lines where the two conduits are located, to form an assembly angle deviation constraint;

[0070] The angle θ between the straight lines where the two tubes are located and the angles α, β, Has the following relationship:

[0071]

[0072] The assembly angle deviation constraints of Agent1 and Agent2 are:

[0073]

[0074] S42, clarifying the upper limit of radial deviation and length deviation between the end conduits assembled at both ends, and establishing a distance expression from the end point of one conduit to the axis of the other conduit and a length deviation expression to form an assembly end position constraint;

[0075] S43, analyzing the discrete action space constraints and other constraints, and establishing corresponding expressions;

[0076] The discrete action space constraints are:

[0077]

[0078] The other constraints are:

[0079]

[0080] That is, in the initial stage t = 0, α at both ends is 0, β and Both are π / 2;

[0081] S44. Based on the defined reward function, discount factor, action space and state space, a reinforcement learning model for optimizing the catheter assembly position is jointly formed.

[0082] Preferably, in step S42, there are radial deviation and length deviation constraints between the terminal tubes assembled at both ends, and the radial deviation is the distance from the end point of one tube to the axis of the other tube, d r1 and d r2 are the radial deviations of the two catheters, and the radial deviation constraints are as follows:

[0083]

[0084] in, and They represent the length of the last conduit assembled from both ends, and τ is the radial deviation per unit length specified in the relevant standards and specifications for conduits;

[0085] d l1 and d l2 Denote the length deviation of the two catheters, d l1 d l2 and d r1 d r2 The following relationship is satisfied:

[0086]

[0087] Preferably, step S5 specifically includes the following sub-steps:

[0088] S51. Design a distributed strategy with two layers of network structure. In each round, the decision of the master agent affects the decision of the sub-agent. At the same time, the decision of the sub-agent will also be fed back to the master agent to guide the decision of the master agent in the next round.

[0089] S52. For the upper network structure, the DQN structure is used to select the discrete actions of the master agent and perform centralized training; for the lower network structure, according to the discrete actions selected by the master agent, the corresponding continuous action parameters of the sub-agent are updated to form a deep multi-agent hierarchical Q network algorithm.

[0090] Preferably, in step S51, tot Noise is introduced into the parameters of the network, and the final TD target and TD error are as follows:

[0091]

[0092] Among them, ω tot It is the upper value network The parameter, ω tot- are the target network parameters, ξ and ξ' are random noises generated according to the standard normal distribution. and I is Agent tot Discrete actions, s tot and tot ′ respectively corresponds to Agent tot The state of the current stage and the next stage. After training is completed, the upper layer Q is maximized tot The output of the network is used to update the upper agent tot Discrete action strategy for:

[0093]

[0094] In step S52, an improved deep deterministic policy gradient algorithm is used to update the corresponding continuous action parameters based on the Actor-Critic framework.

[0095] Preferably, step S6 specifically includes the following sub-steps:

[0096] S61, using random numbers to generate sample data for the simulation experiment, simulating catheter length parameters, and setting hyperparameters in the simulation model;

[0097] S62. Conduct simulation experiments based on the sample data.

[0098] Compared with the prior art, the present invention has the following beneficial effects:

[0099] (1) The catheter assembly sequence optimization method based on multi-agent reinforcement learning provided by the present invention can adapt to complex environments. By introducing a multi-agent reinforcement learning system, it can handle dynamically changing complex assembly environments. In the entire system, the master agent and sub-agents can better adapt to complex and dynamic assembly processes through their own independent strategy learning and the fusion of global strategies, thereby optimizing the catheter assembly and meeting the catheter assembly work.

[0100] (2) The method of the present invention can handle non-convex and complex objective functions, and multi-agent reinforcement learning can handle non-convex optimization problems in catheter assembly. Since the catheter assembly sequence problem usually has multiple local optimal solutions, traditional optimization methods may fall into local minima, while the multi-agent reinforcement learning algorithm MARL can explore a wider search space and avoid falling into local optimality. Through iterative optimization and self-adjustment of strategies, MARL has the ability to approach the global optimal solution.

[0101] (3) The method of the present invention can cope with complex constraints and physical limitations. The multi-agent reinforcement learning algorithm MARL can handle various constraints in catheter assembly, such as pipe length, assembly angle, collision detection, etc., through a flexible reward mechanism. Through a carefully designed reward function, the sub-agents can learn how to optimize their actions under these complex constraints and complete the assembly task.

[0102] (4) The method of the present invention can be continuously improved through the experience pool mechanism, and the multi-agent reinforcement learning algorithm MARL can learn and improve from historical experience. In the process of performing the task, each agent stores its actions, states and rewards in the experience pool, and continuously optimizes its behavior by updating the policy network. In this way, the system can gradually optimize the agent's action selection and assembly sequence for the catheter, ensuring that the next action is the optimal catheter assembly sequence, thereby improving the overall assembly efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0104] Figure 2 It is a schematic diagram of the catheter assembly of the present invention;

[0105] Figure 3 It is a schematic diagram of the structure of the lower layer network algorithm of the present invention;

[0106] Figure 4 Schematic diagram of the Deep MAHQN algorithm structure of the present invention;

[0107] Figure 5 It is a schematic diagram of the training iteration curve of the present invention. DETAILED DESCRIPTION

[0108] The exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0109] In this embodiment, Figure 2As shown, there are n catheters, L is the length vector, and they are connected from the fixed end points on both sides. Start assembling. Each time you assemble, first select a conduit, then decide whether to assemble the conduit from the left end or the right end, and so on, until all conduits are assembled. The goal is to minimize the distance between the end of the conduit assembled from the first end, i.e., the left end, and the end of the conduit assembled from the second end, i.e., the right end, after all conduits are assembled. For example, Figure 2 As shown, the arrow indicates the assembly direction of the catheter. The left end point in this embodiment is the first end, and the right end point is the second end.

[0110]

[0111] The optimization goal of the catheter sequence is to select the optimal catheter assembly sequence and assembly angle so that Minimum.

[0112] Specifically, an embodiment of the present invention provides a catheter assembly sequence optimization method based on multi-agent reinforcement learning, such as Figure 1 As shown, the specific steps of this method are as follows:

[0113] S1. Introduce a multi-agent reinforcement learning framework based on catheter assembly and define the agents and their state spaces.

[0114] Step S1 specifically includes the following sub-steps:

[0115] S11. Define the overall agent and sub-agents respectively, where the overall agent Agent tot It is responsible for selecting the sequence number of the duct to be assembled, i.e., which duct to assemble, and the position to start assembling the duct, i.e., which end point to start assembling the duct from; sub-agent Agent1 and sub-agent Agent2 are responsible for assembling the duct from one end respectively. Sub-agent Agent1 is responsible for assembling the duct from the left end point, and sub-agent Agent2 is responsible for assembling the duct from the right end point.

[0116] S12. For each intelligent agent, define its state space at stage t, wherein the state space of the sub-agent includes: the position coordinates of the end of the catheter after the assembly is completed at stage t when assembling starts from a certain end, and the length of the catheter selected at the current stage; the state space of the overall intelligent agent includes: the position coordinates of the end of the catheter of each sub-agent after the assembly is completed at stage t, and the angle between the straight line where the catheter is located and the positive directions of the x, y, and z axes after the assembly of each sub-agent is completed at stage t.

[0117] Among them, the state space s1(t) of the sub-agent Agent1 is:

[0118]

[0119] The state space s2(t) of the sub-agent Agent2 is:

[0120]

[0121] in, It indicates the position coordinates of the end of the catheter after the assembly is completed at stage t when the assembly starts from the left end point, i.e. the first end. Indicates the length of the catheter selected at the current stage; It indicates the position coordinates of the end of the catheter after the assembly is completed at stage t when the assembly starts from the right end point, i.e. the second end.

[0122] Total Agent tot The state space is:

[0123]

[0124] in,

[0125]

[0126] in, Respectively represent the actions of the two sub-agents in the previous stage; α, β and Respectively represent the angles between the line where the catheter is located and the positive directions of the x, y, and z axes; superscript 1 represents Agent1, and superscript 2 represents Agent2; in the initial stage, α at both ends is 0, β and Both are π / 2; in the subsequent stage, the two sub-agents Agent1 and Agent2 make decision updates.

[0127] S2. Based on the defined intelligent agent and its state space, define the action space of the intelligent agent in the state space.

[0128] Step S2 specifically includes the following sub-steps:

[0129] S21. For each agent, define its action at stage t; wherein the action information of the overall agent includes: the length of the i-th catheter to be assembled at stage t and the information used to decide whether the selected catheter should be assembled from the first end or the second end; the action information of the sub-agent includes the angle between the straight line where the catheters at both ends are located and the positive directions of the x, y, and z axes.

[0130] S22. Update the state of the agent at stage t+1 based on the state and action of the agent at stage t.

[0131] Total Agent tot Action tot (t) is:

[0132]

[0133] in, Indicates that the i-th catheter is selected for assembly at stage t. is the length of the i-th tube, I(t) is used to decide whether the selected tube should be assembled from the first end or the second end:

[0134]

[0135] Execute a in the current stage tot (t) after, s tot (t) converted to s tot (t+1):

[0136]

[0137] If I(t) = 1, then only update and Remain unchanged; if I(t)=0, only update and Remain unchanged.

[0138] and They are all updated by the sub-agent's policy network.

[0139] The actions a1(t) and a2(t) of the two sub-agents are defined as follows:

[0140]

[0141] a1(t) and a2(t) represent the angles between the two end tubes and the positive directions of the x, y, and z axes, respectively. In the initial stage,

[0142]

[0143]

[0144] At the same time, a1(t) and a2(t) satisfy:

[0145]

[0146] After executing a1(t) in the current stage, s1(t) is transformed to s1(t+1):

[0147]

[0148] After executing a2(t) in the current stage, s2(t) is transformed to s2(t+1):

[0149]

[0150] in, The length of the catheter selected for the overall agent.

[0151] S3. Based on the defined agent, state space, and action space, define the reward function corresponding to the agent taking a specified action in each state, as well as the discount factor that reflects the impact of future returns on the current decision.

[0152] Step S3 specifically includes the following sub-steps:

[0153] S31. When assembling from both ends, establish an optimization function to minimize the distance between the end points of the two middle catheters:

[0154]

[0155] Where d is the distance between the end points of the two middle tubes, is the coordinate of the end point of the left catheter at stage t, is the coordinate of the distal end point of the right catheter at stage t.

[0156] and the maximum angular deviation of all conduits during assembly Δ A The smallest optimized function:

[0157]

[0158] S32. Construct the global reward function of the overall agent and the local reward function of the sub-agent:

[0159] The global reward function is:

[0160] r tot (t) = r(t) - r(t+1)

[0161]

[0162] The local reward function is:

[0163]

[0164] in, and are the local reward functions of the two sub-agents, (x m ,y m , z m ) is the midpoint position coordinate of the line connecting the two fixed endpoints, and ω is the weight coefficient of position deviation and angle deviation.

[0165] S33. Define the discount factor based on the impact of future returns on current decisions, and combine it with the reward function to construct the total return of the agent in stage t.

[0166] The discount factor γ is a number between 0 and 1, and the total reward G of the agent in stage t is t Defined as:

[0167] G(r tot (t)) = r tot (t+1)+γr tot (t+2)+γr tot (t+3)+...

[0168]

[0169]

[0170] Among them, G(r tot (t))、 and Represent the total rewards of the total agent and the two sub-agents respectively.

[0171] S4. Define the constraints for each agent when making action decisions. The constraints include assembly angle deviation constraints, assembly end position constraints, and discrete action space constraints to form a reinforcement learning model for optimizing the catheter assembly position.

[0172] Step S4 specifically includes the following sub-steps:

[0173] S41. Clarify the upper limit of the angle deviation between adjacent conduits, and establish an expression for the angle between the straight lines where the two conduits are located to form an assembly angle deviation constraint.

[0174] The angle θ between the straight lines where the two tubes are located and the angles α, β, Has the following relationship:

[0175]

[0176] The assembly angle deviation constraints of Agent1 and Agent2 are:

[0177]

[0178] S42. Clarify the upper limits of radial deviation and length deviation between the end conduits assembled at both ends, and establish the distance expression from the end point of one conduit to the axis of the other conduit and the length deviation expression to form the assembly end position constraint.

[0179] In step S42, there are radial deviation and length deviation constraints between the terminal tubes assembled at both ends. The radial deviation is the distance from the end point of one tube to the axis of the other tube. Figure 2 As shown, d r1 and dr2 are the radial deviations of the two catheters. According to relevant standards, the radial deviation of each unit length of the catheter cannot exceed τ unit lengths. The radial deviation constraint is as follows:

[0180]

[0181] in, and They respectively represent the length of the last conduit assembled from both ends.

[0182] d l1 and d l2 Denote the length deviation of the two catheters, d l1 d l2 and d r1 d r2 The following relationship is satisfied:

[0183]

[0184]

[0185] S43. Analyze the discrete action space constraints and other constraints, and establish corresponding expressions.

[0186] The discrete action space constraints are:

[0187]

[0188] The other constraints are:

[0189]

[0190] That is, in the initial stage t = 0, α at both ends is 0, β and Both are π / 2.

[0191] S44. Based on the defined reward function, discount factor, action space and state space, a reinforcement learning model for optimizing the catheter assembly position is jointly formed.

[0192] S5. Based on the reinforcement learning model for optimizing the catheter assembly position, a deep multi-agent hierarchical Q network algorithm is proposed to optimize the catheter assembly sequence.

[0193] Step S5 specifically includes the following sub-steps:

[0194] S51. Design a distributed strategy with two layers of network structure. In each round, the decision of the master agent affects the decision of the sub-agent. At the same time, the decision of the sub-agent will be fed back to the master agent to guide the decision of the master agent in the next round.

[0195] In step S51,tot Noise is introduced into the parameters of the network, and the final TD target and TD error are as follows:

[0196]

[0197] Among them, ω tot It is the upper value network The parameter, ω tot- are the target network parameters, ξ and ξ' are random noises generated according to the standard normal distribution. and I is Agent tot Discrete actions, s tot and tot 'Corresponding to Agent tot The state of the current stage and the next stage. After training is completed, the upper layer Q is maximized tot The output of the network is used to update the upper agent tot Discrete action strategy for:

[0198]

[0199] S52. For the upper network structure, the DQN structure is used to select the discrete actions of the master agent and perform centralized training; for the lower network structure, according to the discrete actions selected by the master agent, the corresponding continuous action parameters of the sub-agent are updated to form a deep multi-agent hierarchical Q network algorithm.

[0200] In step S52, an improved deep deterministic policy gradient algorithm, hereinafter referred to as TD3 algorithm, is used to update the corresponding continuous action parameters based on the Actor-Critic framework. After the discrete action is selected, the sub-agent selects the discrete action k i Generate new state observations s i Among them, when I * =1, select Agent1 to make the decision; when I * = 0, Agent 2 is selected to make decisions. The corresponding sub-agent will update the corresponding continuous action parameters based on the lower network. The training process of the TD3 algorithm is as follows:

[0201] For each sub-agent, it is trained using truncated double Q-learning with two value networks and one policy network:

[0202]

[0203] in, and is the parameter of the individual value network of each agent in the lower network, φ i are the parameters of the continuous action policy network of each agent.

[0204] There are three other target networks corresponding to each one:

[0205]

[0206] and are the parameters corresponding to the target network.

[0207] First, the continuous action parameters are calculated through the target policy network:

[0208]

[0209] Among them, ξ i is random noise drawn from a truncated normal distribution. The truncated normal distribution is denoted by represents a normal distribution with a mean of 0 and a standard deviation of σ, but the probability that the variable falls outside [-c, c] is 0. This setting can avoid noise ξ i Too large, thus affecting the training effect.

[0210] Use two target value networks to calculate the TD target y respectively i :

[0211]

[0212] Update the lower value network:

[0213]

[0214] Every K rounds, the low-level policy network μ is updated by the policy gradient method. i and three target networks:

[0215]

[0216]

[0217] Since the Actor-Critic framework uses the value network to guide the update of the policy network, if the value network itself is not reliable enough, it will not guide the update of the policy network. Therefore, we need to reduce the frequency of updating the policy network and the target network to make it slower than the update of the value network. We set the value network to be updated once per round, but the policy network and the target network to be updated every K rounds. This ensures that when the value network is used to evaluate the quality of the policy, the value network itself is stable enough and has been trained for a certain period of time. The algorithm structure of the entire lower network is as follows: Figure 3 shown.

[0218] At the beginning of training, the lower network is trained separately for a period of time to stabilize its policy output, and then the upper and lower networks are trained jointly. During joint training, the output of the lower network is used as the input of the higher network, so that the higher network can learn a better discrete action selection strategy. The structure of the overall algorithm is as follows Figure 4 The overall update process of the algorithm is shown in Table 1.

[0219] Table 1 Overall update process of the algorithm

[0220]

[0221]

[0222]

[0223] S6. Based on the deep multi-agent hierarchical Q network algorithm, numerical simulation analysis is carried out to verify the catheter assembly position optimization method.

[0224] Step S6 specifically includes the following sub-steps:

[0225] S61. Generate sample data of the simulation experiment using random numbers, simulate catheter length parameters, and set hyperparameters in the simulation model.

[0226] S62. Conduct simulation experiments based on the sample data.

[0227] In this embodiment, in the numerical simulation, 10 samples are randomly selected from the N~(5,0.05) distribution as the length of the catheter to be assembled, and the right end point is taken. The training was performed based on the Deep MAHQN algorithm. The hyperparameter settings used in the training process are shown in Table 2 below.

[0228] Table 2 Hyperparameter settings

[0229]

[0230] The training iteration curve is as follows Figure 5 As shown in the figure, it can be seen that the algorithm can converge well. In order to test the performance of the model, two groups of catheter data were randomly generated in the same way for testing. The final calculation results are shown in Table 3 below.

[0231] Table 3 Simulation results

[0232]

[0233] As can be seen from Table 3 above, the algorithm shows good performance on both test sets. The calculated results have very small error accumulation and also meet the standard requirements for radial deviation and length deviation.

[0234] In summary, the catheter assembly sequence optimization method based on multi-agent reinforcement learning provided by the present invention can adapt to complex environments. By introducing a multi-agent reinforcement learning system, it can handle dynamically changing complex assembly environments. In the entire system, the master agent and sub-agents can better adapt to complex and dynamic assembly processes through their own independent strategy learning and the fusion of global strategies, thereby optimizing the catheter assembly and meeting the catheter assembly work.

[0235] The embodiments described above are only descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A catheter assembly sequence optimization method based on multi-agent reinforcement learning, characterized in that: The specific steps of this method are as follows: S1. Introduce a multi-agent reinforcement learning framework based on catheter assembly and define the agents and their state spaces; S2. Based on the defined agent and its state space, define the action space of the agent in the state space; S3. Based on the defined agent, state space and action space, define the reward function corresponding to the agent taking a specified action in each state, as well as the discount factor that reflects the impact of future returns on the current decision; S4. Define the constraints of each agent when making action decisions, including assembly angle deviation constraints, assembly end position constraints, and discrete action space constraints, to form a reinforcement learning model for optimizing the catheter assembly position; S5. Based on the reinforcement learning model for optimizing the catheter assembly position, a deep multi-agent hierarchical Q network algorithm is obtained to optimize the catheter assembly sequence; S6. Based on the deep multi-agent hierarchical Q network algorithm, numerical simulation analysis is carried out to verify the catheter assembly position optimization method.

2. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S1 specifically includes the following sub-steps: S11. Define the overall agent and sub-agents respectively, where the overall agent Agent tot Responsible for selecting the serial number of the catheter and the position to start assembling the catheter; Agent 1 and Agent 2 are responsible for assembling the catheter from one end respectively; S12. For each agent, define its state space at stage t, wherein the state space of the sub-agent includes: the position coordinates of the end of the catheter after the assembly is completed at stage t when assembling starts from a certain end, and the length of the catheter selected at the current stage; the state space of the overall agent includes: the position coordinates of the end of the catheter of each sub-agent after the assembly is completed at stage t, and the angle between the straight line where the catheter is located and the positive direction of the x, y, and z axes after the assembly of each sub-agent is completed at stage t; The state space s1(t) of the sub-agent Agent1 is: The state space s2(t) of the sub-agent Agent2 is: in, Indicates the position coordinates of the catheter end after the assembly is completed at stage t when the assembly starts from the first end point. Indicates the length of the catheter selected at the current stage; Indicates the position coordinates of the catheter end after the assembly is completed at stage t when the assembly starts from the second end point. Total Agent tot The state space s tot (t) is: in, in, They represent the actions of the two sub-agents in the previous stage; α, β and θ represent the angles between the straight line where the catheter is located and the positive directions of the x, y and z axes respectively; the subscript last represents the previous stage, the superscript 1 represents the sub-agent Agent1, and the superscript 2 represents the sub-agent Agent2; in the initial stage, α at both ends is 0, and β and θ are both π / 2; in the subsequent stages, the two sub-agents Agent1 and Agent2 make decisions and update them.

3. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S2 specifically includes the following sub-steps: S21, for each agent, define its action at stage t; wherein the action information of the master agent includes the length of the i-th catheter selected for assembly at stage t and the information used to decide whether the selected catheter should be assembled from the first end or the second end; the action information of the sub-agent includes the angle between the straight line where the catheters at both ends are located and the positive directions of the x, y, and z axes; S22. Update the state of the agent at stage t+1 based on the state and action of the agent at stage t.

4. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S2, the general agent Agent tot Action tot (t) is: in, Indicates that the i-th catheter is selected for assembly at stage t. is the length of the i-th tube, I(t) is used to decide whether the selected tube should be assembled from the first end or the second end: Execute a in the current stage tot (t) after, s tot (t) converted to s tot (t+1): If I(t) = 1, then only update and Remain unchanged; if I(t)=0, only update and remain unchanged; and All are updated by the sub-agent’s policy network; The actions a1(t) and a2(t) of the two sub-agents are defined as follows: a1(t) and a2(t) represent the angles between the two end tubes and the positive directions of the x, y, and z axes, respectively. In the initial stage, At the same time, a1(t) and a2(t) satisfy: After executing a1(t) in the current stage, s1(t) is transformed to s1(t+1): After executing a2(t) in the current stage, s2(t) is transformed to s2(t+1): in, The length of the catheter selected for the overall agent.

5. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S3 specifically includes the following sub-steps: S31. When assembling from both ends, establish an optimization function to minimize the distance between the end points of the two middle catheters: Where d is the distance between the end points of the two middle catheters; and the maximum angular deviation of all conduits during assembly Δ A The smallest optimized function: S32. Construct the global reward function of the overall agent and the local reward function of the sub-agent: Global reward function r tot (t) is: r tot (t)=r(t)-r(t+1) The local reward function is: Among them, r(t) and r(t+1) are the reward functions of stage t and stage t+1 respectively. and are the local reward functions of the two sub-agents, x m ,y m , z m is the midpoint coordinate of the line connecting the two fixed endpoints, ω is the weight coefficient of position deviation and angle deviation; r1(t), r1(t+1), r2(t), r2(t+1) are the reward functions of the two sub-agents at stage t and stage t+1 respectively; S33. Define the discount factor based on the impact of future returns on current decisions, and combine it with the reward function to construct the total return of the agent in stage t; The discount factor γ is a number between 0 and 1, and the total reward G of the agent in stage t is t Defined as: G(r tot (t))=r tot (t+1)+γr tot (t+2)+γr tot (t+3)+... Among them, G(r t ot(t))、 and Represent the total rewards of the total agent and the two sub-agents respectively.

6. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S4 specifically includes the following sub-steps: S41, clarifying the upper limit of the angle deviation between two adjacent conduits, and establishing an angle expression of the straight lines where the two conduits are located, to form an assembly angle deviation constraint; The angle θ between the straight lines where the two tubes are located and their respective angles α, β, and θ with the positive directions of the x, y, and z axes have the following relationship: cosθ=cosα1cosα2+cosβ1cosβ2+cosθ1cosθ2; The assembly angle deviation constraints of Agent1 and Agent2 are: S42, clarifying the upper limit of radial deviation and length deviation between the end conduits assembled at both ends, and establishing a distance expression from the end point of one conduit to the axis of the other conduit and a length deviation expression to form an assembly end position constraint; S43, analyzing the discrete action space constraints and other constraints, and establishing corresponding expressions; The discrete action space constraints are: The other constraints are: That is, in the initial stage t = 0, α at both ends is 0, and β and θ are both π / 2; S44. Based on the defined reward function, discount factor, action space and state space, a reinforcement learning model for optimizing the catheter assembly position is formed based on the neural network model.

7. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S42, the end tubes assembled at both ends have radial deviation and length deviation constraints. The radial deviation is the distance from the end point of one tube to the axis of the other tube. r1 and d r2 are the radial deviations of the two catheters respectively, and the radial deviation constraints are as follows: in, and They represent the length of the last conduit assembled from both ends, respectively, and τ is the radial deviation per unit length specified in the conduit standard specification; d l1 and d l2 Denote the length deviation of the two catheters, d l1 ,d l2 and d r1 ,d r2 The following relationship is satisfied: but 8. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S5 specifically includes the following sub-steps: S51. Design a distributed strategy with two layers of network structure. In each round, the decision of the master agent affects the decision of the sub-agent. At the same time, the decision of the sub-agent is fed back to the master agent to guide the decision of the master agent in the next round. S52. For the upper network structure, the DQN structure is used to select the discrete actions of the master agent and perform centralized training; for the lower network structure, according to the discrete actions selected by the master agent, the corresponding continuous action parameters of the sub-agent are updated to form a deep multi-agent hierarchical Q network algorithm.

9. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S51, in the upper layer network structure Q tot The noise is introduced into the parameters of the network, and the final TD target L(ω tot ) and TD error y tot as follows: Among them, ω tot It is the upper value network The parameter, ω tot- are the target network parameters, ξ and ξ' are random noises generated according to the standard normal distribution; and I is the total agent tot Discrete actions, s tot and tot ' respectively correspond to the total agent Agent tot The state of the current stage and the next stage; after training is completed, the upper layer Q is maximized tot The output of the network is used to update the overall agent tot Discrete action strategy for: In step S52, the lower network structure uses an improved deep deterministic policy gradient algorithm based on the Actor-Critic framework to update the corresponding continuous action parameters.

10. The catheter assembly sequence optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Step S6 specifically includes the following sub-steps: S61, using random numbers to generate sample data for a simulation experiment, simulating catheter length parameters, and setting hyperparameters in a simulation model; S62. Conduct simulation experiments based on the sample data.

Images