Electric arc additive molten pool size model prediction control system based on reinforcement learning and control method thereof

By adopting a model prediction control system based on reinforcement learning in arc additive manufacturing, combined with MPC and state estimation module, the problem of unstable molten pool size control is solved, and the precise control of molten pool size is achieved, which meets the complex environment real-time control requirements.

CN118915460APending Publication Date: 2024-11-08NANJING NORMAL UNIVERSITY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411039085.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The molten pool size control is unstable in arc additive manufacturing, and existing control methods are difficult to meet the needs of complex environments and changes in system characteristics, resulting in reduced forming accuracy and risk of forming failure.

Method used

A model prediction and control system based on reinforcement learning is adopted, combined with the MPC controller, state estimation module and reinforcement learning module, precise control of the melt pool size is achieved by predicting changes in the melt pool size, optimizing the control amount and adjusting the control error weight.

Benefits of technology

It reduces control errors, improves the control accuracy and stability of the melt pool size, meets the real-time control needs in complex environments, and reduces the dependence on mathematical models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915460A_ABST
    Figure CN118915460A_ABST
Patent Text Reader

Abstract

The invention discloses an electric arc additive molten pool size model prediction control system based on reinforcement learning and a control method thereof, in the system, a prediction module receives a set value and real-time state information of a molten pool, characterizes dynamic characteristics of the molten pool in an electric arc additive manufacturing process, and predicts changes of a melting width and a melting height; the optimization solving module is used for solving the objective function and constraint condition module, calculating the optimal control quantity and sending the optimal control quantity to an electric arc additive manufacturing system; the target function and constraint condition module calculates the numerical range of the optimal control quantity and the increment of the control quantity; the state estimation module receives molten pool state information fed back by the system and calculates molten pool deviation, so that an MPC controller corrects a prediction model error; and the reinforcement learning module updates a control error weight parameter of the MPC controller according to the control error between the actual molten pool size and the reference value to adapt to the change of the additive manufacturing process. The method can solve the problems that a model prediction controller depends on model accuracy and the parameter processing process is complex, and control errors are reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of arc additive manufacturing, and in particular relates to a model prediction control system for arc additive molten pool size based on reinforcement learning and a control method thereof. Background Art

[0002] Arc additive manufacturing uses an electric arc as a heat source to melt metal materials through an arc between an electrode and a workpiece. The molten metal is fed into the molten pool through a wire feeding system, deposited layer by layer, and formed into the desired metal parts after cooling.

[0003] At present, weak anti-interference and poor stability are the main challenges faced by arc fuse additive manufacturing technology. In the process of arc additive manufacturing, the molten pool is easily affected by factors such as increased deposition height, changes in heat dissipation conditions, and increased heat accumulation, which makes it unstable. The molten pool size changes accordingly, reducing the component forming accuracy, and even leading to forming failure in severe cases. Therefore, controlling the molten pool size has become a key technical issue in arc additive manufacturing. In order to accurately control it, advanced control methods are usually used to adjust the process parameters. Traditional control methods mainly include PID control, fuzzy control, adaptive control, etc., but in the face of complex and changing environments or changes in system characteristics, it is often necessary to manually redesign or adjust the control parameters, which cannot meet today's control needs. With the development of control theory, model predictive control (MPC) has gradually become an advanced control strategy. It predicts future outputs based on the mathematical model of the system and calculates the optimal control strategy through optimization algorithms. Compared with other traditional methods, MPC can process multiple input and output variables at the same time, and realize the optimal control of complex systems by considering the mutual influence between the variables. However, due to factors such as high computational complexity and difficulty in optimization, it is still a big challenge to achieve accurate real-time predictive control of the molten pool size. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a model predictive control system for arc additive molten pool size based on reinforcement learning and a control method thereof, which can solve the problem that the model predictive controller is more dependent on the accuracy of the model and the parameter processing process is complicated, reduce the control error, and realize precise control of the molten pool size.

[0005] Technical solution: A model predictive control system for arc additive molten pool size based on reinforcement learning of the present invention comprises an MPC controller, and a state estimation module and a reinforcement learning module connected to the input end of the MPC controller; wherein the MPC controller comprises a prediction model, an optimization solution module and an objective function and constraint condition module connected to each other;

[0006] The prediction module obtains the optimized melt width and melt height parameters through preliminary experimental analysis, which are denoted as W setand H set , and characterize the dynamic characteristics of the molten pool in the arc additive manufacturing process based on the real-time status information of the molten pool, and predict the changes in the molten pool width and height in the future time domain;

[0007] The optimization solution module is used to receive the prediction results from the prediction model, solve the objective function and constraint condition module, calculate the optimal control amount and send it to the arc additive manufacturing system for execution;

[0008] The objective function and constraint condition module is used to calculate the numerical range of the optimal control quantity issued by the MPC controller to the arc additive manufacturing system and the increment of the control quantity in the control time domain during the process of the MPC controller controlling the molten pool size; set constraints and objectives for the optimization solution module;

[0009] The state estimation module is used to receive the molten pool state information fed back by the arc additive manufacturing system, and correct the error between the predicted molten pool size model and the model in the actual process based on the molten pool state information, and input the calculation result into the MPC controller to correct the error of its prediction model;

[0010] The reinforcement learning module is used to receive the molten pool state information fed back by the arc additive manufacturing system, and timely adjust the control error weight parameter of the MPC controller according to the control error between the actual molten pool size and the reference value of the prediction model, so as to optimize the control strategy and adapt to the changes in the additive manufacturing process.

[0011] Furthermore, the MPC controller is bidirectionally connected to the arc additive manufacturing system, and input ends of the state estimation module and the reinforcement learning module are both connected to the arc additive manufacturing system.

[0012] Based on the same inventive concept, a model predictive control method for arc additive molten pool size based on reinforcement learning of the present invention is applied to the above-mentioned model predictive control system for arc additive molten pool size based on reinforcement learning, comprising:

[0013] The arc additive manufacturing system is used to obtain the real-time status information of the molten pool during the experiment, and the real-time status information of the molten pool is sent to the prediction module in the MPC controller;

[0014] The prediction module receives the real-time status information of the molten pool, and characterizes the dynamic characteristics of the molten pool in the arc additive manufacturing process based on the real-time status information of the molten pool, and predicts the changes of the molten pool width and the molten pool height in the future time domain;

[0015] The objective function and constraint condition module calculates the numerical range of the optimal control quantity issued by the MPC controller to the arc additive manufacturing system and the increment of the control quantity in the control time domain during the process of the MPC controller controlling the molten pool size;

[0016] The optimization solution module receives the prediction results from the prediction model and the dynamic characteristics of the molten pool in the arc additive manufacturing process represented by the prediction module, solves the objective function and constraint condition modules, calculates the optimal control amount and sends it to the arc additive manufacturing system for execution;

[0017] After the arc additive manufacturing system executes the optimal control amount, the molten pool state information at the next moment is fed back to the state estimation module and the reinforcement learning module;

[0018] After receiving the molten pool state information fed back by the arc additive manufacturing system, the state estimation module calculates the deviation between the predicted molten pool size model and the actual molten pool based on the molten pool state information, and inputs the calculation result into the MPC controller to correct the error of its prediction model;

[0019] After receiving the molten pool state information fed back by the arc additive manufacturing system, the reinforcement learning module updates the control error weight parameters of the MPC controller according to the control error between the actual molten pool size and the predicted model output to adapt to the changes in the additive manufacturing process.

[0020] Furthermore, the objective function in the objective function and constraint condition module is defined as follows:

[0021]

[0022] Among them, N p is the prediction time domain; N c To control the time domain; Q W and Q H is the weight matrix of width and height deviation; W(t+i|t) and H(t+i|t) represent the molten pool width and height at the future i-th moment predicted at time t, respectively; W ref (t+i|t) and H ref (t+i|t) represents the reference values ​​of the molten pool width and height respectively; Δu represents the control increment sequence between time t and time t+i predicted at time t; R represents the control weight matrix.

[0023] Furthermore, the objective function and the constraints in the constraint module are defined as follows:

[0024] Welding current constraints:

[0025] I min ≤I≤I max

[0026] Among them, I min is the minimum value of welding current during welding; I max It is the maximum value of welding current during welding;

[0027] Wire feeding speed constraints:

[0028] v min ≤v≤v max

[0029] Among them, v min is the minimum wire feeding speed during welding; v max It is the maximum wire feeding speed during welding;

[0030] Output state constraints

[0031] W min ≤W≤W max

[0032] Among them, W min is the minimum value of the molten pool width during welding, W max It is the maximum value of the molten pool width during the welding process.

[0033] Furthermore, the design of the reinforcement learning module is as follows:

[0034] The observation state of the reinforcement learning module is defined as the melt pool width W and the melt pool height H, the action set is the melt pool width error weight and the melt pool height error weight, and the reward function R is set, which is expressed as follows:

[0035] R(t)=-[αΔW(t) 2 +βΔH(t) 2 ]-γ||Δd(t)|| 2

[0036] Among them, ΔW is the deviation between the actual melt width and the target melt width; ΔH is the deviation between the actual melt width and the target melt width; α and β are weight coefficients used to balance the proportion of melt width and melt height errors in the reward function; γ is the regularization coefficient used to punish the severity of the control input change; Δd(t) represents the change between the current action and the previous action.

[0037] Furthermore, the reinforcement learning module is provided with a DQN network, and the collected data is used to train the DQN network to complete the strategy update; wherein the industrial camera in the arc additive manufacturing system obtains the real-time width and height of the molten pool, the welder in the arc additive manufacturing system obtains the welding current and voltage values, and the robotic arm in the arc additive manufacturing system obtains the wire feeding speed; the MPC controller obtains the deviation value of the molten width and the molten height according to the real-time size and the predicted value; all the above data are used as training data for the DQN network.

[0038] Furthermore, the DQN network is trained using the collected data to complete the strategy update, including:

[0039] Initialize the Q network and target network, and select the initial control strategy;

[0040] Execute the current network strategy, record the current state s at each time step, select and apply action a;

[0041] Record the next state s', use the reward function R(t) to calculate the immediate reward r, and store the sample sequence (s, a, r, s') in the experience replay buffer;

[0042] A batch of data is randomly sampled from the experience replay buffer, and then the target network is used to calculate the target value of the value function; the target value is expressed as follows:

[0043] y=r+ηmax a' Q'(s',a',θ')

[0044] Where η is the discount factor, and 0<η<1; Q' represents the Q value of the target network; s' and a' are the state and action of the next time step; θ' represents the parameters of the target network;

[0045] Update the network by minimizing the loss function, which is formulated as follows:

[0046] L(θ)=E[(yQ(s,a,θ)) 2 ]

[0047] Where E represents the expected value of all possible state-action pairs (s, a); y is the target value of the value function;

[0048] Use the mini-batch stochastic gradient descent algorithm to update the weight parameter θ of the loss function L(θ) to obtain the optimal action value. The specific formula is as follows:

[0049]

[0050] Furthermore, the optimal control quantity is the optimal input value of the wire feeding speed and the welding current.

[0051] Furthermore, the dynamic characteristics of the molten pool during the arc additive manufacturing process are the dynamic relationship between the molten pool size and the control input, and the control input includes the wire feeding speed and the welding current.

[0052] Beneficial effects: Compared with the prior art, the significant technical effects of the present invention are:

[0053] The present invention controls the size of the molten pool through an MPC controller, establishes an objective function to achieve preliminary control of the width and height of the molten pool, and then updates the coefficients in the objective function through reinforcement learning to provide a better control strategy.

[0054] The reinforcement learning module can provide fast decision feedback to meet the needs of real-time control. The DQN algorithm uses the experience replay buffer to store historical experience, which can be reused for learning and improves data utilization.

[0055] The control of the molten pool size by the present invention does not rely too much on the mathematical model of the molten pool control system, and the adopted method can optimize the performance of the molten pool in a dynamic environment while satisfying complex constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A structural block diagram of a model predictive control system for arc additive molten pool size based on reinforcement learning disclosed in an embodiment of the present invention;

[0057] Figure 2 It is a schematic structural diagram of an arc additive manufacturing system disclosed in an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of a DQN network disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described in detail below in conjunction with specific implementation methods and the accompanying drawings of the specification.

[0060] Example 1

[0061] like Figure 1 As shown, the model predictive control system of arc additive molten pool size based on reinforcement learning of the present invention includes an MPC controller (model predictive controller), a state estimation module, and a reinforcement learning module, and the state estimation module and the reinforcement learning module are connected to the input end of the MPC controller. The MPC controller is bidirectionally connected to the arc additive manufacturing system, and the input ends of the state estimation module and the reinforcement learning module are both connected to the arc additive manufacturing system. Among them, the MPC controller includes a prediction model, an optimization solution module, and an objective function and constraint condition module that are interconnected.

[0062] The prediction module is used to receive the set value of the molten pool size and the real-time status information of the molten pool obtained by the arc additive manufacturing system. The real-time status information of the molten pool includes the optimized parameters such as the molten width and molten height obtained through the previous experimental analysis, which are respectively denoted as W set and H set , and characterize the dynamic characteristics of the molten pool in the arc additive manufacturing process based on the real-time status information of the molten pool, and predict the changes in the molten pool width and height in the future time domain. The optimization solution module is used to receive the prediction results from the prediction model, solve the objective function and constraint condition module, calculate the optimal control amount and send it to the arc additive manufacturing system for execution. The objective function and constraint condition module is used to calculate the numerical range of the optimal control amount issued by the MPC controller to the arc additive manufacturing system and the incremental size of the control amount in the control time domain during the process of the MPC controller controlling the molten pool size, and set constraints and goals for the optimization solution module.

[0063] The state estimation module is used to receive the molten pool state information fed back by the arc additive manufacturing system, and calculate the deviation between the predicted molten pool size model and the actual molten pool based on the molten pool state information, and input the calculation result into the MPC controller to correct the error of its prediction model so that the theoretical model is close to the actual process. The reinforcement learning module is used to receive the molten pool state information fed back by the arc additive manufacturing system, and adjust the control error weight parameter of the MPC controller in time according to the control error between the actual molten pool size and the reference value of the prediction model, adapt to the changes in the additive manufacturing process, and improve the control accuracy of the molten pool size.

[0064] In the present invention, the prediction model establishes a dynamic relationship between the molten pool size and the control input (wire feeding speed and welding current) based on the physical characteristics of the arc additive process. The optimization solution module receives the prediction results from the prediction model, solves the objective function and constraints, calculates the optimal control quantity, and sends it to the arc additive system for execution. After the arc additive system executes the command, the molten pool state quantity is obtained and fed back to the state estimation module and the reinforcement learning module. The state estimation module corrects the prediction model error according to the fedback state quantity to improve the prediction accuracy. The reinforcement learning module adjusts the control error weight parameter of the model prediction controller according to the difference between the reference value and the actual molten pool size to achieve precise control of the molten pool size. Through continuous learning and adaptation, the reinforcement learning module can optimize the control strategy and improve the accuracy and stability of the molten pool size control. The control method of the present invention can effectively combine the high precision of the model predictive control and the adaptive ability of reinforcement learning to achieve precise control of the molten pool width and height in the arc additive process, and improve manufacturing efficiency and part quality.

[0065] Example 2

[0066] A model predictive control method for arc additive molten pool size based on reinforcement learning of the present invention is applied to the model predictive control system for arc additive molten pool size based on reinforcement learning in Example 1. The method comprises the following steps:

[0067] S1. Use the arc additive manufacturing system to obtain the real-time status information of the molten pool during the experiment, and send the real-time status information of the molten pool to the prediction module in the MPC controller.

[0068] like Figure 2 As shown, the arc additive manufacturing system for the experiment includes a host computer 1, a robot arm 2, a welding machine 3, an industrial camera 4, a workbench 5, a substrate 6, a welding gun 7, a wire feeder 8, and a shielding gas device 9. The welding gun 7 is fixed on the robot arm 2, and the substrate 6 is fixed on the workbench 5. The industrial camera 4 is fixed on the left side of the workbench 5 with a bracket. The host computer 1 is connected to the welding machine 3 and the industrial camera 4, and the wire feeder 8 is connected to the welding gun 7 to continuously supply the welding wire; the shielding gas device 9 is connected to the welding gun 7 to prevent chemical reactions such as oxidation.

[0069] When the experiment starts, the welding program is first loaded and the welding parameters are set through the host computer 1. Among them, the welding parameters include: welding current 220A, welding voltage 27V, wire feeding speed 15mm / s, scanning speed 10mm / s. The substrate 6 is fixed on the workbench 5, the robot arm 2 moves to the starting position, and the welding gun 7 and the wire feeder 8 are ready. The shielding gas device 9 is turned on. The gas of the shielding gas device 9 is 100% pure argon gas, and the flow rate is set to 15L / min. The industrial camera 4 starts monitoring and captures the real-time image of the molten pool during the welding process. After receiving the parameters, the welder 3 starts to generate an arc, and the robot arm 2 drives the welding gun 7 to move along the predetermined path for welding. According to the real-time molten width and molten height of the molten pool image transmitted back by the industrial camera 4, the host computer 1 controls and adjusts the welding parameters in real time and transmits them to the welder 3. By continuously adjusting the parameters, the predetermined additive manufacturing path is completed, the molten pool width and molten pool height are controlled, and the welding is completed. The host computer 1 is controlled by the model predictive control system of arc additive molten pool size based on reinforcement learning.

[0070] S2. The prediction module receives the real-time status information of the molten pool, and characterizes the dynamic characteristics of the molten pool in the arc additive manufacturing process based on the real-time status information of the molten pool, and predicts the changes of the molten pool width and the molten pool height in the future time domain.

[0071] In this embodiment, the prediction module includes the following units:

[0072] Data receiving unit: used to receive set values ​​and real-time status information of the molten pool;

[0073] Feature extraction unit: extracts dynamic features that are helpful for prediction from real-time status information;

[0074] Model building unit: building a prediction model for the size of the molten pool;

[0075] Prediction algorithm: The model predictive control method used in this paper;

[0076] Interface unit: interacts with other system modules and provides prediction data of melt pool size.

[0077] In this embodiment, the dynamic characteristics of the molten pool in the arc additive manufacturing process are the dynamic relationship between the molten pool size and the control input, and the control input includes the wire feeding speed and the welding current.

[0078] S3, the objective function and constraint condition module calculates the numerical range of the optimal control quantity issued by the MPC controller to the arc additive manufacturing system and the increment of the control quantity in the control time domain during the process of the MPC controller controlling the molten pool size.

[0079] In this embodiment, the objective function in the objective function and constraint condition module is defined as follows:

[0080]

[0081] Among them, N p is the prediction time domain; N c To control the time domain; Q W and Q H is the weight matrix of width and height deviation; W(t+i|t) and H(t+i|t) represent the molten pool width and height at the future i-th moment predicted at time t, respectively; W ref (t+i|t) and H ref (t+i|t) represents the reference values ​​of the molten pool width and height respectively; Δu represents the control increment sequence between time t and time t+i predicted at time t; R represents the control weight matrix, which is a one-dimensional matrix, and the output weight matrix Q W and Q H is a two-dimensional matrix.

[0082] In this embodiment, the constraints in the objective function and constraint condition module are defined as follows:

[0083] (1) Welding current constraints:

[0084] I min ≤I≤I max

[0085] Among them, I min is the minimum value of welding current during welding; I max It is the maximum value of welding current during welding;

[0086] (2) Wire feeding speed constraints:

[0087] v min ≤v≤v max

[0088] Among them, v min is the minimum wire feeding speed during welding; v max It is the maximum wire feeding speed during welding;

[0089] (3) Output state constraints:

[0090] W min ≤W≤W max

[0091] Among them, W min is the minimum value of the molten pool width during welding, W max It is the maximum value of the molten pool width during the welding process.

[0092] In step S3, the optimal control variables are the optimal input values ​​of the wire feeding speed and the welding current.

[0093] In this embodiment, the controller control quantity u(t) is the welding current I and the wire feeding speed v, the state quantity includes the welding gun moving speed, welding voltage, and shielding gas flow rate, and the output of the prediction model is the molten pool width and the molten pool height.

[0094] S4. The optimization solution module receives the prediction results from the prediction model and the dynamic characteristics of the molten pool in the arc additive process characterized by the prediction module, solves the objective function and constraint condition module, calculates the optimal control quantity and sends it to the arc additive manufacturing system for execution.

[0095] In this embodiment, the optimization solution module is composed of the following units:

[0096] Optimization unit: including optimization algorithm, used to solve the state estimation problem of molten pool;

[0097] Parameter Tuner: used to adjust parameters to improve solution performance and result quality;

[0098] Constraint Processor: Processes and converts constraints to ensure they are suitable for the melt pool width and height solution algorithm.

[0099] Integration interface: used to integrate the module with other systems or modules (such as data collection, state estimation, control execution, etc.);

[0100] S5. After the arc additive manufacturing system executes the optimal control quantity, the molten pool state information at the next moment is fed back to the state estimation module and the reinforcement learning module.

[0101] S6. After receiving the molten pool state information fed back by the arc additive manufacturing system, the state estimation module calculates the deviation between the predicted molten pool size model and the actual molten pool based on the molten pool state information, and inputs the calculation result into the MPC controller to correct the error of its prediction model. The deviation refers to the deviation value of the molten width and the molten height.

[0102] In this embodiment, the state estimation module is composed of the following units:

[0103] Data receiving unit: used to receive the state observation value of the molten pool at the current moment;

[0104] State observation unit: uses observed melt pool data and possible observation models to estimate the system state;

[0105] Kalman filter: a commonly used state estimation algorithm used for state estimation and data fusion of linear or nonlinear systems;

[0106] Interface unit: used to input the estimated state quantity into the optimization solution module and perform optimization solution again.

[0107] S7. After receiving the molten pool state information fed back by the arc additive manufacturing system, the reinforcement learning module updates the control error weight parameter of the MPC controller according to the control error between the actual molten pool size and the reference value of the prediction model to adapt to the changes in the additive manufacturing process, thereby improving the control accuracy of the molten pool size.

[0108] In this embodiment, the design of the reinforcement learning module is as follows:

[0109] The observation state of the reinforcement learning module is defined as the melt pool width W and the melt pool height H, and the action set is the melt pool width error weight and the melt pool height error weight. According to this condition, the reward function R is set, and its expression is as follows:

[0110] R(t)=-[αΔW(t) 2 +βΔH(t) 2 ]-γ||Δd(t)|| 2

[0111] Among them, ΔW is the deviation between the actual melt width and the target melt width; ΔH is the deviation between the actual melt width and the target melt width; α and β are weight coefficients used to balance the proportion of melt width and melt height errors in the reward function; γ is the regularization coefficient used to punish the severity of the control input change; Δd(t) represents the change between the current action and the previous action.

[0112] The reinforcement learning module is provided with a DQN network. In this embodiment, a DQN network is constructed, which consists of an input layer, two convolutional layers, two pooling layers, a fully connected layer and an output layer; after the state input layer, there is a convolutional layer 1, including 32 convolutional kernels, with a size of 8×8 and a step length of 4, followed by a pooling layer 1 with a size of 2×2 and a step length of 2, and after the pooling layer 1 is connected a convolutional layer 2, including 64 convolutional kernels, with a size of 4×4 and a step length of 2, followed by a pooling layer 2 with a size of 2×2 and a step length of 2. Further down is a fully connected layer containing 512 neurons, and finally connected to the output layer.

[0113] The DQN network is trained using the collected data to complete the strategy update. Figure 3 As shown, the specific update steps include the following:

[0114] S7.1. Determine the initial Q network and target network, and select the initial control strategy.

[0115] Initialize an experience playback buffer D to store the observed experience; initialize the Q network Q(s,a,θ) and the target Q network Q'(s,a,θ'), and set the parameters of the target network to be the same as the main network, that is, θ'=θ. Select an initial control strategy.

[0116] S7.2. Execute the current network strategy. At each time step, record the current state s, select and apply action a.

[0117] S7.3. Record the next state s', use the reward function R(t) to calculate the immediate reward r, and store the sample sequence (s, a, r, s') in the experience replay buffer.

[0118] S7.4. Randomly sample a batch of data from the experience replay buffer, and then use the target network to calculate the target value of the value function; the target value is expressed as follows:

[0119] y=r+ηmax a' Q'(s',a',θ')

[0120] Among them, η is the discount factor, and 0<η<1; Q' represents the Q value of the target network; s' and a' are the state and action of the next time step; θ' represents the parameters of the target network.

[0121] S7.5. Update the network by minimizing the loss function, which is as follows:

[0122] L(θ)=E[(yQ(s,a,θ)) 2 ]

[0123] Among them, E represents the expected value of all possible state-action pairs (s, a), and y is the target value of the value function.

[0124] S7.6. Use the mini-batch stochastic gradient descent algorithm (SGD) to update the weight parameter θ of the loss function L(θ) to obtain the optimal action value. The specific formula is as follows:

[0125]

[0126] At the same time, the reinforcement learning module is also responsible for learning the dynamic characteristics of the environment to adapt to the nonlinear factors that may occur in the arc additive process.

[0127] The technical solution of the present invention aims to solve the problem that the model predictive controller is more dependent on the accuracy of the model and the parameter processing process is complicated. The wire feeding speed and welding current are used as control inputs, the reinforcement learning method is used to improve the decision-making process, and the DQN network is used to train and learn the parameters, so that the controller can adjust the control parameters in time according to the actual state information and reference values ​​during the welding process, reduce the control error, and achieve precise control of the width and height of the molten pool size.

Claims

1. A model predictive control system for arc additive molten pool size based on reinforcement learning, characterized in that: It includes an MPC controller, and a state estimation module and a reinforcement learning module connected to the input end of the MPC controller; wherein the MPC controller includes a prediction model, an optimization solution module, and an objective function and constraint condition module that are interconnected; The prediction module obtains the optimized melt width and melt height parameters through preliminary experimental analysis, which are denoted as W set and H set , and characterize the dynamic characteristics of the molten pool in the arc additive manufacturing process based on the real-time status information of the molten pool, and predict the changes in the molten pool width and height in the future time domain; The optimization solution module is used to receive the prediction results from the prediction model, solve the objective function and constraint condition module, calculate the optimal control amount and send it to the arc additive manufacturing system for execution; The objective function and constraint condition module is used to calculate the numerical range of the optimal control quantity issued by the MPC controller to the arc additive manufacturing system and the increment of the control quantity in the control time domain during the process of the MPC controller controlling the molten pool size; set constraints and objectives for the optimization solution module; The state estimation module is used to receive the molten pool state information fed back by the arc additive manufacturing system, calculate the deviation between the predicted molten pool size model and the actual molten pool based on the molten pool state information, and input the calculation result into the MPC controller to correct the error of its prediction model; The reinforcement learning module is used to receive the molten pool state information fed back by the arc additive manufacturing system, and timely adjust the control error weight parameter of the MPC controller according to the control error between the actual molten pool size and the reference value of the prediction model, so as to optimize the control strategy and adapt to the changes in the additive manufacturing process.

2. The model predictive control system for arc additive molten pool size based on reinforcement learning according to claim 1, characterized in that: The MPC controller is bidirectionally connected to the electric arc additive manufacturing system, and the input ends of the state estimation module and the reinforcement learning module are both connected to the electric arc additive manufacturing system.

3. A model predictive control method for arc additive molten pool size based on reinforcement learning, characterized in that: The method is applied to the model predictive control system of arc additive molten pool size based on reinforcement learning as described in claim 1, comprising: The arc additive manufacturing system is used to obtain the real-time status information of the molten pool during the experiment, and the real-time status information of the molten pool is sent to the prediction module in the MPC controller; The prediction module receives the real-time status information of the molten pool, and characterizes the dynamic characteristics of the molten pool in the arc additive manufacturing process based on the real-time status information of the molten pool, and predicts the changes of the molten pool width and molten pool height in the future time domain; The objective function and constraint condition module calculates the numerical range of the optimal control quantity issued by the MPC controller to the arc additive manufacturing system and the increment of the control quantity in the control time domain during the process of the MPC controller controlling the molten pool size; The optimization solution module receives the prediction results from the prediction model and the dynamic characteristics of the molten pool in the arc additive manufacturing process represented by the prediction module, solves the objective function and constraint condition modules, calculates the optimal control amount and sends it to the arc additive manufacturing system for execution; After the arc additive manufacturing system executes the optimal control amount, the molten pool state information at the next moment is fed back to the state estimation module and the reinforcement learning module; After receiving the molten pool state information fed back by the arc additive manufacturing system, the state estimation module calculates the deviation between the predicted molten pool size model and the actual molten pool based on the molten pool state information, and inputs the calculation result into the MPC controller to correct the error of its prediction model; After receiving the molten pool state information fed back by the arc additive manufacturing system, the reinforcement learning module updates the control error weight parameters of the MPC controller according to the control error between the actual molten pool size and the reference value of the prediction model to adapt to the changes in the additive manufacturing process.

4. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 3, characterized in that: The objective function in the objective function and constraint condition module is defined as follows: Among them, N p is the prediction time domain; N c To control the time domain; Q W and Q H is the weight matrix of width and height deviation; W(t+i|t) and H(t+i|t) represent the molten pool width and height at the future i-th moment predicted at time t, respectively; W ref (t+i|t) and H ref (t+i|t) represents the reference values ​​of the molten pool width and height respectively; Δu represents the control increment sequence between time t and time t+i predicted at time t; R represents the control weight matrix.

5. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 3, characterized in that: The objective function and the constraints in the constraint module are defined as follows: Welding current constraints: I min ≤I≤I max Among them, I min is the minimum value of welding current during welding; I max It is the maximum value of welding current during welding; Wire feeding speed constraints: in min ≤v≤v max Among them, v min is the minimum wire feeding speed during welding; v max It is the maximum wire feeding speed during welding; Output state constraints: IN min ≤W≤W max Among them, W min is the minimum value of the molten pool width during welding, W max It is the maximum value of the molten pool width during the welding process.

6. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 3, characterized in that: The design of the reinforcement learning module is as follows: The observation state of the reinforcement learning module is defined as the melt pool width W and the melt pool height H, the action set is the melt pool width error weight and the melt pool height error weight, and the reward function R is set, which is expressed as follows: R(t)=-[αΔW(t) 2 +βΔH(t) 2 ]-γ||Δd(t)|| 2 Among them, ΔW is the deviation between the actual melt width and the target melt width; ΔH is the deviation between the actual melt width and the target melt width; α and β are weight coefficients used to balance the proportion of melt width and melt height errors in the reward function; γ is the regularization coefficient used to punish the severity of the control input change; Δd(t) represents the change between the current action and the previous action.

7. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 6, characterized in that: The reinforcement learning module is provided with a DQN network, and the collected data is used to train the DQN network to complete the strategy update; the industrial camera in the arc additive manufacturing system obtains the real-time width and height of the molten pool, the welder in the arc additive manufacturing system obtains the welding current and voltage values, and the robotic arm in the arc additive manufacturing system obtains the wire feeding speed. The MPC controller obtains the deviation value of the molten width and the molten height according to the real-time size and the predicted value; all the above data are used as the training data of the DQN network.

8. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 7, characterized in that: The DQN network is trained using the collected data to complete the strategy update, including: Set the initial Q network and target network, and select the initial control strategy; Execute the current network strategy, record the current state s at each time step, select and apply action a; Record the next state s', use the reward function R(t) to calculate the immediate reward r, and store the sample sequence (s, a, r, s') in the experience replay buffer; A batch of data is randomly sampled from the experience replay buffer, and then the target network is used to calculate the target value of the value function; the target value is expressed as follows: y=r+ηmax a' Q'(s',a',θ') Where η is the discount factor, and 0<η<1; Q' represents the Q value of the target network; s' and a' are the state and action of the next time step; θ' represents the parameters of the target network; Update the network by minimizing the loss function, which is formulated as follows: L(θ)=E[(yQ(s,a,θ)) 2 ] Where E represents the expected value of all possible state-action pairs (s, a); y is the target value of the value function; Use the mini-batch stochastic gradient descent algorithm to update the weight parameter θ of the loss function L(θ) to obtain the optimal action value. The specific formula is as follows:

9. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 3, characterized in that: The optimal control quantity is the optimal input value of wire feeding speed and welding current.

10. The model predictive control method for arc additive molten pool size based on reinforcement learning according to claim 3, characterized in that: The dynamic characteristics of the molten pool in the arc additive manufacturing process are the dynamic relationship between the molten pool size and the control input, and the control input includes the wire feeding speed and the welding current.

Citation Information

Cited By

  • Multi-arc parallel additive manufacturing arc length self-adaptive detection and intelligent feedback control system and method

    CN120868925A

  • Welding wire production control system and method based on visual feedback

    CN121956873A