Laser welding penetration stability closed-loop control method based on deep reinforcement learning

Abstract

The application discloses a laser welding penetration stability closed-loop control method and system based on deep reinforcement learning, based on the online collected welding plasma signal, multi-period scale time series modeling is carried out by using a TimesNet model, high-precision real-time monitoring of the penetration change is realized; then a risk-sensitive control model is constructed based on the proximal policy optimization, which fuses the value function distribution modeling and the entropy risk measurement, the policy network outputs the laser power regulation action online according to the real-time penetration state, the value network introduces the quantile distribution value head to depict the potential value distribution corresponding to the regulation action, and the risk is quantified by combining the entropy risk measurement. The application quantifies the risk of the control action, avoids the instability induced by the coupling of the regulation action and the keyhole oscillation, significantly improves the robustness and precision of the penetration closed-loop control, and is suitable for high-dynamic and strong nonlinear laser welding processes.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology for laser welding, specifically to a closed-loop control method for laser welding penetration stability based on deep reinforcement learning. Background Technology

[0002] Laser welding, as an advanced joining technology with high energy density and high processing efficiency, has been widely used in the processing of key metal components in aerospace, rail transportation, automobile manufacturing, and energy equipment. In laser welding, the penetration depth (i.e., the weld seam penetration depth) is a core indicator determining the mechanical properties and forming quality of the weld joint. However, due to the complex multi-physics coupling effects involved in laser welding, including steam backpressure fluctuations, keyhole periodic oscillations, metal flow within the molten pool, dynamic changes in plasma plume, and material thermal property inhomogeneities, the evolution of penetration depth exhibits significant dynamic, nonlinear, and random fluctuation characteristics. In the welding of lightweight metals such as aluminum alloys, due to their high thermal conductivity and low boiling point, keyhole stability is even worse, easily leading to defects such as abrupt penetration changes, spatter, porosity, and undercut, severely restricting the reliability of weld quality.

[0003] Existing penetration depth control methods are mainly divided into two categories: open-loop and closed-loop. Open-loop control relies on pre-set welding process parameters (such as laser power, welding speed, defocusing amount, etc.), and cannot respond to disturbances during the welding process in real time, making it difficult to ensure penetration depth consistency across a wide range of working conditions. Closed-loop control methods can dynamically adjust process parameters based on real-time penetration depth monitoring results, improving welding accuracy. Traditional closed-loop control methods include PID control, fuzzy logic control, and sliding mode control, but these methods are usually based on linearization or empirical models, and their control effect is limited when faced with the highly nonlinear and strongly coupled characteristics of laser welding. In recent years, data-driven control methods based on deep learning have been attempted to be applied to the laser welding process, using neural networks to learn the mapping relationship between penetration depth and process parameters from a large amount of data. However, they still rely on offline training or preset strategies and lack the ability to learn autonomously and make adaptive decisions in complex dynamic environments.

[0004] Reinforcement learning (RL) achieves adaptive optimization of control strategies through continuous interaction between an agent and its environment, aiming to maximize long-term cumulative rewards. It has already achieved success in fields such as robot control and game theory. Laser welding processes exhibit typical sequential decision-making characteristics: continuous state perception, dynamic parameter adjustment, and real-time feedback updates, making them suitable for reinforcement learning frameworks. Proximal policy optimization (PPO), a representative algorithm in reinforcement learning, has attracted attention due to its training stability and data efficiency. However, conventional PPO methods typically focus on maximizing average control performance or expected rewards, lacking sufficient characterization of the potential differences in outcome distribution and risks arising from different regulatory actions. In laser welding, due to the stochastic oscillation characteristics of weld depth evolution under multi-physics coupling, certain regulatory actions (such as suddenly increasing laser power at the moment of keyhole instability) may resonate with the keyhole's inherent oscillation frequency, thereby inducing or even amplifying process instability, leading to spatter or weld penetration. This means that optimization strategies that only pursue average rewards may implicitly contain significantly high-risk actions, and conventional RL cannot effectively quantify and avoid these risks.

[0005] In summary, the existing technology has the following drawbacks:

[0006] (1) There is a lack of means to monitor the evolution of melting depth in real time and with high precision under strong dynamic disturbances, especially the ability to extract multi-period time-series features of plasma spectral signals.

[0007] (2) Traditional reinforcement learning methods do not introduce risk-sensitive mechanisms and cannot identify and avoid high-risk control actions that may lead to welding instability;

[0008] (3) The transient stability, power smoothness and melting depth tracking accuracy of the melting depth closed-loop control were not uniformly incorporated into the reward function, which may cause the control strategy to experience severe power fluctuations or melting depth oscillations during long-term operation. Summary of the Invention

[0009] The main objective of this application is to provide a closed-loop control method for laser welding penetration stability based on deep reinforcement learning, comprising the following steps:

[0010] Step S1: Real-time acquisition of plasma spectral signals generated during the welding process, and selection of several wavelengths of the plasma spectral signals as monitoring data;

[0011] Step S2: Input the monitoring data within the latest few frames of the time window into the pre-trained TimesNet melt depth monitoring model. The TimesNet model extracts the temporal features of the spectral data based on multi-period scale transformation and outputs the melt depth prediction value at the latest moment as the real-time melt depth status.

[0012] Step S3: Construct a deep reinforcement learning controller based on near-end policy optimization. The controller includes a policy network and a value network. The policy network is used to output process parameters online to adjust actions according to the real-time melt depth state input. The value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network adopt a long short-term memory network structure and use the state sequence of the past period as input.

[0013] Step S4: Introduce a quantile distribution value head into the value network to characterize the potential value distribution corresponding to different control actions; and quantify the risk of the value distribution based on the quantile distribution value head and the entropy risk metric to obtain the action value after risk assessment, so as to optimize the controller's strategy.

[0014] Step S5: Set the reward function of the deep reinforcement learning controller, wherein the reward function includes a depth tracking error term, a depth stability penalty term, and a power smoothness penalty term;

[0015] Step S6: A round-based strategy learning mechanism is used for multi-pass laser welding. During each welding process, the strategy network executes closed-loop control in real time and records the status, action, and reward data. After each welding is completed, the value network is used to perform distributed value modeling and risk assessment on the status, action, and reward data, and the strategy network parameters are updated.

[0016] Step S7: Repeat step S6 until convergence, and obtain the trained policy network, which will be used for subsequent closed-loop control of laser welding penetration stability.

[0017] In one embodiment, the plasma spectral signal in step S1 includes one or more of the following: Mg-I atomic spectral lines, Al-I atomic spectral lines, AlO molecular spectral lines, and Cr-I atomic spectral lines, specifically selected from at least five wavelengths including but not limited to 382.90 nm, 394.35 nm, 396.26 nm, 403.07 nm, 427.44 nm, 484.25 nm, 486.62 nm, 517.27 nm, and 518.33 nm.

[0018] In one embodiment, the step of inputting monitoring data within the latest few frames of time window into a pre-trained TimesNet melt depth monitoring model, wherein the TimesNet model extracts temporal features from the spectral data based on multi-period scale transformation and outputs the melt depth prediction value at the latest moment as the real-time melt depth status, includes:

[0019] Multiple laser welding experiments were conducted using a benchmark welding power to obtain metallographic images of the longitudinal section of the weld and extract actual weld penetration data; a training dataset was constructed by pairing spectral monitoring data with actual weld penetration data.

[0020] Using the Adam optimizer, with a learning rate of 0.0001, a batch size of 64, and 200 iterations, a TimesNet model for multi-period-scale time series modeling was obtained.

[0021] In one embodiment, the construction of a deep reinforcement learning controller based on proximal policy optimization includes a policy network and a value network. The policy network is used to adjust actions by outputting process parameters online based on the input real-time melt depth state, and the value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network adopt a long short-term memory network structure, and the step of using a sequence of states over a past period as input includes:

[0022] The continuous N-frame melt depth state sequence obtained in real time through the TimesNet melt depth monitoring model is used as the controller input state quantity, and the continuous adjustment of laser power is used as the action space.

[0023] A strategy network based on a long short-term memory network is constructed. The continuous N frames of melt depth state sequence are used as input. The temporal dependence features of melt depth evolution are extracted through the long short-term memory network layer, and then the laser power control action is mapped and output through a fully connected layer.

[0024] A value network based on a long short-term memory network is constructed. The input is a continuous N-frame melt depth state sequence, which is the same as the policy network. Temporal features are extracted through the long short-term memory network layer, and then connected to the quantile distribution value head to output the quantile distribution of the state value, providing a distribution data basis for entropy risk measurement.

[0025] In one embodiment, the step of introducing a quantile distribution value head into the value network to characterize the potential value distribution corresponding to different regulatory actions; and quantifying the risk of the value distribution based on the quantile distribution value head combined with an entropy risk metric to obtain the action value after risk assessment, so as to optimize the controller's strategy, includes:

[0026] At the end of the value network, a quantile distribution value head is constructed. The quantile distribution value head contains N quantile outputs, each quantile corresponding to an independent fully connected branch. The temporal hidden features extracted by the long short-term memory network are mapped to N quantile points to form the quantile distribution of state value, where N is an integer greater than or equal to 8 and less than or equal to 64.

[0027] Based on the quantile distribution, the action value after risk assessment is calculated using the entropy risk measurement formula, which is:

[0028]

[0029] in, The value sample corresponding to the i-th quantile in the quantile distribution value head output, where β is a risk-sensitive parameter;

[0030] The value of the action after the risk assessment As the target value of the value network, it is substituted into the advantage function calculation of the near-end policy optimization, and the parameters of the policy network and value network under the near-end policy optimization framework are iteratively updated accordingly.

[0031] In one embodiment, the step of setting the reward function of the deep reinforcement learning controller, the reward function including a depth-of-sight tracking error term, a depth-of-sight stability penalty term, and a power smoothness penalty term, includes:

[0032] Define a melt depth tracking error term to penalize the deviation between the real-time melt depth and the target melt depth. The calculation formula is as follows:

[0033]

[0034] Where, d t Let d be the real-time melt depth at time t. target The target melting depth is preset, and k1 is the contribution weight of the tracking error term;

[0035] A melt depth stability penalty term is defined to suppress drastic jumps in melt depth between adjacent time steps. The calculation formula is as follows:

[0036]

[0037] Where, d t-1 Let k be the real-time melting depth at time t-1, and k2 be the contribution weight of the stability penalty term;

[0038] A power smoothness penalty term is defined to suppress drastic adjustments in laser power; its calculation formula is as follows:

[0039] R smooth =

[0040] Among them, P t-1 Let P be the laser power at time t-1. t Let k be the laser power at time t-1, and k3 be the contribution weight of the power smoothness penalty term;

[0041] Adding the above three items together yields the reward function.

[0042] In one embodiment, the target melt depth d target Determined in the following ways:

[0043] Multiple welding experiments were conducted using the reference welding power to determine the actual penetration depth, and the average penetration depth under stable welding conditions was taken as the target penetration depth.

[0044] In one embodiment, the step of employing a round-based policy learning mechanism for multi-pass laser welding, wherein the policy network executes closed-loop control in real time during each welding pass, recording state, action, and reward data; and after each welding pass, using the value network to perform distributed value modeling and risk assessment on the state, action, and reward data, and updating the policy network parameters, includes:

[0045] Using the benchmark welding power as the initial power for each weld, during the welding process, every preset control cycle, the strategy network outputs a laser power adjustment amount based on the real-time melt depth status at the current moment, controls the laser to adjust the actual output power, and simultaneously records the status, actions and rewards of each time step online to form a complete round trajectory data.

[0046] After a single welding pass is completed, based on the round trajectory data, the quantile distribution value head of the value network is used to perform distributed modeling for each state, outputting a value distribution consisting of 32 quantiles; then, the state value after risk assessment is calculated using the entropy risk measurement formula. And the advantage function is calculated using the generalized advantage estimation method;

[0047] Based on the aforementioned advantage function and the state value after risk assessment The objective function for near-end policy optimization is calculated, the policy update magnitude is limited by the shear probability ratio, and the parameters of the policy network and value network are updated using stochastic gradient descent. After completing one round of updates, the next welding process begins until the total reward converges.

[0048] In one embodiment, the step of repeating step S6 until convergence to obtain a trained policy network, used for subsequent closed-loop control of laser welding penetration stability, includes:

[0049] After each welding pass is completed, the total reward value of the reward function is recorded, and it is determined whether the total reward value of N consecutive welding passes meets the convergence condition. The convergence condition is: the fluctuation range of the total reward value of N consecutive welding passes is less than a preset threshold, and the total reward value no longer increases significantly, where N is an integer greater than or equal to 5.

[0050] When the convergence condition is met, training is stopped, the weight parameters of the current policy network are saved as the trained policy network, and a comparative welding experiment without control is conducted using the benchmark welding power. The weld penetration data of the comparative welding experiment is compared with the weld penetration data controlled by the trained policy network to verify the degree of improvement in the average error and standard deviation of the penetration.

[0051] The trained strategy network is embedded into the industrial control system. In subsequent laser welding tasks, the trained strategy network is directly used to respond online to the real-time weld depth status and output laser power adjustment actions, thereby achieving closed-loop control of weld depth stability without the need for further training.

[0052] Therefore, this application has the following beneficial effects:

[0053] High-precision real-time melt depth monitoring capability: This application employs the TimesNet model for multi-period-scale time-series modeling of plasma spectral signals. Compared to traditional LSTM or CNN, TimesNet extracts multiple main periods through Fast Fourier Transform, folding one-dimensional time-series data into a two-dimensional tensor, while simultaneously capturing details within the period and the evolution patterns during the period, significantly improving the ability to extract complex oscillation features during laser welding. Experiments demonstrate that the average melt depth prediction error is only 0.15 mm, with an average error accuracy of 2.05%, providing accurate and reliable state feedback for closed-loop control.

[0054] Risk-Sensitive Robust Control Strategy: This application innovatively introduces a quantile distributed value head and entropy risk measurement into a deep reinforcement learning framework. Conventional PPO only focuses on maximizing expected return and cannot identify high-risk control actions. This application uses a quantile head to characterize the uncertainty of the value distribution and combines it with an entropy risk measurement to quantify the risk of the value distribution, automatically avoiding high-risk actions coupled with keyhole oscillations that may induce instability. Experimental results show that the standard deviation of weld penetration is reduced by 42.1% after training, significantly improving the stability of the welding process.

[0055] Multi-objective integrated optimization reward mechanism: This application designs a triple reward function that includes penetration depth tracking error, penetration depth stability, and power smoothness. This design not only pursues penetration depth close to the target value, but also suppresses keyhole instability by penalizing penetration depth jumps between adjacent time moments, and protects the laser and reduces heat-affected zone fluctuations by penalizing drastic power adjustments, thus achieving synergistic optimization of welding quality and equipment life.

[0056] Adaptive Learning and Engineering Generalization Capabilities: This application utilizes a reinforcement learning framework based on proximal policy optimization. It does not rely on precise physical models or pre-set policies, but autonomously optimizes control policies through turn-based interaction. Experiments show that this method is not only applicable to aluminum alloys but also effective for various materials such as stainless steel, demonstrating good process adaptability and promising engineering deployment prospects.

[0057] Significant quality improvement: Comparative verification experiments show that after control by this application, the average error of weld penetration depth decreased from 0.693 mm to 0.310 mm (a reduction of 55.3%), and the standard deviation decreased from 0.694 mm to 0.402 mm (a reduction of 42.1%). The uniformity of weld penetration depth in the longitudinal section was significantly improved, and defects such as spatter and porosity were greatly reduced, resulting in a significant increase in the welding yield. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart illustrating a closed-loop control method for laser welding penetration stability based on deep reinforcement learning, provided for an embodiment of this application;

[0060] Figure 2 This is a schematic diagram of the input / output structure of the melt depth monitoring model provided in the embodiments of this application;

[0061] Figure 3 This is a schematic diagram illustrating the training and testing results of the melt depth monitoring model provided in the embodiments of this application;

[0062] Figure 4 This is a schematic diagram of the deep reinforcement learning controller structure provided in an embodiment of this application;

[0063] Figure 5 A schematic diagram of the value network structure provided in the embodiments of this application;

[0064] Figure 6 A schematic diagram illustrating the training process of a deep reinforcement learning controller provided in this application embodiment;

[0065] Figure 7 A schematic diagram showing the comparison results of laser welding penetration stability control provided in the embodiments of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0068] To address the shortcomings of existing technologies, this application proposes a closed-loop control method for laser welding weld penetration stability based on deep reinforcement learning. First, this method utilizes a TimesNet model to perform multi-period-scale time-series modeling of the plasma spectral signal, achieving high-precision real-time monitoring of weld penetration. Second, a deep reinforcement learning controller is constructed based on a near-end policy optimization framework. A quantile-distributed value head is introduced into the value network to characterize the potential value distribution of control actions, and entropy risk measurement is combined to quantify the risk of the value distribution, obtaining the action value after risk assessment, and optimizing the control strategy accordingly. Finally, a comprehensive reward function incorporating weld penetration tracking error, weld penetration stability, and power smoothness is designed, and adaptive control is achieved through round-robin policy learning. This application can effectively avoid high-risk control actions, improve the robustness of weld penetration closed-loop control, and overcome the shortcomings of existing technologies in quantifying and controlling action risk.

[0069] This application provides a closed-loop control method for laser welding penetration stability based on deep reinforcement learning, including steps S1 to S7, as described above. Figure 1 , Figure 1 This is a system flowchart of a closed-loop control method for laser welding penetration stability based on deep reinforcement learning.

[0070] Step S1: Real-time acquisition of plasma spectral signals generated during the welding process, and selection of several wavelengths of the plasma spectral signals as monitoring data;

[0071] Step S2: Input the monitoring data within the latest few frames of the time window into the pre-trained TimesNet melt depth monitoring model. The TimesNet model extracts the temporal features of the spectral data based on multi-period scale transformation and outputs the melt depth prediction value at the latest moment as the real-time melt depth status.

[0072] Step S3: Construct a deep reinforcement learning controller based on near-end policy optimization. The controller includes a policy network and a value network. The policy network is used to output process parameters online to adjust actions according to the real-time melt depth state input. The value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network adopt a long short-term memory network structure and use the state sequence of the past period as input.

[0073] Step S4: Introduce a quantile distribution value head into the value network to characterize the potential value distribution corresponding to different control actions; and quantify the risk of the value distribution based on the quantile distribution value head and the entropy risk metric to obtain the action value after risk assessment, so as to optimize the controller's strategy.

[0074] Step S5: Set the reward function of the deep reinforcement learning controller, wherein the reward function includes a depth tracking error term, a depth stability penalty term, and a power smoothness penalty term;

[0075] Step S6: A round-based strategy learning mechanism is used for multi-pass laser welding. During each welding process, the strategy network executes closed-loop control in real time and records the status, action, and reward data. After each welding is completed, the value network is used to perform distributed value modeling and risk assessment on the status, action, and reward data, and the strategy network parameters are updated.

[0076] Step S7: Repeat step S6 until convergence, and obtain the trained policy network, which will be used for subsequent closed-loop control of laser welding penetration stability.

[0077] Specifically, in this embodiment, referring to Figure 1 , Figure 1 This is a system flowchart of a closed-loop control method for laser welding penetration stability based on deep reinforcement learning.

[0078] Step S1: Real-time acquisition of plasma spectral signals generated during the welding process, and selection of several wavelengths of the plasma spectral signals as monitoring data.

[0079] A high-resolution fiber optic spectrometer mounted on the laser welding head was used, with a sampling frequency set to 1000 Hz and an integration time of 1 ms. Before welding, the spectrometer underwent wavelength calibration and intensity calibration to ensure that the signal's temporal resolution and signal-to-noise ratio met requirements. The acquired raw spectral signals were sequentially processed through moving average filtering (window length of 5 points) and baseline correction to eliminate background noise and random interference. Based on the elemental composition of the aluminum alloy, 10 candidate wavelengths were pre-selected, and specific wavelengths highly correlated with changes in melt depth were selected as monitoring data through melt depth correlation analysis. The 10 wavelengths selected in this embodiment include: Mg-I atomic spectral line 382.90 nm, Al-I atomic spectral line 394.35 nm, Al-I atomic spectral line 396.26 nm, Mn-I atomic spectral line 403.07 nm, Cr-I atomic spectral line 427.44 nm, AlO molecular spectral line 484.25 nm, AlO molecular spectral line 486.62 nm, Mg-I atomic spectral line 517.27 nm, Mg-I atomic spectral line 518.33 nm, and AlO molecular spectral line 518.33 nm.

[0080] Step S2: Input the monitoring data within the latest few frames of the time window into the pre-trained TimesNet melt depth monitoring model. The TimesNet model extracts the temporal features of the spectral data based on multi-period scale transformation and outputs the melt depth prediction value at the latest moment as the real-time melt depth status.

[0081] In this embodiment, the time window length is set to 32 frames. The specific processing procedure of the TimesNet model is as follows: First, a Fast Fourier Transform is performed on the 32 frames × 10 channels of spectral data to extract the 8 main periods with the highest spectral energy (K=8); second, for each main period, the original one-dimensional time series data is folded into a two-dimensional tensor according to the period length; then, a two-dimensional convolutional neural network is used to extract convolutional features from each two-dimensional tensor, while capturing the time pattern within the period and the evolution pattern between periods; finally, the two-dimensional convolutional feature maps corresponding to the 8 periods are concatenated at the channel level, and the latest melt depth prediction value is output through a fully connected layer. In this embodiment, the TimesNet model uses the Adam optimizer with a learning rate of 0.0001, a training batch size of 64, and 200 iteration steps. After training, the model's mean absolute error (MAE) for melt depth prediction on the test set is 0.15 mm, and the average error accuracy is 2.05%.

[0082] Step S3: Construct a deep reinforcement learning controller based on proximal policy optimization.

[0083] The deep reinforcement learning controller comprises a policy network and a value network. The policy network is used to adjust actions based on the real-time melt depth state input, outputting process parameters online. The value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network adopt a Long Short-Term Memory (LSTM) network structure, using a sequence of states over a past period as input. In this embodiment, the state space is defined as the real-time melt depth state, and the action space is defined as the laser power adjustment amount, with the adjustment range limited to ±20% of the base power of 4400 W (i.e., 3520 W to 5280 W). The policy network consists of one LSTM layer and two fully connected layers. The LSTM layer uses the melt depth state sequence within the past 8 frames as input to extract temporal hidden features. The fully connected layers map the hidden features to the mean and standard deviation of a Gaussian distribution and output the laser power adjustment amount at the current moment through sampling. The value network consists of one LSTM layer and a quantized distributed value head. The LSTM layer uses the same 8-frame melt depth state sequence as input to extract temporal hidden features. The learning rate for the policy network was set to 0.0001, the learning rate for the value network was set to 0.001, and the discount factor γ = 0.9.

[0084] Step S4: Introduce a quantile distribution value head into the value network to characterize the potential value distribution corresponding to different control actions; and quantify the risk of the value distribution based on the quantile distribution value head and the entropy risk metric to obtain the action value after risk assessment, so as to optimize the controller's strategy.

[0085] A quantile-distributed value head is introduced at the output of the value network to replace the traditional single-point value estimation. This value head outputs value estimates corresponding to 32 uniformly distributed quantiles, forming a complete value distribution curve that can accurately characterize the range of returns and uncertainties that different regulatory actions may bring. Based on this, the value distribution is quantified for risk by combining an entropy risk measurement function. By setting a risk sensitivity parameter β=-0.8, the controller has risk aversion capabilities, actively filtering out high-risk regulatory actions that may induce keyhole oscillations and process instability.

[0086] In this embodiment, the quantile distribution value head contains 32 quantile outputs, each corresponding to an independent fully connected branch. The temporal hidden features extracted by LSTM are mapped to these 32 quantile points, forming the quantile distribution of state values. Based on this, the entropy risk metric formula is used to calculate the action value after risk assessment. This risk-assessed action value is then used as the target value of the value network and substituted into the advantage function calculation for near-end policy optimization, replacing the conventional scalar value estimation. This allows the policy network to prioritize high-value actions with more stable value distributions and lower tail risk during updates.

[0087] Step S5: Design a multi-objective integrated reward function to guide the controller to learn the optimal control strategy.

[0088] In this embodiment, the reward function consists of three parts: a melt depth tracking error term, a melt depth stability penalty term, and a power smoothness penalty term. The melt depth tracking error term penalizes the degree to which the melt depth deviates from the target value, the melt depth stability penalty term suppresses drastic fluctuations in melt depth, and the power smoothness penalty term avoids frequent and large adjustments to the laser power. Based on the control priority, weighting coefficients k1=5, k2=5, and k3=2 are assigned to the three penalty terms, respectively.

[0089] Step S6: Use a round-based strategy learning mechanism to perform multi-pass laser welding.

[0090] A baseline power of 4400 W was used as the initial power for each welding pass. During the welding process, every 15 ms control cycle, the policy network outputs a laser power adjustment based on the real-time weld depth, controlling the laser to adjust its actual output power. Simultaneously, the state, actions, and rewards at each time step are recorded online, forming complete round trajectory data. After a single welding pass is completed, the quantile distribution value head of the value network is used to perform distributed modeling for each state, outputting a value distribution consisting of 32 quantiles. The state value after risk assessment is calculated using entropy risk measurement, and the advantage function is calculated using the generalized advantage estimation method. Based on the advantage function and the state value after risk assessment, the objective function for near-end policy optimization is calculated. The policy update amplitude is limited by the shear probability ratio, and the parameters of the policy network and value network are updated using stochastic gradient descent.

[0091] Step S7: Repeat step S6 until convergence, and obtain the trained policy network.

[0092] In this embodiment, the convergence condition is set as follows: the fluctuation range of the total reward value for 10 consecutive welds is less than a preset threshold, and the total reward value no longer increases significantly. The training reached convergence after 84 rounds of updates, with a total reward value of -4124.02. Alternatively, the controller is considered converged when the fluctuation range of the total reward value for 10 consecutive rounds is less than 5% and the standard deviation of the weld penetration depth tends to stabilize. After training, a comparative verification experiment was conducted: the average error of the weld penetration depth without regulation was 0.693 mm, and the standard deviation was 0.694 mm; the average error of the weld penetration depth regulated by the controller of this application was 0.310 mm (a reduction of 55.3%), and the standard deviation was 0.402 mm (a reduction of 42.1%). The trained strategy network was embedded into the industrial control system for subsequent online closed-loop control of the laser welding penetration depth stability.

[0093] In one embodiment, the plasma spectral signal in step S1 includes one or more of the following: Mg-I atomic spectral lines, Al-I atomic spectral lines, AlO molecular spectral lines, and Cr-I atomic spectral lines, specifically selected from at least five wavelengths including but not limited to 382.90 nm, 394.35 nm, 396.26 nm, 403.07 nm, 427.44 nm, 484.25 nm, 486.62 nm, 517.27 nm, and 518.33 nm.

[0094] Specifically, in this embodiment, the theoretical basis for selecting the plasma spectral signals of the above-mentioned specific wavelengths as monitoring data for the aluminum alloy laser welding process is as follows: Mg-I atomic spectral lines (382.90 nm, 517.27 nm, 518.33 nm) are sensitive to the concentration changes of metal vapor in the keyhole and can reflect the transient fluctuations in the keyhole opening state; Al-I atomic spectral lines (394.35 nm, 396.26 nm) are the main emission spectral lines of the aluminum alloy matrix elements, and their intensity is strongly correlated with the laser energy absorption efficiency and the molten pool volume; AlO molecular spectral lines (484.25 nm, 486.62 nm, 518.33 nm) originate from the chemical reaction between aluminum and oxygen in the plasma plume, and their intensity indirectly characterizes the density and temperature of the plasma cloud above the keyhole, which helps to determine whether the keyhole is in a stable open state; Cr-I atomic spectral lines (427.44 nm), as characteristic spectral lines of trace alloying elements, can assist in identifying changes in the molten pool flow state.

[0095] In practical applications, considering the limitations of spectrometer resolution and signal-to-noise ratio, this embodiment preferably selects five of the aforementioned wavelengths as monitoring features: Mg-I 382.90 nm, Al-I 394.35 nm, Al-I 396.26 nm, AlO 484.25 nm, and Mg-I 517.27 nm. These five wavelengths cover both atomic and molecular spectral lines, taking into account both keyhole depth sensitivity and plasma plume characteristics. During signal acquisition, the spectrometer sampling frequency is set to 1000 Hz. The raw signal of each wavelength channel is first subjected to a third-order Butterworth low-pass filter (cutoff frequency 5 kHz) to eliminate high-frequency electronic noise, then smoothed by a moving average filter (window length 5 points), and finally normalized to the maximum and minimum values, mapping the signal amplitude of each channel to the [0, 1] interval. The multi-channel spectral signal after the above preprocessing is used as input to the TimesNet model for subsequent real-time melt depth monitoring.

[0096] To ensure the effectiveness of the selected wavelengths, this embodiment also conducted a wavelength correlation pre-screening experiment: 10 welding experiments were performed under a baseline welding power of 4400 W, and the spectral signals of the candidate wavelengths and the corresponding actual weld depth data were collected simultaneously. The Pearson correlation coefficient between the signal intensity of each wavelength and the weld depth was calculated. Wavelengths with an absolute correlation coefficient greater than 0.6 were selected as the final monitoring features, and wavelengths with weak correlation were eliminated to reduce the model input dimension and improve computational efficiency. In this embodiment, the five wavelengths ultimately retained all met the above correlation requirements.

[0097] In one embodiment, the step of inputting monitoring data within the latest several frames of time window into a pre-trained TimesNet melt depth monitoring model, wherein the TimesNet model extracts temporal features from the spectral data based on multi-period scale transformation and outputs the melt depth prediction value at the latest moment as the real-time melt depth status, includes:

[0098] Multiple laser welding experiments were conducted using a benchmark welding power to obtain metallographic images of the longitudinal section of the weld and extract actual weld penetration data; a training dataset was constructed by pairing spectral monitoring data with actual weld penetration data.

[0099] Using the Adam optimizer, with a learning rate of 0.0001, a batch size of 64, and 200 iterations, a TimesNet model for multi-period-scale time series modeling was obtained.

[0100] Specifically, in this embodiment, a training dataset is first constructed. Eight aluminum alloy laser welding experiments were conducted with a baseline welding power of 4400 W and a weld length of 150 mm. During the welding process, a spectrometer simultaneously acquired spectral signals of 10 specific wavelengths at a sampling frequency of 1000 Hz. After welding, the weld was longitudinally cut along the centerline, and after grinding, polishing, and Keller's reagent etching, metallographic images of the longitudinal section of the weld were obtained. The actual weld depth data corresponding to each sampling time was extracted through image processing and matched with the spectral data according to time alignment, resulting in 32,000 training samples and 8,000 validation samples.

[0101] Next, the TimesNet model was trained. The model structure is as follows: the input layer receives 32 frames of spectral data with 10 channels; 8 dominant periods are extracted through Fast Fourier Transform; each period is folded into a 2D tensor, and features are extracted through 2D convolution; finally, the melt depth prediction value is output through a fully connected layer. The training parameters are as follows: the optimizer is Adam, the learning rate is 0.0001, the training batch size is 64, and the number of iterations is 200. The loss function is mean squared error, and Dropout (dropout rate 0.2) is added before the fully connected layer to prevent overfitting. Figure 2This paper illustrates the real-time melt depth monitoring process based on TimesNet in the embodiments of this application. The upper part of the figure shows the time series diagrams of plasma spectral signals at three typical wavelengths, corresponding to characteristic spectral lines such as MgI 382.90 nm (blue), AlI 394.35 nm (green), and AlO 539.43 nm (red). The horizontal axis represents time frames (0~2000 frames), and the vertical axis represents signal intensity. The gray shaded area in the figure represents the monitoring data within the latest few frames of the time window, which is input into the TimesNet model. TimesNet extracts multiple main periods through Fast Fourier Transform, folds the one-dimensional time series data into a two-dimensional tensor, and uses a two-dimensional convolutional neural network to simultaneously capture features within and during the period, finally outputting the real-time melt depth prediction value at the current moment (see...). Figure 2 (See the orange curve below). This process achieves high-precision online mapping from multi-channel spectral signals to real-time melt depth, providing accurate state feedback for subsequent deep reinforcement learning controllers.

[0102] In one embodiment, the construction of a deep reinforcement learning controller based on proximal policy optimization includes a policy network and a value network. The policy network is used to adjust actions by outputting process parameters online based on the input real-time melt depth state, and the value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network adopt a long short-term memory network structure, and the step of using a sequence of states over a past period as input includes:

[0103] The continuous N-frame melt depth state sequence obtained in real time through the TimesNet melt depth monitoring model is used as the controller input state quantity, and the continuous adjustment of laser power is used as the action space.

[0104] A strategy network based on a long short-term memory network is constructed. The continuous N-frame melt depth state sequence is used as input. The temporal dependency features of melt depth evolution are extracted through the long short-term memory network layer, and then the laser power control action is mapped and output through a fully connected layer.

[0105] A value network based on a long short-term memory network is constructed. The input is a continuous N-frame melt depth state sequence, which is the same as the policy network. Temporal features are extracted through the long short-term memory network layer, and then connected to the quantile distribution value head to output the quantile distribution of the state value, providing a distribution data basis for entropy risk measurement.

[0106] Specifically, in this embodiment, the state space and action space are first defined. The sequence of eight consecutive frames of melt depth state obtained in real time through the TimesNet melt depth monitoring model is used as the controller input state quantity, i.e., state s. t =[d t-7 ,d t-6 ,…,d t ], where dt This represents the predicted melt depth at the current moment. The continuous adjustment of the laser power is used as the action space, with the adjustment range limited to ±20% of the base power of 4400W to ensure the laser operates within a safe power range. The action output adopts a Gaussian distribution strategy, and the strategy network outputs the action mean and standard deviation.

[0107] Secondly, a policy network based on a long short-term memory network is constructed. Specifically, it is represented as follows:

[0108]

[0109] in The hidden feature vectors extracted by the policy network. Represents the policy network, The power adjustment factor is used; the policy network consists of an input layer, an LSTM layer, a fully connected layer, and an output layer. The input layer receives 8 consecutive frames of melt depth state sequences, with a data dimension of 8×18×1. The LSTM layer contains 128 hidden neurons, which take the state sequences as input, extract the temporal dependency features of melt depth evolution, and output a hidden state vector. .

[0110] Next, construct a value network based on a long short-term memory network. Specifically, this is represented as:

[0111]

[0112] in The hidden feature vectors extracted from the value network, where For a value network, the state is represented. The corresponding value estimation. The value network and policy network share the same input structure, namely, a sequence of 8 consecutive melt-depth states. The LSTM layer also contains 128 hidden neurons, which extract temporal features and output a hidden state vector. Unlike conventional value networks, the value network in this embodiment connects a quantile-distributed value head after the LSTM layer. This quantile head contains 32 independent fully connected branches, each branch will... Mapped to a quantile output (i=1,2,…,32), forming the probability distribution of state values. The learning rate of the value network is set to 0.001, and the quantile regression loss function adopts the quantile Huber loss to enhance the stability of training. Figure 3This paper presents the training and testing results of the TimesNet weld depth monitoring model in this embodiment. The top-left subplot shows the training history. As the number of iterations (0-60 steps) increases, both the training loss and validation loss decrease rapidly and tend to converge, with the final training loss being 0.074 and the validation loss being 0.053. Simultaneously, the training MAE and validation MAE also continuously decrease, with the final mean absolute error (MAE) obtained from training being 0.205 mm and from validation being 0.147 mm. The top-right subplot shows the error statistics matrix, which intuitively reflects the model's prediction accuracy in different weld depth ranges. Within the actual weld depth range of 6.7-8.0 mm, the model's prediction accuracy reaches 90.9%, and it also achieves 87.8% and 82.0% in the 5.4-6.7 mm and 8.0-9.3 mm ranges, respectively, demonstrating the model's good performance across the entire range. The bottom subplot shows the variation of the weld depth prediction accuracy of the complete test weld over time frames (0-5000 frames). The red curve represents the predicted weld penetration value, and the blue curve represents the actual weld penetration value. The two curves highly overlap, indicating that the model can accurately track the dynamic fluctuations in weld penetration. Statistical analysis shows that the mean absolute error (MAE) of the weld penetration prediction for this test weld is only 0.150 mm, with an average error accuracy of 2.05%, verifying the high-precision real-time monitoring capability of the TimesNet model. Through the collaborative work of the policy network and value network, the controller can output control actions based on the time-series weld penetration state and provide a complete value distribution data foundation for subsequent entropy risk measurement, thereby achieving risk-sensitive deep reinforcement learning control.

[0113] Figure 4This paper illustrates the closed-loop control agent architecture for laser welding weld penetration depth based on deep reinforcement learning in this embodiment. The architecture consists of two parts: a laser welding environment and a closed-loop control agent. In the laser welding environment, a spectrometer collects plasma signals and outputs the weld penetration state in real time through a TimesNet weld penetration monitoring model. This state is input to the control agent. The core of the agent includes a policy network and a value network. The policy network, based on a long short-term memory network, outputs power-adjusted actions according to the current state, forming a closed loop through power adjustment arrows. The value network on the right uses distributed modeling, characterizing the value distribution of each state through quantile distribution value heads (as shown in the green bar chart). Based on this, an entropy risk measurement module is introduced to quantify the risk of the value distribution, obtaining the action value after risk assessment. Subsequently, the advantage function is calculated using generalized advantage estimation, and finally, the policy network parameters are updated through proximal policy optimization. The shear probability ratio (1-ε to 1+ε) diagram in the lower right corner of the figure illustrates the mechanism of proximal policy optimization in preventing training oscillations by limiting the policy update amplitude. Through the architecture of state perception-risk quantification-strategy optimization, this method can effectively avoid high-risk control actions and improve the robustness of closed-loop control of laser welding penetration depth.

[0114] Figure 5 This application demonstrates a value network structure based on distributed modeling, specifically designed to output a multi-quantile value distribution. The network comprises two parallel paths: an upper LSTM fully connected layer for extracting temporal features of the melt depth state sequence; and a lower quantile fully connected layer that independently learns the value features corresponding to different quantiles. The two feature paths are combined by a fusion module to output a value distribution composed of multiple quantiles, rather than a single scalar value. This structure provides a distributed data foundation for subsequent entropy risk measurement, enabling the controller to quantify the uncertainty of control actions, thereby avoiding high-risk actions and improving the robustness of melt depth closed-loop control.

[0115] In one embodiment, the step of introducing a quantile distribution value head into the value network to characterize the potential value distribution corresponding to different regulatory actions; and quantifying the risk of the value distribution based on the quantile distribution value head combined with an entropy risk metric to obtain the action value after risk assessment, so as to optimize the controller's strategy, includes:

[0116] At the end of the value network, a quantile distribution value head is constructed. The quantile distribution value head contains N quantile outputs, each quantile corresponding to an independent fully connected branch. The temporal hidden features extracted by the long short-term memory network are mapped to N quantile points to form the quantile distribution of state value, where N is an integer greater than or equal to 8 and less than or equal to 64.

[0117] Based on the quantile distribution, the action value after risk assessment is calculated using the entropy risk measurement formula, which is:

[0118]

[0119] in, The value sample corresponding to the i-th quantile in the quantile distribution value head output, where β is a risk-sensitive parameter;

[0120] The value of the action after the risk assessment As the target value of the value network, it is substituted into the advantage function calculation of the near-end policy optimization, and the parameters of the policy network and value network under the near-end policy optimization framework are iteratively updated accordingly.

[0121] Specifically, in this embodiment, a quantile-distributed value head is first constructed. After the LSTM layer of the value network, a quantile-distributed value head is constructed. This quantile head contains 32 quantile outputs (N=32), with each quantile corresponding to an independent fully connected branch. Specifically, the 128-dimensional hidden state vector output from the LSTM layer is simultaneously input into the 32 independent fully connected branches. Each branch contains a hidden layer (64 neurons) and an output node, outputting the value sample corresponding to the quantile. (i=1,2,…,32). These 32 quantiles are evenly distributed within the interval [0.01, 1], covering the complete value distribution range from low risk to high risk. The weight parameters of each branch are initialized and updated independently, enabling each quantile to adaptively learn different regions of the value distribution. The quantile head is trained using the quantile Huber loss function, which combines the robustness of Huber loss with the unbiasedness of quantile regression, effectively handling outliers in the value distribution.

[0122] Secondly, risk quantification is performed using entropy risk measurement. This is based on 32 value samples output from the quantile distribution value head. The entropy risk measurement formula is used to calculate the action value after risk assessment. The entropy risk measurement formula is:

[0123]

[0124] in, The quantile is the value sample corresponding to the i-th quantile in the quantile distribution value head output, and β is a risk-sensitive parameter. In this embodiment, β = -0.8. This value makes the entropy risk measure exhibit risk aversion, that is, for actions with a large variance in the value distribution, their risk assessment value will be lowered, thus inhibiting them in strategy optimization. When β approaches 0, the measure degenerates into the expected value (i.e., the value assessment of a conventional PPO); when β is positive, it exhibits risk preference. This embodiment selects β = -0.8 to maintain the ability to avoid high-risk actions while avoiding excessive conservatism that leads to a slow control response.

[0125] Furthermore, the risk assessment results are integrated into the PPO optimization framework. The action value calculated based on the risk assessment is used as the target value of the value network and substituted into the advantage function calculation for near-end policy optimization. This advantage function is then used to calculate the policy loss and value loss of PPO, and the parameters of the policy network and value network are iteratively updated. In this way, the policy network prioritizes actions with more stable value distributions and lower tail risks during updates, thereby automatically avoiding high-risk control actions coupled with keyhole oscillations in the weld penetration closed-loop control, thus improving the robustness of the welding process.

[0126] In one embodiment, the step of setting the reward function of the deep reinforcement learning controller, the reward function including a depth-of-sight tracking error term, a depth-of-sight stability penalty term, and a power smoothness penalty term, includes:

[0127] Define a melt depth tracking error term to penalize the deviation between the real-time melt depth and the target melt depth. The calculation formula is as follows:

[0128]

[0129] Where, d t Let d be the real-time melt depth at time t. target The target melting depth is preset, and k1 is the contribution weight of the tracking error term;

[0130] A melt depth stability penalty term is defined to suppress drastic jumps in melt depth between adjacent time steps. The calculation formula is as follows:

[0131]

[0132] Where, d t-1 Let k be the real-time melting depth at time t-1, and k2 be the contribution weight of the stability penalty term;

[0133] A power smoothness penalty term is defined to suppress drastic adjustments in laser power; its calculation formula is as follows:

[0134] R smooth =

[0135] Among them, P t-1 Let P be the laser power at time t-1. t Let k be the laser power at time t-1, and k3 be the contribution weight of the power smoothness penalty term;

[0136] Adding the above three items together yields the reward function.

[0137] Specifically, in this embodiment, a melt depth tracking error term is first defined. This term penalizes the deviation between the real-time melt depth and the target melt depth, prompting the controller to stabilize the melt depth near the target value. The calculation formula is:

[0138]

[0139] Where, d t Let d be the real-time melt depth at time t. target The target melting depth is defined as k1, where k1 is the contribution weight of the tracking error term; in this embodiment, the target melting depth d is... target Through multiple experiments at the reference welding power, the average penetration depth under stable welding conditions was determined, i.e., d. target =6.702. The weight k1 is set to 5, so that when the melt depth deviation is 0.5 mm, the tracking error term penalty is about -1.25, giving the controller a clear optimization direction.

[0140] Next, a melt depth stability penalty term is defined. This term is used to suppress drastic jumps in melt depth between adjacent time points, preventing discontinuities in melt depth caused by keyhole oscillation or sudden changes in molten pool flow. Its calculation formula is:

[0141]

[0142] Where, d t-1 Let be the real-time melt depth at time t-1, and k2 be the contribution weight of the stability penalty term; in this embodiment, k2 is set to 5. This penalty term adopts an absolute value form, which can apply equal penalty to melt depth jumps in any direction, thereby suppressing high-frequency oscillations and promoting a smooth transition of melt depth.

[0143] Next, a power smoothness penalty term is defined. This term suppresses drastic adjustments to the laser power, protecting the laser's optical components from thermal shock caused by frequent, large-amplitude adjustments, while also reducing the negative impact of power fluctuations on welding stability. Its calculation formula is as follows:

[0144] A power smoothness penalty term is defined to suppress drastic adjustments in laser power; its calculation formula is as follows:

[0145] R smooth =

[0146] Among them, P t-1 Let P be the laser power at time t-1. t Let k be the laser power at time t-1, and k3 be the contribution weight of the power smoothness penalty term; in this embodiment, k3 is set to 2. This penalty term takes a squared form, imposing a larger penalty on large power adjustments, thus encouraging the controller to use gradual power adjustment.

[0147] Finally, adding the above three items together, we obtain the comprehensive reward function. :

[0148]

[0149] By optimizing the mesh search, the welding depth tracking accuracy was ensured while effectively suppressing abrupt changes in weld depth and power fluctuations, thus achieving comprehensive optimization of welding quality and equipment operation stability.

[0150] In one embodiment, the target melt depth d target Determined in the following ways:

[0151] Multiple welding experiments were conducted using the reference welding power to determine the actual penetration depth, and the average penetration depth under stable welding conditions was taken as the target penetration depth.

[0152] Specifically, in this embodiment, multiple welding experiments were first conducted using a reference welding power to determine the actual penetration depth. The average penetration depth under stable welding conditions was then used as the target penetration depth. The specific operation was as follows: Under the process parameters of a reference welding power of 4400 W, a welding speed of 10 mm / s, a decoking amount of 3 mm, and a shielding gas flow rate of 15 L / min, eight aluminum alloy laser welding experiments were performed, with each weld length being 150 mm. All process parameters were kept constant during the welding process, and no closed-loop control was applied to obtain the natural penetration depth characteristics of the material under stable conditions.

[0153] After welding, all eight welds were longitudinally cut along the weld centerline using electrical discharge machining (EDM), and the longitudinal sections were successively ground and polished. Keller's reagent was used to etch the longitudinal sections for 20 seconds to clearly reveal the weld fusion line outline. Subsequently, high-resolution scanners were used to acquire longitudinal section images of the welds, and image processing software was used to extract the penetration depth data along the length of each weld. For each weld, 10 penetration depth measurement points were selected at each of the starting, middle, and ending sections under stable welding conditions, for a total of 30 measurement points. The average value was taken as the representative penetration depth of that weld.

[0154] The eight representative penetration values ​​obtained from eight experiments were statistically analyzed. After removing outliers, the average value was calculated to obtain the average penetration under stable welding conditions. In this embodiment, for aluminum alloy material, when the reference power is 4400W, the welding speed is 10 mm / s, and the decoking amount is 3mm, the target penetration depth d was determined by the above method. target =6.702mm. This value serves as the target melting depth for subsequent deep reinforcement learning controller training and is used in the calculation of the tracking error term in the reward function.

[0155] It should be noted that the above-mentioned method for determining the target penetration depth is universal. When the welding material, plate thickness, or process parameters change, it is only necessary to repeat the above-mentioned benchmark welding experiment and penetration depth measurement procedure to obtain the target penetration depth value under the corresponding working conditions, thereby ensuring the adaptability of the control method of this application in different application scenarios.

[0156] In one embodiment, the step of employing a round-based policy learning mechanism for multi-pass laser welding, wherein the policy network executes closed-loop control in real time during each welding pass, recording state, action, and reward data; and after each welding pass, using the value network to perform distributed value modeling and risk assessment on the state, action, and reward data, and updating the policy network parameters, includes:

[0157] Using the benchmark welding power as the initial power for each weld, during the welding process, every preset control cycle, the strategy network outputs a laser power adjustment amount based on the real-time melt depth status at the current moment, controls the laser to adjust the actual output power, and simultaneously records the status, actions and rewards of each time step online to form a complete round trajectory data.

[0158] After a single welding pass is completed, based on the round trajectory data, the quantile distribution value head of the value network is used to perform distributed modeling for each state, outputting a value distribution consisting of 32 quantiles; then, the state value after risk assessment is calculated using the entropy risk measurement formula. And the advantage function is calculated using the generalized advantage estimation method;

[0159] Based on the aforementioned advantage function and the state value after risk assessment The objective function for near-end policy optimization is calculated, the policy update magnitude is limited by the shear probability ratio, and the parameters of the policy network and value network are updated using stochastic gradient descent. After completing one round of updates, the next welding process begins until the total reward converges.

[0160] Specifically, in this embodiment, a baseline welding power of 4400 W is initially used as the initial power for each weld pass. During the welding process, every 15 ms control cycle, the strategy network outputs a laser power adjustment amount ΔP based on the real-time weld depth status at the current moment.t This controls the laser to adjust its actual output power. Specifically, the output power at the current moment is P. t =P t-1 +ΔP t , where ΔP t The result is obtained from a Gaussian distribution sampled by the policy network. Simultaneously, the state s at each time step is recorded online. t (i.e., a continuous 8-frame melt depth sequence), Action a t (Power regulation amount) and bonus R t Under the condition of communication delays that exist in the high-frequency asynchronous update process, a complete round trajectory data containing approximately 400 time steps is formed.

[0161] After a single welding pass is completed, based on the aforementioned round trajectory data, a distributed model of each state is performed using the quantile distribution value head of the value network. Specifically, the quantile head outputs 32 quantiles. This constitutes the value distribution under this state. Subsequently, the state value after risk assessment is calculated using the entropy risk measurement formula.

[0162] Simultaneously, the advantage function is calculated using the generalized advantage estimation method. Based on the aforementioned advantage function and the state value after risk assessment... The objective function for near-end policy optimization is calculated. The policy update magnitude is limited by setting a pruning probability ratio to avoid training oscillations caused by excessive policy changes. The value network uses quantile Huber loss as its loss function, simultaneously optimizing the parameters of both the policy and value networks. Stochastic gradient descent is used for parameter updates, with a learning rate of 0.0001 for both the policy and value networks. After one round of updates, the next welding process begins, repeating the above steps.

[0163] Figure 6 This paper illustrates the convergence curve of the total reward and the improvement process of weld depth control stability during the training process of the deep reinforcement learning controller in this embodiment. The line graph above shows that as the number of training rounds increases (horizontal axis 0~80+), the total reward (vertical axis) gradually rises from around -7000 and converges, finally reaching -4124.02 in the 84th round, indicating that the controller strategy has stabilized. The three metallographic images of the weld longitudinal section below visually reflect the gradual improvement of the control effect: in the 21st round of training, the weld depth profile fluctuates violently, and the blue line fluctuates significantly; in the 39th round, the fluctuation is alleviated and the stability is improved; while in the 84th round (after convergence), the weld depth profile is uniform and smooth, with minimal fluctuations.

[0164] In one embodiment, the step of repeating step S6 until convergence to obtain a trained policy network, used for subsequent closed-loop control of laser welding penetration stability, includes:

[0165] After each welding pass is completed, the total reward value of the reward function is recorded, and it is determined whether the total reward value of N consecutive welding passes meets the convergence condition. The convergence condition is: the fluctuation range of the total reward value of N consecutive welding passes is less than a preset threshold, and the total reward value no longer increases significantly, where N is an integer greater than or equal to 5.

[0166] When the convergence condition is met, training is stopped, the weight parameters of the current policy network are saved as the trained policy network, and a comparative welding experiment without control is conducted using the benchmark welding power. The weld penetration data of the comparative welding experiment is compared with the weld penetration data controlled by the trained policy network to verify the degree of improvement in the average error and standard deviation of the penetration.

[0167] The trained strategy network is embedded into the industrial control system. In subsequent laser welding tasks, the trained strategy network is directly used to respond online to the real-time weld depth status and output laser power adjustment actions, thereby achieving closed-loop control of weld depth stability without the need for further training.

[0168] Specifically, in this embodiment, the convergence condition is first determined. After each weld is completed, the total reward value corresponding to that weld process is recorded. In this embodiment, the convergence condition is set as follows: the fluctuation range of the total reward value for 10 consecutive welds (N=10) is less than 5% of the average total reward, and the total reward value no longer shows a significant upward trend. Specifically, the total reward value of the most recent 10 welds is calculated. When the fluctuation range of the total reward value is less than 5% of the average total reward and the increment of the reward value for 5 consecutive welds is less than 1%, the convergence condition is considered met. In this embodiment, the above convergence condition is achieved during the 84th round of training, at which time the total reward value is -4124.02.

[0169] Next, training is stopped and the model is saved. When the convergence condition is met, training is stopped, and the weight parameters of the current policy network are saved as the trained policy network. To verify the control effect, a baseline welding power of 4400 W is used to conduct a comparative welding experiment without any control applied. This experiment is performed three times, and the average value is taken. The weld penetration data from the comparative welding experiment is compared and analyzed with the weld penetration data controlled by the trained policy network. The experimental results show that the average error of the weld penetration without control applied is 0.693 mm, and the standard deviation is 0.694 mm; the average error of the weld penetration controlled by the controller in this embodiment is 0.310 mm (a reduction of 55.3%), and the standard deviation is 0.402 mm (a reduction of 42.1%). Figure 7This paper presents a comparative study of the effects of a deep reinforcement learning controller employing entropy risk measurement on the stability of laser welding weld penetration depth in embodiments of this application. The left-hand "Entropy Risk Measurement" bar chart represents the risk quantification of action value using quantile distribution value heads, guiding the control strategy to execute the "regulation" action. The right-hand side compares the longitudinal cross-sectional images of weld penetration depth and the corresponding penetration depth curves (blue represents actual penetration depth, orange dashed line represents target penetration depth) under "regulation" and "unregulation" conditions. The unregulated weld (section 2) exhibits drastic penetration depth fluctuations, with the actual penetration depth deviating significantly from the target value; its mean absolute error (MAE) is 0.693 mm, and its standard deviation (σ) is 0.694 mm. In contrast, the weld (section 1) regulated by the method of this application exhibits an actual penetration depth closely aligned with the target value, with a uniform profile; its MAE is reduced to 0.310 mm, and its σ is reduced to 0.402 mm. This indicates that after applying risk-sensitive control, the average penetration depth error is reduced by 55.3%, and the standard deviation is reduced by 42.1%, significantly improving the stability and consistency of laser welding penetration depth. Figure 7 The metallographic cross-sections of the uncontrolled weld and the controlled weld of this application are shown. The controlled weld has significantly improved the uniformity of the weld penetration and greatly reduced defects such as spatter and porosity.

[0170] Finally, the model is solidified into the industrial control system. The weight parameters of the trained policy network are exported in ONNX format and deployed to the control computer of the industrial laser welding equipment. In subsequent laser welding tasks, the system directly calls the trained policy network: every 4ms, the real-time weld depth status output by the TimesNet model is acquired, input into the policy network to calculate the laser power adjustment, and output to the laser controller to achieve closed-loop control. This process requires no further training, and the policy network can be directly applied to welding tasks of aluminum alloys with different batches and different plate thicknesses. Testing showed that the solidified policy network consistently maintained a weld depth error within 0.35 mm when welding 100 weld seams continuously, demonstrating good generalization ability and engineering adaptability.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] It should be particularly noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, or of course, by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A closed-loop control method for laser welding penetration stability based on deep reinforcement learning, characterized in that, Includes the following steps: Step S1: Real-time acquisition of plasma spectral signals generated during the welding process, and selection of several wavelengths of the plasma spectral signals as monitoring data; Step S2: Input the monitoring data within the latest few frames of the time window into the pre-trained TimesNet melt depth monitoring model. The TimesNet model extracts the temporal features of the spectral data based on multi-period scale transformation and outputs the melt depth prediction value at the latest moment as the real-time melt depth status. Step S3: Construct a deep reinforcement learning controller based on near-end policy optimization. The controller includes a policy network and a value network. The policy network is used to output process parameters online to adjust actions according to the real-time melt depth state input. The value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network adopt a long short-term memory network structure and use the state sequence of the past period as input. Step S4: Introduce a quantile distribution value head into the value network to characterize the potential value distribution corresponding to different control actions; and quantify the risk of the value distribution based on the quantile distribution value head and the entropy risk metric to obtain the action value after risk assessment, so as to optimize the controller's strategy. Step S5: Set the reward function of the deep reinforcement learning controller, wherein the reward function includes a depth tracking error term, a depth stability penalty term, and a power smoothness penalty term; Step S6: A round-based strategy learning mechanism is used for multi-pass laser welding. During each welding process, the strategy network executes closed-loop control in real time and records the status, action, and reward data. After each welding is completed, the value network is used to perform distributed value modeling and risk assessment on the status, action, and reward data, and the strategy network parameters are updated. Step S7: Repeat step S6 until convergence, and obtain the trained policy network, which will be used for subsequent closed-loop control of laser welding penetration stability.

2. The method according to claim 1, characterized in that, The plasma spectral signal mentioned in step S1 includes one or more of the following: Mg-I atomic spectral lines, Al-I atomic spectral lines, AlO molecular spectral lines, and Cr-I atomic spectral lines, specifically selected from at least five wavelengths including but not limited to 382.90 nm, 394.35 nm, 396.26 nm, 403.07 nm, 427.44 nm, 484.25 nm, 486.62 nm, 517.27 nm, and 518.33 nm.

3. The method according to claim 1, characterized in that, The step of inputting monitoring data within the latest few frames of time window into a pre-trained TimesNet melt depth monitoring model, wherein the TimesNet model extracts temporal features from the spectral data based on multi-period scale transformation, and outputs the melt depth prediction value at the latest moment as the real-time melt depth status, includes: Multiple laser welding experiments were conducted using a benchmark welding power to obtain metallographic images of the longitudinal section of the weld and extract actual weld penetration data; a training dataset was constructed by pairing spectral monitoring data with actual weld penetration data. Using the Adam optimizer, with a learning rate of 0.0001, a batch size of 64, and 200 iterations, a TimesNet model for multi-period-scale time series modeling was obtained.

4. The method according to claim 1, characterized in that, The deep reinforcement learning controller based on near-end policy optimization is constructed. The controller includes a policy network and a value network. The policy network is used to adjust the action by outputting process parameters online according to the input real-time melt depth state. The value network is used to evaluate the value reward corresponding to the current state. Both the policy network and the value network employ a long short-term memory network structure. The steps, which take a sequence of states over a past period as input, include: The continuous N-frame melt depth state sequence obtained in real time through the TimesNet melt depth monitoring model is used as the controller input state quantity, and the continuous adjustment of laser power is used as the action space. A strategy network based on a long short-term memory network is constructed. The continuous N-frame melt depth state sequence is used as input. The temporal dependency features of melt depth evolution are extracted through the long short-term memory network layer, and then the laser power control action is mapped and output through a fully connected layer. A value network based on a long short-term memory network is constructed. The input is a continuous N-frame melt depth state sequence, which is the same as the policy network. Temporal features are extracted through the long short-term memory network layer, and then connected to the quantile distribution value head to output the quantile distribution of the state value, providing a distribution data basis for entropy risk measurement.

5. The method according to claim 1, characterized in that, The steps of introducing quantile distribution value heads into the value network to characterize the potential value distribution corresponding to different control actions, and quantifying the risk of the value distribution based on the quantile distribution value heads combined with entropy risk measurement to obtain the action value after risk assessment, in order to optimize the controller's strategy, include: At the end of the value network, a quantile distribution value head is constructed. The quantile distribution value head contains N quantile outputs, each quantile corresponding to an independent fully connected branch. The temporal hidden features extracted by the long short-term memory network are mapped to N quantile points to form the quantile distribution of state value, where N is an integer greater than or equal to 8 and less than or equal to 64. Based on the quantile distribution, the action value after risk assessment is calculated using the entropy risk measurement formula, which is: ； in, The value sample corresponding to the i-th quantile in the quantile distribution value head output, where β is a risk-sensitive parameter; The value of the action after the risk assessment As the target value of the value network, it is substituted into the calculation of the advantage function of the near-end policy optimization, and the parameters of the policy network and value network under the near-end policy optimization framework are iteratively updated accordingly.

6. The method according to claim 1, characterized in that, The step of setting the reward function for the deep reinforcement learning controller, wherein the reward function includes a depth-of-sight tracking error term, a depth-of-sight stability penalty term, and a power smoothness penalty term, includes: Define a melt depth tracking error term to penalize the deviation between the real-time melt depth and the target melt depth. The calculation formula is as follows: ； Where, d t Let d be the real-time melt depth at time t. target The target melting depth is preset, and k1 is the contribution weight of the tracking error term; A melt depth stability penalty term is defined to suppress drastic jumps in melt depth between adjacent time steps. The calculation formula is as follows: ； Where, d t-1 Let k be the real-time melting depth at time t-1, and k2 be the contribution weight of the stability penalty term; A power smoothness penalty term is defined to suppress drastic adjustments in laser power; its calculation formula is as follows: R smooth = ； Among them, P t-1 Let P be the laser power at time t-1. t Let k be the laser power at time t-1, and k3 be the contribution weight of the power smoothness penalty term; Adding the above three items together yields the reward function.

7. The method according to claim 6, characterized in that, The target melting depth d target Determined in the following ways: Multiple welding experiments were conducted using the reference welding power to determine the actual penetration depth, and the average penetration depth under stable welding conditions was taken as the target penetration depth.

8. The method according to claim 1, characterized in that, The method employs a round-based strategy learning mechanism for multi-pass laser welding. During each welding pass, the strategy network executes closed-loop control in real time, recording state, action, and reward data. After each welding pass, the value network is used to perform distributed value modeling and risk assessment on the state, action, and reward data, and the strategy network parameters are updated. The steps include: Using the benchmark welding power as the initial power for each weld, during the welding process, every preset control cycle, the strategy network outputs a laser power adjustment amount based on the real-time melt depth status at the current moment, controls the laser to adjust the actual output power, and simultaneously records the status, actions and rewards of each time step online to form a complete round trajectory data. After a single welding pass is completed, based on the round trajectory data, the quantile distribution value head of the value network is used to perform distributed modeling for each state, outputting a value distribution consisting of 32 quantiles; then, the state value after risk assessment is calculated using the entropy risk measurement formula. And the advantage function is calculated using the generalized advantage estimation method; Based on the aforementioned advantage function and the state value after risk assessment The objective function for near-end policy optimization is calculated, the policy update magnitude is limited by the shear probability ratio, and the parameters of the policy network and value network are updated using stochastic gradient descent. After completing one round of updates, the next welding process begins until the total reward converges.

9. The method according to claim 1, characterized in that, The step of repeating step S6 until convergence is achieved, resulting in a trained policy network used for subsequent closed-loop control of laser welding penetration stability, includes: After each welding pass is completed, the total reward value of the reward function is recorded, and it is determined whether the total reward value of N consecutive welding passes meets the convergence condition. The convergence condition is: the fluctuation range of the total reward value of N consecutive welding passes is less than a preset threshold, and the total reward value no longer increases significantly, where N is an integer greater than or equal to 5. When the convergence condition is met, training is stopped, the weight parameters of the current policy network are saved as the trained policy network, and a comparative welding experiment without control is conducted using the benchmark welding power. The weld penetration data of the comparative welding experiment is compared with the weld penetration data controlled by the trained policy network to verify the degree of improvement in the average error and standard deviation of the penetration. The trained strategy network is embedded into the industrial control system. In subsequent laser welding tasks, the trained strategy network is directly used to respond online to the real-time weld depth status and output laser power adjustment actions, thereby achieving closed-loop control of weld depth stability without the need for further training.

Images