A reinforcement learning-based position strategy optimization method for a stitch welding torch

Abstract

The present application belongs to the technical field of welding control, and particularly relates to a welding gun position strategy optimization method based on reinforcement learning, comprising: acquiring an action vector and extracting features to form a basic state vector, combining an instant reward to build a welding gun position basic decision model; extracting the ratio of amplitude and frequency from the molten pool brightness time sequence signal as a welding mode index, combining the mode critical reference value of online latching to obtain an augmented state vector; building a conduction type sub-evaluation network and a convection type sub-evaluation network, calculating the gating weight fusion action value estimation based on the mode index and the critical reference value, and calculating the mode drift compensation term to be added to the Bellman update formula to correct the strategy evaluation; partition management is performed on the experience replay pool to update the evaluation sub-network independently, the critical reference value is refreshed online, and the welding gun position incremental instruction is output. The present application effectively suppresses the deepening divergence trend after the critical pass, and realizes the stable regulation and control of the consistency of multi-pass depth.

Description

Technical Field

[0001] This invention belongs to the field of welding control technology, specifically relating to a method for optimizing the position strategy of welding torches based on reinforcement learning. Background Technology

[0002] Mesh welding is widely used in multi-pass close-packing welding of pressure vessel heads, thick-walled pipes, and large structural components. One of its core quality indicators is the consistency of weld penetration between passes. The lateral offset of the welding torch relative to the workpiece, its height from the workpiece, and its travel angle are key control variables that determine the shape and penetration of the weld pool. Even a small drift in any of these variables can cause significant fluctuations in penetration, reinforcement height, and fusion line, directly affecting the mechanical properties and density of the weld.

[0003] For dynamic control of welding torch position in mesh welding, the industry generally introduces reinforcement learning methods on the basis of traditional PID path planning. The typical approach is to use the molten pool image features, the current pass number, and the previous pass's weld depth as state inputs, and the torch position fine-tuning amount as action outputs. The immediate reward is constructed based on the deviation between the weld depth and the nominal reference. After offline training using a continuous action reinforcement learning algorithm such as DDPG or PPO, it is deployed to the position servo loop. Adaptive position correction between passes is achieved through continuous state observation and action feedback.

[0004] However, under the continuous heat accumulation, multi-pass welding will abruptly switch from a conductive molten pool mode to a convective molten pool mode near a certain critical pass. The dominant heat transfer mode of the molten pool changes from solid heat conduction to liquid circulation dominated by Marangoni convection. The response function of weld penetration to welding torch height and tilt angle changes from an approximately linear relationship to a relationship with strong nonlinear amplification characteristics. At this point, the reinforcement learning policy network that originally converged under the conductive condition undergoes structural drift in its state transition dynamics. A systematic mismatch occurs between the Q-value estimate and the actual reward. Every fine-tuning of the policy after the critical pass is amplified into a monotonic shift in weld penetration, ultimately causing the weld penetration to deviate from the baseline value and continue to diverge. Summary of the Invention

[0005] This invention provides a reinforcement learning-based method for optimizing the position strategy of welding torches in multi-pass welding to solve the technical problem of state transition dynamic drift caused by the switching from conduction to convection welding modes during multi-pass welding, which leads to Q-value mismatch and penetration divergence in reinforcement learning strategies.

[0006] This invention provides a reinforcement learning-based method for optimizing the position strategy of a welding torch, comprising the following steps: S1. Obtain the motion vector of the welding torch relative to the workpiece, and extract the visual features of the molten pool, historical melt depth and pass number to form the basic state vector. Combine the real-time reward to build the basic decision model of the welding torch position. S2, extract the amplitude-to-frequency ratio from the molten pool brightness time-series signal as the welding mode index, and combine it with the mode critical reference value based on the online latching of the weld penetration residual. Then, add the welding mode index and the mode critical reference value to the basic state vector to obtain the augmented state vector. S3. Construct a conduction-type sub-evaluation network and a convection-type sub-evaluation network. Calculate the gating weights based on the welding mode index and the mode critical reference value. Fuse the action value estimates of the augmented state vector and calculate the mode drift compensation term to be added to the Bellman update formula to correct the strategy valuation. S4, based on the relationship between the welding mode index and the mode critical reference value, divides the experience playback pool into zones to independently update the conduction type sub-evaluation network and the convection type sub-evaluation network, and refreshes the mode critical reference value online based on the predicted penetration depth residual, and outputs the welding torch position increment command.

[0007] Its effects are as follows: by introducing a mode index based on melt pool brightness oscillation into the reinforcement learning state vector, and with the critical reference value determined by physical inflection point adaptively, the policy can effectively identify the mode switching from conduction to convection; furthermore, by improving the Bellman update through dual evaluation subnet gating fusion and mode drift compensation term, the direction and intensity of Q-value estimation are pre-compensated synchronously before and after the critical trace, thus avoiding policy mismatch caused by abrupt changes in state transition dynamics from a mechanistic perspective.

[0008] Furthermore, the motion vector of the welding torch relative to the workpiece is obtained, and the visual features of the molten pool, historical melt depth, and pass number are extracted to form a basic state vector. Combined with the immediate reward, a basic decision-making model for the welding torch position is constructed, including: obtaining the lateral offset of the welding torch relative to the workpiece, the fine-tuning increments of the height from the workpiece and the travel angle to form the motion vector; extracting the aspect ratio of the molten pool from the molten pool brightness time-series signal collected by the high-speed camera of the molten pool, and combining it with the melt depth of the previous pass and the current pass number to form a basic state vector; obtaining the immediate reward by taking the negative sign of the absolute value of the deviation between the measured melt depth of this pass and the nominal reference melt depth; and completing the pre-training of the basic decision-making model for the welding torch position based on the basic state vector, motion vector, and immediate reward.

[0009] Its effects are as follows: it clarifies the physical construction rules of actions, states, and reward vectors; it uses the three-dimensional fine-tuning of the welding torch as continuous actions; it integrates information on the weld pool, weld depth, and pass number to construct the state; and it uses the negative value of the absolute value of the weld depth deviation as the immediate reward. Utilizing historical data from the conduction process, the network is pre-trained offline, enabling adaptive torch control under conduction conditions based solely on the basic model. The weld depth converges around the benchmark, providing a stable initial strategy to support subsequent cross-mode compensation. The continuous action output can also be directly integrated with a servo system, demonstrating good engineering adaptability.

[0010] Furthermore, the amplitude-to-frequency ratio is extracted from the molten pool brightness timing signal as a welding mode index, including: Within a sliding time window with the current trace duration as its length, the mean value of the melt pool brightness timing signal is removed. The brightness range within the sliding time window is calculated as the oscillation amplitude, and the zero-crossing rate is extracted to calculate the frequency as the main oscillation frequency; The welding mode index is obtained by calculating the ratio of oscillation amplitude to main oscillation frequency.

[0011] Its effect is that by extracting the amplitude-to-frequency ratio from the molten pool brightness time-series signal as the welding mode index, the state space of the strategy network includes, for the first time, a physical feature dimension that can distinguish the boundary between conduction and convection working conditions, providing an accurate and adaptive basis for subsequent dual-subnet gating fusion and compensation term construction.

[0012] Furthermore, in conjunction with the mode critical reference value based on online latching of melt depth residuals, including: During the current batch of welding, the first-order difference sign of the weld penetration residual is continuously monitored; When the sign of the first-order difference changes from alternating positive and negative to consecutive identical signs, it is determined that the divergence inflection point has been reached. The welding mode index of the pass where the divergence inflection point is located is locked as the critical reference value for the mode in this batch.

[0013] Furthermore, gating weights are calculated based on the welding mode index and the mode critical reference value, and the motion value estimation of the augmented state vector is fused, including: The ratio of the welding mode index to the sum of the welding mode index and the critical reference value of the mode is calculated and used as the convection type weight. The difference between 1 and the convection weight is used as the conduction weight; By using transmission weights and convection weights, the action value estimates output by the transmission sub-evaluation network and the convection sub-evaluation network are weighted and summed respectively to obtain the fused action value estimate.

[0014] Furthermore, a mode drift compensation term is added to the Bellman update formula to correct the policy estimate, including: Calculate the absolute value of the difference between the welding mode index and the critical reference value of the mode, and divide it by the sum of the two to obtain the mode compensation coefficient. The difference between the estimates of the convection-type sub-evaluation network and the conduction-type sub-evaluation network is calculated and multiplied by the difference between the welding mode index and the mode critical reference value to obtain the basic compensation term; The mode drift compensation term is obtained by multiplying the mode compensation coefficient by the basic compensation term, and then the mode drift compensation term is added to the Bellman update formula to complete the correction of the policy valuation.

[0015] Its effect is that by improving the Bellman equation through dual-evaluation subnet gating fusion and mode drift compensation term, the direction and intensity of Q-value estimation are synchronously pre-compensated before and after the critical pass, thus eliminating the strategy mismatch caused by the sudden change in state transition dynamics from a mechanistic perspective, so that the melt depth can still be maintained within the controlled range near the nominal reference after cross-mode switching.

[0016] Furthermore, based on the relationship between the welding mode index and the mode critical reference value, the experience playback pool is divided into zones for independent updating of the conduction-type sub-evaluation network and the convection-type sub-evaluation network, including: Empirical elements with a welding mode index less than the mode critical reference value of a first preset proportion are routed to the conduction pool, empirical elements with a welding mode index greater than the second preset proportion are routed to the convection pool, and the rest are routed to the transition pool. Inside the transition pool, samples with welding mode indices less than the critical reference value are classified as the conduction side, and the rest are classified as the convection side. The conduction-type evaluation network is updated by sampling from the conduction side of the conduction pool and the transition pool, and the convection-type evaluation network is updated by sampling from the convection side of the convection pool and the transition pool.

[0017] Furthermore, based on the critical reference value of the online refresh mode for predicted penetration depth residual, the welding torch position increment command is output, including: After each welding pass is completed, the residual difference between the measured penetration depth and the predicted penetration depth of that pass is read. If consecutive residuals with the same sign are detected to reappear, the critical reference value of the mode is refreshed with the welding mode index of the latest divergence inflection point, and the transition pool samples are re-divided simultaneously. The updated policy network outputs welding torch position increment commands to the position servo controller.

[0018] Its effect is that by managing the experience playback pool by mode partitioning and combining it with the online refresh of critical values, it ensures that the two evaluation subnets of conduction and convection are always updated on their respective pure sample distributions. At the same time, it enables the critical reference value to be dynamically tracked and corrected as the heat accumulation within the batch continues to evolve, thus ensuring the continuous effectiveness of the entire cross-mode compensation method.

[0019] Furthermore, before latching the welding mode index of the pass where the divergence inflection point is located as the mode critical reference value for this batch, the following steps are also included: Before the divergence inflection point is triggered, calculate the rolling average of the welding mode index of the completed passes in the current batch; The rolling mean is used as a temporary initial value for the critical reference value of the pattern until the divergence inflection point is triggered, after which the latched value overwrites the temporary initial value.

[0020] Furthermore, the pre-training of the basic decision-making model for welding torch position is completed based on the basic state vector, action vector, and immediate reward, including: An offline training set was constructed based on historical track data of conduction conditions; The policy network and evaluation network are iteratively updated using the offline training set until the melting depth oscillates and converges near the nominal benchmark value, forming an initial usable policy.

[0021] The beneficial effects are: This invention, supplemented by mode-based partitioning, experience playback, and online critical value refresh, ensures that the above methods remain effective throughout the entire welding batch. With the combined effect of these three factors, the penetration depth control remains within a controlled range near the nominal reference even after cross-mode switching. Furthermore, it effectively suppresses the monotonic divergence trend of existing methods after the critical pass, achieving stable control of penetration depth consistency across multiple passes, and improving the adaptability of the welding process under complex heat accumulation conditions and the final weld quality. Attached Figure Description

[0022] Figure 1 This is a flowchart of the reinforcement learning-based welding torch position strategy optimization method in this invention.

[0023] Figure 2 This is a comparison diagram of the consistency of weld penetration depth in multiple passes in this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] An embodiment of the reinforcement learning-based welding torch position strategy optimization method provided by this invention: like Figure 1 As shown, the reinforcement learning-based optimization method for welding torch position strategy includes the following steps: S1, construct a basic decision model to obtain basic state vectors and action vectors.

[0026] In an optional embodiment, a reinforcement learning decision module is deployed above the position servo control layer of the welding equipment to determine the lateral offset of the welding torch relative to the workpiece. Height from workpiece with the angle of travel The fine-tuning increments constitute the continuous action vector Brightness timing signals are acquired by a high-speed camera in the molten pool. Extract the aspect ratio of the molten pool from it. Compared to the previous melting depth Combined with the current track number Together they constitute the basic state vector Instant rewards are based on the actual melting depth measured in this round. With nominal reference penetration depth The absolute value of the deviation is obtained by taking the negative sign, that is... The closer the welding torch is to the optimal point where the penetration depth hits the reference, the closer the bonus value is to zero.

[0027] Furthermore, a continuous action reinforcement learning framework is employed to deploy the policy network and the evaluation network. Output deterministic position fine-tuning actions to evaluate the network. The value of the action is estimated, and the parameter updates are iteratively completed using the Bellman equation:

[0028] in For learning rate, The discount factor is obtained by reverse lookup based on the convergence speed of the first few iterations, following conventional engineering practices in reinforcement learning, and does not require manual fixation. It should be noted that there are many algorithmic frameworks in this field capable of achieving continuous action space exploration and deterministic policy output. Those skilled in the art can choose different algorithm implementations according to their actual needs. This invention uses the DDPG framework as a typical example but is not limited thereto.

[0029] The basic model is pre-trained offline based on historical pass data for conduction conditions, forming an initial usable strategy that provides a foundation for subsequent in-depth corrections for mode switching. At this stage, the strategy can achieve stable oscillation and convergence of melt depth around the benchmark value under conduction conditions. However, since the state vector does not yet contain any feature dimensions reflecting the physical mode of the melt pool, the strategy network is essentially unable to perceive the boundary between conduction and convection conditions. Once heat accumulation triggers a mode abrupt change, the strategy will continue to output actions according to the original kinetic assumptions without any awareness, thus laying the groundwork for subsequent melt depth divergence.

[0030] By constructing a complete basic state representation that includes visual features of the weld pool, historical weld depth of each pass, and pass number, and using the instantaneous weld depth deviation as a reward signal to drive the parameter learning of the evaluation network, the welding position control has the basic ability to stably adapt under conductive working conditions, and establishes a reliable strategic starting point for the superposition of subsequent cross-mode compensation methods.

[0031] S2, extract the mode index to obtain the augmented state vector.

[0032] In an optional embodiment, a welding mode index is introduced as an augmented state dimension in addition to the basic state vector. The welding mode index is derived from the brightness time-series signal acquired by the high-speed camera of the molten pool. Extracted from the surface brightness signal. Due to the thin and slow-flowing liquid region of the conductive molten pool, the surface brightness exhibits high-frequency, small-amplitude fluctuations; while the surface brightness of the convective molten pool exhibits low-frequency, large-amplitude fluctuations due to the intense agitation caused by the Marangoni circulation. This physical difference can be extracted and characterized in a very simple way: within a sliding time window of duration equal to the current pass duration, the brightness signal is analyzed. After removing the mean, the oscillation amplitude within the window is calculated. (Taking the extreme difference in brightness within the window) and the main oscillation frequency (The frequency is calculated using the zero-crossing rate, eliminating the need for FFT overload calculations), and the welding mode index is defined as:

[0033] in The physical meaning of is the amplitude energy carried per unit oscillation frequency. A larger value indicates that the molten pool tends to operate under convective conditions. A larger value means a higher ratio between the amplitude and frequency of the brightness signal, and thus a stronger disturbance of the convective circulation on the surface of the molten pool. A smaller value means that the brightness signal is mainly characterized by high-frequency, small-amplitude fluctuations, and the molten pool is still in the conduction-type range dominated by solid heat conduction.

[0034] Next, a critical reference value for the model is introduced. As a basis for distinguishing between conduction and convection types. The value is not determined by a fixed empirical constant, but rather by an online adaptive method strongly bound to the actual heat accumulation characteristics: during the current batch of welding, the weld penetration residual sequence is continuously monitored. The first-order difference sign, when the sign changes from alternating positive and negative around zero to consecutive identical signs, is considered a divergence inflection point, and the trace number where the inflection point is located is... The value is latched for this batch. The number of consecutive identical-sign judgment passes is determined based on the actual bevel geometry and the thermal conductivity of the base material. For example, the number of consecutive identical-sign judgment passes can be 3. This method of setting the threshold naturally binds it to the actual heat accumulation characteristics, bevel geometry, and thermal conductivity of the base material, enabling customized solutions for each case.

[0035] Considering that the inflection point has not yet been triggered in the initial stage of the batch, In the pending state, to avoid missing denominators in subsequent fusion equations, before the inflection point is latched, the rolling average of the welding mode index of the completed passes in the current batch is used as... The temporary initial value, where the number of completed passes is denoted as . Once the inflection point is triggered, the latched value overwrites the temporary initial value, and subsequent rolling updates cease. The pattern index... With critical reference value Together they are added to the basic state vector to obtain the augmented state vector. This provides pattern-aware input for subsequent strategy adjustments.

[0036] By extracting the amplitude-to-frequency ratio from the molten pool brightness time-series signal as the welding mode index, and using the inflection point of the weld depth residual sign to latch the critical reference value of the mode online, the state space of the strategy network for the first time includes physical feature dimensions that can distinguish between the boundary of conduction and convection working conditions. This provides an accurate and adaptive mode judgment basis for the next step of gating fusion and compensation term construction of the two subnets.

[0037] S3 calculates the gating weights and constructs compensation terms to correct the strategy valuation.

[0038] In an alternative embodiment, instead of using a single evaluation network to address cross-modal drift, a transmission-type sub-evaluation network is constructed. With convection sub-evaluation network Both parameters are updated independently on their respective empirical samples, sharing the same parameter structure but independent weights. In the action decision and Q-value estimation stages, gating weights are calculated and fused based on the welding mode index and the mode critical reference value, defining convection type weights. With transmission weights for:

[0039] Since both the welding mode index and the mode critical reference value are ratios of brightness oscillation amplitude to frequency, and their values ​​are always positive real numbers, the sum of the two is always greater than 0. exist much smaller When the Q-value approaches 0, the fused Q-value is mainly in the conduction subnet; in Much larger When the Q-value approaches 1, the fused Q-value is dominated by the convection subnet; in the critical transition region, the two subnets smoothly take over proportionally. Compared to the step-like abrupt change at the threshold of hard handover, this gating method... A gradual transition is provided when the policy is in the critical range, avoiding the oscillation problem of repeated jumps due to small fluctuations near the threshold. The fused action value estimate is:

[0040] Furthermore, the established Bellman update formula is improved to calculate the mode drift compensation term. To correct the policy valuation, the formula is added to the Bellman update formula. The improved update formula is as follows:

[0041] Where the mode compensation coefficient The relation is:

[0042] exist When the value is 0, the valuations of the two networks are already aligned, and no compensation is needed; and As the gap widens, it approaches 1; the greater the deviation, the stronger the compensation. Basic compensation item. The relation is:

[0043] The symbols and directions are fully covered by four combinations: when and hour, The compensation term pulls down the Q-value estimate, and the strategy tends to be more conservative in terms of position increments, consistent with the physical requirement to suppress the amplitude of motion in the convection amplification zone; when and hour, Under convective conditions, this action is actually more advantageous than under conductive conditions; the compensation term should be appropriately increased to raise the Q value to encourage the strategy to continue in this direction. and hour, Compensation provides early warning of the Q-value contraction trend approaching the convection region; when and hour, The compensation term slightly suppresses the tendency to drift prematurely towards the convective side. The above four combinations ensure that the compensation direction is automatically determined by the difference in physical state under any operating condition, and there are no uncovered scenarios where the direction is reversed.

[0044] Because traditional Bellman updates only reflect instantaneous TD errors and cannot predict structural abrupt changes in state transition dynamics, the policy will continue to accumulate mismatch along erroneous dynamic assumptions after a mode jump. This invention addresses this issue by... and The difference characterizes the value gap between two modes under the same state action, and then... The actual direction and intensity of this drop are given, allowing the strategy to pre-compensate along the mode-switching direction the moment it crosses the critical stage, thus breaking the transmission chain of mismatch accumulation. For example, suppose a certain stage... , , , ,but , Compensation items This negative compensation is equivalent to informing the strategy in advance that it is entering the convection amplification zone and needs to be dealt with by a more conservative position increment, so as to avoid the action being amplified by the nonlinear response into a monotonic shift in the melting depth.

[0045] By improving the Bellman equation through dual-evaluation subnet gating fusion and mode drift compensation terms, the direction and intensity of Q-value estimation are simultaneously pre-compensated before and after the critical pass, thus eliminating the strategy mismatch caused by the sudden change in state transition dynamics from a mechanistic perspective. This allows the melt depth to remain within a controlled range near the nominal reference after cross-mode switching.

[0046] S4 manages the experience replay pool and outputs incremental instructions.

[0047] In an optional embodiment, an experience playback pool partitioning management method is designed to be strictly aligned with pattern recognition. Based on the relationship between the welding mode index and the mode critical reference value, the experience tuple... When storing in the playback pool, it depends on the storage time. relatively The relationships are routed to three independent partitions: the conduction pool, the convection pool, or the transition pool. Less than The sample product of the first preset ratio is routed to the conduction cell. Greater than Samples whose product with the second preset ratio are routed to the convection pool, and the remaining samples are routed to the transition pool; for example, the first preset ratio can be 0.9, and the second preset ratio can be 1.1. The transition pool is then used for... and The size relationship is the sole criterion for judgment. The sample is designated as the conduction side. The samples were classified as convective. (Transduction-type sub-evaluation network) The convection sub-evaluation network is updated only by sampling samples from the conduction side of the conduction pool and the transition pool. The sampling and updating are done only from the convective side samples of the convection pool and the transition pool, thus preventing the samples from the two modes from contaminating each other's evaluation networks.

[0048] Next, during online execution, the residual between the measured and predicted penetration depths is read after each welding pass. This residual is used to reverse-calibrate the accuracy of the mode routing for that pass. If consecutive residuals with the same sign reappear, it indicates that the current... It may fail due to further evolution caused by thermal accumulation. Therefore, the critical reference value of the online refresh mode based on the predicted melt depth residual is used, i.e., the latest inflection point. Value refresh And simultaneously re-divide the transition pool samples.

[0049] On the equipment side, this method is embedded into the existing position servo controller host computer in the form of a decision module. Based on the updated strategy network, it outputs the welding gun position increment command to the position servo controller, sends the welding gun position increment command downward, and receives the weld pool vision and weld depth detection data upward. Without changing the original hardware configuration, it only completes the strategy-level optimization and adjustment of the multi-pass welding process through the data processing layer, which is feasible for engineering implementation.

[0050] Thus, by managing the experience replay pool by pattern partitioning and combining it with online updating of pattern critical reference values, it is ensured that the two evaluation subnets, conduction-type and convection-type, are always updated on their respective pure sample distributions, while also ensuring... It can dynamically track and correct the continuous evolution of heat accumulation within the batch, ensuring the continuous effectiveness and adaptive robustness of the entire cross-mode compensation method throughout the entire life cycle of multi-pass welding.

[0051] like Figure 2 As shown, it can be observed that the existing method exhibits a continuous monotonic shift in melt depth after the critical pass, while the method of the present invention maintains the melt depth within a controlled range before and after the critical pass, indicating that the present invention effectively suppresses the trend of melt depth divergence after the critical pass.

[0052] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A reinforcement learning-based optimization method for welding torch position strategy, characterized in that, Includes the following steps: S1. Obtain the motion vector of the welding torch relative to the workpiece, and extract the visual features of the molten pool, historical melt depth and pass number to form the basic state vector. Combine the real-time reward to build the basic decision model of the welding torch position. S2, extract the amplitude-to-frequency ratio from the molten pool brightness time-series signal as the welding mode index, and combine it with the mode critical reference value based on the online latching of the weld penetration residual. Then, add the welding mode index and the mode critical reference value to the basic state vector to obtain the augmented state vector. S3. Construct a conduction-type sub-evaluation network and a convection-type sub-evaluation network. Calculate the gating weights based on the welding mode index and the mode critical reference value. Fuse the action value estimates of the augmented state vector and calculate the mode drift compensation term to be added to the Bellman update formula to correct the strategy valuation. S4, based on the relationship between the welding mode index and the mode critical reference value, divides the experience playback pool into zones to independently update the conduction type sub-evaluation network and the convection type sub-evaluation network, and refreshes the mode critical reference value online based on the predicted penetration depth residual, and outputs the welding torch position increment command.

2. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, The motion vector of the welding torch relative to the workpiece is obtained, and the visual features of the molten pool, historical melt depth, and pass number are extracted to form a basic state vector. A basic decision-making model for the welding torch position is constructed by combining the immediate reward. This includes: obtaining the lateral offset of the welding torch relative to the workpiece, the fine-tuning increments of the height from the workpiece and the travel angle to form the motion vector; extracting the aspect ratio of the molten pool from the time-series signal of the molten pool brightness acquired by the high-speed camera, and combining it with the melt depth of the previous pass and the current pass number to form a basic state vector; obtaining an immediate reward by taking the negative sign of the absolute value of the deviation between the measured melt depth and the nominal reference melt depth of the current pass; and completing the pre-training of the basic decision-making model for the welding torch position based on the basic state vector, motion vector, and immediate reward.

3. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, The amplitude-to-frequency ratio is extracted from the weld pool brightness time-series signal as a welding mode index, including: Within a sliding time window with the current trace duration as its length, the mean value of the melt pool brightness timing signal is removed. The brightness range within the sliding time window is calculated as the oscillation amplitude, and the zero-crossing rate is extracted to calculate the frequency as the main oscillation frequency; The welding mode index is obtained by calculating the ratio of oscillation amplitude to main oscillation frequency.

4. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, Combining the mode critical reference values ​​based on online latching of melt depth residuals, including: During the current batch of welding, the first-order difference sign of the weld penetration residual is continuously monitored; When the sign of the first-order difference changes from alternating positive and negative to consecutive identical signs, it is determined that the divergence inflection point has been reached. The welding mode index of the pass where the divergence inflection point is located is locked as the critical reference value for the mode in this batch.

5. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, Gating weights are calculated based on the welding mode index and the mode critical reference value, and the motion value estimation of the augmented state vector is fused, including: The ratio of the welding mode index to the sum of the welding mode index and the critical reference value of the mode is calculated and used as the convection type weight. The difference between 1 and the convection weight is used as the conduction weight; By using transmission weights and convection weights, the action value estimates output by the transmission sub-evaluation network and the convection sub-evaluation network are weighted and summed respectively to obtain the fused action value estimate.

6. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, The calculation of mode drift compensation terms is added to the Bellman update formula to correct the policy valuation, including: Calculate the absolute value of the difference between the welding mode index and the critical reference value of the mode, and divide it by the sum of the two to obtain the mode compensation coefficient. The difference between the estimates of the convection-type sub-evaluation network and the conduction-type sub-evaluation network is calculated and multiplied by the difference between the welding mode index and the mode critical reference value to obtain the basic compensation term; The mode drift compensation term is obtained by multiplying the mode compensation coefficient by the basic compensation term, and then the mode drift compensation term is added to the Bellman update formula to complete the correction of the policy valuation.

7. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, Based on the relationship between the welding mode index and the mode critical reference value, the experience playback pool is divided into zones for independent updating of the conduction-type sub-evaluation network and the convection-type sub-evaluation network, including: Empirical elements with a welding mode index less than the mode critical reference value of a first preset proportion are routed to the conduction pool, empirical elements with a welding mode index greater than the second preset proportion are routed to the convection pool, and the rest are routed to the transition pool. Inside the transition pool, samples with welding mode indices less than the critical reference value are classified as the conduction side, and the rest are classified as the convection side. The conduction-type evaluation network is updated by sampling from the conduction side of the conduction pool and the transition pool, and the convection-type evaluation network is updated by sampling from the convection side of the convection pool and the transition pool.

8. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 1, characterized in that, Based on the critical reference value of the online refresh mode for predicted penetration depth residual, the welding torch position increment command is output, including: After each welding pass is completed, the residual difference between the measured penetration depth and the predicted penetration depth of that pass is read. If consecutive residuals with the same sign are detected to reappear, the critical reference value of the mode is refreshed with the welding mode index of the latest divergence inflection point, and the transition pool samples are re-divided simultaneously. The updated policy network outputs welding torch position increment commands to the position servo controller.

9. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 4, characterized in that, Before locking the welding mode index of the pass where the divergence inflection point is located as the mode critical reference value for this batch, the following steps are also included: Before the divergence inflection point is triggered, calculate the rolling average of the welding mode index of the completed passes in the current batch; The rolling mean is used as a temporary initial value for the critical reference value of the pattern until the divergence inflection point is triggered, after which the latched value overwrites the temporary initial value.

10. The reinforcement learning-based method for optimizing the position strategy of welding torches according to claim 2, characterized in that, Pre-training of the basic decision-making model for welding torch position based on the fundamental state vector, action vector, and immediate reward includes: An offline training set was constructed based on historical track data of conduction conditions; The policy network and evaluation network are iteratively updated using the offline training set until the melting depth oscillates and converges near the nominal benchmark value, forming an initial usable policy.

Images