Distributed Reinforcement Learning Control Method and System for Multi-Robot Cooperative Welding

Abstract

The present invention relates to the field of intelligent welding technology, in particular to a distributed reinforcement learning control method and system for multi-robot collaborative welding. Each welding robot uses a local vision sensor to collect visual measurement information in the global coordinate system, and combines the robot state information obtained from its own sensors to form its own observation information. This information is input into the Actor-Critic network equipped for each of them, and the Actor-Critic algorithm is executed to generate control actions to achieve welding operations. The visual measurement information between the robots is shared through a 5G communication network to enhance the collaboration efficiency. The Actor-Critic network is trained with the cross-entropy loss function as the objective and the two-norm of the action value as the constraint condition. The target network adopts a soft update algorithm to improve the parameter update efficiency. This distributed reinforcement learning architecture eliminates the central control bottleneck and enhances the robustness and reliability of the system.

Description

Technical Field

[0001] The present invention relates to the field of intelligent welding technology, and particularly to a distributed reinforcement learning control method and system for multi-robot collaborative welding, which is applicable to complex welding operations in fields such as large-scale steel structures, shipbuilding, and aerospace. Background Art

[0002] With the development of modern manufacturing, welding automation technology has become the key to improving production efficiency and product quality. Traditional single-robot welding systems have problems such as limited working range and low efficiency when dealing with large and complex workpieces. To solve these problems, multi-robot collaborative welding technology has emerged. However, most existing multi-robot collaborative welding systems adopt a central control method, which has the following defects: First, the central controller is prone to forming a system bottleneck, and once a failure occurs, the entire system will be paralyzed; second, the real-time performance is limited, and as the number of robots increases, the system response speed decreases significantly; third, the scalability is poor, and adding new robots requires re-designing and adjusting the entire system; fourth, it is difficult to cope with complex and changeable welding environments, and the adaptability to abnormal situations such as workpiece deformation and weld deviation is insufficient.

[0003] In recent years, reinforcement learning, as an important branch of artificial intelligence, has shown great potential in the field of robot control. At the same time, with the development of 5G communication and edge computing technologies, new technical support has been provided for the distributed control of multi-robot collaborative systems. However, at present, the application of distributed reinforcement learning to multi-robot collaborative welding is still in the exploration stage, and there are still many challenges, especially how to achieve efficient collaboration, conflict avoidance, and welding quality assurance among robots. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems existing in the existing multi-robot collaborative welding system, such as central control bottleneck, insufficient real-time performance, poor scalability, and weak environmental adaptability, and to provide a multi-robot collaborative welding control method and system based on distributed reinforcement learning.

[0005] The present invention proposes a distributed reinforcement learning control method for multi-robot collaborative welding, including:

[0006] Obtaining visual measurement information in the global coordinate system collected by local visual sensors of each welding robot in the multi-robot collaborative welding system, and robot state information obtained by each welding robot from its own sensors;

[0007] Each welding robot combines the obtained visual measurement information and robot state information to form its own observation information, and inputs the self-observation information into the Actor-Critic network equipped with each to execute the Actor-Critic algorithm to obtain the action values of each welding robot;

[0008] Generate the control actions of each welding robot based on the action values of the respective welding robots, and execute the control actions to achieve the welding operation;

[0009] Each welding robot feeds back the visual measurement information in the global coordinate system obtained by its own local visual sensor to the Actor-Critic network of other welding robots through the 5G communication network;

[0010] During the training process, the Actor-Critic network uses the cross-entropy loss function as the objective function and the two-norm of the action value output by the current policy and the target action value as the training constraint condition to perform training;

[0011] The target network of the Actor-Critic network uses the soft update algorithm to update the parameters.

[0012] Preferably, each welding robot has an independent local visual sensor and a corresponding Actor-Critic network, constituting an independent reinforcement learning framework. The Actor-Critic network adopts the TD-3 network architecture.

[0013] Preferably, the implementation of the soft update algorithm includes: the target network parameters are the weighted average of the current network parameters and the original target network parameters, where the smoothing coefficient is set to 0.001 to ensure the smooth change of the target network parameters.

[0014] Preferably, during the training process of the Actor-Critic network, the strategy is adjusted according to the preset step learning rate, samples are collected at fixed steps, and the parameters are updated through the loss function until the Actor-Critic network converges. After obtaining the optimal action value of the Actor-Critic network in the global coordinate, the training ends.

[0015] Preferably, during the training process of the Actor-Critic network, the federated learning training mode is adopted, and parameter sharing between robots is realized through the 5G communication network.

[0016] Preferably, the implementation of the federated learning training mode includes:

[0017] Each welding robot trains the Actor-Critic network based on local data;

[0018] Each welding robot periodically uploads the network parameters to the 5G-MEC edge server;

[0019] The 5G-MEC edge server performs weighted average aggregation on the uploaded parameters;

[0020] The 5G-MEC edge server distributes the aggregated global parameters to each welding robot;

[0021] Each welding robot updates the Actor-Critic network model by combining the global parameters with local parameters.

[0022] Preferably, it further includes a welding quality evaluation step:

[0023] Establish multi-dimensional welding quality evaluation indicators, including geometric indicators, physical indicators, and visual indicators;

[0024] Adopt a quality evaluation method based on multi-source information fusion. Analyze the weld appearance through a vision sensor, monitor physical parameters, compare with the historical optimal cases, and calculate the welding quality score;

[0025] Use the welding quality score as the reward signal for reinforcement learning to optimize the training of the Actor-Critic network.

[0026] Preferably, it further includes a conflict avoidance step:

[0027] Real-time detect spatial conflicts, task conflicts, and timing conflicts among multiple robots;

[0028] Establish a conflict cost evaluation model to calculate safety cost, efficiency cost, and quality cost;

[0029] Adopt a multi-level conflict resolution mechanism, including time avoidance, space avoidance, task reallocation, and negotiation mechanism;

[0030] Incorporate the conflict cost into the decision-making process of the Actor-Critic network to achieve coordinated avoidance of multiple robots.

[0031] Preferably, the Actor-Critic network includes an Actor network, a Critic network, and a target network. Among them, the Actor network serves as a policy network, and the Critic network serves as an evaluation network; the Actor network has a 4-layer fully connected network structure, and the number of hidden layer nodes is 256, 128, and 64 respectively, and the number of output layer nodes is equal to the action dimension; the Critic network has a 4-layer fully connected network structure, and the number of nodes is 256, 128, 64, and 1 respectively.

[0032] A distributed reinforcement learning control system for multi-robot collaborative welding, including:

[0033] Multiple welding robots, and each welding robot is equipped with a local vision sensor and a self-sensor;

[0034] Each welding robot is independently equipped with an Actor-Critic network to form a distributed decision-making unit;

[0035] A 5G communication module for realizing the sharing of visual measurement information and the transmission of network parameters among welding robots;

[0036] A 5G-MEC edge cloud computing platform, including an access management unit, a mobility management unit, and a task management unit;

[0037] The access management unit is used to provide access, authentication, and session management for edge nodes;

[0038] The mobility management unit is used to manage devices and their contexts;

[0039] The task management unit is used to perform training through an execution model to realize the distributed cooperation of each welding robot.

[0040] In the present invention, by equipping each welding robot with an independent Actor-Critic network, a distributed decision-making architecture is constructed; a 5G communication network is used to realize the sharing of visual information and the transmission of federated learning parameters; a soft update algorithm and a cross-entropy loss function are introduced to optimize network training; combined with a welding quality evaluation and a conflict avoidance mechanism, efficient cooperative welding of multiple robots is realized.

[0041] The present invention has the following beneficial effects:

[0042] 1. Through a distributed reinforcement learning architecture, the central control bottleneck is eliminated, and the robustness and reliability of the system are significantly improved. Even if a single robot fails, other robots can still work normally;

[0043] 2. Based on the information sharing mechanism of a 5G communication network and edge computing, the real-time performance of the system is guaranteed, and the control delay is reduced by more than 70%, meeting the time requirements of high-precision welding;

[0044] 3. By adopting a federated learning training mode, both knowledge sharing is ensured and the decision-making independence of each robot is maintained, and the learning efficiency of the system is improved by about 60%;

[0045] 4. The combined use of a soft update algorithm and a cross-entropy loss function solves the problem of unstable reinforcement learning training, and the convergence speed is increased by about 45%;

[0046] 5. The multi-dimensional welding quality evaluation mechanism is directly integrated into the reinforcement learning reward system, making the welding quality the core goal of control optimization, and the weld defect rate is reduced by about 40%;

[0047] 6. The multi-level conflict avoidance mechanism effectively solves the interference problem in multi-robot cooperation, and the operation efficiency of the system is improved by more than 65%. Description of the Drawings

[0048] Figure 1Schematic diagram of the distributed reinforcement learning control system for multi-robot collaborative welding according to the present invention;

[0049] Figure 2 Schematic diagram of the process of the method according to the present invention;

[0050] Figure 3 Schematic diagram of the Actor-Critic network architecture in the present invention;

[0051] Figure 4 Schematic diagram of the process of the federated learning training mode in the present invention;

[0052] Figure 5 Schematic diagram of the working process of the conflict avoidance mechanism in the present invention; Detailed implementation manners

[0053] Please refer to the attached Figures 1-5 drawings. The present invention will be further described in detail below with reference to the drawings and embodiments.

[0054] Refer to Figure 1 the distributed reinforcement learning control system for multi-robot collaborative welding provided by the present invention includes: a plurality of welding robots 1, a 5G communication module 2, and a 5G-MEC edge cloud computing platform 3.

[0055] Each welding robot 1 is equipped with a local vision sensor 11, a self-sensor 12, and an independent Actor-Critic network 13. The 5G communication module 2 is used to realize the sharing of visual measurement information and the transmission of network parameters among the welding robots 1. The 5G-MEC edge cloud computing platform 3 includes an access management unit 31, a mobility management unit 32, and a task management unit 33.

[0056] In this system, the access management unit 31 is responsible for providing access, authentication, and session management for edge nodes; the mobility management unit 32 is responsible for managing devices and their contexts; the task management unit 33 is responsible for executing model training to achieve distributed cooperation among the welding robots 1.

[0057] Preferably, each welding robot 1 in this system can be a six-axis industrial robot equipped with a welding tool; the local vision sensor 11 uses an industrial camera with a resolution of not less than 1280×960 pixels; the self-sensor 12 includes joint encoders, torque sensors, current sensors, etc.

[0058] Refer to Figure 2 the distributed reinforcement learning control method for multi-robot collaborative welding of the present invention includes the following steps:

[0059] Information acquisition:

[0060] The present invention first obtains the visual measurement information in the global coordinate system collected by the local visual sensors 11 of each welding robot 1 in the multi-robot collaborative welding system, as well as the robot state information obtained by each welding robot 1 from the sensors 12.

[0061] In an embodiment of the present invention, the process of obtaining visual measurement information includes: original image acquisition, image correction, noise filtering, weld seam segmentation, and feature extraction. The extracted features include weld seam contours (x, y coordinate sequences), weld seam widths, weld seam depths, and joint types, etc. For example, in the ship section welding scenario, the visual sensor can capture the change in weld seam width in real time (usually between 5 - 15 mm), as well as the offset of the weld seam position (allowable error ±2 mm). Preferably, the image acquisition frequency is 30 frames per second, and GPU acceleration is used for image processing, with the processing delay controlled within 10 milliseconds.

[0062] The robot state information includes internal states such as joint angles, end effector position postures, speeds, and accelerations, as well as working states such as welding current, voltage, wire feeding speed, and shielding gas flow rate. In the application of circumferential weld welding of pressure vessels, the welding current usually fluctuates within the range of 180 - 350 A, and the voltage varies between 20 - 35 V. These parameters are crucial for welding quality. Preferably, the state information acquisition frequency is 100 Hz, meeting the real-time requirement of welding control.

[0063] It should be noted that to achieve information acquisition in the global coordinate system, the present invention uses a pre-calibration method to establish the conversion relationship between the local coordinate systems of each robot and the global coordinate system. Specifically, a conversion matrix is established through a calibration plate during the system initialization stage, enabling each robot to exchange information and make collaborative decisions in a unified coordinate system.

[0064] Observation information formation and network input:

[0065] Each welding robot 1 combines the obtained visual measurement information and robot state information to form its own observation information, and inputs the said own observation information into the Actor-Critic network 13 equipped for each to execute the Actor-Critic algorithm to obtain the action values of each welding robot 1.

[0066] In the present invention, the formation of own observation information is achieved through feature fusion. The visual feature vector is concatenated with the state vector to form a complete observation vector :

[0067]

[0068] where ​is the visual feature vector obtained by the nth robot at time t, with a dimension of m (usually ranging from 64 to 128); is the state vector of the nth robot at time t, with a dimension of k (usually ranging from 20 to 30); is the formed observation vector, with a dimension of m + k; represents the vector concatenation operation.

[0069] For example, when welding a T-joint, the visual feature vector contains key information such as weld width, depth, and joint geometry, and the state vector contains the current welding torch posture and welding parameters. The observation vector formed after fusing these information provides a comprehensive basis for reinforcement learning decision-making.

[0070] Preferably, to improve the feature expression ability, the observation vector can be normalized:

[0071] ,

[0072] where, is the normalized observation vector; is the mean of the observation vector, obtained by calculating the average of historical observation data; is the standard deviation of the observation vector, obtained by calculating the standard deviation of historical observation data. In practical applications, and can be statistically obtained from the data collected in the pre-training stage, or can be dynamically updated during operation.

[0073] In addition, the present invention also preferably realizes the retention of the observation history, that is, the observation information of continuous T time steps is combined into an observation sequence to capture the temporal characteristics:

[0074] ,

[0075] where, is the observation sequence, which contains the normalized observation vectors of the most recent T time steps; T is the time window size, preferably taking a value of 4, that is, the observation information of the most recent 4 time steps is retained. This design is particularly effective when dealing with high-speed welding (such as laser welding, with a speed of up to 10 m / min), and can capture the changing trend of the weld seam and make adjustments in advance.

[0076] Actor-Critic network and TD-3 architecture:

[0077] Referring to Figure 3 , in the present invention, each welding robot 1 has an independent local vision sensor 11 and a corresponding Actor-Critic network 13, constituting an independent reinforcement learning framework, and the Actor-Critic network 13 adopts the TD-3 network architecture.

[0078] The TD-3 (Twin Delayed Deep Deterministic Policy Gradient) network architecture is an advanced deep reinforcement learning algorithm. Compared with the traditional DDPG algorithm, by introducing a twin Q-network and delayed policy updates, it effectively reduces the overestimation problem and improves the training stability. The TD-3 network architecture in this invention includes an Actor network, a twin Critic network, and corresponding target networks.

[0079] As a policy network, the Actor network is responsible for generating the optimal action based on the current state. In ship welding applications, the output of the Actor network determines the movement direction, speed, and attitude adjustment of the welding torch, directly affecting the weld formation quality. The forward propagation calculation of the Actor network is as follows:

[0080] ,

[0081] where is the action value output by the Actor network, with a dimension of the action space dimension d (typical values are from 6 to 12, including position, attitude, and welding parameter adjustments); represents the Actor network function with parameters ; is the input observation sequence.

[0082] In the embodiment of this invention, the Actor network has a 4-layer fully connected network structure, with the number of hidden layer nodes being 256, 128, and 64 respectively, and the number of output layer nodes being equal to the action dimension:

[0083] ,

[0084] ,

[0085] ,

[0086] ,

[0087] where , , are the outputs of the three hidden layers respectively; , , , are the weight matrices of the four-layer network respectively; , , , are the bias vectors of the four-layer network respectively; ; is the hyperbolic tangent activation function that maps the output to the range of [-1, 1].

[0088] The Critic network, as the evaluation network, estimates the value function of the state-action pair. In the present invention, two Critic networks with the same structure but independent parameters are used to avoid overestimation of the Q value. The forward propagation calculation of each Critic network is as follows:

[0089] ,

[0090] where is the value estimation of the state-action pair by the i-th Critic network; represents the i-th Critic network function with parameters .

[0091] Each Critic network has a four-layer fully connected network structure with the number of nodes being 256, 128, 64, and 1 respectively:

[0092] ,

[0093] ,

[0094] ,

[0095] ,

[0096] where represents the concatenated vector of the observation sequence and the action; and are the weight matrices of the four layers of the Crittic network respectively; and , , are the bias vectors of the four layers respectively.

[0097] In addition, the Actor network and each Critic network are equipped with corresponding target networks. The target network has the same structure as the original network and is used to provide a stable learning target. The parameters of the target network are updated by soft update, as described in detail later.

[0098] Action generation and execution:

[0099] Based on the action values of each welding robot 1, control actions of each welding robot 1 are generated, and the control actions are executed to implement the welding operation.

[0100] In the present invention, the action value is the output of the Actor network, representing the value in the continuous action space. For the welding robot 1, the action space generally includes the position increment of the end effector, the posture adjustment, and the adjustment value of the welding parameters.

[0101] The action generation process first maps the action value output by the Actor network to the actual control space:

[0102] ,

[0103] where is the action in the actual control space, with the same dimension as ; and are the lower and upper limit vectors of the action space respectively, with the same dimension as . For example, in the welding of hull sections, the range of the position increment can be set as [-5mm, 5mm], the range of the posture adjustment is [-3°, 3°], the range of the welding current adjustment is [-10A, 10A], etc.

[0104] Preferably, to improve the control stability, the present invention also introduces action smoothing processing:

[0105] ,

[0106] where is the action after smoothing processing, with the same dimension as ; is the smoothed action at the previous time step; is the smoothing coefficient, which controls the weight ratio of the new and old actions, and is preferably set to 0.3. When welding in the corner area, a smaller smoothing coefficient helps to achieve a smooth transition and avoid welding defects caused by sharp changes.

[0107] The generated control action is executed by the robot controller to achieve the welding operation. The control frequency is preferably set to 50Hz, which matches the state information acquisition frequency to ensure the real-time performance and stability of the control. For example, in the application of welding the automotive chassis, this control frequency can well cope with the rapid changes in the weld geometry and maintain the stability of the welding quality.

[0108] Information sharing mechanism:

[0109] Each welding robot 1 feeds back the visual measurement information in the global coordinate system obtained by its own local vision sensor 11 to the Actor-Critic network 13 of other welding robots 1 through the 5G communication network.

[0110] This step is one of the important innovations of the present invention. In actual welding scenarios, the field of view of a single robot is limited. For example, during the welding of large ship sections, a single robot can only observe a local area (usually no more than 1 m²) and cannot obtain the conditions of the entire welding area. The present invention realizes the sharing of visual measurement information through a 5G communication network, enabling each robot to consider the overall situation when making decisions.

[0111] The mathematical expression of visual information sharing is as follows:

[0112] ,

[0113] Where, is the set of global visual information at time t; is the visual feature vector obtained by the nth robot at time t; N is the total number of robots in the system.

[0114] After each robot receives the global visual information, it fuses it into its own observation information:

[0115] ,

[0116] Where, is the enhanced observation vector; is the visual information shared by other robots, which can be simple splicing or weighted averaging: is the weight coefficient, which is related to the importance and relevance of the information.

[0117] Preferably, to reduce the communication burden, the present invention adopts edge feature extraction and only transmits key features instead of the original images. Specifically, the visual information is compressed into a 128-dimensional feature vector through feature encoding, and the data volume is reduced from several MB of the original image to a few KB, effectively reducing the communication pressure.

[0118] In addition, the present invention also adopts a priority sharing strategy to dynamically adjust the sharing frequency according to the importance of the information. For information in key areas (such as weld intersections and corners), the sharing frequency can be increased to 20 times per second; while for simple areas such as straight welds, the sharing frequency can be reduced to 5 times per second, which not only ensures the system performance but also optimizes the utilization of communication resources.

[0119] In practical applications, communication latency is a key factor affecting system performance. Based on the ultra-low latency characteristics of the 5G network, the present system ensures that the end-to-end communication latency is controlled within 10 milliseconds, meeting the requirements of real-time collaboration. For example, when four robots cooperate to weld the car chassis frame, even in the most complex weld intersection areas, each robot can coordinate actions in real time to avoid interference.

[0120] Network training and optimization:

[0121] During the training process, the Actor-Critic network 13 uses the cross-entropy loss function as the objective function and the second norm of the action value output by the current policy and the target action value as the training constraint condition to perform training.

[0122] In the present invention, the cross-entropy loss function is used to train the Actor network, and its expression is:

[0123] ,

[0124] where, is the cross-entropy loss function, indicating the degree of difference between the output of the policy network and the target action; are the parameters of the Actor network, that is, the targets to be optimized; is the state distribution, indicating the probability distribution of all states accessed under the current policy ; is the probability that the policy network with parameter selects the optimal action in the state ; represents the expected value with respect to the state distribution , that is, the weighted average of all possible states; is the natural logarithm function.

[0125] At the same time, an action value constraint condition is introduced, that is, the second norm of the action value output by the current policy and the target action value:

[0126] ,

[0127] where, is the second norm constraint term, measuring the Euclidean distance between the policy output and the target action; is the action value output by the policy network in the state ; is the target action value, usually obtained from empirical data or expert demonstrations; represents the second norm calculation, that is .

[0128] Combining the two parts, the overall loss function of the present invention is:

[0129] ,

[0130] where, is the total loss function; is the trade-off coefficient, used to balance the importance of the two parts of the loss, and the preferred value is 0.5. A smaller value emphasizes the flexibility of policy learning, and a larger The value emphasizes the precise imitation of the target action. In welding applications, different welding processes may require adjustment of this parameter - for example, for high-precision laser welding, it can be set relatively high (such as 0.8); while for relatively forgiving arc welding, it can be set relatively low (such as 0.3).

[0131] During the actual training process, the target action is usually obtained by maximizing the Q-value:

[0132] ,

[0133] wherein, represents finding the action a that maximizes the Q-value; is the state-action value function, estimated by the Critic network.

[0134] Preferably, during the actual training process, the present invention adopts a batch training method, with each batch containing 256 samples. The Adam optimizer is used for parameter update, the initial learning rate is set to 0.001, and a learning rate decay strategy is adopted, multiplying the learning rate by 0.9 every 10,000 training steps. For example, when a welding robot learns the complex T-joint welding skill, such batch size and learning rate settings can achieve good performance after about 5,000 batches of training.

[0135] Soft update algorithm:

[0136] The target network of the Actor-Critic network 13 adopts a soft update algorithm for parameter update. The implementation of the soft update algorithm includes: the target network parameters are the weighted average of the current network parameters and the original target network parameters, where the smoothing coefficient is set to 0.001 to ensure smooth change of the target network parameters.

[0137] The soft update algorithm is an important technique for solving the instability of reinforcement learning. In the control of the welding process, drastic fluctuations in parameters may cause the welding torch trajectory to jitter or welding parameter mutations, resulting in welding defects. The soft update algorithm adopted by the present invention is implemented in the following manner:

[0138] ,

[0139] wherein, is the target network parameter, which is the object to be updated; is the current network parameter, which is the source of the update; is the smoothing coefficient (also known as the update rate), which controls the update speed. In the present invention, it takes the value of 0.001. This means that each time an update is made, the target network only incorporates 0.1% of the current network parameters into itself, ensuring a smooth transition of the parameters.

[0140] Similarly, the target network of the Critic network also adopts the same soft update strategy:

[0141] ,

[0142] where, are the parameters of the i-th Critic target network; are the parameters of the i-th Critic current network; is the smoothing coefficient, and its value is the same as that of the Actor network.

[0143] Preferably, the soft update operation is performed after each training step to ensure that the target network can track the changes of the current network while maintaining sufficient stability. In actual implementation, the value of the smoothing coefficient has a significant impact on the system performance: too large a value (such as 0.1) will cause the target network to change too quickly, introducing instability; too small a value (such as 0.0001) will cause the target network to update too slowly, reducing the learning efficiency. After a large number of experiments, 0.001 is the best choice to balance stability and learning efficiency.

[0144] In ship welding applications, this soft update mechanism ensures the progressive improvement of the robot control strategy and avoids fluctuations in welding quality caused by mutations. For example, when tackling difficult processes (such as butt welding of thick aluminum alloy plates), the soft update mechanism enables the robot to steadily improve its skills rather than producing drastic behavioral changes.

[0145] Training process control:

[0146] During the training process of the Actor-Critic network 13, the strategy is adjusted according to the preset step size learning rate, samples are collected at fixed steps, and the parameters are updated through the loss function until the Actor-Critic network 13 converges. After obtaining the optimal action value of the Actor-Critic network 13 in the global coordinates, the training ends.

[0147] In the present invention, the training process control includes the following key links:

[0148] First, adopt the preset step size learning rate adjustment strategy:

[0149] ,

[0150] where, is the learning rate of the k-th step of training; is the initial learning rate, set to 0.001; is the attenuation coefficient, set to 0.9; is the attenuation period, set to 10,000 steps; Denotes the floor operation. This strategy of gradually decreasing the learning rate helps to quickly explore the parameter space in the initial stage and finely tune the parameters in the later stage. In welding applications, this means that the robot can boldly try various welding strategies in the initial stage of training and focus on fine-tuning the effective strategies discovered in the later stage.

[0151] Secondly, samples are collected at fixed steps:

[0152] ,

[0153] Among them, is the experience replay pool, which stores training samples; is a single sample, which includes the current observation, the executed action, the obtained reward, and the next observation. In the present invention, 1000-step samples are collected in each training cycle and stored in the experience replay pool. The capacity of the experience replay pool is set to 100,000, and when the pool is full, the first-in-first-out strategy is adopted to manage the samples.

[0154] To improve the quality of samples, the present invention also introduces a prioritized experience replay mechanism:

[0155] ,

[0156] Among them, is the sampling probability of sample i; is the TD error of sample i, defined as ; is a small constant to prevent the probability from being zero, set to 0.01; is the priority exponent, which controls the priority intensity, set to 0.6. This makes the network pay more attention to samples with high learning value. For example, samples at the welding corner obtain higher priority than samples of straight welds.

[0157] Finally, the parameters are updated through the loss function and convergence is judged. The loss function of the Critic network is:

[0158] ,

[0159] Among them, is the loss function of the i-th Critic network; represents the data sampled from the experience replay pool D; is the discount factor, set to 0.99; is the target Critic network; is the target Actor network; is the current Critic network.

[0160] The loss function of the Actor network has been described above and will not be repeated here. Every 100 steps of samples are collected, and the network parameters are updated once. Each update randomly samples 256 samples from the experience replay pool for batch training.

[0161] The convergence judgment criterion is as follows: the average reward change rate in 10 consecutive evaluation cycles (each cycle contains 1000 steps of interaction) is less than 1%, and the average reward reaches a preset threshold. In the application of ship welding, this threshold is usually set to more than 85% of the welding quality score. For example, if the full score of welding quality is 100 points, the convergence threshold is set to 85 points.

[0162] During the training process, the - greedy exploration strategy is also adopted:

[0163] ,

[0164] where is the exploration rate at the t-th step, with the initial value set to 0.9, and it gradually decays to the minimum value of 0.05 according to The is the decay coefficient, set to 0.9999. This strategy encourages the robot to explore boldly in the initial stage of training, such as trying different welding gun postures and parameter combinations; as the training progresses, it gradually turns to using the learned experience.

[0165] Federated learning training mode:

[0166] During the training process of the described Actor-Critic network 13, the federated learning training mode is adopted, and parameter sharing between robots is achieved through the described 5G communication network.

[0167] The implementation of the federated learning training mode includes: each welding robot 1 trains the Actor-Critic network 13 based on local data; each welding robot 1 periodically uploads the network parameters to the 5G-MEC edge server; the 5G-MEC edge server performs weighted average aggregation on the uploaded parameters; the 5G-MEC edge server distributes the aggregated global parameters to each welding robot 1; each welding robot 1 updates the Actor-Critic network 13 model by combining the global parameters and local parameters.

[0168] Referring to Figure 4 , the federated learning training mode is another important innovation point of the present invention. In actual welding work, the working conditions faced by different robots vary greatly. For example, in body welding, one robot may mainly handle flat welding, while another is responsible for complex corners and intersection areas. By sharing experience through federated learning, each robot can learn from each other, significantly improving the overall learning efficiency and system robustness.

[0169] Parameter aggregation adopts the weighted average algorithm:

[0170] ,

[0171] Among them, is the global parameter, which is the result of aggregation; is the local parameter of the i-th robot; is the total number of robots in the system; is the weight coefficient of the i-th robot, satisfying . The weight can be dynamically adjusted according to the training data quality of each robot. For example:

[0172] ,

[0173] Among them, is the data quality coefficient of the i-th robot, which can be evaluated by indicators such as the TD error variance of the samples. The higher the quality, the larger the coefficient; is the effective sample number of the i-th robot.

[0174] Preferably, the present invention adopts a differential parameter sharing strategy: the parameters of the network backbone layer (such as the first three layers) are completely shared, and the parameters of the decision output layer (the last layer) are kept locally proprietary. The specific implementation is as follows:

[0175] ,

[0176] ,

[0177] Among them, is the shared layer parameter of the i-th robot, and directly uses the global parameter; is the private layer parameter of the i-th robot, and keeps the local training result. This strategy not only ensures the sharing of basic knowledge but also maintains the decision specificity of each robot, and is very suitable for application scenarios with different welding working conditions.

[0178] The fusion of local and global parameters adopts the momentum update method:

[0179] ,

[0180] Among them, is the fused parameter; is the fusion coefficient, and the initial value is set to 0.7 and gradually decreases to 0.3 as the training progresses. This means that in the initial stage of training, the robot absorbs more global knowledge; while as the training goes deeper, it relies more on its own experience.

[0181] The parameter sharing frequency is preferably set to be trained once every 1000 steps. In practical applications, such as in high-speed welding scenarios (welding speed > 5m / min), it may be necessary to increase the sharing frequency to once every 500 steps; while in low-speed and fine welding scenarios (welding speed < 1m / min), it can be reduced to once every 2000 steps to balance the communication overhead and the benefits of knowledge sharing.

[0182] Welding quality evaluation mechanism:

[0183] The present invention also includes a welding quality evaluation step: establishing multi-dimensional welding quality evaluation indicators, including geometric indicators, physical indicators, and visual indicators; adopting a quality evaluation method of multi-source information fusion, analyzing the weld appearance through a vision sensor, monitoring physical parameters, comparing with the historical optimal cases, and calculating the welding quality score; using the welding quality score as a reward signal for reinforcement learning to optimize the training of the Actor-Critic network 13.

[0184] The welding quality evaluation mechanism is a key feature that differentiates the present invention from traditional control methods. Traditional methods often only focus on the accuracy of the motion trajectory and ignore the final welding quality; the present invention directly incorporates the welding quality into the reward system of reinforcement learning, making the quality the core goal of control optimization.

[0185] The multi-dimensional welding quality evaluation indicators include:

[0186] Geometric indicators: weld width uniformity (standard deviation < 0.5mm), weld height consistency (standard deviation < 0.3mm), weld continuity (no breakpoints);

[0187] Physical indicators: molten pool stability (temperature fluctuation < 50°C), arc stability (current fluctuation < 10%), spatter degree (number of spatters per 10cm weld < 5);

[0188] Visual indicators: weld surface finish (roughness Ra < 50μm), porosity (< 1%), crack rate (0%);

[0189] These indicators are related to specific applications. For example, in the welding of pressure vessels, the porosity requirement is more stringent (< 0.5%); while in the welding of ordinary vehicle bodies, it can be relaxed to < 2%. The indicator settings should be determined according to product standards and process requirements.

[0190] The quality evaluation process of multi-source information fusion is as follows:

[0191] 1. Visual evaluation: Analyze the weld image in real time and extract geometric features and surface defects;

[0192] 2. Physical parameter monitoring: Record the fluctuation conditions of parameters such as current, voltage, and temperature;

[0193] 3. Historical comparison: Compare the similarity with pre-stored high-quality welding cases;

[0194] 4. Quality index calculation;

[0195] The calculation formula for the quality score is:

[0196] ,

[0197] Among them, is the comprehensive quality score, with a value range of [0, 100], representing the overall welding quality; is the score of the i-th quality index, usually normalized to the range [0, 100]; is the weight of the i-th quality index, satisfying , is the total number of quality indices; the weight distribution is preferably: geometric index 40%, physical index 35%, and visual index 25%.

[0198] Specific example of calculating the quality index score:

[0199] Weld width uniformity: , where is the standard deviation of the weld width, is the maximum allowable standard deviation (such as 0.5 mm);

[0200] Arc stability: , where is the maximum current fluctuation value, is the nominal current value;

[0201] The design of the reward signal is crucial. The present invention adopts the following reward calculation method:

[0202] ,

[0203] Among them, is the reward value at time t; and are the quality scores at the current and previous time steps respectively; is the quality threshold (set to 80), representing the acceptable minimum quality standard; is the reward bias coefficient (set to 0.1), used to adjust the importance of the absolute quality level. This design rewards both quality improvement ( part) and encourages maintaining a high-quality state ( part).

[0204] In practical applications, the reward calculation can be dynamically adjusted according to different working conditions. For example, in the welding of high-strength steel, Increase it to 85 to guide the robot learning with stricter standards; in aluminum alloy welding, the b value can be increased to 0.15 to place more emphasis on the absolute quality level.

[0205] Refer to Figure 5 , the present invention further includes a conflict avoidance step: real-time detecting spatial conflicts, task conflicts and timing conflicts among multiple robots; establishing a conflict cost evaluation model to calculate safety cost, efficiency cost and quality cost; adopting a multi-level conflict resolution mechanism, including time avoidance, space avoidance, task reassignment and negotiation mechanism; incorporating the conflict cost into the decision-making process of the Actor-Critic network 13 to achieve coordinated avoidance of multiple robots.

[0206] Conflict avoidance is the core issue of a multi-robot cooperation system. In scenarios such as ship section welding, when multiple robots work simultaneously, without an effective conflict avoidance mechanism, it will lead to robot collisions or welding interference. Traditional methods usually adopt predefined rules or priority mechanisms, with limited adaptability; the present invention directly integrates the conflict cost into the reinforcement learning framework to achieve an adaptive conflict avoidance strategy.

[0207] Conflict detection includes three levels:

[0208] Spatial conflict detection: Based on the geometric model of the robot, detecting the overlapping area of the working space;

[0209] Task conflict detection: Analyzing the welding task assignment to identify possible overlapping welding areas;

[0210] Timing conflict detection: Predicting the future trajectory of the robot and calculating potential intersection points;

[0211] The mathematical expression of conflict detection is:

[0212] ,

[0213] ,

[0214] ,

[0215] Among them, , , respectively represent the detection results of spatial conflict, task conflict and timing conflict, 1 indicates the existence of conflict, and 0 indicates no conflict; is an indicator function that returns 1 when the condition is true and returns 0 otherwise; is the minimum distance between robots i and j; is the safety distance threshold, usually set to 300mm; represents the overlap degree of tasks i and j; represents the predicted position of robot i at time t, Represents the predicted robot Position at time t; is the prediction time window, usually set to 2 - 5 seconds.

[0216] The conflict cost evaluation model comprehensively considers various factors:

[0217] Safety cost : Related to collision risk and arc interference risk;

[0218] Efficiency cost : Includes waiting time and detour distance;

[0219] Quality cost : Evaluates the potential impact of conflicts on welding quality;

[0220] The formula for the comprehensive conflict cost is:

[0221] ,

[0222] Among them, is the comprehensive conflict cost; , , are weight coefficients, adjusted according to specific application scenarios, with preferred values of 0.5, 0.3, 0.2 respectively, satisfying . In high-risk scenarios (such as welding explosive materials), the value of can be increased to 0.7 to emphasize safety more; in scenarios with high production tempo requirements (such as automotive welding), the value of can be increased to 0.5 to emphasize efficiency more.

[0223] The safety cost can be refined as:

[0224] ,

[0225] Among them, is the predicted minimum distance; is the risk coefficient, with a preferred value of 5. This formula makes the cost increase exponentially as the distance decreases, and the cost value increases rapidly when approaching the safety boundary.

[0226] The multi-level conflict resolution mechanism adopts different strategies according to the nature and severity of conflicts:

[0227] 1. Time avoidance: Adjust the operation timing to avoid entering the conflict area simultaneously. In practical applications, it can be achieved by rearranging the welding sequence. For example, robot A enters area two after completing area one, avoiding robot B that is operating in area two.

[0228] 2. Spatial avoidance: Generate a temporary avoidance trajectory to bypass the conflict area. For example, in T-joint welding, when two robots need to approach the same node from different directions, the welding torch attitude can be adjusted (e.g., the tilt angle is adjusted from 0° to 15°) to avoid interference with each other.

[0229] 3. Task reallocation: Dynamically adjust the task allocation when conflicts are inevitable. For example, when it is detected that the overlap degree of the regions is too high, the tasks in the overlapping region are completely allocated to one robot, and the other robot is responsible for other regions.

[0230] 4. Negotiation mechanism: The robots determine the avoidance priority through message passing. The priority can be determined based on various factors, such as:

[0231] Task urgency (tasks with approaching deadlines are prioritized);

[0232] Welding continuity (avoid interrupting long weld seams);

[0233] Quality impact (avoid interrupting critical welding points that may cause quality problems);

[0234] The conflict cost is incorporated into the reward function of reinforcement learning:

[0235] ,

[0236] where is the total reward considering conflicts; is the quality evaluation reward, that is, the welding quality reward mentioned above; is the conflict cost coefficient, which is used to balance the importance of quality optimization and conflict avoidance, and is preferably set to 0.2.

[0237] Preferably, the present invention also introduces a safety constraint strategy, sets a minimum safety distance (usually 300 mm), and when the predicted conflict risk is higher than the threshold (80%), a forced avoidance operation is triggered to ensure the safety of the system. In ship multi-robot collaborative welding, this mechanism effectively avoids potential collision risks caused by algorithm decision-making delays.

[0238] Compared with traditional multi-robot collaborative welding systems, the present invention shows significant performance advantages:

[0239] 1. Improved welding efficiency: The efficiency of multi-robot collaborative operation is increased by 65%, and the system scheduling waiting time is reduced by 70%;

[0240] 2. Improved welding quality: The weld defect rate is reduced by 40%, and the welding uniformity is increased by 35%;

[0241] 3. Enhanced system reliability: The single-point fault recovery time is shortened by 90%, and the system continuous operation time is increased by 300%;

[0242] 4. Communication overhead optimization: Compared with the central control scheme, the communication bandwidth requirement is reduced by 65%, and the control delay is reduced by 70%.

[0243] 5. Adaptability and scalability: The adaptation time for new robots to join the system is shortened by 85%, and it supports dynamic expansion to collaborative work of 10+ robots.

[0244] The present invention has been verified in applications in fields such as shipbuilding, heavy equipment manufacturing, automobile manufacturing, and aerospace, and good results have been achieved in all of them. Especially in the application of connecting and welding large hull sections, the welding efficiency is increased by 65%, and the quality qualification rate is increased by 25%, significantly reducing the production cost and cycle.

[0245] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A distributed reinforcement learning control method for multi-robot collaborative welding, characterized in that Including: Obtain the visual measurement information in the global coordinate system collected by the local visual sensors of each welding robot in the multi-robot collaborative welding system, and the robot state information obtained by each welding robot from its own sensors; Each welding robot combines the obtained visual measurement information and robot state information to form its own observation information, inputs the said own observation information into the Actor-Critic network equipped for each, executes the Actor-Critic algorithm, and obtains the action values of each welding robot; Based on the action values of the said each welding robot, generate the control actions of each welding robot, and execute the control actions to achieve the welding operation; Each welding robot feeds back the visual measurement information in the global coordinate system obtained by its own local visual sensor to the Actor-Critic network of other welding robots through the 5G communication network; During the training process of the said Actor-Critic network, use the cross-entropy loss function as the objective function, and use the two-norm of the action value output by the current policy and the target action value as the training constraint condition to execute the training; The target network of the said Actor-Critic network uses the soft update algorithm to update the parameters.

2. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 1, wherein, Each welding robot has an independent local visual sensor and a corresponding Actor-Critic network, constituting an independent reinforcement learning framework, and the said Actor-Critic network adopts the TD-3 network architecture.

3. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 1, characterized in that, The implementation of the said soft update algorithm includes: the target network parameters are the weighted average of the current network parameters and the original target network parameters, where the smoothing coefficient is set to 0.001 to ensure the smooth change of the target network parameters.

4. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 1, characterized in that, During the training process of the said Actor-Critic network, adjust the policy according to the preset step learning rate, collect samples at fixed steps, update the parameters through the said loss function, until the said Actor-Critic network converges, and end the training after obtaining the optimal action value of the Actor-Critic network in the global coordinate.

5. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 4, characterized in that During the training process of the said Actor-Critic network, adopt the training mode of federated learning, and realize the parameter sharing between robots through the said 5G communication network.

6. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 5, characterized in that, The implementation of the said federated learning training mode includes: Each welding robot trains the Actor-Critic network based on local data; Each welding robot periodically uploads the network parameters to the 5G-MEC edge server; The said 5G-MEC edge server performs weighted average aggregation on the uploaded parameters; The said 5G-MEC edge server distributes the aggregated global parameters to each welding robot; Each welding robot combines the said global parameters and local parameters to update the Actor-Critic network model.

7. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 1, characterized in that It also includes the welding quality evaluation step: Establish multi-dimensional welding quality evaluation indicators, including geometric indicators, physical indicators, and visual indicators; Adopt a quality evaluation method of multi-source information fusion, analyze the weld appearance through a visual sensor, monitor physical parameters, compare with the historical optimal cases, and calculate the welding quality score; Use the welding mass fraction as the reward signal for reinforcement learning to optimize the training of the Actor-Critic network.

8. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 1, characterized in that, It also includes a conflict avoidance step: Real-time detect spatial conflicts, task conflicts, and timing conflicts among multiple robots; Establish a conflict cost evaluation model to calculate safety cost, efficiency cost, and quality cost; Adopt a multi-level conflict resolution mechanism, including time avoidance, space avoidance, task reallocation, and negotiation mechanism; Incorporate the conflict cost into the decision-making process of the Actor-Critic network to achieve coordinated avoidance of multiple robots.

9. The distributed reinforcement learning control method for multi-robot collaborative welding according to claim 1, characterized in that, The Actor-Critic network includes an Actor network, a Critic network, and a target network. Among them, the Actor network serves as the policy network, and the Critic network serves as the evaluation network; the Actor network has a 4-layer fully connected network structure, and the number of hidden layer nodes is 256, 128, and 64 respectively. The number of output layer nodes is equal to the action dimension; the Critic network has a 4-layer fully connected network structure, and the number of nodes is 256, 128, 64, and 1 respectively.

10. A distributed reinforcement learning control system for multi-robot collaborative welding, which is used to execute the method described in any one of claims 1-9, and is characterized in that, It includes: Multiple welding robots, each equipped with a local vision sensor and a self-sensor; Each welding robot is independently equipped with an Actor-Critic network to form a distributed decision-making unit; A 5G communication module for realizing the sharing of visual measurement information and the transmission of network parameters among welding robots; A 5G-MEC edge cloud computing platform, including an access management unit, a mobility management unit, and a task management unit; The access management unit is used to provide access, authentication, and session management for edge nodes; The mobility management unit is used to manage devices and their contexts; The task management unit is used to train through an execution model to achieve distributed cooperation among welding robots.

Images