Deep learning and multi-source information fusion weld penetration prediction method and system

Abstract

The invention provides a weld penetration prediction method and system based on deep learning and multi-source information fusion. The method comprises the steps that S1, multi-source data are collected and preprocessed; the multi-source data comprises groove splicing information, welding electric signals and molten pool image information; s2, establishing a data set according to the preprocessed multi-source data information; s3, constructing a multi-source information fusion penetration prediction network based on the data set, and obtaining a penetration prediction model; and S4, inputting real-time processing data into the fusion depth prediction model, and outputting a current fusion depth prediction value. According to the method, multi-information fusion is carried out on pool shape geometrical characteristics closely related to the molten pool, welding electric signals, welding gaps, misalignment and other splicing states to predict the fusion depth, and compared with a single sensing information molten pool prediction method only considering molten pool images, welding electric signals or splicing states, the reliability and accuracy are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of robotic intelligent welding technology, specifically to a method and system for predicting weld penetration depth based on deep learning and multi-source information fusion, and more particularly to a real-time prediction method for weld penetration depth based on deep learning and multi-source information fusion. Background Technology

[0002] In welding manufacturing, penetration depth directly determines the mechanical properties of the weld joint. Penetration depth, stability, and consistency are among the most important control indicators in intelligent welding. However, the welding process is exceptionally complex, being a multivariable, nonlinearly coupled time-varying (time-delay) system. During welding, the change in penetration depth is directly related to the bevel angle, splice gap, misalignment, and fluctuations in welding parameters. Current methods for penetration depth prediction mainly include the following: Patent document CN111738369A addresses CMT welding by using a color CCD to capture images of the molten pool, extracting its area, length, and width, and then using a Support Vector Machine (SVM) or Backpropagation Neural Network (BPNN) to predict the weld penetration depth. The method predicts the penetration state and uses a BPNN for weld penetration prediction. It was trained at a current range of 90A and validated at 70A and 100A. However, this method suffers from insufficient image processing adaptability, focusing only on the molten pool, making it susceptible to interference and challenging its reliability.

[0003] Patent document CN118447332A proposes a deep residual convolutional network to predict weld depth for fillet welds, using the molten pool image as input. While the proposed model can predict not only weld depth but also penetration state, it relies heavily on pixel values ​​from the molten pool image, resulting in a large input volume and high computational demands. Furthermore, it only considers a single molten pool image variable, which may pose significant challenges in demanding application scenarios.

[0004] Patent document CN115906674A takes the welding voltage, welding current, welding gap, welding speed and the penetration depth h at time t-1 as input parameters for the first 6 time points and uses a recurrent neural network to predict the penetration depth at time t+1. Although the model proposed in this patent document considers the influence of multiple variables such as current, voltage, welding gap and welding speed and also considers the system time delay, it does not consider the influence of the local features reflected by the current parameters on the prediction results.

[0005] Patent document CN116275420A uses deep learning to predict weld penetration based on the pre-weld fillet weld width and the angle between the two plates, as well as the welding current, voltage, moving speed, and molten pool temperature during welding. However, this patent document does not propose a specific deep learning method. Furthermore, the weld width and angle are collected before welding, and thinner weldments may warp and deform during welding, potentially leading to significant prediction errors.

[0006] Therefore, changes in weld penetration are caused by a variety of factors, including fluctuations in welding current and voltage. Relying on a single feature signal is clearly insufficient to accurately determine these changes. Multi-sensor information fusion is undoubtedly a better solution. Furthermore, deep learning based on large datasets offers better adaptability and generalization capabilities compared to traditional machine learning algorithms, thus possessing greater potential for engineering applications. During welding, an increased weld gap leads to increased weld penetration and even molten pool collapse, along with increased welding stress, which easily causes welding deformation. Conversely, a too-small weld gap makes penetration difficult. Increased welding current means increased heat input and thus increased weld penetration, while decreased welding current means decreased heat input and decreased weld penetration. The dynamic characteristics of the molten pool image directly reflect changes in actual weld penetration. Experienced welders can accurately determine the penetration status on the reverse side by observing the flow behavior, size, and color changes of the molten pool on the front side, allowing for precise parameter adjustment strategies to ensure adequate weld penetration. Undoubtedly, fusing multi-source information such as workpiece processing and splicing status, changes in welding current and voltage signals, and the dynamic geometric features of the molten pool, combined with deep learning algorithms, provides better prediction accuracy and adaptability for weld penetration prediction.

[0007] Through a search and analysis of existing technical literature, it was found that there is no method for real-time prediction of weld penetration by fusing the geometric feature signals of the molten pool image, welding electrical signals, and splicing status from multiple sources and using deep learning. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for predicting weld penetration depth using deep learning and multi-source information fusion.

[0009] A method for predicting weld penetration depth based on deep learning and multi-source information fusion, provided by the present invention, includes: Step S1: Collect multi-source data and perform preprocessing; The multi-source data includes bevel splicing information, welding electrical signals, and molten pool image information; Step S2: Establish a dataset based on the preprocessed multi-source data information; Step S3: Construct a multi-source information fusion melting depth prediction network based on the dataset to obtain a melting depth prediction model; Step S4: Input the real-time processed data into the melt depth prediction model and output the current melt depth prediction value.

[0010] Preferably, step S1 includes: A camera is used to acquire images of the molten pool, and a laser 3D vision sensor is used to scan the bevel or the bevel splicing status to obtain bevel splicing information. Hall sensors are used to collect changes in current and voltage in real time during the welding process to obtain welding electrical signals.

[0011] Preferably, the preprocessing includes: Extract bevel splicing information: The collected contour point cloud data is fitted with bevel feature lines using a random sampling consensus algorithm to obtain the left and right workpiece lines, the left and right bevel slope lines, and the left and right blunt edge lines respectively. The intersection points of the lines are obtained by finding the intersection points, resulting in 4 feature points. The bevel angle, gap, and misalignment information are calculated based on the feature points. Welding electrical signal preprocessing: Kalman filtering is used for preprocessing, and the state equation and observation equation are as follows:

[0012]

[0013] In the formula, for n 3D system state vector; The state transition matrix is ​​used to model the state transition from... k -1 hour arrives k Linear relationship over time; To control the input vector, Its control matrix; This is process noise; for m 3D observation vector; The observation matrix maps the state space to the observation space. To observe noise; Molten pool shape feature identification: The molten pool is separated from the background, and its geometric features are calculated. The geometric features include the molten pool area, molten pool length, molten pool width, molten pool half-length, and molten pool trailing angle.

[0014] Preferably, step S2 includes the following sub-steps: Step S2.1: After welding, a 3D laser displacement sensor is used to scan the back of the weld to obtain the point cloud data of the weld reinforcement height on the back. Based on the contour characteristics, the Ransac straight line fitting method is used to extract the straight line segments of the test plate plane, and the boundary points of the straight line segment fitting model are used as the feature points of the weld width on the back. Step S2.2: Align multi-source data by downsampling to the lowest sampling frequency; Step S2.3: Construct the dataset using a sliding window time-series slice.

[0015] Preferably, the multi-source information fusion melting depth prediction network includes a temporal convolution branch and a long short-term memory branch, and the temporal convolution branch includes an SE compressed excitation block.

[0016] Preferably, the training process of the melt depth prediction model includes: The time series data is sliced ​​into time windows and normalized, and divided into training, validation and test sets according to time order. An MLSTM-FCN model is constructed, in which the LSTM branch captures long-term temporal dependencies and the FCN branch extracts local features. After fusion, the melt width is predicted through a fully connected layer. The MSE loss function and Adam optimizer are used for training. Dropout and early stopping mechanisms are combined to prevent overfitting and the learning rate is dynamically adjusted.

[0017] Preferably, the validation set is always located after the training set; the hyperparameters are optimized by grid search, and the feature contribution is analyzed using SHAP values ​​to verify the model's ability to capture the time delay characteristics of the melt pool.

[0018] Preferably, the generalization performance is evaluated by calculating indicators on an independent test set, and the error distribution pattern under different working conditions is analyzed; the prediction accuracy is verified by time series comparison curves and error heatmaps.

[0019] A weld penetration prediction system based on deep learning and multi-source information fusion, provided by the present invention, includes: Module M1: Collects multi-source data and performs preprocessing; The multi-source data includes bevel splicing information, welding electrical signals, and molten pool image information; Module M2: Creates a dataset based on the preprocessed multi-source data. Module M3: Construct a multi-source information fusion melting depth prediction network based on the dataset to obtain a melting depth prediction model; Module M4: Inputs real-time processed data into the melt depth prediction model and outputs the current melt depth prediction value.

[0020] Preferably, the module M1 includes: A camera is used to acquire images of the molten pool, and a laser 3D vision sensor is used to scan the bevel or the bevel splicing status to obtain bevel splicing information. Hall sensors are used to collect changes in current and voltage in real time during the welding process to obtain welding electrical signals.

[0021] Preferably, the preprocessing includes: Extract bevel splicing information: The collected contour point cloud data is fitted with bevel feature lines using a random sampling consensus algorithm to obtain the left and right workpiece lines, the left and right bevel slope lines, and the left and right blunt edge lines respectively. The intersection points of the lines are obtained by finding the intersection points, resulting in 4 feature points. The bevel angle, gap, and misalignment information are calculated based on the feature points. Assume the coordinates of the four feature points are as follows: upper left feature point P1 ( x 1,y 1), Lower left feature point P2 ( x 2, y 2), upper right feature point P3 ( x 3, y 3), lower right feature point P4 ( x 4, y 4).

[0022] The formula for calculating the bevel angle is:

[0023] The bevel gap is: G=( x 4 -x 2) The bevel offset is: M=( y 3- y 1) Welding electrical signal preprocessing: Kalman filtering is used for preprocessing, and the state equation and observation equation are as follows:

[0024]

[0025] In the formula, for n 3D system state vector; The state transition matrix is ​​used to model the state transition from... k -1 hour arrives k Linear relationship over time; To control the input vector, Its control matrix; This is process noise; for m 3D observation vector; The observation matrix maps the state space to the observation space. To observe noise; Molten pool shape feature identification: The molten pool is separated from the background, and its geometric features are calculated. The geometric features include the molten pool area, molten pool length, molten pool width, molten pool half-length, and molten pool trailing angle.

[0026] Preferably, module M2 includes the following sub-modules: Module M2.1: After welding, a 3D laser displacement sensor is used to scan the back of the weld to obtain the point cloud data of the weld reinforcement height on the back. Based on the contour characteristics, the Ransac straight line fitting method is used to extract the straight line segments of the test plate plane, and the boundary points of the straight line segment fitting model are used as the feature points of the weld width on the back. Module M2.2: Aligns multi-source data by downsampling to the lowest sampling frequency; Module M2.3: Constructs a dataset using time-series slices via a sliding window.

[0027] Preferably, the multi-source information fusion melting depth prediction network includes a temporal convolution branch and a long short-term memory branch, and the temporal convolution branch includes an SE compressed excitation block.

[0028] Preferably, the training process of the melt depth prediction model includes: The time series data is sliced ​​into time windows and normalized, and divided into training, validation and test sets according to time order. An MLSTM-FCN model is constructed, in which the LSTM branch captures long-term temporal dependencies and the FCN branch extracts local features. After fusion, the melt width is predicted through a fully connected layer. The MSE loss function and Adam optimizer are used for training. Dropout and early stopping mechanisms are combined to prevent overfitting and the learning rate is dynamically adjusted.

[0029] Preferably, the validation set is always located after the training set; the hyperparameters are optimized by grid search, and the feature contribution is analyzed using SHAP values ​​to verify the model's ability to capture the time delay characteristics of the melt pool.

[0030] Preferably, the generalization performance is evaluated by calculating indicators on an independent test set, and the error distribution pattern under different working conditions is analyzed; the prediction accuracy is verified by time series comparison curves and error heatmaps.

[0031] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention integrates multi-source information closely related to the weld pool, such as the pool shape geometry, welding electrical signals, and splicing states like weld gaps and misalignments, to predict the weld depth. Compared to single-sensor information weld pool prediction methods that only consider the weld pool image, welding electrical signals, or splicing states, this invention greatly improves reliability and accuracy.

[0032] 2. This invention employs a composite deep learning model, MLSTM-FCN, which integrates temporal convolutional networks and long short-term memory networks, and incorporates an attention mechanism module and an SE compression-excitation module. This model considers both the local features of the time series and long-term historical dependency information, conforming to the characteristics of a melt pool system, which is a multivariable, nonlinearly coupled, and time-delayed system. As a result, it can obtain more accurate prediction results, better adaptability, and better generalization ability.

[0033] 3. This invention replaces the molten pool image with dynamic geometric features and integrates them with welding electrical signals, bevel splicing gaps, misalignments, bevel angles, etc., as input. Compared with deep learning methods that use molten pool image information as input, this invention significantly reduces input data, has low computing power requirements, fast calculation speed, and stronger real-time performance, laying a good foundation for realizing online real-time closed-loop control of weld depth. Attached Figure Description

[0034] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of real-time melt depth prediction based on deep learning and multi-source information fusion.

[0035] Figure 2 A schematic diagram defining the four feature points of the bevel.

[0036] Figure 3 This is a semantic segmentation map of the melt pool contour based on VGG16-UNET.

[0037] Figure 4 This is a geometric feature extraction diagram of the molten pool shape based on the molten pool classification mask.

[0038] Figure 5 This is a diagram of a real-time prediction model for a fused melt pool based on MLTSM-FCN using multi-source information fusion.

[0039] Figure 6 This is a verification diagram of the model's real-time prediction effect on melting depth.

[0040] The diagram shows: The image of the molten pool represents the image of the molten pool.

[0041] Pixel-level classification refers to classification at the pixel level.

[0042] Convolution refers to the process of creating a convolutional network.

[0043] Max Pooling represents maximum pooling.

[0044] Skip connection indicates a skip connection.

[0045] Up Sampling refers to upsampling.

[0046] Block copied means block copying.

[0047] Workpiece groove model refers to the bevel model of the workpiece.

[0048] Current represents electric current.

[0049] Ransac stands for Random Sampling Consensus Algorithm.

[0050] Misalignment indicates misaligned edges.

[0051] Encode means encoding.

[0052] Decode means to decode.

[0053] Output means output.

[0054] Kalman filter refers to the Kalman filter.

[0055] Voltage represents voltage.

[0056] Feature fusion refers to the fusion of features.

[0057] MLSTM stands for Multivariate Long Short-Term Memory Network.

[0058] LSTM stands for Long Short-Term Memory Network.

[0059] FCN stands for Fully Convolutional Network.

[0060] Data Acquisition refers to data collection.

[0061] Feature Extraction refers to the extraction of features.

[0062] Slicing and alignment refer to slicing and alignment.

[0063] Back Bead Width Prediction refers to the prediction of the width of the back bead.

[0064] Ground truth represents the actual value.

[0065] MiniRocket stands for Mini Rocket Algorithm.

[0066] Length indicates the length. Detailed Implementation

[0067] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0068] The purpose of this invention is to address the shortcomings and defects in existing technologies (such as high computational power requirements, complex models, low accuracy and reliability, and insufficient generalization ability in multi-sensor information fusion) by providing a multi-source information fusion method that combines molten pool characteristics, welding electrical signals, splicing status, etc., and adopts a temporal convolution + long short-term memory fusion network prediction algorithm to reduce computational power requirements, improve accuracy and generalization ability, realize online real-time prediction of welding molten pool, and lay the foundation for real-time closed-loop control of weld depth.

[0069] Reference Figure 1 As shown, this invention provides a quantitative real-time prediction method for weld depth based on deep learning and the fusion of multi-source information such as molten pool images and welding electrical signals. During the welding process, molten pool images, weldment splicing status, and welding current and voltage signals are first acquired. The data are preprocessed to form time-series data pairs synchronized with the welding process in real time, which are then input into a deep learning model. This allows for real-time prediction of weld depth through multi-sensor information fusion, and includes the following steps: Step S1: Design welding experiments and simultaneously collect information from multiple sources. Specifically: During welding, a camera is used to capture images of the molten pool, a laser 3D vision sensor is used to scan the bevel or bevel splicing status, and Hall effect sensors are used to collect real-time data on changes in current and voltage during the welding process. The diversity of data must be fully considered when designing the experiments. Different bevel splicing states (gap, misalignment, etc.), different current parameter ranges, random current variations within the same weld bead, and other scenarios that fully reflect actual working conditions should be designed.

[0070] Step S2: Multi-source sensor information data processing Step S201: Extraction of bevel splicing status information The collected contour point cloud data is first fitted with the bevel feature lines using the Random Sample Consensus (Ransac) algorithm to obtain the left and right workpiece lines, the left and right bevel slope lines, and the left and right blunt edge lines. Then, the intersection points of the lines are obtained by finding the four feature points. Based on the four feature points of the bevel, the bevel angle, gap, misalignment and other information can be obtained.

[0071] Step S202: Welding electrical signal filtering During the welding process, electrical signals are distorted due to power supply noise and electromagnetic interference from surrounding equipment. Therefore, further filtering preprocessing is required before inputting the signal into the model to remove noise and obtain a clean electrical signal waveform. Kalman filtering, also known as linear quadratic estimation, estimates the joint probability distribution of variables for each time range. By reducing noise in the voltage signal through two stages of prediction and verification, it is more accurate than prediction based on a single observation.

[0072] Its state equation and observation equation are shown below.

[0073]

[0074]

[0075] In the formula for n A dimensional system state vector (such as position, velocity, etc.), the target to be estimated; The state transition matrix is ​​used to model the state transition from... k -1 hour arrives k Linear relationship over time; To control the input vector (optional). Its control matrix; This is process noise; for m 3D observation vector; The observation matrix maps the state space to the observation space. To observe noise.

[0076] Step S203: Identification of Molten Pool Shape Features (1) Molten pool contour segmentation like Figure 3 As shown, the deep learning VGG16-UNet semantic network is used as the melt pool segmentation model to separate the melt pool from the background. The input is a [...] n , n The image of the molten pool [3] is used to output two categories: molten pool and background. The value of each channel represents the probability that the pixel belongs to the corresponding category, and the sum of the probabilities is 1. The segmentation clearly shows the category of each pixel, and the category with the highest predicted probability is taken as the classification result. The output image size remains [3]. n × n .

[0077] (2) Calculation of geometric characteristics of the molten pool like Figure 4As shown, a bicubic interpolation algorithm is used to upsample the molten pool segmentation image to restore it to the original image size, obtaining the molten pool mask image. Then, the shape and geometric features of the molten pool contour are extracted. The geometric features include: molten pool area, molten pool length, width, molten pool half-length, and molten pool trailing angle, etc.

[0078] Step S3: Dataset Labeling Step S301: Extraction of back weld width After welding, a 3D laser displacement sensor was used to scan the back of the weld to obtain point cloud data of the weld reinforcement height. Based on the contour characteristics, the Ransac straight line fitting method was used to extract straight line segments from the test plate plane, and the boundary points of the straight line segment fitting model were used as the feature points of the weld width on the back side.

[0079] Step S302: Alignment of multi-source timing input data Since the splicing status, welding current and voltage signals, and molten pool images are acquired by laser scanning sensors, Hall current and voltage sensors, and molten pool monitoring cameras, respectively, and the signal acquisition frequencies are different, and the splicing status can be acquired before welding or in real time during welding depending on the system configuration, the data is aligned according to the following principles: 1. When the stitching process uses a laser 3D vision sensor for pre-welding data acquisition, the higher acquisition frequency should be set to an integer multiple of the lowest sampling frequency, and uniformly set to 1 / of the lowest sampling frequency. t Hz is used for downsampling alignment. The values ​​of current, voltage, and gap misalignment are determined by left and right sides respectively. t The mean is calculated within a 2-second window, and the geometric parameters of the molten pool are replaced by linear interpolation results.

[0080] 2. When the splicing process is achieved by using a laser 3D vision sensor to collect data in real time during welding, the data extraction frequency should be aligned. The higher sampling frequency should be set to 1 / ... t Hz as an integer multiple, and uniformly based on the lowest sampling frequency 1 / t Alignment is achieved through two methods: first, downsampling alignment; and second, alignment of multiple signals with the welding position. During welding, real-time scanning is performed, and the information collected by the sensors is ahead of the welding position. Therefore, a program delay method should be used to align the splicing state with the actual welding position. The values ​​of current, voltage, and gap misalignment are determined by the left and right sides respectively. t The mean is calculated within a 2-second window, and the geometric parameters of the molten pool are replaced by linear interpolation results.

[0081] Step S302: Align the multi-source timing input data set with the output data. After data downsampling, time-based data was obtained. ts For time series with a step size, further time series slicing is needed, using a sliding window method with a sliding window length of [value missing]. lThe sliding step size is t (The length of the data sequence is) l / t The window slides sequentially, extracting the time-series data of the feature quantities within the sliding window as input, and extracting the target value at the front of the window, i.e., the width of the back weld at the current moment, as output, to construct a dataset composed of time series of multi-source feature information.

[0082] Step S4: Construct a multi-source information fusion melting depth prediction network Step S401: Design a composite network structure of temporal convolution + long short-term memory The model consists of two branches: a temporal convolution branch and a long short-term memory (LSTM) branch. The temporal convolution branch employs a fully convolutional network model, containing three FCN convolutional blocks. The basic structure of the FCN includes a Conv1D one-dimensional convolutional layer, a BatchNorm1d batch normalization layer, and a ReLU activation function layer. A squeeze-and-excite block (SE block) is added after the first two convolutional blocks. The number of input channels in the first convolutional layer is the same as the dimension of the time series data. The number of output channels in the three convolutional layers are set to 128, 256, and 128, respectively. A global pooling layer is added after the last convolutional block to reduce model parameters. The LSTM branch dimensionalizes the time series data and then passes it to the LSTM block. The LSTM block consists of either a regular LSTM layer or an LSTM layer with attention. Dropout is then added to alleviate overfitting. The outputs of the global pooling layer and the LSTM block are concatenated and passed to the SoftMax classification layer to finally output the prediction result.

[0083] Step S402: Design the Squeeze-and-Excite Block (SE Block) The Squeeze compression operation will... Perform global average pooling. N × Q All values ​​in the space are compressed into a single scalar to obtain information about the channel dimension. z Then, the Excite module is used to perform excitation operations to obtain channel-dimensional features.

[0084] This is achieved using two fully connected layers, the first of which will be 1×1× F The tensor of dimension 1 is compressed to 1×1× F / rThe parameters are reduced (r is the compression ratio), and the second fully connected layer restores the features to 1×1×. F dimension.

[0085] Step S5: Training, validation, and testing of the melt depth prediction network (1) Model parameter configuration: Before training the network, the training method is configured, including setting the learning rate, decay method and loss function, as well as the batch size, number of iterations, epochs, etc.

[0086] (2) Data Preparation: After the network is constructed, the collected multi-source information is preprocessed. First, the scanned weld point cloud is processed to obtain splicing status information such as bevel gap and misalignment of the weldment. Kalman filtering is used to remove noise from the welding current and voltage signals. The molten pool image is segmented and identified, and then the geometric features such as the length, width, half-length and tail angle of the molten pool are calculated. The width is extracted from the back weld width point cloud data after welding. The preprocessed welding voltage and current signals, bevel splicing status, and geometric features of the molten pool are used as inputs, and the width of the molten pool on the back is used as output. Timing alignment is performed according to the lowest sampling frequency. (3) Model Training: First, the time series data is sliced ​​into time windows and normalized, then divided into training, validation, and test sets according to time sequence. For example... Figure 5 As shown, an MLSTM-FCN model is constructed, where the LSTM branch captures long-term temporal dependencies and the FCN branch extracts local features. After fusion, the melt width is predicted through a fully connected layer. The MSE loss function and Adam optimizer are used for training, combined with Dropout and early stopping mechanisms to prevent overfitting, and the learning rate is dynamically adjusted to improve convergence efficiency. (4) Model Validation: Cross-validation based on rolling time windows ensures temporal causality, and the validation set is always located after the training data. Hyperparameters (such as the number of hidden units in LSTM and the kernel size) are optimized through grid search, and the contribution of features is analyzed using SHAP values ​​to verify the model's ability to capture the characteristics of the melt pool delay (such as the matching degree between the output and the actual delay of the melt width), ensuring the robustness and physical consistency of the model on historical data; (5) Model testing: Calculate MAE, RMSE and other indicators on independent test sets to evaluate generalization performance and analyze the error distribution under different working conditions. Visually verify the prediction accuracy through time series comparison curves and error heatmaps, test whether the model inference speed meets the real-time monitoring requirements (e.g., single prediction time < 50ms), and verify whether the response under input disturbances conforms to the physical laws of welding process (e.g., increased current leads to increased penetration).

[0087] Step S6: Multi-source information fusion to achieve real-time prediction of melting depth like Figure 6As shown, during welding, the molten pool image is acquired by the molten pool monitoring camera equipped with the intelligent welding system, the welding current and voltage signals are acquired in real time by the Hall sensor, and the splicing information of the welding groove is acquired synchronously by the laser 3D sensor. Then, after molten pool geometric feature extraction, electrical signal filtering and noise reduction, and groove splicing status information extraction, and after time alignment, the data is input into a pre-trained multivariate long short-term memory network-temporal fully convolutional network for discrimination. This enables real-time identification of the weld depth during the welding process, laying the foundation for closed-loop real-time control of the weld penetration.

[0088] Example 1 Reference Figure 1 As shown, a method for predicting weld penetration depth using deep learning and multi-source information fusion includes the following steps: Step S1: Collect multi-source data information for the welding process. During welding, cameras are used to capture images of the molten pool, laser 3D vision sensors are used to scan the bevel or bevel splicing status, and Hall effect sensors are used to collect real-time data on current and voltage changes during welding. The diversity of data must be fully considered when designing experiments. Different working conditions should be designed, including different bevel splicing states (gap, misalignment, etc.), different current parameter ranges, and random current variations within the same weld bead, to fully reflect the changes in actual working conditions.

[0089] Specifically, the experiments were conducted with the following parameters: Fixed-parameter experiments: welding current 140A-220A, welding speed 4mm / s-8mm / s, step size 1mm / s; Variable-parameter experiments: welding speed 8mm / s, with three conditions: fixed gap 1mm, gap randomly increasing from 0.5mm to 1.5mm, and gap randomly decreasing from 1.5mm to 0.5mm; no misalignment, uniform increase from -1mm to 1mm, and uniform decrease from 1mm to -1mm; and a current with a mean of 200A and a variance of 20A following a normal distribution, uniform increase from 175A to 225A, and uniform decrease from 225A to 175A. Based on these experimental parameters, a total of 27 sets of combined welding current and welding speed experiments, and 27 sets of experiments with cross-variable parameters for gap, misalignment, and current were designed, for a total of 54 sets of experiments.

[0090] Step S2: Multi-source sensor information data processing Step S201: Extraction of bevel splicing status information The collected contour point cloud data is first fitted with the bevel feature lines using the Random Sample Consensus (Ransac) algorithm to obtain the left and right workpiece lines, the left and right bevel slope lines, and the left and right blunt edge lines. Then, the intersection points of the lines are obtained by finding the four feature points. Based on the four feature points of the bevel, the bevel angle, gap, misalignment and other information can be obtained.

[0091] like Figure 2 As shown, assume the coordinates of the four feature points are as follows: upper left feature point P1 ( x 1, y 1), Lower left feature point P2 ( x 2, y 2), upper right feature point P3 ( x 3, y 3), lower right feature point P4 ( x 4, y 4).

[0092] The formula for calculating the bevel angle is:

[0093] The bevel gap is: G=( x 4- x 2) The bevel offset is: M=( y 3- y 1) Step S202: Welding electrical signal preprocessing During the welding process, electrical signals are distorted due to power supply noise and electromagnetic interference from surrounding equipment. Therefore, further filtering preprocessing is required before inputting the signal into the model to remove noise and obtain a clean electrical signal waveform. Kalman filtering, also known as linear quadratic estimation, estimates the joint probability distribution of variables for each time range. By reducing noise in the voltage signal through two stages of prediction and verification, it is more accurate than prediction based on a single observation.

[0094] Its state equation and observation equation are shown below.

[0095]

[0096]

[0097] In the formula for n A dimensional system state vector (such as position, velocity, etc.), the target to be estimated; The state transition matrix is ​​used to model the state transition from... k -1 hour arrives k Linear relationship over time; To control the input vector (optional). Its control matrix; This is process noise; for m 3D observation vector; The observation matrix maps the state space to the observation space. To observe noise.

[0098] Step S203: Identification of Molten Pool Shape Features (1) Segmentation of molten pool region A deep learning VGG16-UNet semantic network is used as the melt pool segmentation model to separate the melt pool from the background. The input is a melt pool image of size [512, 512, 3]. The output is two categories: melt pool and background. The value of each channel represents the probability that the pixel belongs to the corresponding category, and the sum of the probabilities is 1. The segmentation can clearly show the category of each pixel. The category with the highest predicted probability is taken as the classification result. The output image size is still 512×512.

[0099] (2) Calculation of geometric characteristics of the molten pool The molten pool segmentation image is upsampled using a bicubic interpolation algorithm to restore it to the original image size of 1024×1024, obtaining the molten pool mask image. Then, the shape and geometric features of the molten pool contour are extracted. Geometric features include: molten pool area, molten pool length, width, molten pool half-length, and molten pool trailing angle, etc.

[0100] Step S3: Dataset Labeling Step S301: Extraction of back weld width After welding, a 3D laser displacement sensor was used to scan the back of the weld to obtain point cloud data of the weld reinforcement height. Based on the contour characteristics, the Ransac straight line fitting method was used to extract straight line segments from the test plate plane, and the boundary points of the straight line segment fitting model were used as the feature points of the weld width on the back side.

[0101] Step S302: Alignment of multi-source timing input data The laser measuring instrument measures the gap, misalignment, and bevel angle of the bevel in real time online. The sampling frequency is set to 500 Hz, the current and voltage sampling frequency is 1000 Hz, and the molten pool camera captures images at approximately 50 fps. Therefore, a uniform downsampling frequency of 50 Hz is used. The values ​​of current, voltage, gap, and misalignment are calculated by averaging over a 0.01 s window on each side, while the molten pool geometric parameters are replaced by linear interpolation results.

[0102] Step S302: Align the multi-source timing input data set with the output data. After data downsampling, a time series with a step size of 0.02s was obtained. Further time series slicing is required. A sliding window method is adopted, with a sliding window length of 1s (i.e., 50 data points) and a sliding step size of 0.02s. The window slides sequentially, and the time series data of the feature quantities within the sliding window are taken out as input. The target value at the front of the window, i.e., the width of the back weld at the current moment, is taken out as output, and a dataset composed of time series of multi-source feature information is constructed.

[0103] Step S4: Construct a multi-source information fusion melting depth prediction network Step S401: Design a composite network structure of temporal convolution + long short-term memory The model consists of two branches: a temporal convolution branch and a long short-term memory (LSTM) branch. The temporal convolution branch employs a fully convolutional network (FCN) model, containing three FCN convolutional blocks. The basic FCN structure includes a Conv1D one-dimensional convolutional layer, a BatchNorm1d batch normalization layer, and a ReLU activation function layer. A squeeze-and-excite (SE) block is added after the first two convolutional blocks. The number of input channels in the first convolutional layer is the same as the dimension of the time series data. The output channels of the three convolutional layers are set to 128, 256, and 128, respectively. A global pooling layer is added after the last convolutional block to reduce model parameters. The LSTM branch dimensionalizes the time series data before passing it to an LSTM block. The LSTM block consists of either a regular LSTM layer or an LSTM layer with attention, and Dropout is then added to alleviate overfitting. The outputs of the global pooling layer and the LSTM block are concatenated and then passed to a SoftMax classification layer to output the final prediction result.

[0104] Step S402: Design the Squeeze-and-Excite Block (SE Block) The Squeeze compression operation will... Perform global average pooling. N × Q All values ​​in the spatial dimension are compressed into a single scalar to obtain channel-dimensional information z. Then, the Excite module is used for excitation operations to obtain channel-dimensional features. This is achieved using two fully connected layers; the first fully connected layer compresses the 1×1×... F The tensor of dimension 1 is compressed to 1×1× F / r Reduce parameters ( r (This is the compression ratio), and the second fully connected layer then restores the features to 1×1×. F dimension.

[0105] Step S5: Training, validation, and testing of the melt depth prediction network (1) Model parameter configuration: Before training the network, the training method is configured, including setting the learning rate, decay method and loss function, as well as the batch size, number of iterations, epochs, etc.

[0106] (2) Data Preparation: After the network is constructed, the collected multi-source information is preprocessed. First, the scanned weld point cloud is processed to obtain splicing status information such as bevel gap and misalignment of the weldment. Kalman filtering is used to remove noise from the welding current and voltage signals. The molten pool image is segmented and identified, and then the geometric features such as the length, width, half-length and tail angle of the molten pool are calculated. The width is extracted from the back weld width point cloud data after welding. The preprocessed welding voltage and current signals, bevel splicing status, and geometric features of the molten pool are used as inputs, and the width of the molten pool on the back is used as output. Timing alignment is performed according to the lowest sampling frequency. (3) Model Training: First, the time-series data was sliced ​​and normalized, resulting in 4800 sets of data. These were divided into a training set (75%), a validation set (10%), and a test set (15%) according to time sequence. An MLSTM-FCN model was constructed, where the LSTM branch captures long-term temporal dependencies and the FCN branch extracts local features. After fusion, the melt width was predicted through a fully connected layer. The MSE loss function and Adam optimizer were used for training. Dropout and early stopping mechanisms were combined to prevent overfitting, and the learning rate was dynamically adjusted to improve convergence efficiency. (4) Model Validation: Cross-validation based on rolling time windows ensures temporal causality, and the validation set is always located after the training data. Hyperparameters (such as the number of hidden units in LSTM and the kernel size) are optimized through grid search, and the contribution of features is analyzed using SHAP values ​​to verify the model's ability to capture the time delay characteristics of the melt pool (such as the matching degree between the output and the actual delay of the melt width), ensuring the robustness and physical consistency of the model on historical data.

[0107] (5) Model testing: Calculate MAE, RMSE and other indicators on independent test sets to evaluate generalization performance and analyze the error distribution under different working conditions. Visually verify the prediction accuracy through time series comparison curves and error heatmaps, test whether the model inference speed meets the real-time monitoring requirements (e.g., single prediction time < 100ms), and verify whether the response under input disturbances conforms to the physical laws of welding process (e.g., increased current leads to increased penetration).

[0108] Step S6: Multi-source information fusion to achieve real-time prediction of melting depth During welding, the molten pool image is acquired by the molten pool monitoring camera equipped with the intelligent welding system, the welding current and voltage signals are acquired in real time by the Hall sensor, and the splicing information of the welding groove is acquired synchronously by the laser 3D sensor. Then, after the geometric features of the molten pool are extracted, the electrical signals are filtered and denoised, the groove splicing status information is extracted, and after time alignment, the data is sliced ​​according to the time window and input into the pre-trained multivariate long short-term memory network-temporal fully convolutional fusion network for discrimination. This enables real-time identification of the weld depth during the welding process, laying the foundation for closed-loop real-time control of the weld penetration.

[0109] Example 2 A weld penetration prediction system based on deep learning and multi-source information fusion, comprising: Module M1: Collects multi-source data and performs preprocessing; The multi-source data includes bevel splicing information, welding electrical signals, and molten pool image information; Module M2: Creates a dataset based on the preprocessed multi-source data. Module M3: Construct a multi-source information fusion melting depth prediction network based on the dataset to obtain a melting depth prediction model; Module M4: Inputs real-time processed data into the melt depth prediction model and outputs the current melt depth prediction value.

[0110] The module M1 includes: A camera is used to acquire images of the molten pool, and a laser 3D vision sensor is used to scan the bevel or the bevel splicing status to obtain bevel splicing information. Hall sensors are used to collect changes in current and voltage in real time during the welding process to obtain welding electrical signals.

[0111] The preprocessing includes: Extract bevel splicing information: The collected contour point cloud data is fitted with bevel feature lines using a random sampling consensus algorithm to obtain the left and right workpiece lines, the left and right bevel slope lines, and the left and right blunt edge lines respectively. The intersection points of the lines are obtained by finding the intersection points, resulting in 4 feature points. The bevel angle, gap, and misalignment information are calculated based on the feature points. Welding electrical signal preprocessing: Kalman filtering is used for preprocessing, and the state equation and observation equation are as follows:

[0112]

[0113] In the formula, for n 3D system state vector; The state transition matrix is ​​used to model the state transition from... k -1 hour arrives k Linear relationship over time; To control the input vector, Its control matrix; This is process noise; for m 3D observation vector; The observation matrix maps the state space to the observation space. To observe noise; Molten pool shape feature identification: The molten pool is separated from the background, and its geometric features are calculated. The geometric features include the molten pool area, molten pool length, molten pool width, molten pool half-length, and molten pool trailing angle.

[0114] Module M2 includes the following sub-modules: Module M2.1: After welding, a 3D laser displacement sensor is used to scan the back of the weld to obtain the point cloud data of the weld reinforcement height on the back. Based on the contour characteristics, the Ransac straight line fitting method is used to extract the straight line segments of the test plate plane, and the boundary points of the straight line segment fitting model are used as the feature points of the weld width on the back. Module M2.2: Aligns multi-source data by downsampling to the lowest sampling frequency; Module M2.3: Constructs a dataset using time-series slices via a sliding window.

[0115] The multi-source information fusion melting depth prediction network includes a temporal convolution branch and a long short-term memory branch, and the temporal convolution branch includes an SE compressed excitation block.

[0116] The training process of the melt depth prediction model includes: The time series data is sliced ​​into time windows and normalized, and divided into training, validation and test sets according to time order. An MLSTM-FCN model is constructed, in which the LSTM branch captures long-term temporal dependencies and the FCN branch extracts local features. After fusion, the melt width is predicted through a fully connected layer. The MSE loss function and Adam optimizer are used for training. Dropout and early stopping mechanisms are combined to prevent overfitting and the learning rate is dynamically adjusted.

[0117] The validation set is always located after the training set; hyperparameters are optimized through grid search, and feature contribution is analyzed using SHAP values ​​to verify the model's ability to capture the time delay characteristics of the melt pool.

[0118] The generalization performance was evaluated by calculating metrics on an independent test set, and the error distribution patterns under different operating conditions were analyzed. The prediction accuracy was verified by time series comparison curves and error heatmaps.

[0119] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0120] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for predicting weld penetration depth using deep learning and multi-source information fusion, characterized in that, include: Step S1: Collect multi-source data and perform preprocessing; The multi-source data includes bevel splicing information, welding electrical signals, and molten pool image information; Step S2: Establish a dataset based on the preprocessed multi-source data information; Step S3: Construct a multi-source information fusion melting depth prediction network based on the dataset to obtain a melting depth prediction model; Step S4: Input the real-time processed data into the melt depth prediction model and output the current melt depth prediction value.

2. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 1, characterized in that, Step S1 includes: A camera is used to acquire images of the molten pool, and a laser 3D vision sensor is used to scan the bevel or the bevel splicing status to obtain bevel splicing information. Hall sensors are used to collect changes in current and voltage in real time during the welding process to obtain welding electrical signals.

3. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 2, characterized in that, The preprocessing includes: Extract bevel splicing information: The collected contour point cloud data is fitted with bevel feature lines using a random sampling consensus algorithm to obtain the left and right workpiece lines, the left and right bevel slope lines, and the left and right blunt edge lines respectively. The intersection points of the lines are obtained by finding the intersection points, resulting in 4 feature points. The bevel angle, gap, and misalignment information are calculated based on the feature points. Welding electrical signal preprocessing: Kalman filtering is used for preprocessing, and the state equation and observation equation are as follows: In the formula, for n 3D system state vector; The state transition matrix is ​​used to model the state transition from... k -1 hour arrives k Linear relationship over time; To control the input vector, Its control matrix; This is process noise; for m 3D observation vector; The observation matrix maps the state space to the observation space. To observe noise; Molten pool shape feature identification: The molten pool is separated from the background, and its geometric features are calculated. The geometric features include the molten pool area, molten pool length, molten pool width, molten pool half-length, and molten pool trailing angle.

4. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S2.1: After welding, a 3D laser displacement sensor is used to scan the back of the weld to obtain the point cloud data of the weld reinforcement height on the back. Based on the contour characteristics, the Ransac straight line fitting method is used to extract the straight line segments of the test plate plane, and the boundary points of the straight line segment fitting model are used as the feature points of the weld width on the back. Step S2.2: Align multi-source data by downsampling to the lowest sampling frequency; Step S2.3: Construct the dataset using a sliding window time-series slice.

5. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 4, characterized in that, The multi-source information fusion melting depth prediction network includes a temporal convolution branch and a long short-term memory branch, and the temporal convolution branch includes an SE compressed excitation block.

6. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 4, characterized in that, The training process of the melt depth prediction model includes: The time series data is sliced ​​into time windows and normalized, and divided into training, validation and test sets according to time order. An MLSTM-FCN model is constructed, in which the LSTM branch captures long-term temporal dependencies and the FCN branch extracts local features. After fusion, the melt width is predicted through a fully connected layer. The MSE loss function and Adam optimizer are used for training. Dropout and early stopping mechanisms are combined to prevent overfitting and the learning rate is dynamically adjusted.

7. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 6, characterized in that, The validation set is always located after the training set; hyperparameters are optimized through grid search, and feature contribution is analyzed using SHAP values ​​to verify the model's ability to capture the time delay characteristics of the melt pool.

8. The weld penetration prediction method based on deep learning and multi-source information fusion according to claim 6, characterized in that, The generalization performance was evaluated by calculating metrics on an independent test set, and the error distribution patterns under different operating conditions were analyzed. The prediction accuracy was verified by time series comparison curves and error heatmaps.

9. A weld penetration prediction system based on deep learning and multi-source information fusion, characterized in that, include: Module M1: Collects multi-source data and performs preprocessing; The multi-source data includes bevel splicing information, welding electrical signals, and molten pool image information; Module M2: Creates a dataset based on preprocessed multi-source data. Module M3: Construct a multi-source information fusion melting depth prediction network based on the dataset to obtain a melting depth prediction model; Module M4: Inputs real-time processed data into the melt depth prediction model and outputs the current melt depth prediction value.

10. The weld penetration prediction system based on deep learning and multi-source information fusion according to claim 9, characterized in that, The module M1 includes: A camera is used to acquire images of the molten pool, and a laser 3D vision sensor is used to scan the bevel or the bevel splicing status to obtain bevel splicing information. Hall sensors are used to collect changes in current and voltage in real time during the welding process to obtain welding electrical signals.

Images