Welding real-time monitoring method based on multi-source information fusion and artificial intelligence

Abstract

The invention relates to the technical field of quality monitoring in the welding process, and provides a real-time welding monitoring method based on multi-source information fusion and artificial intelligence, which comprises the following steps of: simultaneously monitoring welding information in the welding process in real time by using multiple sensors, preprocessing the welding information, and extracting multi-dimensional features from the welding information in parallel; standardizing the multi-dimensional features, and splicing the standardized multi-dimensional features to form a high-dimensional fusion feature vector; a maximum correlation minimum redundancy criterion is adopted to screen the high-dimensional fusion features, and a group of optimized fusion feature subsets with high discrimination and low redundancy are screened out; state recognition based on a double-layer fusion decision model, wherein the optimized fusion feature subset is input into the double-layer fusion decision model, and the final welding state category and the corresponding confidence coefficient are obtained; through the dynamic weighting strategy of the double-layer fusion decision model, the prediction results of the heterogeneous machine learning model are adaptively integrated, the differentiated analysis advantages of different models on features are fully utilized, and the accuracy and robustness of defect identification are significantly improved.

Description

Technical Field

[0001] This invention relates to the technical field of welding process quality monitoring, and in particular to a real-time welding monitoring method based on multi-source information fusion and artificial intelligence. Background Technology

[0002] Welding, as a key core process in modern manufacturing, is widely used in the manufacturing of ships, bridges, pressure vessels and various heavy equipment. Its connection quality directly determines the structural integrity and service safety of the above-mentioned equipment, and plays an irreplaceable role in ensuring the manufacturing quality of major equipment.

[0003] Traditional welding quality control relies heavily on the individual skills and experience of welders, primarily through post-weld non-destructive testing (such as X-ray inspection and ultrasonic testing) for quality verification. This method suffers from significant lag, failing to detect and correct defects in real time during manufacturing, leading to high rework costs and low production efficiency. To achieve online monitoring of the welding process, various sensing technologies have been introduced. Single-sensor-based monitoring methods are currently the mainstream, mainly falling into three categories: First, visual sensing, which infers the penetration state by analyzing the shape, size, and texture of the molten pool image, but is susceptible to interference from strong arc light, spatter, and fumes; second, arc sensing, which assesses arc stability by monitoring the waveform characteristics of welding current and voltage, but lacks sensitivity to weld surface defects (such as undercut and weld beads); and third, acoustic sensing, which identifies process anomalies by analyzing the spectral characteristics of arc sound, but environmental noise easily contaminates the signal. In complex real-world welding environments, these single-signal-source methods have limited information dimensions, weak anti-interference capabilities, and unreliable reliability. To overcome the limitations of single sensors, information fusion technology has been introduced into the field of welding monitoring. Existing technologies typically involve simply splicing together signals from a limited number of sensors (e.g., two or three), then inputting the data into a classifier (e.g., support vector machine, BP neural network) for state determination. However, these methods have significant drawbacks: First, feature extraction relies heavily on human experience and lacks adaptability to different working conditions; second, to achieve high recognition accuracy, computationally complex deep models are often used, resulting in large system response delays and failing to meet the stringent real-time requirements of industrial environments (typically requiring diagnosis within hundreds of milliseconds); finally, simple feature or decision-level fusion fails to deeply explore the inherent complementary and synergistic mechanisms between multiple signal sources, leading to performance bottlenecks. This is especially true in highly efficient and advanced welding processes such as double-sided double-arc welding and laser-arc hybrid welding, where the processes are more complex and multiple signals experience strong coupling interference. Existing monitoring methods, whether based on single sensors or simple fusion, struggle to achieve stable and accurate real-time defect identification in such scenarios.

[0004] Therefore, there is an urgent need for a real-time monitoring method for the welding process that can adaptively extract highly discriminative multi-source features, adopt a lightweight and efficient model architecture, and achieve intelligent fusion decision-making. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a real-time welding monitoring method based on multi-source information fusion and artificial intelligence. The real-time monitoring method includes:

[0006] Step 1, Multi-source signal synchronous acquisition: Using multiple sensors to simultaneously and in real time monitor welding information during the welding process, including visual images of the molten pool, welding current, arc voltage, and arc sound signals;

[0007] Step 2, welding information preprocessing: filtering, noise reduction and time alignment of welding current, arc voltage and arc sound signals; inputting the visual image of the molten pool into a lightweight target detection network model to automatically and in real time locate and capture the image of the molten pool area;

[0008] Step 3, Parallel Adaptive Feature Extraction: Extract multi-dimensional features in parallel from the preprocessed welding information;

[0009] Step 4, Feature Fusion and Optimization: After standardizing the multidimensional features, they are concatenated to form a high-dimensional fusion feature vector; and the maximum correlation minimum redundancy (mRMR) criterion is used to screen the high-dimensional fusion features, and a subset of optimized fusion features with strong discriminative power and low redundancy is selected.

[0010] Step 5, State recognition based on a two-layer fusion decision model: Input the optimized fusion feature subset into a two-layer fusion decision model to obtain the final welding state category and the corresponding confidence level;

[0011] Step 6, Monitoring Result Output and Response: Real-time output of identification results and confidence levels. When an abnormal state is identified and the confidence level exceeds the preset threshold, an alarm is triggered and recorded.

[0012] Furthermore, in step two, the lightweight target detection network model includes:

[0013] Input unit, used to receive visual images of the molten pool;

[0014] The encoder unit includes four GhostBottleneck modules cascaded in sequence. The GhostBottleneck module is the core basic module of the lightweight network GhostNet, which has lightweight and efficient feature extraction characteristics. The first GhostBottleneck module receives the melt pool visual image input and outputs multi-scale encoded features after being processed by each GhostBottleneck module in the encoder unit.

[0015] The adaptive feature fusion unit includes a first adaptive spatial feature fusion module, a second adaptive spatial feature fusion module, and a third adaptive spatial feature fusion module. The output features of the first GhostBottleneck module are connected to the first adaptive spatial feature fusion module, the output features of the second GhostBottleneck module are connected to the second adaptive spatial feature fusion module, and the output features of the third GhostBottleneck module are connected to the third adaptive spatial feature fusion module.

[0016] The decoder unit includes GhostBottleneck modules cascaded in sequence. The output features of the fourth GhostBottleneck module and the output features of the third adaptive spatial feature fusion module are jointly connected to the first GhostBottleneck module. The output features of the first GhostBottleneck module and the output features of the second adaptive spatial feature fusion module are jointly connected to the second GhostBottleneck module. The output features of the second GhostBottleneck module and the output features of the first adaptive spatial feature fusion module are jointly connected to the third GhostBottleneck module.

[0017] The output unit includes a multi-scale feature fusion module and a candidate bounding box and confidence score generation module. The output features of the third GhostBottleneck module are processed by the multi-scale feature fusion module and then input to the candidate bounding box and confidence score generation module to output the candidate bounding box and confidence score corresponding to the melt pool visual image.

[0018] Furthermore, in step two, the step of acquiring the molten pool region image is as follows:

[0019] The normalized visual image of the molten pool is then input into the lightweight target detection network model being loaded; the normalization process includes unifying the image scale and eliminating interference caused by changes in illumination and differences in equipment.

[0020] The lightweight object detection network model outputs several candidate bounding boxes and their confidence scores.

[0021] The candidate bounding boxes and their confidence scores are processed by the non-maximum suppression (NMS) algorithm. The candidate bounding boxes with the highest confidence scores are filtered and retained by setting a confidence score threshold, while redundant and overlapping bounding boxes are removed.

[0022] Based on the coordinate information of the bounding box with the highest confidence, the image of the molten pool region is accurately extracted from the original visual image of the molten pool, providing accurate target area data for subsequent feature extraction and welding status recognition.

[0023] Furthermore, in step three, the multidimensional features include time-frequency joint features, time-frequency pattern features, and deep semantic features.

[0024] Furthermore, the extraction steps for the time-frequency joint features are as follows:

[0025] (311) Extract time-domain statistical features from the pre-processed welding current and arc voltage respectively. The time-domain statistical features include root mean square, peak-to-peak value, mean, variance, kurtosis, and waveform factor.

[0026] (312) Wavelet packet decomposition was performed on the welding current and arc voltage respectively, and the percentage of energy of each sub-band after decomposition was calculated to account for the total energy.

[0027] (313) The time-domain statistical characteristics of the welding current signal and its corresponding sub-band energy percentage are combined to form the welding current time-frequency joint characteristics; the time-domain statistical characteristics of the arc voltage signal and its corresponding sub-band energy percentage are combined to form the welding current time-frequency joint characteristics and the arc voltage time-frequency joint characteristics. The welding current time-frequency joint characteristics and the arc voltage time-frequency joint characteristics are collectively referred to as the time-frequency joint characteristics.

[0028] Furthermore, the extraction steps for the time-frequency pattern features are as follows:

[0029] (321) Perform pre-emphasis, framing, and windowing processing on the pre-processed arc sound signal;

[0030] (322) Calculate the gamma-pass frequency cepstral coefficients of each frame of signal, and solve the mean and variance of the gamma-pass frequency cepstral coefficients of all frames respectively to obtain the mean characteristics and variance characteristics of the gamma-pass frequency cepstral coefficients. The two are combined to form the statistical characteristics of the gamma-pass frequency cepstral coefficients.

[0031] (323) Simultaneously generate the Mel spectrum of the electric arc sound signal, input the Mel spectrum into a preset two-dimensional convolutional layer, extract the key frequency domain texture information of the spectrum through convolution operation, and output the Mel spectrum convolutional features;

[0032] (324) The time-frequency pattern features of the arc sound signal are obtained by combining the statistical features of the cepstral coefficients of the gamma-pass frequency with the convolutional features of the Mel spectrum by dimension splicing.

[0033] Furthermore, the extraction steps for the deep semantic features are as follows:

[0034] The captured image of the molten pool region is input into the PoolFeatNet lightweight network, which outputs deep semantic features.

[0035] The PoolFeatNet lightweight network includes a feature input unit, a backbone network unit, and a feature output unit. The backbone network unit adopts the ShuffleNetV2 architecture and consists of multiple cascaded ShuffleNetV2 basic units. During the feature processing of each ShuffleNetV2 basic unit, an efficient channel attention mechanism is introduced to adaptively learn channel weights, enabling the network to adaptively focus on key regions in the image with extremely low computational cost. After collaborative processing by the backbone network and the attention mechanism, the feature output unit outputs deep semantic features.

[0036] Furthermore, in step four, during the feature selection training phase, the maximum relevance and minimum redundancy criterion is adopted. The mutual information between all features in the high-dimensional fused feature vector and the welding state category is calculated as the relevance score, and the mutual information between each pair of features is calculated as the redundancy measure. The feature with the highest comprehensive score of "relevance-redundancy" is selected iteratively to obtain the optimal feature subset. The feature index of the optimal feature subset is recorded to form a feature selection index table.

[0037] Furthermore, in step five, the welding status categories include normal, incomplete penetration, weld leakage, undercut, and weld deviation.

[0038] Furthermore, in step five, the two-layer fusion decision model includes:

[0039] The base learner layer integrates multiple heterogeneous and computationally efficient machine learning models, including at least Extreme Gradient Boosting Tree (XGBoost), Linear Kernel Support Vector Machine (SVM), and One-Dimensional Convolutional Neural Network (CNN). Each machine learning model takes the fused features as input, performs preliminary analysis and judgment on the fused features based on different mathematical principles, and outputs its corresponding probability vector.

[0040] The dynamic fusion layer includes a dynamic weight allocation network (meta-network). The input of the dynamic weight allocation network is the probability vector of each machine learning model in the base learner layer and the optimized fusion feature subset. The dynamic weight allocation network learns a weight allocation strategy through training and automatically assigns corresponding weights to each machine learning model in the base learner layer for different input feature patterns. Then, the probability vectors of each machine learning model are fused by weighted summation to output the final welding state category and the corresponding confidence level.

[0041] The present invention has the following beneficial effects:

[0042] (1) This invention achieves a comprehensive representation of the welding process status by integrating multi-source information such as visual images of the molten pool, welding current, arc voltage, and arc sound. At the same time, through the dynamic weighting strategy of the dual-layer fusion decision model, it adaptively integrates the prediction results of heterogeneous machine learning models, makes full use of the advantages of different models in different feature analysis, significantly improves the accuracy and robustness of defect identification, effectively avoids the judgment deviation caused by the limitation of a single information source or fixed fusion rules, and provides accurate and reliable technical support for welding quality control.

[0043] (2) In this invention, the molten pool positioning stage adopts a lightweight target detection network based on GhostBottleneck and adaptive spatial feature fusion, which greatly compresses the number of parameters and achieves millisecond-level inference; the feature extraction stage adopts a parallel processing architecture combined with a lightweight PoolFeatNet network and an efficient time-frequency feature extraction method to improve feature extraction efficiency; the feature optimization stage simplifies the feature dimensions by using the maximum correlation and minimum redundancy criterion to reduce the computational complexity of the subsequent decision model; this design makes the single processing delay of the entire process much lower than the monitoring cycle of hundreds of milliseconds, ensuring that the entire chain processing from signal acquisition to status recognition is completed within the monitoring cycle, realizing the real-time detection and response of welding defects, and avoiding the expansion of defects.

[0044] (3) This invention does not bind to the exclusive logic of a specific welding process. Through the idea of ​​multi-source information fusion, lightweight processing strategy and adaptive decision-making mechanism, it can flexibly adapt to the monitoring needs of other hot processing processes. Only by adjusting the sensor type (matching the signal characteristics of different hot processing), feature extraction parameters and model training data, the status monitoring of other hot processing processes can be realized. This universal design greatly expands the application scenarios of the technology and enhances the industrialization value and promotion potential of the technology.

[0045] (4) This invention focuses on key features by adaptively learning channel weights to adapt to the differences in molten pool features under different working conditions; the dynamic fusion layer learns weight allocation strategies through meta-networks to dynamically adjust the weights of the machine learning model for different input feature patterns and match the feature distribution patterns of different welding conditions. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the present invention.

[0047] Figure 2 This is a schematic diagram of the lightweight target detection network model in this invention.

[0048] Figure 3 This is a comparison chart of the average contribution of different signal source features to state recognition in this invention.

[0049] Figure 4This is a schematic diagram of the structure of the two-layer fusion decision model in this invention.

[0050] Figure 5 This is a comparison chart of the key performance indicators of the present invention and existing methods. Detailed Implementation

[0051] The technical solution of the present invention will be further described in detail below with reference to specific embodiments. However, these embodiments are not intended to limit the present invention. Any similar structures and similar variations of the present invention should be included in the protection scope of the present invention. The commas in the present invention all indicate the relationship between and. The English letters in the present invention are case-sensitive.

[0052] like Figure 1 As shown, this invention provides a real-time welding monitoring method based on multi-source information fusion and artificial intelligence. The real-time monitoring method includes:

[0053] Step 1, Multi-source signal synchronous acquisition: Multiple sensors are used to simultaneously and in real time monitor welding information during the welding process. The welding information includes visual images of the molten pool, welding current, arc voltage, and arc sound signals. The multi-source sensor synchronous acquisition design can comprehensively capture multi-dimensional status information of the welding process, avoiding the limitations of monitoring a single information source, and laying a data foundation for subsequent accurate identification.

[0054] Step 2, Welding Information Preprocessing: The welding current, arc voltage, and arc sound signals are filtered, denoised, and time-aligned (signal slice alignment based on a synchronous trigger clock). Filtering and denoising effectively remove environmental interference signals and equipment noise, improving the signal-to-noise ratio of the original signal. Time alignment ensures consistency of multi-source signals in the time dimension, avoiding feature misalignment caused by timing deviations. The molten pool visual image is then input into a lightweight target detection network model to automatically and in real-time locate and extract the molten pool region image (ROI).

[0055] like Figure 2 As shown, the lightweight target detection network model includes:

[0056] Input unit, used to receive visual images of the molten pool;

[0057] The encoder unit includes four cascaded GhostBottleneck modules. The GhostBottleneck module is the core module of the lightweight network GhostNet, which has lightweight and efficient feature extraction characteristics. Compared with traditional convolutional modules, it can significantly reduce the number of parameters and computational overhead, while ensuring excellent multi-scale melt pool feature extraction capabilities. The first GhostBottleneck module receives the melt pool visual image input, and after being processed by each GhostBottleneck module in the encoder unit, it outputs multi-scale encoded features.

[0058] The adaptive feature fusion unit includes a first adaptive spatial feature fusion module, a second adaptive spatial feature fusion module, and a third adaptive spatial feature fusion module. The output features of the first GhostBottleneck module are connected to the first adaptive spatial feature fusion module, the output features of the second GhostBottleneck module are connected to the second adaptive spatial feature fusion module, and the output features of the third GhostBottleneck module are connected to the third adaptive spatial feature fusion module. The adaptive spatial feature fusion mechanism adopted by this unit can dynamically adjust the fusion weights of features at different scales, effectively solving the spatial inconsistency problem of multi-scale melt pool features and improving feature utilization.

[0059] The decoder unit comprises cascaded GhostBottleneck modules. The output features of the fourth GhostBottleneck module and the output features of the third adaptive spatial feature fusion module are connected to the first GhostBottleneck module. The output features of the first GhostBottleneck module and the output features of the second adaptive spatial feature fusion module are connected to the second GhostBottleneck module. The output features of the second GhostBottleneck module and the output features of the first adaptive spatial feature fusion module are connected to the third GhostBottleneck module. The decoder unit adopts a cascaded GhostBottleneck module design, which can achieve progressive upsampling and fine reconstruction of features while ensuring lightweight design, and accurately restore the spatial location information of the melt pool region.

[0060] The output unit includes a multi-scale feature fusion module and a candidate bounding box and confidence score generation module. The output features of the third GhostBottleneck module are processed by the multi-scale feature fusion module and then input to the candidate bounding box and confidence score generation module to output the candidate bounding boxes and confidence scores corresponding to the melt pool visual image. The multi-scale feature fusion module can integrate feature information from different levels, improving adaptability to melt pools of different sizes, while the candidate bounding box and confidence score generation module can quickly output accurate localization results and reliability assessments.

[0061] The core improvement of this network lies in replacing standard convolutional layers with Ghost modules (a lightweight convolutional method that generates redundant feature maps through inexpensive linear operations) to significantly reduce the number of parameters and computational cost, thereby lowering hardware deployment costs. Furthermore, it introduces an adaptive spatial feature fusion (ASFF) mechanism to enhance the network's efficiency in utilizing melt pool features at different scales, thus improving localization accuracy. This model achieves millisecond-level inference speeds while maintaining high detection accuracy, meeting the real-time requirements of online monitoring.

[0062] The steps for acquiring the molten pool region image are as follows:

[0063] The normalized visual image of the molten pool is then input into the lightweight target detection network model being loaded. The normalization process includes unifying the image scale and eliminating interference caused by changes in illumination and differences in equipment. This can improve the model's adaptability to different shooting conditions and reduce the impact of environmental and equipment factors on the detection results.

[0064] The lightweight object detection network model outputs several candidate bounding boxes and their confidence scores.

[0065] The candidate bounding boxes and their confidence scores are processed by the Non-Maximum Suppression (NMS) algorithm. By setting a confidence threshold, the candidate bounding boxes with the highest confidence scores are filtered and retained, while redundant and overlapping bounding boxes are removed. The Non-Maximum Suppression (NMS) algorithm can effectively remove redundant bounding boxes, avoid repeated localization, and improve the accuracy of melt pool region localization.

[0066] Based on the coordinate information of the bounding box with the highest confidence, the image of the molten pool region is accurately extracted from the original visual image of the molten pool, providing accurate target area data for subsequent feature extraction and welding status recognition.

[0067] Step 3, Parallel Adaptive Feature Extraction: Multi-dimensional features are extracted in parallel from the preprocessed welding information; these features include joint time-frequency features, time-frequency pattern features, and deep semantic features. This parallel extraction design significantly improves feature extraction efficiency, avoids time delays caused by serial processing, and ensures real-time monitoring requirements are met.

[0068] The extraction steps for the joint time-frequency features are as follows:

[0069] (311) Extract time-domain statistical features from the pre-processed welding current and arc voltage respectively. The time-domain statistical features include root mean square, peak-to-peak value, mean, variance, kurtosis, and waveform factor, which can intuitively reflect the time-domain amplitude distribution and fluctuation characteristics of the signal and accurately capture the instantaneous change law of current and voltage during the welding process.

[0070] (312) Wavelet packet decomposition is performed on welding current and arc voltage respectively, and the percentage of energy of each sub-band to the total energy is calculated (sub-band energy percentage). Wavelet packet decomposition can accurately decompose different frequency components, effectively capture the energy distribution differences of welding current and voltage signals in different frequency bands, and provide rich frequency domain information for subsequent feature fusion.

[0071] (313) The time-domain statistical characteristics of the welding current signal and its corresponding sub-band energy percentage are combined to form the welding current time-frequency joint characteristics; the time-domain statistical characteristics of the arc voltage signal and its corresponding sub-band energy percentage are combined to form the welding current time-frequency joint characteristics and the arc voltage time-frequency joint characteristics. The welding current time-frequency joint characteristics and the arc voltage time-frequency joint characteristics are collectively referred to as time-frequency joint characteristics, which can integrate time-domain and frequency-domain information, comprehensively characterize the dynamic characteristics of welding current and voltage signals, and improve the discriminative power of the characteristics.

[0072] The extraction steps for the time-frequency pattern features are as follows:

[0073] (321) The pre-processed arc sound signal is pre-emphasized, framed, and windowed. First, the arc sound signal is pre-emphasized to improve the identification of high-frequency components and compensate for high-frequency attenuation during signal transmission. Then, the signal is framed according to the preset frame length and frame shift to divide the continuous sound signal into several overlapping frame segments to avoid incomplete information in a single frame. Finally, a Hanning window or Hamming window is applied to each frame signal to suppress spectral leakage at the frame edges and ensure the accuracy of subsequent frequency domain analysis. Pre-emphasis can effectively enhance the identification of high-frequency defect features, overlapping frame design can avoid loss of feature information, and windowing operation can significantly reduce spectral leakage and improve the accuracy of frequency domain analysis.

[0074] (322) Calculate the gamma-pass frequency cepstral coefficients of each frame of signal, and solve the mean and variance of the gamma-pass frequency cepstral coefficients of all frames respectively to obtain the mean characteristics and variance characteristics of the gamma-pass frequency cepstral coefficients. The two are combined to form the statistical characteristics of the gamma-pass frequency cepstral coefficients. This characteristic can accurately characterize the frequency distribution law and inter-frame fluctuation characteristics of the arc sound signal, and has high sensitivity to the sound frequency changes caused by welding defects.

[0075] The formula for calculating the gamma-pass frequency cepstral coefficients (GFCC) is as follows:

[0076]

[0077] in, For the first First-order gamma-pass frequency cepstral coefficients, This represents the number of filters in a gamma-pass filter bank. For the first The output energy of each filter The order of the cepstral coefficients is usually taken as... ,in This represents the number of GFCC coefficients to be extracted.

[0078] (323) Simultaneously generate the Mel spectrum of the arc sound signal. The Mel spectrum can convert linear frequencies into Mel frequencies that conform to the characteristics of human hearing, and intuitively present the time-frequency distribution texture of the signal. Input the Mel spectrum into a preset two-dimensional convolutional layer, extract the key frequency domain texture information of the spectrum through convolution operation, and output the Mel spectrum convolution feature. This feature belongs to the frequency domain texture representation feature, which can capture the local feature patterns of the arc sound signal in different time-frequency regions. The Mel spectrum can better match the characteristics of human hearing and highlight the sound features related to the welding state. The two-dimensional convolutional layer can efficiently extract the local texture features of the spectrum and improve the abstract representation ability of the features.

[0079] (324) The statistical features of the cepstral coefficients of the gamma-pass frequency and the convolutional features of the mell spectrum are combined and dimensionally concatenated to obtain the time-frequency pattern features of the arc sound signal, thereby achieving comprehensive coverage of the time-domain fluctuations, frequency-domain distribution and local texture information of the arc sound signal.

[0080] The extraction steps for the deep semantic features are as follows:

[0081] The cropped image of the molten pool region is input into the PoolFeatNet lightweight network, which outputs deep semantic features; the cropped image of the molten pool region is then enlarged to 224. 224 pixels, input to PoolFeatNet, passed through the network to the last convolutional layer and then global average pooling is performed to output a 128-dimensional feature vector;

[0082] The PoolFeatNet lightweight network comprises a feature input unit, a backbone network unit, and a feature output unit. The backbone network unit adopts the ShuffleNetV2 architecture, consisting of multiple cascaded ShuffleNetV2 basic units. An efficient channel attention mechanism is introduced during the feature processing of each ShuffleNetV2 basic unit. By adaptively learning channel weights, the network can adaptively focus on key regions in the image with extremely low computational cost. The ShuffleNetV2 architecture offers lightweight and efficient feature extraction advantages, significantly reducing network computational overhead and adapting to the hardware deployment requirements of real-time monitoring. The attention mechanism achieves channel weight learning solely through one-dimensional convolution and global average pooling, accurately focusing on key defect features in the melt pool region without complex calculations, significantly improving the discriminative power of deep semantic features. After collaborative processing by the backbone network and the ECA module, the feature output unit outputs deep semantic features.

[0083] Step 4, Feature Fusion and Optimization: Standardize the multi-dimensional features and concatenate them to form a high-dimensional fused feature vector; compare the average contribution of features from different signal sources in state recognition, for example... Figure 3 As shown; and the maximum relevance minimum redundancy (mRMR) criterion is used to screen the high-dimensional fusion features, and a subset of optimized fusion features with strong discriminative power and low redundancy is selected;

[0084] During the feature selection and training phase, the maximum relevance and minimum redundancy criterion is adopted. The mutual information between all features in the high-dimensional fused feature vector and the welding state category is calculated as a relevance score, and the pairwise mutual information between features is calculated as a redundancy measure. The optimal feature subset is obtained by iteratively selecting the features with the highest combined relevance-redundancy score. The feature index of this optimal feature subset is recorded to form a feature selection index table. This criterion can eliminate redundant features to the greatest extent while ensuring high relevance between features and welding state categories. This not only significantly reduces the computational complexity of subsequent models but also effectively avoids overfitting, improving the generalization performance and operational efficiency of the state recognition model.

[0085] The mathematical expression for the maximum correlation minimum redundancy (mRMR) criterion is as follows:

[0086]

[0087] in, Let be the criterion function value, and the objective be to maximize it. For the selected feature subset, The number of features in the feature subset. Features in a feature subset The welding condition category (e.g., normal, incomplete penetration, leak). Features With category Mutual information between features represents the correlation between features and categories. Features With features Mutual information between features indicates the redundancy between two features.

[0088] For example, the 200ms current signal and the 200ms voltage signal are decomposed into 3-level 'db4' wavelet packets, and the energy percentage of each of the eight sub-bands is calculated. These are then combined with the six time-domain statistics of each signal to form 14-dimensional features, for a total of 28-dimensional features.

[0089] After pre-emphasis, framing, and windowing of the 200ms audio signal, the 13-dimensional GFCC coefficients of each frame are calculated, and their mean and variance are taken to obtain 26-dimensional features. At the same time, the Mel spectrogram (80x20) of the segment is generated, and after processing by an 8-core 3x3 convolutional layer, 8-dimensional features are obtained. The two are merged to obtain a total of 34 dimensions.

[0090] After standardizing the above 28-dimensional, 34-dimensional, and 128-dimensional features, they are concatenated into a 190-dimensional vector. Using a pre-stored "feature selection index table", N corresponding values ​​(e.g., 40) are extracted to form the final fused feature vector.

[0091] Step 5, State recognition based on the two-layer fusion decision model: Input the optimized fusion feature subset into a two-layer fusion decision model to obtain the final welding state category and the corresponding confidence level; the welding state categories include normal, incomplete penetration, weld leakage, undercut, and weld deviation.

[0092] like Figure 4 As shown, the two-layer fusion decision model includes:

[0093] The base learner layer integrates multiple heterogeneous and computationally efficient machine learning models, including at least Extreme Gradient Boosting Tree (XGBoost), Linear Kernel Support Vector Machine (SVM), and One-Dimensional Convolutional Neural Network (CNN). Each machine learning model takes the fused features as input, performs preliminary analysis and judgment on the fused features based on different mathematical principles, and outputs its corresponding probability vector. Integrating heterogeneous models can fully utilize the advantages of different models (XGBoost is good at handling nonlinear features, SVM is good at high-dimensional small sample classification, and one-dimensional CNN is good at capturing temporal features), analyze the fused features from multiple dimensions, and improve the comprehensiveness and reliability of the preliminary prediction results.

[0094] The dynamic fusion layer includes a dynamic weight allocation network (meta-network). The input to this network is the probability vectors of each machine learning model in the base learner layer and an optimized fusion feature subset. The network learns a weight allocation strategy through training, automatically assigning corresponding weights to each machine learning model in the base learner layer for different input feature patterns. Then, the probability vectors of each machine learning model are fused using a weighted summation method to output the final welding state category and its corresponding confidence score. The dynamic weight allocation network can dynamically adjust the weights of each machine learning model based on the differences in input feature patterns, adaptively matching different welding conditions. Compared to fixed-weight fusion methods, this significantly improves the robustness and classification accuracy of the model.

[0095] This design enables the model to adaptively adjust its dependence on different information sources and machine learning models, thereby achieving robustness and classification performance superior to single-model or fixed-rule fusion methods.

[0096] The weighted fusion calculation formula for the two-layer fusion decision model is as follows:

[0097]

[0098] in, This is the final weighted and fused probability vector. The number of machine learning models, For the first Dynamic weights of a machine learning model For the first The probability vector output by the machine learning model. Step 6, monitoring results output and response: output the identification results and confidence level in real time. When an abnormal state is identified and the confidence level exceeds the preset confidence level threshold (e.g., 85%, which can be adjusted according to actual application needs), trigger an alarm and record it.

[0099] To verify the comprehensive performance advantages of this invention, several typical existing welding monitoring methods, including single-vision method, traditional fusion method, and single-arc method, were used as comparison objects. Accuracy, recall, F1 score, real-time performance, and robustness (noise resistance) were identified as key performance evaluation indicators. Comparative experiments were conducted between this invention and various existing methods based on these indicators. The performance scores of each method under different indicators are presented in a multi-dimensional comprehensive performance comparison chart, such as... Figure 5 As shown in the figure, it is clear that the present invention achieves high performance scores in terms of recognition accuracy, recall, F1 score, and robustness (noise resistance), which are significantly better than the single vision method, traditional fusion method, and single arc method. In terms of real-time performance, although its score is slightly lower than the single arc method, it is significantly higher than the single vision method and traditional fusion method. Considering the performance of each key indicator, the overall performance advantage of the present invention is quite prominent.

[0100] This invention achieves a comprehensive characterization of the welding process by fusing multi-source information such as visual images of the molten pool, welding current, arc voltage, and arc sound. Simultaneously, through a dynamic weighting strategy of a two-layer fusion decision model, it adaptively integrates the prediction results of heterogeneous machine learning models, fully leveraging the differentiated analytical advantages of different models for features. This significantly improves the accuracy and robustness of defect identification, effectively avoiding judgment biases caused by limitations of a single information source or fixed fusion rules, and providing precise and reliable technical support for welding quality control.

[0101] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

Claims

1. A real-time welding monitoring method based on multi-source information fusion and artificial intelligence, characterized in that, Real-time monitoring methods include: Step 1, Multi-source signal synchronous acquisition: Using multiple sensors to simultaneously and in real time monitor welding information during the welding process, including visual images of the molten pool, welding current, arc voltage, and arc sound signals; Step 2, welding information preprocessing: filtering, noise reduction and time alignment of welding current, arc voltage and arc sound signals; inputting the visual image of the molten pool into a lightweight target detection network model to automatically and in real time locate and capture the image of the molten pool area; Step 3, Parallel Adaptive Feature Extraction: Extract multi-dimensional features in parallel from the preprocessed welding information; Step 4, Feature Fusion and Optimization: After standardizing the multidimensional features, they are concatenated to form a high-dimensional fusion feature vector; and the maximum correlation and minimum redundancy criterion is used to screen the high-dimensional fusion features to select a subset of optimized fusion features with strong discriminative power and low redundancy. Step 5, State recognition based on a two-layer fusion decision model: Input the optimized fusion feature subset into a two-layer fusion decision model to obtain the final welding state category and the corresponding confidence level; Step 6, Monitoring Result Output and Response: Real-time output of identification results and confidence levels. When an abnormal state is identified and the confidence level exceeds the preset threshold, an alarm is triggered and recorded.

2. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 1, characterized in that, In step two, the lightweight target detection network model includes: Input unit, used to receive visual images of the molten pool; The encoder unit includes four GhostBottleneck modules cascaded in sequence. The first GhostBottleneck module receives the melt pool visual image input and outputs multi-scale encoded features after being processed sequentially by each GhostBottleneck module within the encoder unit. The adaptive feature fusion unit includes a first adaptive spatial feature fusion module, a second adaptive spatial feature fusion module, and a third adaptive spatial feature fusion module. The output features of the first GhostBottleneck module are connected to the first adaptive spatial feature fusion module, the output features of the second GhostBottleneck module are connected to the second adaptive spatial feature fusion module, and the output features of the third GhostBottleneck module are connected to the third adaptive spatial feature fusion module. The decoder unit includes GhostBottleneck modules cascaded in sequence. The output features of the fourth GhostBottleneck module and the output features of the third adaptive spatial feature fusion module are jointly connected to the first GhostBottleneck module. The output features of the first GhostBottleneck module and the output features of the second adaptive spatial feature fusion module are jointly connected to the second GhostBottleneck module. The output features of the second GhostBottleneck module and the output features of the first adaptive spatial feature fusion module are jointly connected to the third GhostBottleneck module. The output unit includes a multi-scale feature fusion module and a candidate bounding box and confidence score generation module. The output features of the third GhostBottleneck module are processed by the multi-scale feature fusion module and then input to the candidate bounding box and confidence score generation module to output the candidate bounding box and confidence score corresponding to the melt pool visual image.

3. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 2, characterized in that, In step two, the steps for acquiring the image of the molten pool region are as follows: The normalized visual image of the molten pool is then input into the lightweight object detection network model being loaded. The lightweight object detection network model outputs several candidate bounding boxes and their confidence scores. The candidate bounding boxes and their confidence scores are processed by a non-maximum suppression algorithm. The candidate bounding boxes with the highest confidence scores are filtered and retained by setting a confidence score threshold, while redundant and overlapping bounding boxes are removed. Based on the coordinate information of the bounding box with the highest confidence, the image of the molten pool region is accurately extracted from the original visual image of the molten pool, providing accurate target area data for subsequent feature extraction and welding status recognition.

4. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 1, characterized in that, In step three, the multidimensional features include time-frequency joint features, time-frequency pattern features, and deep semantic features.

5. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 4, characterized in that, The extraction steps for the time-frequency joint features are as follows: (311) Extract time-domain statistical features from the pre-processed welding current and arc voltage respectively. The time-domain statistical features include root mean square, peak-to-peak value, mean, variance, kurtosis, and waveform factor. (312) Wavelet packet decomposition was performed on the welding current and arc voltage respectively, and the percentage of energy of each sub-band after decomposition was calculated to account for the total energy. (313) The time-domain statistical characteristics of the welding current signal are combined with the corresponding sub-band energy percentage to form the time-frequency joint characteristics of the welding current; The time-domain statistical characteristics of the arc voltage signal are combined with the energy percentage of its corresponding sub-band to form the time-frequency joint characteristics of welding current and arc voltage. The time-frequency joint characteristics of welding current and arc voltage are collectively referred to as time-frequency joint characteristics.

6. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 4, characterized in that, The steps for extracting the time-frequency pattern features are as follows: (321) Perform pre-emphasis, framing, and windowing processing on the pre-processed arc sound signal; (322) Calculate the gamma-pass frequency cepstral coefficients of each frame of signal, and solve the mean and variance of the gamma-pass frequency cepstral coefficients of all frames respectively to obtain the mean characteristics and variance characteristics of the gamma-pass frequency cepstral coefficients. The two are combined to form the statistical characteristics of the gamma-pass frequency cepstral coefficients. (323) Simultaneously generate the Mel spectrum of the electric arc sound signal, input the Mel spectrum into a preset two-dimensional convolutional layer, extract the key frequency domain texture information of the spectrum through convolution operation, and output the Mel spectrum convolutional features; (324) The time-frequency pattern features of the arc sound signal are obtained by combining the statistical features of the cepstral coefficients of the gamma-pass frequency with the convolutional features of the Mel spectrum by dimension splicing.

7. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 4, characterized in that, The steps for extracting the deep semantic features are as follows: The captured image of the molten pool region is input into the PoolFeatNet lightweight network, which outputs deep semantic features. The PoolFeatNet lightweight network includes a feature input unit, a backbone network unit, and a feature output unit. The backbone network unit adopts the ShuffleNetV2 architecture and is composed of multiple cascaded ShuffleNetV2 basic units. An efficient channel attention mechanism is introduced in the feature processing of each ShuffleNetV2 basic unit. By adaptively learning the channel weights, the network can adaptively focus on key regions in the image with extremely low computational cost. After being processed by the backbone network and the attention mechanism, the deep semantic features are output by the feature output unit.

8. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 1, characterized in that, In step four, during the feature selection training phase, the maximum relevance and minimum redundancy criterion is adopted. The mutual information between all features in the high-dimensional fused feature vector and the welding state category is calculated as the relevance score, and the mutual information between each pair of features is calculated as the redundancy measure. The feature with the highest comprehensive score of "relevance-redundancy" is selected iteratively to obtain the optimal feature subset. The feature index of the optimal feature subset is recorded to form a feature selection index table.

9. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 1, characterized in that, In step five, the welding status categories include normal, incomplete penetration, leaky weld, undercut, and weld deviation.

10. The welding real-time monitoring method based on multi-source information fusion and artificial intelligence according to claim 1, characterized in that, In step five, the two-layer fusion decision model includes: The base learner layer integrates multiple heterogeneous and computationally efficient machine learning models, including at least a limit gradient boosting tree, a linear kernel support vector machine, and a one-dimensional convolutional neural network. Each machine learning model takes the fused features as input, performs preliminary analysis and judgment on the fused features based on different mathematical principles, and outputs its corresponding probability vector. The dynamic fusion layer includes a dynamic weight allocation network. The input of the dynamic weight allocation network is the probability vector of each machine learning model in the base learner layer and the optimized fusion feature subset. The dynamic weight allocation network learns a weight allocation strategy through training and automatically assigns corresponding weights to each machine learning model in the base learner layer for different input feature patterns. Then, the probability vectors of each machine learning model are fused by weighted summation to output the final welding state category and the corresponding confidence level.

Images