Infrared and acoustic emission multi-modal fusion welding online quality monitoring method
By using a multimodal fusion method combining infrared and acoustic emission for online welding quality monitoring, the real-time and accuracy issues of traditional welding quality control have been resolved. This method enables real-time defect detection and control during the welding process, reducing false alarms and missed alarms, and improving production efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAN THERMAL POWER RES INST CO LTD
- Filing Date
- 2026-01-26
- Publication Date
- 2026-06-09
AI Technical Summary
Traditional welding quality control methods mainly rely on post-weld non-destructive testing, which cannot achieve real-time intervention. Single sensing methods are easily interfered with and have limited defect detection capabilities, leading to increased production costs and low efficiency.
An online quality monitoring method for welding using a multimodal fusion of infrared and acoustic emission is adopted. By synchronously collecting multimodal sensor data, extracting feature parameters, and using a support vector machine model for fusion judgment, the method outputs prediction results and confidence scores, thereby achieving real-time defect detection.
It significantly reduces false alarm and false alarm rates, enables immediate detection and control of defects when they occur, improves the accuracy and real-time performance of welding quality inspection, and forms a closed loop of "perception-decision-action" to prevent defects from escalating.
Smart Images
Figure CN122165084A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of welding process quality control and non-destructive testing technology, specifically to a method for online quality monitoring of welding using a multimodal fusion of infrared and acoustic emission. Background Technology
[0002] Welding quality is a key factor in ensuring the safety and reliability of various welded structures, directly affecting the service performance of equipment under harsh conditions such as high temperature, high pressure, heavy load, or alternating load. Traditional welding quality control methods mainly rely on post-weld non-destructive testing technologies, such as radiographic testing, ultrasonic testing, magnetic particle testing, and penetrant testing. While these methods can identify typical defects such as porosity, cracks, lack of fusion, and slag inclusions to some extent, they are post-weld inspection methods and cannot intervene in the welding process in real time. Once defects are detected, secondary repairs or even component scrapping are often required, leading to increased production costs and extended production cycles, severely restricting the improvement of quality and efficiency in high-end equipment manufacturing.
[0003] In recent years, welding monitoring technology based on process signals has been gradually applied, among which infrared thermography and acoustic emission monitoring are two promising methods. However, single sensing methods often have limitations: infrared monitoring is easily affected by welding fumes, spatter, and surface conditions; acoustic emission signals are easily affected by mechanical noise and environmental vibration, and have limited ability to distinguish defect types.
[0004] Therefore, in order to overcome the shortcomings of a single sensing mode and improve the accuracy and robustness of welding process status identification and defect diagnosis, there is an urgent need for a monitoring system that can integrate multiple information and achieve high-precision, high-reliability online judgment, so as to detect and take measures immediately when defects occur. Summary of the Invention
[0005] To address the problems existing in the background technology, this invention proposes an online quality monitoring method for welding that integrates infrared and acoustic emission multimodal methods. By complementing multimodal information, the accuracy and real-time performance of defect detection are significantly improved.
[0006] To achieve the above objectives, the present invention adopts the following solution: The online quality monitoring method for multimodal welding combining infrared and acoustic emission includes the following steps: Step 1: Synchronously acquire multimodal sensing data obtained by different sensing devices during the welding process. The multimodal sensing data includes at least infrared thermal sequence images and acoustic emission signals. Step 2: Extract feature parameters from the infrared thermal sequence image and the acoustic emission signal, respectively; Step 3: Combine the feature parameters into a fused feature vector and input it into a pre-trained classification model for fusion judgment, and output the prediction result and confidence level representing the welding quality status; Step 4: Make a decision based on the prediction results and confidence level. If it is determined that there is a welding defect, output an alarm signal.
[0007] Optionally, before step 1, a system preparation and startup phase is also included, which includes: installing and connecting the device: fixing the infrared thermal imager to the side and rear of the welding torch and adjusting its field of view to cover the molten pool and heat-affected zone; fixing the acoustic emission sensor to the workpiece; configuring monitoring parameters: inputting welding process parameters, and the system loads the corresponding pre-trained classification model and monitoring threshold accordingly; system calibration: collecting data during defect-free welding and establishing a characteristic baseline for normal welding.
[0008] Optionally, the characteristic baseline includes at least a reference range for the molten pool cooling rate and a reference range for the acoustic emission signal energy.
[0009] Optionally, in step 2, the feature parameters extracted from the infrared thermal sequence image include the cooling rate, which is quantified by the following method: calculate the highest temperature within the region of interest (ROI), select isotherms of 800℃ and 500℃, calculate the time Δt_(8 / 5) required for the distance and temperature difference between them, and substitute it into the cooling rate formula CR = 300 / Δt_(8 / 5)(℃ / s); The method for extracting feature parameters from acoustic emission signals includes: performing bandpass filtering on the acoustic emission signal acquired in step 1, performing wavelet packet transform on the filtered effective signal, decomposing the effective signal to the fourth layer to filter out the target frequency band, and extracting the absolute energy and ringing count features within the target frequency band.
[0010] Optionally, in step 3, the classification model is a support vector machine model, and its output prediction results include at least four types: normal, porosity, lack of fusion, and crack.
[0011] Optionally, in step 4, the preset decision logic for the decision judgment is as follows: the result is valid only when the confidence level is higher than the preset confidence threshold; the welding defect is determined and an alarm is triggered only when N consecutive valid results are all of the same abnormal defect type.
[0012] Optionally, in step 4, the output alarm signal includes sending an emergency stop control signal to the welding power supply or robot controller.
[0013] An online quality monitoring system for infrared and acoustic emission multimodal fusion welding includes: a multi-sensor module for synchronously acquiring infrared thermal sequence images and acoustic emission signals; a synchronous acquisition module connected to the multi-sensor module for receiving and synchronously packaging data from the multi-sensor module; a data processing module connected to the synchronous acquisition module and configured to perform feature extraction, fusion judgment, and decision output; and an alarm control module connected to the data processing module for responding to alarm signals and performing corresponding output operations.
[0014] Optionally, the multi-sensor module includes: at least one high-speed infrared thermal imager for acquiring the infrared thermal sequence images; and at least one acoustic emission sensor coupled to the welded workpiece via a waveguide rod for acquiring the acoustic emission signals.
[0015] Optionally, the synchronous acquisition module includes a high-speed data acquisition card, which assigns a unified timestamp to the infrared thermal sequence image and acoustic emission data; the data processing module uses an industrial control computer as a carrier and has a built-in core processing unit, which includes an infrared feature extraction unit, an acoustic emission feature extraction unit, and a multi-source information fusion unit.
[0016] The beneficial effects of this invention are as follows: First, this solution applies two sensing technologies based on different physical principles—infrared thermal imaging reflecting changes in thermal state and acoustic emission detection reflecting stress wave release—online and synchronously to welding process monitoring. This solves the problem of precise alignment between the two technologies in terms of time (microsecond-level synchronization) and space (correlation with the same event). Through the complementarity and fusion of the two technologies, and by using a support vector machine (SVM) classification model for intelligent decision-making, the limitations of single-sensor technology are effectively overcome, and the false alarm and false negative rates are significantly reduced.
[0017] Furthermore, this scheme employs targeted feature extraction. For infrared features, it extracts the cooling rate at the tail of the molten pool, which is strongly correlated with defects, and directly quantifies the cooling rate using a pre-defined formula CR=300 / Δt_(8 / 5)(℃ / s). This provides accurate thermal indicators for subsequent quality assessment, offering a more intuitive approach compared to general temperature field qualitative analysis and ensuring accurate isotherm identification under strong welding interference. For acoustic emission features, it uses time-frequency analysis methods such as wavelet packet transform to extract the absolute energy and ring count of specific frequency bands from complex background noise. The combination of these two methods enables multi-dimensional quantitative characterization of welding quality, distinguishing it from single-feature or general acoustic emission analysis methods.
[0018] Furthermore, as a controller, the system completes the entire process from signal acquisition, analysis, judgment to output within milliseconds, ultimately generating a physical signal to directly control the emergency stop of the welding equipment. This forms a complete "perception-decision-action" closed loop, preventing defects from escalating. During this process, it can simultaneously output predicted results and confidence levels representing the welding quality status, providing not only alarms but also preliminary defect type assessments and richer diagnostic information for the operator. Attached Figure Description
[0019] Figure 1 This is a flowchart of the monitoring method of the present invention. Detailed Implementation
[0020] To make the present invention clearer and more understandable, the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the given embodiments are only one implementation method and do not represent all embodiments.
[0021] Example 1 Combination Figure 1 This embodiment provides an online quality monitoring method for multimodal welding fusion of infrared and acoustic emission. Before implementing the monitoring method, a system preparation and startup phase is required, which specifically includes: Install and connect the equipment, using high-quality shielded cables to connect the synchronous acquisition card, industrial control computer, audible and visual alarm, and welding robot I / O interface to form a complete system.
[0022] Configure monitoring parameters and input welding process parameters. The system will then load the corresponding pre-trained classification model and monitoring thresholds. The process parameters include base material type (Q235B), welding wire type (ER50-6), shielding gas (82%Ar+18%CO2), preset current (250A), voltage (28V), and welding speed (30cm / min).
[0023] The system is calibrated to simultaneously acquire infrared and acoustic emission data during defect-free welding to establish a characteristic baseline for normal welding. The characteristic baseline includes at least a reference range for the molten pool cooling rate and a reference range for the acoustic emission signal energy.
[0024] After completing the above steps, start monitoring to perform online quality monitoring of infrared and acoustic emission multimodal fusion welding, including the following steps: Step 1: Synchronously acquire multimodal sensing data from different sensing devices during the welding process. This multimodal sensing data includes at least infrared thermal sequence images and acoustic emission signals. Infrared thermal sequence images are acquired at a frame rate of 500Hz, and each frame is accurately timestamped (microsecond level). Simultaneously, analog voltage signals from the acoustic emission sensor are acquired at a sampling rate of 5MS / s. Every 50,000 points acquired (corresponding to a 10ms time interval) are bundled into a data packet and timestamped with a time stamp aligned with the infrared thermal sequence image frame.
[0025] Step 2: Extract feature parameters from the infrared thermal sequence image and the acoustic emission signal, respectively. The feature parameters extracted from the infrared thermal sequence image include features related to the cooling rate extracted from the temperature field of the weld pool and heat-affected zone; the feature parameters extracted from the acoustic emission signal include performing wavelet packet transform on the signal and extracting energy and ringing count features within a specific high-frequency band.
[0026] Specifically, when extracting features from infrared thermal sequence images, for each received frame of thermal image, the molten pool and heat-affected zone are selected according to a preset region of interest (ROI), eliminating irrelevant background interference, focusing on the core analysis object, and improving the accuracy of feature parameters. For the selected ROI region, the highest temperature T_max within the ROI is calculated based on an algorithm combining adaptive threshold segmentation and region traversal: For the selected ROI region, the high-temperature area is first highlighted using adaptive threshold segmentation technology, and then all pixels within the region are traversed to select the pixel with the highest temperature value. The temperature of this pixel is T_max, providing basic temperature data for subsequent thermal analysis.
[0027] In this embodiment, for the welding process of Q235B base material + ER50-6 welding wire, based on the critical temperature range experiment of the solidification stage of the molten pool, isotherms at 800℃ and 500℃ were selected, which can accurately reflect the key process of molten pool cooling. The time Δt_(8 / 5) required for the distance and temperature difference between them was calculated and substituted into the cooling rate formula CR = 300 / Δt_(8 / 5) (℃ / s) to directly quantify the cooling rate, providing accurate thermal indicators for subsequent quality judgment. This is more intuitive than the general qualitative analysis of the temperature field and ensures the accuracy of isotherm identification under strong interference environment in welding.
[0028] When extracting acoustic emission signal characteristics, for each 10ms long raw acoustic emission signal data packet received, a bandpass filter is first performed. The filter frequency range is set to 20kHz-1MHz to remove low-frequency mechanical and high-frequency electrical noise while retaining the effective frequency range of the acoustic emission signal. The effective signal band is determined based on experiments on the frequency distribution of noise during the welding process. This maximizes the retention of acoustic emission energy from welding defects while filtering out noise such as mechanical vibration and electrical interference.
[0029] Then, wavelet packet transform is performed on the filtered effective signal to decompose it to the fourth level, resulting in 16 frequency band sub-signals, achieving fine-grained segmentation of the signal in different frequency ranges. Next, two target frequency bands, 125-250kHz and 250-500kHz, are selected from the 16 frequency band sub-signals. These target frequency bands are sensitive bands determined through spectral analysis experiments of acoustic emission signals from Q235B welding defects, and can accurately capture the differences in acoustic emission energy distribution among different types of welding defects.
[0030] Finally, the absolute energy E_AE of the signals within these two target frequency bands is calculated, and the number of pulses with amplitudes exceeding the 45 dB threshold in this signal segment is counted. Low-amplitude interference pulses are excluded, and the ringing count RC is obtained. It is important to understand that the joint extraction of absolute energy E_AE and ringing count RC is a feature set designed for infrared-acoustic emission multimodal fusion monitoring scenarios in this embodiment. E_AE reflects the energy intensity of the defect source, and RC reflects the activity level of the defect event. The combination of the two can achieve multi-dimensional quantitative characterization of welding quality, which is different from single feature or general acoustic emission analysis methods.
[0031] Step 3: Combine all feature values (CR, T_max, E_AE, RC) extracted in the past 10ms into a feature vector F. Input the feature vector F into the pre-loaded support vector machine (SVM) classification model for real-time inference and output the prediction result Y and confidence level P representing the welding quality status.
[0032] As one embodiment, the prediction result Y and its confidence level P can be expressed as: Y=0, P=0.92, normal welding (confidence level 92%); Y=1, P=0.87, suspected stomata (confidence level 87%); Y=2, P=0.96, highly suspected non-fusion (96% confidence level); Y=3, P=0.78, suspected crack (78% confidence level).
[0033] Step 4: Based on the prediction results and confidence levels, a decision is made. If a welding defect is determined to exist, an alarm signal is output. The preset decision logic for the decision is as follows: the result is valid only when the confidence level is higher than a preset confidence threshold; a welding defect is determined and an alarm is triggered only when N consecutive valid results are of the same abnormal defect type, thus improving the reliability of the defect inference result and the anti-interference capability of the alarm logic.
[0034] As one embodiment, the confidence threshold P_threshold is set to 0.85. The inference can only be considered valid if the confidence P > P_threshold. The continuous alarm threshold N=3 is set. To prevent instantaneous interference, an alarm can only be triggered when three consecutive valid inference results are all non-zero (i.e., defects) and the defect types are consistent.
[0035] Upon triggering the alarm, the system immediately executes the following multi-dimensional alarm output operations to ensure timely detection and handling by on-site personnel: First, visual alarm on the interface: The alarm information is displayed in the most prominent area of the operation interface in a flashing red light. The alarm information includes a defect type identifier, for example, "Alarm: Incomplete fusion!". Simultaneously, the precise occurrence time of the defect alarm is automatically recorded and displayed next to the alarm information. Second, audible and visual alarm: The audible and visual alarm is triggered, emitting a high-decibel buzzer and flashing red light to alert the on-site operator to the defect anomaly in a dual warning manner. Emergency stop control signal output: A manual operation trigger interface is reserved to send a 24V TTL standard high-level "emergency stop" signal to the welding robot controller via a digital I / O card, forcing the welding robot to suspend the current welding operation and preventing further expansion of the defect.
[0036] Furthermore, to achieve full-cycle data control of the welding process and support for subsequent process optimization, all raw data, feature data, alarm events, and operation logs throughout the welding process are automatically saved to the SQL database in the form of "time-data" and can be exported at any time for generating quality reports or conducting long-term process analysis.
[0037] Therefore, the monitoring method in this embodiment effectively overcomes the limitations of single-sensor technology and significantly reduces false alarms and false negatives by complementing and fusing infrared and acoustic emission information and using a support vector machine (SVM) classification model for intelligent decision-making. Furthermore, this method employs targeted feature extraction. For infrared features, it does not simply monitor temperature but extracts the molten pool tail cooling rate (CR), which is strongly correlated with defects. For acoustic emission features, it uses time-frequency analysis methods such as wavelet packet transform to extract energy and ring counts in specific frequency bands (e.g., 125-500kHz) from complex background noise. These features are closely related to events such as material fracture and porosity formation, and can effectively suppress on-site noise interference such as arc light, smoke, and mechanical vibration.
[0038] Example 2 This embodiment provides an online quality monitoring system for infrared and acoustic emission multimodal fusion welding, comprising: a multi-sensor module for synchronously acquiring infrared thermal sequence images and acoustic emission signals; a synchronous acquisition module connected to the multi-sensor module for receiving and synchronously packaging data from the multi-sensor module; a data processing module connected to the synchronous acquisition module and configured to perform feature extraction, fusion judgment, and decision output; and an alarm control module connected to the data processing module for responding to alarm signals and performing corresponding output operations.
[0039] The multi-sensor module includes: at least one high-speed infrared thermal imager for acquiring the infrared thermal sequence images; and at least one acoustic emission sensor coupled to the welded workpiece via a waveguide rod for acquiring the acoustic emission signals.
[0040] Specifically, the infrared thermal imager is a high-speed mid-wave infrared thermal imager (such as FLIR A6751sc), with a resolution of no less than 640x512, a frame rate of no less than 500 Hz, and a thermal sensitivity (NETD) of <25 mK. It is fixed to the side and rear of the welding torch of the welding robot using a universal bracket, with its optical axis forming a 30-45 degree angle with the welding direction, ensuring its field of view completely covers the molten pool and the heat-affected zone within a 100mm radius behind it. A narrow-band filter with a center wavelength of 3.9μm is added to the lens to effectively suppress interference from the welding arc light. The acoustic emission sensor (such as PAC Nano30) is coupled to the workpiece via a waveguide rod. The installation position is selected on the back or side of the weld, within a range of 50-100mm from the center of the molten pool. Ensure the mounting surface is flat and smooth, and apply a layer of ultrasonic coupling agent (such as silicone grease) to ensure sound wave transmission efficiency. The sensor is connected to a preamplifier via a coaxial cable, and then to a data acquisition card.
[0041] Specifically, the synchronous acquisition module includes a high-speed data acquisition card, which assigns a unified timestamp to the infrared thermal sequence images and acoustic emission data. The data processing module uses an industrial control computer as a carrier and incorporates a core processing unit, specifically including an infrared feature extraction unit, an acoustic emission feature extraction unit, and a multi-source information fusion unit. Further, the infrared feature extraction unit processes the thermal sequence in real time, calculating the cooling rate (CR) at the tail of the molten pool, the peak temperature (T_p) of the heat-affected zone, and the area change rate of a specific isotherm; the acoustic emission feature extraction unit performs wavelet packet transform on the acoustic emission signal, extracting signal energy, amplitude, ring count, and absolute energy from two frequency bands: 125-250 kHz and 250-500 kHz; the multi-source information fusion unit uses a support vector machine (SVM)-based classifier for decision-level fusion.
[0042] The above modules complement each other through multimodal information, forming a monitoring system that integrates multiple types of information and achieves high-precision and high-reliability online judgment. It can detect defects immediately when they occur and take measures, greatly improving the accuracy and real-time performance of defect detection.
[0043] The specific embodiments of the present invention have been described in detail above with reference to the figures, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.
Claims
1. A method for online quality monitoring of multimodal welding combining infrared and acoustic emission, characterized in that: Includes the following steps: Step 1: Synchronously acquire multimodal sensing data obtained by different sensing devices during the welding process. The multimodal sensing data includes at least infrared thermal sequence images and acoustic emission signals. Step 2: Extract feature parameters from the infrared thermal sequence image and the acoustic emission signal, respectively; Step 3: Combine the feature parameters into a fused feature vector and input it into a pre-trained classification model for fusion judgment, and output the prediction result and confidence level representing the welding quality status; Step 4: Make a decision based on the prediction results and confidence level. If it is determined that there is a welding defect, output an alarm signal.
2. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission as described in claim 1, characterized in that: Before step 1, there is a system preparation and startup phase, which includes: installing and connecting the device: fixing the infrared thermal imager to the side and rear of the welding torch and adjusting its field of view to cover the molten pool and heat-affected zone; fixing the acoustic emission sensor to the workpiece; configuring monitoring parameters: inputting welding process parameters, and the system loads the corresponding pre-trained classification model and monitoring threshold accordingly; system calibration: collecting data during defect-free welding and establishing a characteristic baseline for normal welding.
3. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission as described in claim 2, characterized in that: The characteristic baseline includes at least a reference range for the molten pool cooling rate and a reference range for the acoustic emission signal energy.
4. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission according to claim 1, characterized in that: In step 2, the feature parameters extracted from the infrared thermal sequence image include the cooling rate, which is quantified by the following method: calculate the highest temperature in the region of interest (ROI), select the 800℃ and 500℃ isotherms, calculate the time Δt_(8 / 5) required for the distance and temperature difference between them, and substitute it into the cooling rate formula CR = 300 / Δt_(8 / 5)(℃ / s); The method for extracting feature parameters from acoustic emission signals includes: performing bandpass filtering on the acoustic emission signal acquired in step 1, performing wavelet packet transform on the filtered effective signal, decomposing the effective signal to the fourth layer to filter out the target frequency band, and extracting the absolute energy and ringing count features within the target frequency band.
5. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission according to claim 1, characterized in that: In step 3, the classification model is a support vector machine model, and its output prediction results include at least four types: normal, porosity, lack of fusion, and crack.
6. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission as described in claim 1, characterized in that: In step 4, the preset decision logic for the decision judgment is as follows: the result is valid only when the confidence level is higher than the preset confidence threshold; the welding defect is determined and an alarm is triggered only when N consecutive valid results are of the same abnormal defect type.
7. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission according to claim 1, characterized in that: In step 4, the output alarm signal includes sending an emergency stop control signal to the welding power supply or robot controller.
8. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission according to claim 1, characterized in that: It also includes an online quality monitoring system for multimodal welding fusion of infrared and acoustic emission, the system comprising: a multi-sensor module for synchronously acquiring the infrared thermal sequence images and acoustic emission signals; and a synchronous acquisition module connected to the multi-sensor module for receiving and synchronously packaging the data from the multi-sensor module; The data processing module, connected to the synchronous acquisition module, is configured to perform the feature extraction, fusion judgment, and decision output. An alarm control module, connected to the data processing module, is used to respond to the alarm signal and perform corresponding output operations.
9. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission according to claim 8, characterized in that, The multi-sensor module includes: at least one high-speed infrared thermal imager for acquiring the infrared thermal sequence images; and at least one acoustic emission sensor coupled to the welded workpiece via a waveguide rod for acquiring the acoustic emission signals.
10. The online quality monitoring method for multimodal welding fusion of infrared and acoustic emission according to claim 8, characterized in that, The synchronous acquisition module includes a high-speed data acquisition card, which assigns a unified timestamp to the infrared thermal sequence image and acoustic emission data; the data processing module uses an industrial control computer as a carrier and has a built-in core processing unit, which includes an infrared feature extraction unit, an acoustic emission feature extraction unit, and a multi-source information fusion unit.