Online visual detection method for welding defects of aluminum row

Abstract

The invention relates to the field of visual inspection, in particular to an online visual inspection method for welding defects of an aluminum row, which comprises the following steps of: performing joint calibration, synchronously acquiring a deformed stripe image and an infrared thermal image, and processing stripe data through phase unwrapping and coordinate calibration to reconstruct a three-dimensional shape of a welding seam; meanwhile, an optical flow method is used for constructing a molten pool dynamic sensing model, an infrared thermal image is converted into a motion vector field, and dynamic changes of the molten pool are captured; then, the system fuses the three-dimensional morphology, the surface temperature and the motion vector field data to form a multi-modal feature image; abnormal data are eliminated through molten pool characteristic distribution verification, and the analysis reliability is ensured. And finally, a three-branch detection head is adopted to carry out deep analysis on the image, so that the types of defects such as air holes, cracks and undercuts can be identified, the severity of the defects can be quantified, the geometric position, shape and contour of the defects can be accurately positioned, and the comprehensive online monitoring of the welding quality is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual detection, in particular to an online visual detection method for aluminum strip welding defects. BACKGROUND

[0002] In the field of machine vision detection, accurate image analysis of high-temperature dynamic processes such as welding and smelting is a key technical problem, and accurate identification of micro abnormalities from multi-modal visual data streams is crucial to improving the quality control level of automated production.

[0003] Existing image analysis methods have obvious limitations in such applications: relying solely on three-dimensional geometric data for analysis can reconstruct the surface topography of an object, but cannot capture defect precursors caused by internal physical processes such as heat conduction anomalies; analyzing thermal imaging data alone can reflect temperature changes, but due to the lack of precise spatial geometric constraints, the positioning and shape characterization of abnormal heat sources are not accurate; in terms of multi-modal data fusion, existing technologies mostly use simple feature layer splicing, which cannot effectively mine the deep non-linear correlations implied by different modal data in the time and space dimensions; in terms of time series analysis, traditional models are susceptible to gradient problems when processing high-speed industrial data streams, and are difficult to effectively capture the dependency relationships between key transient events with long spans in dynamic processes, resulting in insufficient understanding of complex processes and inadequate anomaly detection capabilities.

[0004] Therefore, an online visual detection method for aluminum strip welding defects is proposed. SUMMARY

[0005] The present application aims to provide an online visual detection method for aluminum strip welding defects, which fuses three-dimensional topography and thermal field spatiotemporal information and uses a deep learning model for dynamic perception analysis to achieve real-time detection and quantitative evaluation of welding defects with high precision and multi-dimensionality. To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] An online visual detection method for aluminum strip welding defects, comprising the following steps:

[0007] Jointly calibrate the stripe projection system and the infrared thermal imager, the stripe projection system projects a preset multi-frequency phase shift sinusoidal stripe to the aluminum strip welding area, and real-time collects the deformed stripe image on the surface of the welding seam; simultaneously collect infrared thermal image data;

[0008] Perform phase unwrapping calculation and coordinate axis calibration on the stripe projection data to obtain three-dimensional topography data; perform temperature field mapping on the infrared thermal image data to obtain surface temperature distribution data, use the optical flow method to construct a molten pool dynamic perception model, create a layered structure of the infrared thermal image data, and combine key point tracking sparse optical flow and full image scanning dense optical flow to convert the infrared thermal image data into a motion vector field;

[0009] The three-dimensional topography data, the surface temperature distribution data and the motion vector field are fused to form a multi-modal feature image; verification calculation is performed based on the molten pool feature distribution; a three-branch detection head is used to analyze the multi-modal feature image after verification, and the defect distribution of the weld surface and the heat affected zone is detected and positioned.

[0010] Preferably, the joint calibration of the fringe projection system and the infrared thermal imager comprises: using a circular dot array and a calibration board of different emissivity materials; acquiring images of the calibration board by using the fringe projection system and the infrared thermal imager respectively; solving the calibration parameters of the infrared thermal imager based on the corresponding point pairs of two-dimensional pixel coordinates and three-dimensional space coordinates; and using the calibration parameters to register the surface temperature distribution data into the coordinate system of the three-dimensional topography data.

[0011] Preferably, the acquisition of the deformed fringe image and the infrared thermal image data comprises: projecting a pre-set coded multi-frequency phase shift sinusoidal fringe to the surface of the aluminum strip welding area by the fringe projection system; setting an external hardware trigger signal source, which synchronously triggers the camera of the fringe projection system and the infrared thermal imager to acquire image frames, so that each frame of fringe projection data strictly corresponds to each frame of infrared thermal image data in time stamp; the fringe projection data is a sequence of deformed fringe images modulated by the three-dimensional topography of the weld surface; the infrared thermal image data acquired by the infrared thermal imager is a sequence of pixel-level temperature radiation intensity data containing the molten pool, the weld and the heat affected zone thereof.

[0012] Preferably, the step of performing phase unwrapping calculation and coordinate axis calibration on the fringe projection data to obtain the three-dimensional topography data comprises: using a time-domain phase unwrapping algorithm to calculate the acquired multiple frames of deformed fringe image sequence to obtain a wrapped phase image, and further to solve a continuous absolute phase distribution image; converting the phase value of each pixel point in the absolute phase distribution image into a three-dimensional space coordinate of the pixel point in the world coordinate system through a mapping relationship between the camera pixel coordinate system and the world three-dimensional coordinate system; organizing the set of three-dimensional space coordinate points calculated from each frame in the continuous acquisition process into a three-dimensional point cloud data stream arranged in time sequence as the three-dimensional topography data.

[0013] Preferably, the conversion of the infrared thermal image data into a motion vector field comprises: acquiring time-series infrared thermal image data, and creating a layered image pyramid based on the infrared thermal image data; identifying key points in the top layer low-resolution image of the layered image pyramid, and tracking the key points across the time sequence to calculate a sparse optical flow field; taking the sparse optical flow field as an initial guide to perform full-pixel dense optical flow calculation in the bottom layer high-resolution image of the layered image pyramid, and converting the infrared thermal image data into a motion vector field.

[0014] Preferably, the step of forming the multi-modal feature image comprises:

[0015] On a single time frame, the profile features of the weld are extracted from the three-dimensional topography data, including the weld width, height and cross-sectional shape parameters; the geometric features of the molten pool are extracted from the surface temperature distribution data, including the molten pool area, length, width and centroid position, and the temperature field features are extracted, including the maximum temperature, average temperature, temperature gradient and isotherm distribution pattern of the key area;

[0016] Between consecutive time frames, the motion vector field is calculated, including the motion speed and acceleration of the molten pool centroid, the change rate of the molten pool area, the change rate of the weld width in the welding direction, and the temperature rise and fall rate of the monitoring point over time;

[0017] The profile features, the geometric features and the temperature field features, together with the motion vector field, form a multi-modal feature image.

[0018] Preferably, the verification calculation based on the molten pool feature distribution comprises: superimposing a virtual grid to divide the area of the multi-modal feature image, mapping the multi-modal data in the area to a feature space based on the molten pool feature distribution verification to obtain corresponding feature vectors; calculating the spatial distance between the feature vectors as the molten pool feature distribution score, and marking the image area as high risk when the consistency score is lower than the preset threshold.

[0019] Preferably, using a three-branch detection head comprises:

[0020] The classification branch performs probability calculation on the defect features through the Softmax layer, and outputs probability values including the types of porosity, crack, surface depression, protrusion, uneven weld, slag inclusion, undercut, and incomplete fusion defects;

[0021] The regression branch maps the spatio-temporal feature sequence to a regression feature representation through a fully connected layer; and converts the regression feature representation to a continuous value through a linear layer, and outputs numerical values of the depth, area and volume of the defect, quantifying the severity of the defect;

[0022] The segmentation branch uses a decoder structure to up-sample the feature maps of the intermediate layers of the Transformer encoder; refines the up-sampling results through a convolutional layer to obtain a defect prediction map; processes the prediction map using a binary function to generate a binary mask for identifying the defect position; and labels the defect position in the weld based on the binary mask to obtain the defect position, shape and contour of the defect in the weld and heat-affected zone;

[0023] The detection results of the classification branch, the regression branch and the segmentation branch are summarized to identify a defect area of the aluminum strip welding, and a structured data report containing defect type, severity, geometric position, shape and contour information is generated.

[0024] Compared with the prior art, the present application has the following advantages:

[0025] 1、The present application constructs a spatio-temporal feature sequence by synchronously collecting the three-dimensional topographic data and dynamic temperature field data of the weld, compared with the single modal detection method, this multi-modal information fusion can capture the deep correlation between geometric deformation and thermal anomaly, and use the Transformer model to deeply learn the dynamic evolution law, which significantly improves the recognition ability and detection accuracy of complex defects such as pores, cracks and incomplete fusion.

[0026] 2、The present application adopts a three-branch detection head of classification, regression and segmentation, which can simultaneously output the category of defects, the quantitative index of severity, and the accurate position and contour in the weld; such comprehensive diagnostic report provides detailed and quantifiable decision basis for subsequent quality evaluation, process tracing and closed-loop control.

[0027] 3、The present application uses the Transformer model to model the dynamic molten pool, which can not only detect the formed defects, but also learn the rules of melt flow and heat conduction, so as to have the ability to identify the abnormal dynamic process leading to defects, and has the potential to develop from post-detection to in-process intervention and even pre-warning, laying a foundation for realizing intelligent closed-loop control of the welding process. BRIEF DESCRIPTION OF DRAWINGS

[0028] Fig. 1 The method flow chart of the online visual detection method for aluminum strip welding defects of the present application;

[0029] Fig. 2 The structural schematic diagram of the online visual detection method for aluminum strip welding defects of the present application;

[0030] Fig. 3 The flowchart of the three-branch detection head of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] Please refer to Figs. 1 to 3The application provides an online visual detection method for welding defects of an aluminum strip, referring to Fig. 1 The application provides an online visual detection method for welding defects of an aluminum strip, referring to Fig. 2 The application provides an online visual detection method for welding defects of an aluminum strip, referring to

[0033] The application provides an online visual detection method for welding defects of an aluminum strip, referring to

[0034] The application provides an online visual detection method for welding defects of an aluminum strip, referring to

[0035] The application provides an online visual detection method for welding defects of an aluminum strip, referring to

[0036] The application provides an online visual detection method for welding defects of an aluminum strip, referring to

[0037] Embodiment one

[0038] First, joint calibration is performed based on Zhang Zhengyou calibration method, and a special calibration board is used in the calibration process; the calibration board is made of high-temperature-resistant base material, has a circular dot array with a diameter of 5 mm on the surface, and is sprayed with materials with different emissivities; for example, part of the circular dots are black coatings with high emissivity, and the other part is metal coatings with low emissivity; before formal welding detection, the calibration board is fixed at the same distance and pose as the actual weld, for example, about 100 mm away from the two acquisition devices;

[0039] The stripe projection system and the infrared thermal imager are used to collect images of the calibration board respectively; in the collected stripe projection camera image and infrared thermal imager image, the two-dimensional pixel coordinates of all the circle point centers are extracted through an image processing algorithm; further, the three-dimensional space coordinates of these common feature points (i.e. circle point centers) in the world coordinate system defined by the stripe projection system are calculated by using the structure light three-dimensional reconstruction principle of the stripe projection system; the rotation and translation relationship between the two coordinate systems is described by using a coordinate system conversion matrix, so as to ensure that the three-dimensional topography data and the two-dimensional temperature data are strictly aligned in space. Based on the corresponding point pairs formed by the known two-dimensional pixel coordinates and three-dimensional space coordinates, a mapping relationship can be established; by using the mapping relationship, the calibration parameters of the infrared thermal imager are solved, including the intrinsic matrix (focal length, principal point coordinates, etc.) and distortion coefficients (radial distortion, tangential distortion, etc.); finally, by using the solved calibration parameters, the subsequent collected infrared thermal image data is corrected and transformed, so that the surface temperature distribution data is registered to the stripe projection world coordinate system in which the three-dimensional topography data is located, and the two kinds of data are strictly aligned in space.

[0040] The described joint calibration method realizes the accurate alignment of the stripe projection system and the infrared thermal imager in space, so that the three-dimensional topography data and the surface temperature data can be fused at the pixel level, and a multi-dimensional and unified data basis is provided for subsequent molten pool dynamic perception and defect detection.

[0041] On an automated welding workbench, an aluminum row workpiece to be welded is fixed at a preset position; above the welding area, a stripe projection system is deployed, and a projector DLP4500 of the stripe projection system projects a series of preset coded, high-frequency sinusoidal stripes of 30 frames per second to the surface of the welding area; at the same time, an industrial camera synchronously collects a deformed stripe image sequence modulated by the three-dimensional topography of the welding seam surface, to obtain stripe projection data.

[0042] At the same time, a high-resolution infrared thermal imager, such as FLIRT1020, is synchronously deployed on one side of the stripe projection system, with a resolution of 1024x768 and a frame rate of 30Hz; the infrared thermal imager synchronously collects infrared thermal image data of the molten pool, the welding seam and the heat-affected zone during the entire welding process; these data are essentially sequences containing pixel-level temperature radiation intensity, for characterizing the thermal dynamics in the welding process; in order to ensure that the two kinds of data are strictly synchronized in time, an external hardware trigger signal source is provided, such as a synchronous trigger started by a welding power source or controlled by a PLC, which synchronously triggers the camera of the stripe projection system and the infrared thermal imager to collect image frames, so as to ensure that each frame of stripe projection data and each frame of infrared thermal image data can strictly correspond in time stamp.

[0043] The collected fringe projection data is a sequence of deformed fringe images sorted by time; each frame image in the sequence records the fringe pattern projected onto the welding area; due to the modulation of the three-dimensional topography of the weld surface, the originally straight fringes will show corresponding deformation or distortion in the image; the degree and mode of these deformations directly reflect the geometric structure information of the weld surface; the collected infrared thermal image data is a sequence of pixel-level temperature radiation intensity data sorted by time; each frame image in the sequence is a two-dimensional data matrix, whose field of view covers the molten pool, the solidified weld, and the surrounding heat-affected zone; the numerical value of each pixel in the matrix represents the thermal radiation energy intensity emitted by the physical point at that time, which is directly related to the surface temperature and used to represent the thermal field distribution and dynamic changes in the welding process.

[0044] Through hardware synchronization triggering, strict correspondence of data on the collection timestamp is achieved, and this accurate timing correlation provides a data basis for in-depth analysis of the instantaneous coupling relationship between the geometric morphology and the thermal field distribution in the welding dynamic process.

[0045] Further, phase unwrapping calculation and coordinate axis calibration are performed on the fringe projection data to obtain three-dimensional topography data, the purpose of which is to convert the collected two-dimensional image data into accurate three-dimensional geometric information.

[0046] Specifically, the fringe projection system projects a series of pre-encoded, multi-frequency phase-shifted sinusoidal fringes onto the surface of the aluminum strip welding area; on an automated welding workbench, a hardware trigger signal is sent by the welding controller, which synchronously triggers the industrial camera and the infrared thermal imager of the fringe projection system at a fixed frequency of 30 frames per second; the projector of the fringe projection system projects a series of pre-encoded 8-frequency phase-shifted sinusoidal fringes onto the surface of the aluminum strip welding area; the fringe sequence contains eight different spatial frequency fringe patterns, each frequency containing four frames of sinusoidal fringe images with a phase shift of 90 degrees, for example, phases of 0°, 90°, 180°, and 270°; this multi-frequency encoding method can effectively solve the phase ambiguity problem of single-frequency fringes on complex surfaces.

[0047] Due to the hardware synchronization triggering, each frame of the deformed fringe image sequence collected by the fringe projection camera, which is modulated by the three-dimensional topography of the weld surface, for example, the 150th fringe image collected at 5 seconds, can strictly correspond to the pixel-level temperature radiation intensity data sequence collected by the infrared thermal imager at the same time point, which contains the molten pool, the weld, and the heat-affected zone; the fringe projection data provides the geometric structure information of the weld, while the data collected by the infrared thermal imager provides the thermal dynamic information in the welding process.

[0048] Furthermore, phase unwrapping and three-dimensional coordinate calculation are performed. Specifically, for each set of acquired deformed stripe image sequences, a temporal phase unwrapping algorithm is used for calculation. This algorithm first uses a four-step phase shift method to calculate the wrapped phase map. For example, for four frames of images at a specific frequency, the wrapped phase is calculated. Then, using the wrapped phase maps of eight stripes at different frequencies, combined with the multi-frequency heterodyne method, phase unwrapping is performed to calculate a continuous and non-jumping absolute phase distribution map.

[0049] Furthermore, by establishing the mapping relationship between the camera pixel coordinate system and the world 3D coordinate system in the joint calibration step, the phase value of each pixel in the absolute phase distribution map is converted into its 3D spatial coordinates in the world coordinate system through the triangulation principle; for example, the phase value at pixel (1024,768) is converted into its 3D coordinates (55.2mm,28.1mm,-3.5mm) in the world coordinate system.

[0050] Furthermore, the method organizes the three-dimensional point cloud data stream by arranging the set of three-dimensional spatial coordinate points calculated in each frame during continuous acquisition into a three-dimensional point cloud data stream in chronological order. This data stream serves as three-dimensional topographic data, accurately recording the geometric evolution of the aluminum busbar weld throughout the welding process. For example, during the entire 30-second welding process, this method generates a sequence containing 900 frames of three-dimensional point cloud data, with each frame containing more than 1 million points, thus providing a detailed geometric basis for subsequent defect detection.

[0051] A temporal phase unwrapping algorithm is used to calculate the wrapped phase map from the acquired multi-frame deformed stripe image sequence, and further calculate the continuous absolute phase distribution map. The calculation includes a deep learning-based denoising and repair algorithm to fill in the phase jump or data missing areas caused by welding fumes or high light reflection. It can intelligently repair the data missing caused by fumes and strong light, ensuring that a complete and accurate phase distribution map can be obtained even in harsh environments.

[0052] This process ensures that each frame of 3D topography data corresponds precisely to each frame of temperature radiation intensity data in terms of timestamps; this synchronization mechanism provides a reliable data foundation for subsequent spatiotemporal feature fusion, accurately capturing the correlation between molten pool dynamics and weld thermodynamic evolution and geometric deformation.

[0053] The system acquires the first and second consecutive infrared thermal images, both with an original resolution of 640x512 pixels. Based on these two images, a corresponding four-layer image pyramid is created through Gaussian blur and downsampling operations. The bottom layer of the pyramid is the original image of 640x512 pixels, the next layer above it has a resolution of 320x256 pixels, the next layer above that has a resolution of 160x128 pixels, and the top layer has a resolution of 80x64 pixels.

[0054] Furthermore, sparse optical flow calculation is performed on the top layer of the layered image pyramid, i.e., on the low-resolution image of 80x64 pixels. On the first frame of the top layer image, the Shi-Tomasi corner detection algorithm is used. The corner quality level threshold is set to 0.01, the minimum spacing is 5 pixels, and about 200 to 300 feature points are identified as key points. The Lucas-Kanade pyramid optical flow method is used to track the position of the identified key points between the first and second frames of the top layer image. This calculation generates a two-dimensional motion vector for each successfully tracked key point, and the set of all these vectors constitutes a sparse optical flow field.

[0055] Guided by the sparse optical flow field, full-pixel dense optical flow calculations are performed in the high-resolution image at the bottom layer of the layered image pyramid. This calculation incorporates three-dimensional topography data as a priori constraints, transforming the infrared thermal image data into a physical motion vector field containing information on the melt flow direction and velocity. The three-dimensional topography data is used to distinguish between surface displacement caused by changes in viewing angle and actual melt flow. For example, if the optical flow shows pixels moving, but the three-dimensional topography shows the area is flat, the system will determine that this may be a change in viewing angle; however, if both the optical flow and the three-dimensional topography show the molten pool surface is concave or convex, the system will determine that this is actual melt movement. This eliminates the depth ambiguity of two-dimensional optical flow, allowing the motion vector field to more accurately reflect the physical flow state of the molten pool, thereby enabling earlier detection of potential defects caused by abnormal melt flow.

[0056] Furthermore, using the sparse optical flow field calculated at the top layer as the initial value, the calculation is performed layer by layer down the pyramid, and finally the dense optical flow field is calculated on the high-resolution image at the bottom layer. Specifically, the sparse optical flow field calculated at the top layer is used as the initial data, and it is upsampled to the size of the next layer (160x128) and used as the initial input for the optical flow calculation at that layer. This process is iterated, and the result of the current layer calculation is passed to the next layer until the bottom layer of the pyramid.

[0057] Furthermore, at the bottom layer of the pyramid, i.e., the original resolution image of 640x512 pixels, Gunnar was used. Dense optical flow algorithm; the initial flow field of this algorithm is the flow field data passed down from the previous layer. Based on this, the algorithm calculates the displacement vector of each pixel in the first frame image to the second frame image.

[0058] The final output is a motion vector field with the same resolution as the original infrared thermal image (640x512); this vector field is a two-dimensional vector matrix, where the vector at each coordinate position corresponds to the direction and velocity of motion of that point on the molten pool surface within a 10-millisecond time interval (unit: pixels / frame);

[0059] This method first performs sparse optical flow calculation on low-resolution images, which not only significantly reduces the amount of computation to meet real-time requirements, but also effectively overcomes the problem of large displacement tracking caused by the high-speed movement of the molten pool. Then, the calculation result is used as the initial value to guide the dense optical flow solution of the original resolution image, effectively identifying the relevant details of macroscopic flow trends and microscopic flow.

[0060] Static feature extraction is performed on a single time frame. Specifically, the system processes the three-dimensional topography data; by performing contour analysis on the three-dimensional point cloud of the weld area, the weld width at that moment is calculated to be 8.2 mm and the weld reinforcement height is 0.5 mm, and the cross-sectional shape parameters of the weld are extracted.

[0061] Furthermore, the system processes the surface temperature distribution data; by setting a temperature threshold (e.g., 660 degrees Celsius), it identifies and segments the molten pool region; it calculates that the area of ​​the molten pool is 95 square millimeters, the length along the welding direction is 15 millimeters, the width is 8 millimeters, and its centroid is located at image coordinates (320, 250); at the same time, it extracts temperature field features in the molten pool and its surrounding key areas, calculates that the highest temperature of the molten pool is 1800 degrees Celsius, the average temperature is 1350 degrees Celsius, and records the magnitude of the temperature gradient and the distribution shape of the isotherms at the tail of the molten pool;

[0062] Furthermore, the system retrieves data from the previous time frame (5.00 seconds, i.e., 10 milliseconds ago) and compares it with the data in the current frame to calculate dynamic features;

[0063] Specifically, by comparing the position of the centroid of the molten pool in two consecutive frames, the velocity of the centroid along the welding direction was calculated to be 10 mm per second, and the acceleration was calculated to be 0.1 mm squared per second. By comparing the area of ​​the molten pool, the rate of change of its area was calculated to be an increase of 2 square millimeters per second. Similarly, the rate of change of the weld width in the welding direction was calculated to be a decrease of 0.1 mm per second. In addition, at a fixed monitoring point, the rate of temperature rise and fall over time was calculated to be a cooling of 3000 degrees Celsius per second.

[0064] The static contour features, geometric features, and temperature field features extracted in a single time frame, together with the dynamic motion vector field features calculated between consecutive time frames, are structured and organized. All these quantified feature data together constitute a multimodal feature data set associated with physical spatial location, which is the multimodal feature image.

[0065] Furthermore, the multimodal feature image is subjected to region segmentation and verification:

[0066] Specifically, the system overlays a 50x50 virtual grid onto the welding area corresponding to the multimodal feature image, dividing it into 2500 independent region units;

[0067] Furthermore, the data within each region cell is processed. Taking a region cell with grid coordinates (25, 30) as an example, the system processes all multimodal data within this cell (such as local weld height, average temperature, motion vectors, etc.) through a pre-trained mapping model, using a neural network with three fully connected layers as the mapping model. It explains that through self-supervised learning, these data are projected onto a unified 128-dimensional feature space, obtaining three corresponding feature vectors belonging to different modalities.

[0068] Specifically, in the 128-dimensional feature space, the spatial distance between each pair of these three feature vectors is calculated, for example, the Euclidean distance between the morphology feature vector and the temperature feature vector is calculated; the calculated distance value (e.g., 0.25) is used as the molten pool feature distribution score of the unit in this region; the molten pool feature distribution is a statistical law followed by the multimodal features (morphology, temperature, motion) of the molten pool and its surrounding area during a large number of good welding processes;

[0069] Furthermore, the score is compared with a preset threshold (e.g., 0.3). Since the calculated score of 0.25 is lower than the preset threshold of 0.3, the system marks the region cell with grid coordinates (25, 30) as a high-risk image region. The system repeats this verification process for all 2500 region cells. When the input data quality is poor (e.g., obscured by smoke or dust) or the model itself lacks confidence in the prediction results, it explicitly marks the region numerically using variance and joint regression methods, and uses this information to adjust the calculation of the melt pool feature distribution score. The coordinate information of the high-risk image regions is stored and collected. Based on the coordinate information of the high-risk image regions, the corresponding regions are highlighted or marked on the original image, and an analysis map marked with high-risk regions is output. Incorporating uncertainty quantification into the melt pool feature distribution verification allows the system to automatically reduce the weight of unreliable data when facing environmental interference such as smoke and dust, avoiding false alarms and helping to distinguish between real defects and data quality problems.

[0070] By verifying the inherent consistency between multimodal data, it is possible to discover and locate latent or early defects that cannot be identified by a single sensor, such as areas with normal geometry but internal thermal anomalies; the spatial location of high-risk areas is output as the focus of attention for defect areas.

[0071] Reference Fig. 3 This is a flowchart illustrating the three-branch detection head of the present invention. The verified multimodal feature image is first processed by the classification branch. This branch network uses a Softmax layer and a "Swin Transformer" as the encoder. Simultaneously, the loss function used by each branch is defined; for example, the classification branch uses cross-entropy loss, the regression branch uses L1 loss, and the segmentation branch uses Dice loss. The total loss is a weighted sum of the various losses calculated on the input features, outputting a probability list covering eight predefined defect types.

[0072] Furthermore, taking the currently detected high-risk area as an example, the probability values ​​output by the classification branch are: {porosity: 0.88, cracks: 0.05, surface depressions: 0.02, protrusions: 0.01, uneven welds: 0.03, slag inclusions: 0.01, undercut: 0.00, lack of fusion: 0.00}; based on the highest value of the "porosity" item in this probability distribution, the system classifies the defect type of this area as "porosity";

[0073] Furthermore, the multimodal feature image is synchronously fed into a regression branch; this branch maps the input features into a regression feature representation via a fully connected layer, and then a linear layer converts this representation into a set of continuous numerical outputs; specifically, the linear layer outputs three values, corresponding to the depth, area, and volume of the defect, respectively; for the defect identified as "pores" above, the quantized values ​​output by the regression branch are: depth 1.2 mm, area 2.1 square millimeters, and volume 2.5 cubic millimeters; these values ​​objectively describe the physical size and severity of the defect;

[0074] Furthermore, the segmentation branch receives and processes feature maps from the intermediate layers of the Transformer encoder; a decoder structure upsamples these feature maps step by step, restoring their spatial dimensions to correspond to the high-risk regions of the input.

[0075] Specifically, the upsampled result is processed through several convolutional layers to refine the features and generate a defect prediction map, where the intensity value of each pixel represents the probability that it belongs to a defect. Further, the prediction map is processed by a binarization function with a preset threshold of 0.5 to generate a binary mask for accurately identifying the defect location.

[0076] Finally, the system marks the location of the weld defect based on the binary mask; by parsing the pixel area with a value of 1 in the mask, the system obtains the geometric location, specific shape (e.g., two independent near-circular areas) and complete set of contour coordinate points of the "porosity" defect in the weld.

[0077] Furthermore, the system summarizes the detection results from the classification branch, regression branch, and segmentation branch. Specifically, the system integrates information from the three branches, identifies the defective area of ​​the aluminum busbar welding, and generates a structured data report.

[0078] The multi-task detection head can output the category, quantified size, and precise location of defects in parallel at the same time. This embodiment realizes a comprehensive characterization from a simple "defect presence or absence" judgment to "what it is, how serious it is, and where it is". This integrated architecture significantly reduces the consumption of computing resources and inference time by sharing the underlying feature extraction network, meeting the real-time requirements of industrial online high-speed inspection scenarios. Finally, all analysis results are summarized into a structured data report, providing direct, quantitative, and comprehensive data support for automated quality assessment, process traceability, and process optimization.

[0079] Example 2

[0080] This embodiment aims to verify the performance of the method of the present invention in handling non-linear welds, complex workpiece surfaces, and external dynamic interference such as welding fumes and strong light reflection; it will focus on a specific scenario: multi-angle collaborative welding of aluminum busbars with curved joints by robots;

[0081] Because the curved joint prevents the calibration board from being completely planarized, the movement of the robot arm causes dynamic changes in the relative positions of the camera and thermal imager, requiring more precise synchronization. Furthermore, the fumes and plasma arc generated during the welding process interfere with the quality of optical and infrared images.

[0082] A custom-designed curved calibration plate matching the shape of the aluminum busbar curved joint is used; the surface of the plate is also decorated with a dot array and coatings with different emissivity. In the joint calibration step, in addition to calibrating the intrinsic and extrinsic parameters of the camera and thermal imager, kinematic calibration is also performed between them and the coordinate system of the robot arm end effector. This enables the system to track the precise position of the camera and thermal imager relative to the workpiece in real time, and ensures the spatial alignment of the three-dimensional shape data and thermal field data even during robot movement.

[0083] Furthermore, the industrial camera adopts a high dynamic range mode, which continuously captures multiple images with different exposure times and fuses them to effectively suppress strong light reflection and overexposed areas, ensuring the clarity of deformed stripes; the infrared thermal imager uses a narrow-band optical filter, which only allows infrared radiation of specific wavelengths to pass through, in order to filter out strong light interference generated by non-thermal sources such as plasma arc light, ensuring the accuracy of temperature data.

[0084] Traditional straight-line scanning models are not suitable for the three-dimensional reconstruction of curved welds. Curved welding causes complex changes in the flow direction and velocity of the molten pool, which are difficult to accurately capture using a single optical flow method. In areas with smoke and dust interference, some modal data may be missing or distorted.

[0085] A coordinate system reconstruction algorithm based on curved paths is adopted. The system first identifies the center line of the weld by a preset robot trajectory or real-time image processing, and uses this center line as a reference to construct a local three-dimensional coordinate system that dynamically moves and rotates with the welding path. All point cloud data are converted to this local coordinate system for analysis, making the extraction of geometric features of curved welds more accurate.

[0086] When constructing a dynamic sensing model of the molten pool using the optical flow method, depth information is introduced to assist in the process. The system can provide depth information for each pixel in the infrared thermal image by using the three-dimensional topography data obtained through fringe projection. When calculating the optical flow, combining depth changes, such as surface depressions or protrusions in the molten pool region, can more accurately determine whether the melt is flowing outward or entrained inward, thereby more accurately capturing potential slag inclusions or porosity precursors.

[0087] Furthermore, when forming a multimodal feature image, an uncertainty weight is added to each feature; for example, in areas where smoke and dust obscure the infrared image quality and cause a decrease in quality, the corresponding temperature field feature weight will be reduced; while in areas where geometric information is clear, the weight of the morphology feature will be increased.

[0088] Traditional rectangular bounding boxes or single masks are difficult to accurately annotate curved cracks or uneven welds. A region may have multiple defects at the same time, such as porosity and surface inhomogeneity. Complex models and data processing increase the computational burden.

[0089] Unlike traditional binary masks, instance segmentation can distinguish and label each individual defect, even if they are multiple closely adjacent pores or cracks extending along a curve. For slender defects, such as cracks and undercuts, the segmentation mask will be post-processed by a curve fitting algorithm to generate one or more B-spline curves to more accurately describe its spatial shape and trend, rather than a simple set of pixels.

[0090] Furthermore, multi-label classification and hierarchical regression are employed:

[0091] The classification branch upgrades the traditional single-label classification to multi-label classification; this allows the model to output the probability of multiple defect types at the same time. For example, a region may be labeled as "porosity" with a probability of 0.85 and "irregular weld" with a probability of 0.7 at the same time, which more realistically reflects the coexistence of complex defects.

[0092] Regression Branch: A graded regression is used to determine the severity of the regression. In addition to outputting continuous depth, area, and volume values, it also maps them to several preset severity levels, such as mild, moderate, and severe. This provides a more intuitive decision-making basis for manual quality inspection and automated sorting.

[0093] Meanwhile, in order to meet real-time requirements, the trained deep learning model is pruned and quantized during the model deployment phase. By removing redundant connections and reducing computational precision, the model size and inference time can be significantly reduced without significantly affecting performance, ensuring the efficient operation of the system on the industrial production line.

[0094] This embodiment employs robot-assisted multi-angle collaborative welding, specifically a scenario involving curved joint aluminum busbars. The resulting weld is a complex, curved process. This embodiment introduces a customized curved calibration plate, verifying the invention's online detection capabilities under irregular weld seams and dynamic interference. It overcomes challenges such as non-planar calibration and environmental interference, ensuring that the system can still achieve high-precision detection of the welding process and defects in complex industrial processing environments.

[0095] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An on-line visual inspection method for detecting defects in a seam weld of aluminum, characterized by, The method comprises the following steps: The fringe projection system and the infrared thermal imager are jointly calibrated, the fringe projection system projects preset multi-frequency phase shift sinusoidal fringes to the welded area of the aluminum strip, and the deformed fringe images of the weld surface are collected in real time; Synchronous acquisition of infrared thermal image data; Phase unwrapping calculation and coordinate axis calibration are performed on the fringe projection data to obtain three-dimensional topographic data; temperature field mapping is performed on the infrared thermal image data to obtain surface temperature distribution data; a molten pool dynamic perception model is constructed using an optical flow method, the model is used for layering the infrared thermal image data, combining key point tracking sparse optical flow and full image scanning dense optical flow, and converting the infrared thermal image data into a motion vector field; The three-dimensional topographic data, the surface temperature distribution data and the motion vector field are fused to form a multi-modal feature image; the multi-modal feature image is verified based on the molten pool feature distribution, and a three-branch detection head is used to analyze the verified multi-modal feature image to detect and locate the defect distribution of the weld surface and the heat affected zone.

2. The method of claim 1, wherein the method is characterized by: The joint calibration of the fringe projection system and the infrared thermal imager comprises: A circular dot array and a calibration board of different emissivity materials are used; the fringe projection system and the infrared thermal imager are used to collect images of the calibration board respectively; the calibration parameters of the infrared thermal imager are solved based on the corresponding point pairs of two-dimensional pixel coordinates and three-dimensional space coordinates; and the surface temperature distribution data is registered into the coordinate system of the three-dimensional topographic data using the calibration parameters.

3. The method of claim 1, wherein the method comprises: It comprises: The collection of the deformed fringe images and the infrared thermal image data comprises: The fringe projection system projects preset coded multi-frequency phase shift sinusoidal fringes to the surface of the aluminum strip welding area; an external hardware trigger signal source is set, the signal source synchronously triggers the camera of the fringe projection system and the infrared thermal imager to collect image frames, and each frame of fringe projection data strictly corresponds to each frame of infrared thermal image data in time stamp; the fringe projection data is a sequence of deformed fringe images modulated by the three-dimensional topography of the weld surface; the infrared thermal imager collects infrared thermal image data which is a sequence of pixel-level temperature radiation intensity data containing the molten pool, the weld and the heat affected zone.

4. The method of claim 1, wherein the method further comprises: The step of performing phase unwrapping calculation and coordinate axis calibration on the fringe projection data to obtain three-dimensional topographic data specifically comprises: A time domain phase unwrapping algorithm is used to calculate the collected sequence of deformed fringe images to obtain a wrapped phase image and further solve a continuous absolute phase distribution image; the phase value of each pixel point in the absolute phase distribution image is converted into a three-dimensional space coordinate in the world coordinate system through the mapping relationship between the camera pixel coordinate system and the world three-dimensional coordinate system; the set of three-dimensional space coordinate points calculated in each frame in the continuous acquisition process is organized into a three-dimensional point cloud data stream arranged in time sequence as the three-dimensional topographic data.

5. The method of claim 1, wherein the method is an on-line visual inspection method for detecting defects in an aluminum strip. The conversion of the infrared thermal image data into a motion vector field comprises: The infrared thermal image data is acquired in time sequence, and a layered image pyramid is created based on the infrared thermal image data; key points are identified in a top layer low-resolution image of the layered image pyramid, and the key points are tracked in time sequence to calculate a sparse optical flow field; the sparse optical flow field is used as an initial guide to perform full-pixel dense optical flow calculation in a bottom layer high-resolution image of the layered image pyramid, and the infrared thermal image data is converted into a motion vector field.

6. The method of claim 1, wherein the method further comprises: The step of forming the multi-modal feature image comprises: On a single time frame, the profile features of the weld are extracted from the three-dimensional topographic data, including weld width, height and cross-sectional shape parameters; the geometric features of the molten pool are extracted from the surface temperature distribution data, including molten pool area, length, width and centroid position, and the temperature field features are extracted, including the highest temperature, average temperature, temperature gradient and isotherm distribution pattern of the key area; Between consecutive time frames, the motion vector field is calculated, including the motion speed and acceleration of the molten pool centroid, the change rate of the molten pool area, the change rate of the weld width in the welding direction, and the temperature rise and fall rate of the monitoring point over time; The profile features, the geometric features and the temperature field features are combined with the motion vector field to form a multi-modal feature image.

7. The method of claim 1, wherein the method further comprises: The verification calculation based on the molten pool feature distribution comprises: The area of the multi-modal feature image is segmented by superimposing a virtual grid, the multi-modal data in the area is mapped to a feature space based on molten pool feature distribution verification to obtain corresponding feature vectors; the spatial distance between the feature vectors is calculated as a molten pool feature distribution score, and when the distribution score is lower than a preset threshold, the current area is marked as a high-risk image area.

8. The method of claim 1, wherein the method further comprises: The three-branch detection head comprises: The classification branch performs probability calculation on the defect features through a Softmax layer, and outputs probability values including the types of porosity, crack, surface depression, protrusion, uneven weld, slag inclusion, undercut and incomplete fusion defects; The regression branch is mapped to a regression feature representation through a fully connected layer; the regression feature representation is converted to a continuous value through a linear layer, and numerical values of the depth, area and volume of the defect are output to quantify the severity of the defect; The segmentation branch uses a decoder structure to up-sample the feature maps of the intermediate layers of the Transformer encoder; the up-sampling results are refined through a convolution layer to obtain a defect prediction map; a binary mask for identifying the position of the defect is generated by processing the prediction map using a binary function; the position of the weld defect is labeled based on the binary mask to obtain the geometric position, shape and contour of the defect in the weld and heat affected zone; The detection results of the classification branch, the regression branch and the segmentation branch are summarized to identify the defect area of the aluminum lap welding, and a structured data report containing the defect type, severity, geometric position, shape and contour information is generated.

Images