Multi-modal sensor fusion based intelligent detection and early warning system for welding seam defects in automobile welding

Abstract

The present application relates to the technical field of intelligent detection of welds, and particularly relates to a multi-modal sensor fusion intelligent detection and early warning system for automobile welding weld defects, which comprises a synchronous acquisition module, a multi-level feature module, an initial classification module, a label propagation module, a step-by-step optimization module and a defect early warning module; by synchronously acquiring HDR visual images and thermal imaging images and constructing multi-level resolution image representations, artifact correlation features and defect physical features are extracted to realize explicit differentiation between artifacts and real defects, and then structural priors are successively transferred to higher resolution levels for label optimization, and in the optimization process, the composite features make the artifacts lacking physical evidence be corrected and the real defects conforming to physical laws be retained, thereby solving the problem of high false alarm rate of weld defects caused by synthetic artifacts introduced by HDR imaging and motion coupling, and significantly improving the accuracy and stability of the automobile welding online detection system.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent weld inspection, and more particularly to an intelligent detection and early warning system for weld defects in automotive welding based on multimodal sensor fusion. Background Technology

[0002] Robotic welding technology is widely used in automated automotive welding production lines, and the welding quality directly determines the strength, safety, and durability of the vehicle body structure. To achieve online quality monitoring, industrial camera vision inspection has become the mainstream method. However, the extremely strong arc light generated by the welding arc causes both overexposed bright areas and underexposed dark areas to exist simultaneously in the acquired images. The dynamic range of a single frame image far exceeds the capabilities of conventional image sensors, resulting in severe loss of weld details.

[0003] To address these issues, existing technologies commonly employ HDR imaging, using dual-gain or multi-exposure synthesis to extend the dynamic range and simultaneously preserve details in both bright and dark areas. However, in actual welding production lines, welding robots inevitably experience slight movements due to limitations in motion control precision and mechanical vibrations. When HDR multi-frame synthesis is coupled with this movement, image artifacts such as ghosting, edge overshoot, and false textures will occur in high-contrast areas like the weld-base material interface and the edges of high-reflectivity zones. These artifacts are morphologically highly similar to real weld defects such as microcracks, undercut, and lack of fusion, causing traditional defect recognition algorithms based on edge detection and texture analysis to misidentify a large number of artifacts as defects, leading to a sharp increase in the false alarm rate and severely weakening the reliability and engineering practicality of online inspection systems. Summary of the Invention

[0004] To address the technical problems existing in the background art, this invention proposes a multimodal sensor fusion-based intelligent detection and early warning system for weld defects in automotive welding, the specific solution of which is as follows:

[0005] A multimodal sensor fusion-based intelligent detection and early warning system for weld defects in automotive welding includes:

[0006] A synchronous acquisition module is used to synchronously acquire multimodal data of the weld area, including HDR visual images and thermal imaging images;

[0007] A multi-level feature module is used to construct multi-level resolution image representations based on the HDR visual image and the thermal imaging image respectively, and extract composite features, including artifact correlation features characterizing HDR imaging artifacts and defect physical features characterizing real physical defects.

[0008] An initial classification module is used to generate an initial classification confidence map based on the composite features, wherein the initial classification confidence map represents the initial confidence that a pixel belongs to a real defect;

[0009] The label propagation module is used to iteratively propagate labels onto the initial classification confidence map at the lowest resolution level of the multi-level resolution image representation to obtain a label matrix of the lowest resolution level structure.

[0010] The step-by-step optimization module is used to optimize the labels of higher resolution layers step by step, using the lowest resolution hierarchical structure label matrix as a priori. The label optimization combines the initial classification confidence map and composite features of the current layer to obtain the label matrix of each hierarchical structure.

[0011] The defect warning module is used to construct a defect mask from the pixels identified as defects in the highest resolution hierarchical label matrix, perform defect pattern recognition, and generate a warning signal.

[0012] Furthermore, in the synchronous acquisition module, the HDR visual image is acquired by an HDR visual camera installed on the end effector of the welding robot, and the thermal imaging image is acquired by an infrared thermal imager installed on the end effector of the welding robot; the HDR visual camera is configured in dual-gain mode or multi-exposure synthesis mode;

[0013] The HDR visual camera and the infrared thermal imager are synchronized and triggered by a high-precision clock.

[0014] Furthermore, in the multi-level feature module, multi-level resolution image representations are constructed, and composite features are extracted, including:

[0015] Gaussian downsampling is performed on the HDR visual image and the thermal imaging image respectively to generate L-layer images with progressively decreasing resolution, where L≥3, and the lowest resolution layer is the top layer and the highest resolution layer is the bottom layer.

[0016] Extract composite features from each layer of images at each resolution level.

[0017] Furthermore, the artifact correlation features include at least one of the following:

[0018] Spectrum proximity, which is the distance between a pixel and the nearest saturated specular region;

[0019] Gradient brightness nonlinearity refers to the degree to which the statistical relationship between local gradient magnitude and local average brightness deviates from the preset natural image statistical model.

[0020] Color space distortion refers to the Euclidean distance of a boundary pixel relative to its neighboring regions in the Lab or YCrCb color space in terms of color chromaticity components.

[0021] The physical characteristics of the defect include abnormal thermal gradients, i.e., abrupt changes in temperature gradient detected at the corresponding location in the thermal imaging image.

[0022] Furthermore, in the initial classification module, an initial classification confidence map is generated based on the composite features. This initial classification confidence map represents the initial confidence level that a pixel belongs to a true defect, including:

[0023] Obtain a training sample set, which includes multiple sets of multi-level resolution image representations and their corresponding composite features, as well as pixel-level labeled category labels, including real defects, HDR imaging artifacts and normal regions;

[0024] Based on the training sample set, an initial classification model is constructed using a micro neural network. The initial classification model takes the composite feature as input and outputs the confidence level that a pixel belongs to the true defect category.

[0025] The composite features corresponding to the images at each resolution level are input into the trained initial classification model, and the initial classification confidence map is output.

[0026] Furthermore, in the label propagation module, at the lowest resolution level of the multi-level resolution image representation, iterative label propagation is performed on the initial classification confidence map to obtain the lowest resolution level structure label matrix, including:

[0027] The initial classification confidence map is downsampled to the resolution of the lowest resolution level;

[0028] Based on the confidence level after downsampling, the initial labels of the pixels are divided into background, weld area and undetermined anomaly area to form the lowest resolution level initial label matrix;

[0029] Iterative label propagation is performed on the initial label matrix of the lowest resolution level. For a pixel labeled as an undetermined anomaly region, if the number of pixels labeled as weld seam region in its neighborhood is greater than the number of pixels labeled as background, the label of the pixel in the undetermined anomaly region is updated to weld seam region; otherwise, it is updated to background. The iteration continues until convergence, and the label matrix of the lowest resolution level structure is obtained.

[0030] Furthermore, in the stepwise optimization module, the label optimization is achieved by constructing an energy function and iteratively minimizing the energy function;

[0031] In the energy function, the total cost of assigning a pixel to a certain label category is composed of a weighted sum of data terms, context terms, guidance terms, and feature terms;

[0032] The data item constrains the label allocation based on the initial classification confidence map of the current level; the context item constrains the label allocation based on the consistency of neighboring pixel labels; the guiding item constrains the label allocation based on the prior structural information passed from the label matrix of the previous layer; and the feature item constrains the label allocation based on the composite feature.

[0033] The feature term is constructed such that when the pixel is assigned to a defect category, the value of the feature term increases with the increase of the artifact correlation feature intensity of the pixel and decreases with the increase of the physical feature intensity of the defect.

[0034] When a pixel is assigned to a non-defect category, the value of the feature term increases with the increase of the physical defect feature strength of the pixel and decreases with the increase of the artifact correlation feature strength.

[0035] Furthermore, the data item is defined as follows: when the label represents a true defect category, the value of the data item decreases as the confidence that the pixel belongs to a true defect increases in the initial classification confidence map of the current level, and the decreasing trend is logarithmic; when the label represents a non-true defect category, the value of the data item decreases as the confidence that the pixel does not belong to a true defect increases, and the decreasing trend is logarithmic.

[0036] The context term is defined as follows: the value of the context term is equal to the number of neighboring pixels whose labels are inconsistent when the label assigned to the current pixel is compared with the labels of each pixel in its preset neighborhood.

[0037] The guiding term is defined as follows: when the label assigned to the current pixel is consistent with the structure category indicated by the upper-level structure label matrix at that pixel position, the value of the guiding term is zero; when the label assigned to the current pixel is inconsistent with the structure category indicated by the upper-level structure label matrix at that pixel position, the value of the guiding term is a preset positive number.

[0038] Furthermore, the defect early warning module performs defect pattern recognition, including:

[0039] Morphological post-processing is performed on the connected regions in the defect mask to obtain each connected region;

[0040] Extract the morphological features of each connected region, as well as the statistics of the composite features within the corresponding region;

[0041] The extracted morphological features and statistics are input into a pre-trained classifier, which outputs the defect pattern category corresponding to each connected region; the defect pattern category includes cracks, pores, undercuts and lack of fusion.

[0042] Furthermore, the defect early warning module generates early warning signals including:

[0043] Based on the identified defect patterns, defect sizes, and defect locations, and in conjunction with preset quality standards, an early warning signal is generated. The early warning signal is used to mark the defect location and type.

[0044] The warning signal is fed back to the production execution system or robot controller.

[0045] Compared with the prior art, the present invention can achieve at least the following beneficial effects:

[0046] 1. This invention simultaneously acquires HDR visual images and thermal imaging images and constructs multi-level resolution image representations. It extracts artifact correlation features and defect physical features to achieve explicit distinction between artifacts and real defects. At the low-resolution level, it establishes macroscopic structural priors that are not affected by artifacts through iterative label propagation. Then, it passes the structural priors to higher resolution levels for label optimization. By utilizing composite features during the optimization process, artifacts lacking physical evidence are corrected, while real defects that conform to physical laws are preserved. This fundamentally solves the problem of high false alarm rate of weld defects caused by synthetic artifacts introduced by HDR imaging and motion coupling, and significantly improves the accuracy and stability of the automotive welding online inspection system.

[0047] 2. This invention explicitly distinguishes between artifacts and defects at the feature level by extracting artifact correlation features that characterize HDR imaging artifacts and defect physical features that characterize real physical defects. Artifact correlation features capture the unique image statistical characteristics of artifacts such as synthetic ghosting and edge overshoot, while defect physical features, such as thermal gradient anomalies, reflect the physical changes caused by real defects. Since HDR imaging artifacts only exist in visual images and do not produce corresponding temperature anomalies in thermal imaging images, the physical information provided by thermal imaging provides a reliable criterion for distinguishing between artifacts and real defects.

[0048] By establishing macroscopic structural priors through iterative label propagation at the lowest resolution level, the interference of artifacts on structural judgment is effectively suppressed. Image details in the low resolution level are smoothed, the high-frequency features of artifacts are suppressed, and the macroscopic structure of the weld area and the background area is more significant. The resulting structural label matrix is ​​stable and reliable.

[0049] By progressively passing the macroscopic structure prior to higher resolution levels for label optimization, and combining the initial classification confidence map and composite features of the current level during the optimization process, pixel-level label correction from coarse to fine is achieved. This progressive guidance mechanism ensures that the label allocation at the high resolution level does not deviate from the established macroscopic structure, so that artifact regions are corrected to non-defect categories due to lack of physical evidence support during the optimization process, while real defects are preserved because they conform to physical laws. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0051] Figure 1 This is a schematic diagram of the overall system principle of the present invention;

[0052] Figure 2 This is a flowchart of the initial classification module of the present invention;

[0053] Figure 3 This is a flowchart of the defect warning module of the present invention. Detailed Implementation

[0054] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0055] Please refer to Figures 1 to 3 This invention provides an intelligent detection and early warning system for weld defects in automotive welding assembly based on multimodal sensor fusion, including a synchronous acquisition module, a multi-level feature module, an initial classification module, a tag propagation module, a step-by-step optimization module, and a defect early warning module.

[0056] The synchronous acquisition module is used to synchronously acquire multimodal data of the weld area, including HDR visual images and thermal imaging images.

[0057] It should be noted that the multimodal data includes HDR visual images and thermal imaging images. These two types of data reflect the state of the weld area from different physical dimensions. HDR visual images are used to capture high dynamic range visible light information of the weld area, which can simultaneously and clearly present the bright arc light area and the darker base material area during the welding process, providing rich texture and edge information for subsequent artifact recognition and defect detection. Thermal imaging images are used to capture temperature field distribution information of the weld area, which can reflect the abnormal heat conduction caused by real physical defects such as porosity, cracks, and lack of fusion inside the weld, providing a physical basis for distinguishing between HDR imaging artifacts and real defects.

[0058] In an optional embodiment, in the synchronous acquisition module, the HDR visual image is acquired by an HDR visual camera mounted on the end effector of the welding robot, and the thermal imaging image is acquired by an infrared thermal imager mounted on the end effector of the welding robot; the HDR visual camera is configured in dual-gain mode or multi-exposure synthesis mode;

[0059] The HDR visual camera and the infrared thermal imager are triggered synchronously by a high-precision clock with a synchronization accuracy better than 100 microseconds.

[0060] It should be noted that the HDR vision camera is configured in dual-gain mode or multi-exposure synthesis mode. Both modes can acquire images with different brightness responses in single or multiple exposures and synthesize them, thereby effectively expanding the dynamic range of the image, avoiding local overexposure or underexposure caused by welding arc light, and preserving the detailed information of key areas such as the junction of the weld and the base material and the edge of the high-gloss reflection zone.

[0061] It should be noted that the synchronization accuracy of better than 100 microseconds means that the trigger time deviation between the HDR visual camera and the infrared thermal imager does not exceed 100 microseconds. This synchronization accuracy ensures that the data collected by the two types of sensors strictly correspond to the same physical location of the weld, enabling the thermal field gradient anomaly features extracted from the thermal imaging image in subsequent steps to be accurately mapped to the corresponding pixel positions in the HDR visual image. Insufficient synchronization accuracy will lead to spatial misalignment of multimodal features, making it impossible to effectively use the physical features of defects based on thermal imaging for energy function constraints of pixel-level labels in the visual image, thereby reducing the ability to distinguish between artifacts and real defects.

[0062] The multi-level feature module is used to construct multi-level resolution image representations based on the HDR visual image and the thermal imaging image respectively, and extract composite features, including artifact correlation features characterizing HDR imaging artifacts and defect physical features characterizing real physical defects.

[0063] It should be noted that constructing multi-level resolution image representations for both the HDR visual image and the thermal imaging image is to analyze the features of the weld region at different spatial scales. At the lowest resolution level, image details are smoothed, and macroscopic structural partitions (such as the weld region and background region) are more prominent, which is beneficial for establishing a stable global structural prior in the subsequent label propagation module. At higher resolution levels, the image retains richer detail information, which is beneficial for fine-tuning pixel-level labels in the subsequent progressive optimization module. Through this multi-level resolution architecture from coarse to fine, it is possible to retain the detailed information of the real defects while suppressing noise and artifact interference.

[0064] In an optional embodiment, the multi-level feature module constructs a multi-level resolution image representation and extracts composite features, including:

[0065] Gaussian downsampling is performed on the HDR visual image and the thermal imaging image respectively to generate L-layer images with progressively decreasing resolution, where L≥3, and the lowest resolution layer is the top layer and the highest resolution layer is the bottom layer.

[0066] Extract composite features from each layer of images at each resolution level.

[0067] It should be noted that Gaussian downsampling is performed on both HDR visual and thermal images to generate L layers of images with progressively decreasing resolution, where L≥3. Gaussian downsampling, which smooths the image using Gaussian filtering before downsampling, effectively suppresses high-frequency noise, ensuring a good signal-to-noise ratio for each layer at its corresponding resolution. Defining the lowest resolution layer as the top layer and the highest resolution layer as the bottom layer is to maintain consistency with the "top-to-bottom" processing order in the subsequent label propagation and progressive optimization modules, facilitating description and understanding. The downsampling factor is typically set to 2, meaning the resolution of each layer is half that of the layer above.

[0068] It should be noted that the composite features include two categories: artifact correlation features and defect physical features. These two types of features are designed to distinguish HDR imaging artifacts from real physical defects from different dimensions. Artifacts such as synthetic ghosting and edge overshoot generated by the coupling of HDR imaging with the welding robot's motion have specific image statistical characteristics, but unlike real physical defects (such as cracks and pores), these artifacts are not accompanied by abnormal changes in the temperature field. Therefore, by extracting artifact correlation features characterizing HDR imaging artifacts and defect physical features characterizing real physical defects, complementary discrimination criteria can be provided for distinguishing between the two during the subsequent label optimization process of the progressive optimization module.

[0069] In an optional embodiment, the artifact correlation feature includes at least one of the following:

[0070] Spectrum proximity, which is the distance between a pixel and the nearest saturated specular region;

[0071] It should be noted that the highlight proximity refers to the distance between a pixel and the nearest saturated highlight area. Since ghosting artifacts typically appear near high-contrast edges during HDR composition, especially around the boundaries of saturated highlight areas, highlight proximity effectively characterizes the probability that a pixel is located in an area prone to artifacts. Pixels closer to saturated highlight areas are more likely to be affected by HDR composition artifacts.

[0072] Gradient brightness nonlinearity refers to the degree to which the statistical relationship between local gradient magnitude and local average brightness deviates from the preset natural image statistical model.

[0073] It should be noted that the gradient brightness nonlinearity refers to the degree to which the statistical relationship between the local gradient magnitude and the local average brightness deviates from the preset statistical model of natural images. In natural images, the relationship between the local gradient magnitude and the local average brightness usually follows certain statistical laws. However, artifacts introduced by HDR synthesis and motion coupling can disrupt these statistical laws, producing abnormally high gradient-brightness combinations. Therefore, gradient brightness nonlinearity can serve as an effective indicator for identifying artifacts.

[0074] Color space distortion refers to the Euclidean distance of a boundary pixel relative to its neighboring regions in the Lab or YCrCb color space in terms of color chromaticity components.

[0075] It should be noted that the chromaticity space distortion refers to the degree of difference in chromaticity components between a boundary pixel and its neighboring region in the Lab or YCrCb color space. HDR composite artifacts not only manifest as abnormal brightness at image edges but are also often accompanied by color channel distortion, such as abnormal color shifts at highlight boundaries. By calculating the Euclidean distance in chromaticity components between the boundary pixel and its neighboring region in the Lab or YCrCb color space, this color distortion feature, which differs from the true physical edge, can be captured.

[0076] The physical characteristics of the defect include abnormal thermal gradients, i.e., abrupt changes in temperature gradient detected at the corresponding location in the thermal imaging image.

[0077] It should be noted that the aforementioned thermal gradient anomaly refers to abrupt changes in temperature gradient detected at the corresponding location in the thermal imaging image. Real weld defects (such as cracks, porosity, and lack of fusion) alter the internal heat conduction path of the material. For example, cracks block heat flow, creating a localized thermal resistance effect and resulting in significant temperature differences on either side of the crack; pores, due to the much lower thermal conductivity of the air inside compared to the metal, also create localized thermal bridging effects. These physical phenomena manifest as abrupt changes in temperature gradient at the corresponding location in the thermal imaging image. HDR imaging artifacts exist only in visual images and do not cause corresponding temperature anomalies in thermal imaging images; therefore, thermal gradient anomalies can serve as a reliable physical criterion for distinguishing real defects from synthetic artifacts.

[0078] Please see Figure 2 The initial classification module is used to generate an initial classification confidence map based on the composite features. The initial classification confidence map represents the initial confidence that a pixel belongs to a real defect.

[0079] It should be noted that the initial classification confidence map refers to the confidence map output by the initial classification model, which characterizes the probability that each pixel belongs to the true defect category. Each pixel value in this confidence map ranges between 0 and 1; the closer the value is to 1, the more likely the pixel is to belong to a true defect, and the closer the value is to 0, the more likely the pixel is to belong to a normal area or an artifact area. The initial classification confidence map provides a preliminary judgment of the defect area in the weld image. However, due to the presence of HDR imaging artifacts, this initial judgment may contain misclassifications and low-confidence areas. Therefore, it needs to be corrected in the subsequent label propagation module and the step-by-step optimization module through a label optimization process guided by a multi-level resolution structure prior.

[0080] In an optional embodiment, the initial classification module generates an initial classification confidence map based on the composite features. The initial classification confidence map represents the initial confidence level that a pixel belongs to a true defect, including:

[0081] Obtain a training sample set, which includes multiple sets of multi-level resolution image representations and their corresponding composite features, as well as pixel-level labeled category labels, including real defects, HDR imaging artifacts and normal regions;

[0082] It should be noted that the construction of the training sample set is the foundation for the initial classification model training. The training sample set comes from HDR visual images and thermal images collected during historical welding processes. Each pixel is assigned a category label through manual or semi-automatic annotation. The category labels are divided into three types: real defects, including actual welding defects such as cracks, porosity, undercut, and lack of fusion; HDR imaging artifacts, including image artifacts introduced by the coupling of imaging and motion, such as synthetic ghosting, edge overshoot, and pseudo-texture; and normal regions, i.e., healthy weld seams or background regions without real defects or artifacts. This three-classification annotation method enables the model to learn discriminative features to distinguish between real defects and artifacts during training.

[0083] Based on the training sample set, an initial classification model is constructed using a micro neural network. The initial classification model takes the composite feature as input and outputs the confidence level that a pixel belongs to the true defect category.

[0084] It should be noted that a miniature neural network is used to build the initial classification model because miniature neural networks have the advantages of small parameter count and fast inference speed, making them suitable for running at multiple levels of multi-resolution image representations. This allows them to meet real-time requirements while maintaining a certain level of classification accuracy. In subsequent steps, this initial classification model will output a confidence map consistent with the resolution of each level, thus providing an initial classification basis for label optimization at each level. The miniature neural network can adopt the MobileNetV2-U-Net structure, which combines the lightweight feature extraction capabilities of MobileNetV2 with the encoder-decoder architecture of U-Net, effectively fusing multi-scale features and outputting pixel-level prediction results. Model training uses the cross-entropy loss function, and the network parameters are optimized through the backpropagation algorithm until the model's classification accuracy on the validation set reaches the preset requirements.

[0085] The composite features corresponding to the images at each resolution level are input into the trained initial classification model, and the initial classification confidence map is output.

[0086] It's important to note that inputting the composite features corresponding to images at each resolution level into the trained initial classification model means that the initial classification model operates independently at each layer of the multi-resolution image representation. Because the image resolutions differ across levels, the information contained in the extracted composite features also varies. Features at lower resolution levels focus more on macroscopic structural information, while features at higher resolution levels focus more on microscopic details. Therefore, the initial classification confidence maps output from each level also differ in their confidence distribution. This multi-scale initial classification result provides rich, multi-level initial information for subsequent progressive label optimization in the progressive optimization module.

[0087] The label propagation module is used to iteratively propagate labels onto the initial classification confidence map at the lowest resolution level of the multi-resolution image representation to obtain a label matrix at the lowest resolution level.

[0088] It's important to note that the lowest resolution level was chosen for iterative label propagation because this level has the lowest resolution. After multiple Gaussian downsampling operations, local high-frequency details (including edge overshoot and pseudo-textures caused by HDR imaging artifacts) are significantly smoothed, while macroscopic structural regions such as weld seams and background areas become more prominent and stable. Processing at this level effectively avoids artifacts interfering with structural judgment, thus establishing a reliable global structural prior for subsequent progressive label optimization.

[0089] In an optional embodiment, the label propagation module performs iterative label propagation on the initial classification confidence map at the lowest resolution level of the multi-resolution image representation to obtain a lowest resolution level structure label matrix, including:

[0090] The initial classification confidence map is downsampled to the resolution of the lowest resolution level;

[0091] It should be noted that downsampling the initial classification confidence map to the resolution of the lowest resolution level is to ensure that the spatial size of the confidence map is consistent with the image representation of that level. The confidence value of each pixel in the downsampled confidence map reflects the overall probability that the corresponding region belongs to a real defect, rather than the fine-grained judgment of a single pixel.

[0092] Based on the confidence level after downsampling, the initial labels of the pixels are divided into background, weld area and undetermined anomaly area to form the lowest resolution level initial label matrix;

[0093] It should be noted that the initial pixel labels are divided into three categories—background, weld area, and undetermined anomaly area—based on the confidence level after downsampling. This is based on the following considerations: at the lowest resolution level, areas with high confidence can be directly identified as weld areas or background, while areas with intermediate confidence are marked as undetermined anomaly areas. Undetermined anomaly areas are regions with uncertain labels; they may contain real defects, artifacts, or areas with blurred boundaries. These need to be corrected based on the structural context information of their neighborhood during subsequent iterative label propagation. This three-part strategy can guide the uncertain regions while retaining the label information of the determined regions.

[0094] Iterative label propagation is performed on the initial label matrix of the lowest resolution level. For a pixel labeled as an undetermined anomaly region, if the number of pixels labeled as weld seam region in its neighborhood is greater than the number of pixels labeled as background, the label of the pixel in the undetermined anomaly region is updated to weld seam region; otherwise, it is updated to background. The iteration continues until convergence, and the label matrix of the lowest resolution level structure is obtained.

[0095] It should be noted that the iterative label propagation of the initial label matrix at the lowest resolution level is based on the assumption of spatial continuity. This means that weld regions are spatially continuous and not isolated, while artifacts or noise often manifest as spatially discontinuous, isolated anomalous regions. Specifically, for a pixel labeled as a potential anomalous region, if the number of pixels labeled as weld regions in its neighborhood is greater than the number of pixels labeled as background, it indicates that the pixel has a stronger spatial connection to the weld region, and its label should be updated to weld region. Conversely, if the number of background pixels in its neighborhood is dominant, it should be updated to background. Through multiple iterations, this propagation process gradually eliminates isolated potential anomalous regions, forming a spatially continuous and structurally sound weld region partition, ultimately converging to obtain the lowest resolution level structure label matrix. This matrix provides reliable initial guidance for passing structural prior information to higher resolution levels in subsequent progressive optimization modules.

[0096] The step-by-step optimization module is used to optimize the labels of higher resolution layers step by step, using the lowest resolution hierarchical structure label matrix as a priori. The label optimization combines the initial classification confidence map and composite features of the current layer to obtain the label matrix of each hierarchical structure.

[0097] It should be noted that using the lowest-resolution hierarchical structure label matrix as a priori, and then progressively optimizing the labels at higher-resolution levels, is the core mechanism by which this method achieves coarse-to-fine and robust correction. The lowest-resolution hierarchical structure label matrix obtained in the label propagation module establishes the macroscopic structural division of the weld region. This macroscopic structure is stable and not easily affected by artifacts. Passing this macroscopic structural information as a priori constraint to higher-resolution levels provides reliable structural guidance during the label optimization process at higher-resolution levels, ensuring that even with a large number of misclassifications in the initial classification confidence map, the optimization process can still converge towards the correct structural direction.

[0098] It should be noted that the specific method of step-by-step propagation is as follows: For the current resolution level, the structural label matrix obtained by the iteration convergence of the previous level is upsampled to match its spatial resolution with the current level, serving as the prior structural information for label optimization at the current level. The structural label matrix of the previous level represents the stable structural partition at a coarser scale. When performing label optimization at the current level, this prior information can constrain the label allocation at the current level to not deviate from the established macroscopic structure.

[0099] In an optional embodiment, in the stepwise optimization module, the label optimization is achieved by constructing an energy function and iteratively minimizing the energy function;

[0100] It should be noted that the label optimization is achieved by constructing an energy function and iteratively minimizing it. Its design principle originates from the conditional random field or Markov random field theory in image segmentation. The energy function assigns a total energy value to the label allocation scheme for all pixels in the entire image; the lower the energy, the more reasonable the label allocation scheme. The optimal pixel-level classification result is obtained by iteratively optimizing to find the label allocation scheme that minimizes the total energy.

[0101] In the energy function, the total cost of assigning a pixel to a certain label category is composed of a weighted sum of data terms, context terms, guidance terms, and feature terms;

[0102] It should be noted that in the energy function, the total cost of assigning a pixel to a certain label category is composed of a weighted sum of data, context, guidance, and feature terms. The weight coefficients of these four terms can be set experimentally or empirically according to different application scenarios. The data term reflects the consistency between the label assignment and the initial classification result, and its role is to incorporate information from the initial classification confidence map into the optimization process. The context term reflects the spatial smoothness of the label assignment, and its role is to promote consistency in the labels of adjacent pixels and avoid isolated classification noise. The guidance term reflects the consistency between the label assignment and the structural prior passed from the previous layer, and its role is to ensure that the label optimization at the current level does not deviate from the established macroscopic structure. The feature term reflects the physical consistency between the label assignment and composite features (artifact correlation features and defect physical features), and its role is to use multimodal physical information to distinguish between synthetic artifacts and real defects. The four constraints work together to determine the final label assignment result.

[0103] The data item constrains the label allocation based on the initial classification confidence map of the current level; the context item constrains the label allocation based on the consistency of neighboring pixel labels; the guiding item constrains the label allocation based on the prior structural information passed from the label matrix of the previous layer; and the feature item constrains the label allocation based on the composite feature.

[0104] The feature term is constructed such that when the pixel is assigned to a defect category, the value of the feature term increases with the increase of the artifact correlation feature intensity of the pixel and decreases with the increase of the physical feature intensity of the defect.

[0105] When a pixel is assigned to a non-defect category, the value of the feature term increases with the increase of the physical defect feature strength of the pixel and decreases with the increase of the artifact correlation feature strength.

[0106] It should be noted that the construction logic of the aforementioned feature terms is key to this method's distinction between HDR imaging artifacts and real physical defects. The design principle is as follows: real physical defects (such as cracks and pores) are usually accompanied by physical features such as abnormal temperature gradients in thermal imaging images, but not necessarily by typical image features of HDR imaging artifacts (such as proximity of highlights, nonlinear gradient brightness, etc.); while HDR imaging artifacts exhibit specific statistical anomalies in visual images, but lack corresponding physical evidence (i.e., no abnormal temperature gradients in thermal imaging images). Therefore, when a pixel is assigned to a defect category, if its artifact correlation feature strength is high while its defect physical feature strength is low, it indicates that the pixel is more likely an artifact than a real defect, and the value of the feature term should be increased to increase the energy cost of this assignment; conversely, if its defect physical feature strength is high, it indicates that the assignment conforms to physical laws, and the value of the feature term should be decreased to reduce the energy cost of this assignment. When a pixel is assigned to the non-defect category, if its defect physical feature intensity is high, it indicates that the pixel possesses the physical characteristics of a real defect and should not be assigned to the non-defect category; therefore, the value of the feature term should be increased. Conversely, if its artifact correlation feature intensity is high, it indicates that the pixel is likely an artifact, and assigning it to the non-defect category is reasonable; therefore, the value of the feature term should be decreased. Through this two-way constraint mechanism, the feature term can effectively suppress the misclassification of artifact regions as defects during the optimization process, while simultaneously preventing the omission of real defect regions.

[0107] In an optional embodiment, the data item is defined as follows: when the label represents a true defect category, the value of the data item decreases as the confidence that the pixel belongs to a true defect in the current level initial classification confidence map increases, and the decreasing trend is logarithmic; when the label represents a non-true defect category, the value of the data item decreases as the confidence that the pixel does not belong to a true defect increases, and the decreasing trend is logarithmic.

[0108] The context term is defined as follows: the value of the context term is equal to the number of neighboring pixels whose labels are inconsistent when the label assigned to the current pixel is compared with the labels of each pixel in its preset neighborhood.

[0109] The guiding term is defined as follows: when the label assigned to the current pixel is consistent with the structure category indicated by the upper-level structure label matrix at that pixel position, the value of the guiding term is zero; when the label assigned to the current pixel is inconsistent with the structure category indicated by the upper-level structure label matrix at that pixel position, the value of the guiding term is a preset positive number.

[0110] It should be noted that the value of the data item decreases logarithmically as the confidence level increases. This design principle originates from the concept of cross-entropy in information theory. When the confidence level is close to 1, the value of the data item approaches 0, indicating that the initial classification model has a high confidence that the pixel belongs to the defect category, and the cost imposed by the data item when assigning it to the defect category is small. When the confidence level is close to 0, the value of the data item increases sharply, indicating that the initial classification model believes that the pixel is unlikely to be a defect, and the cost imposed by the data item when assigning it to the defect category is large. This logarithmic relationship can reasonably reflect the constraint strength of the confidence level of the initial classification result on label assignment.

[0111] It should be noted that the value of the context term is equal to the number of neighboring pixels whose labels do not match the labels of the currently assigned pixel when compared with the labels of pixels in the preset neighborhood. This definition method is called the Potts model, a commonly used smooth constraint model in image segmentation. The neighborhood range is usually a four-neighborhood (top, bottom, left, and right) or an eight-neighborhood (including the diagonal direction). When the label of a neighboring pixel matches the label of the current pixel, no energy cost is added; the more inconsistent neighboring pixels, the higher the energy cost. This design encourages adjacent pixels to have the same label, thereby achieving a smooth transition of labels in space and effectively suppressing isolated classification noise.

[0112] It should be noted that the value of the guiding term is designed as follows: it takes a value of zero when the label assigned to the current pixel matches the structure category indicated by the label matrix of the previous layer at that position; otherwise, it takes a preset positive value. This binary penalty method is simple and effective: label assignments that conform to the structural prior do not increase the energy cost, while label assignments that violate the structural prior are subject to a fixed penalty. The value of the preset positive value can be adjusted according to the strength requirements of the prior constraints in actual applications. The larger the value, the stronger the prior constraints, and the more the optimization result depends on the structural information of the previous layer.

[0113] Please see Figure 3 The defect warning module is used to construct a defect mask from the pixels identified as defects in the highest resolution hierarchical label matrix, perform defect pattern recognition, and generate a warning signal.

[0114] It should be noted that the pixels identified as defects in the highest resolution hierarchical structure label matrix are used to form the defect mask because the highest resolution level, after being optimized level by level, not only retains the consistency of the macroscopic structure established by the label propagation module, but also finely corrects the pixel labels in each level by combining the initial classification confidence map and composite features of the current level. Therefore, the structure label matrix of this level has the highest positioning accuracy and the clearest boundary for the defect region, making it the best choice to form the final defect mask.

[0115] In an optional embodiment, the defect warning module performs defect pattern recognition, including:

[0116] Morphological post-processing is performed on the connected regions in the defect mask to obtain each connected region;

[0117] It should be noted that morphological post-processing of connected regions in the defect mask aims to eliminate isolated noise and minor breaks that may exist within the mask. Morphological post-processing typically includes opening and closing operations. Opening, through erosion followed by dilation, can remove small, isolated false-detection regions; closing, through dilation followed by erosion, can fill in small voids within the defect region and connect adjacent broken regions. Morphological post-processing yields connected regions with more complete boundaries and more regular shapes, providing more accurate region boundaries for subsequent feature extraction and pattern recognition.

[0118] Extract the morphological features of each connected region, as well as the statistics of the composite features within the corresponding region;

[0119] It should be noted that extracting morphological and composite feature statistics from each connected region is to obtain discriminative feature descriptions that can distinguish different defect types. Morphological features include geometric characteristics such as area, perimeter, aspect ratio, roundness, and rectangularity of the connected region. For example, cracks typically present as elongated lines with a large aspect ratio but a small area; pores typically appear as approximately circular or elliptical shapes with high roundness; undercut usually appears at the weld edge, presenting a strip-like shape along the weld direction; and non-fused areas are typically irregular in shape with tortuous boundaries. Composite feature statistics refer to the mean, variance, maximum, and minimum values ​​of artifact correlation features and defect physical features extracted from the multi-level feature modules within the connected region. Combining morphological and composite feature statistics allows for a comprehensive description of the defect region's characteristics from both geometric and physical dimensions, providing rich discriminative information for the classifier.

[0120] The extracted morphological features and statistics are input into a pre-trained classifier, which outputs the defect pattern category corresponding to each connected region; the defect pattern category includes cracks, pores, undercuts and lack of fusion.

[0121] It's important to note that a pre-trained classifier is used for defect pattern recognition in connected regions because different types of defects exhibit significant differences in morphology and physical characteristics. Supervised machine learning methods can effectively distinguish defect patterns such as cracks, pores, edge bites, and lack of fusion. The pre-trained classifier can be a decision tree classifier or a support vector machine classifier. The advantage of a decision tree classifier lies in its strong model interpretability, which can intuitively demonstrate the importance of different features in classification decisions. The advantage of a support vector machine classifier is its good generalization ability to small sample data, making it suitable for scenarios with a limited number of defect samples. The training process of the classifier is similar to the training of the initial classification model in the initial classification module. It requires pre-collecting sample data with defect pattern labels, extracting morphological features and composite feature statistics, and then training the classifier until the classification accuracy reaches the preset requirements.

[0122] In an optional embodiment, the defect warning module generates a warning signal by including:

[0123] Based on the identified defect patterns, defect sizes, and defect locations, and in conjunction with preset quality standards, an early warning signal is generated. The early warning signal is used to mark the defect location and type.

[0124] The warning signal is fed back to the production execution system or robot controller.

[0125] It should be noted that the warning signal is generated based on the identified defect pattern, size, and location, combined with preset quality standards. These preset quality standards refer to weld quality acceptance criteria established according to automotive manufacturing industry standards or internal control standards. They typically specify the maximum permissible size, allowed area, and prohibited defect types for different defect types. For example, welds in critical load-bearing areas of the vehicle body are not allowed to have cracks or lack of fusion defects; the diameter of porosity must not exceed a certain threshold; and the depth and length of undercut must not exceed specified values. When an identified defect exceeds the permissible range of the quality standards, a corresponding warning signal is generated.

[0126] It should be noted that the warning signal is used to mark the location and type of defects, facilitating operators to quickly locate and address problematic welds. Feeding the warning signal back to the production execution system allows defect information to be recorded in the production database for product quality traceability and process statistical analysis. Feeding the warning signal back to the robot controller can trigger online adjustments to the welding robot's process parameters, such as welding current, welding speed, or shielding gas flow rate, to eliminate similar defects in subsequent welding processes, achieving closed-loop control of welding quality. This real-time warning and feedback mechanism is a key element in the practical application of this method in real-world production environments, effectively reducing the outflow rate of defective products and minimizing rework and scrap costs.

[0127] In summary, this invention solves the false alarm problem caused by HDR imaging artifacts, significantly improves the accuracy and stability of weld defect identification, and ensures the reliability and engineering practicality of the online inspection system for automotive welding.

[0128] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0129] In the embodiments provided by this invention, it should be understood that the disclosed system or method can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for instance, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0130] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.

[0132] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0133] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal sensing fusion-based intelligent detection and early warning system for weld defects in automotive welding, characterized in that, include: A synchronous acquisition module is used to synchronously acquire multimodal data of the weld area, including HDR visual images and thermal imaging images; A multi-level feature module is used to construct multi-level resolution image representations based on the HDR visual image and the thermal imaging image respectively, and extract composite features, including artifact correlation features characterizing HDR imaging artifacts and defect physical features characterizing real physical defects. An initial classification module is used to generate an initial classification confidence map based on the composite features, wherein the initial classification confidence map represents the initial confidence that a pixel belongs to a real defect; The label propagation module is used to iteratively propagate labels onto the initial classification confidence map at the lowest resolution level of the multi-level resolution image representation to obtain a label matrix of the lowest resolution level structure. The step-by-step optimization module is used to optimize the labels of higher resolution layers step by step, using the lowest resolution hierarchical structure label matrix as a priori. The label optimization combines the initial classification confidence map and composite features of the current layer to obtain the label matrix of each hierarchical structure. The defect warning module is used to construct a defect mask from the pixels identified as defects in the highest resolution hierarchical label matrix, perform defect pattern recognition, and generate a warning signal.

2. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: In the synchronous acquisition module, the HDR visual image is acquired by an HDR visual camera installed on the end effector of the welding robot, and the thermal imaging image is acquired by an infrared thermal imager installed on the end effector of the welding robot; the HDR visual camera is configured in dual-gain mode or multi-exposure synthesis mode; The HDR visual camera and the infrared thermal imager are synchronized and triggered by a high-precision clock.

3. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: In the multi-level feature module, multi-level resolution image representations are constructed, and composite features are extracted, including: Gaussian downsampling is performed on the HDR visual image and the thermal imaging image respectively to generate L-layer images with progressively decreasing resolution, where L≥3, and the lowest resolution layer is the top layer and the highest resolution layer is the bottom layer. Extract composite features from each layer of images at each resolution level.

4. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 3, characterized in that: The artifact correlation features include at least one of the following: Spectrum proximity, which is the distance between a pixel and the nearest saturated specular region; Gradient brightness nonlinearity refers to the degree to which the statistical relationship between local gradient magnitude and local average brightness deviates from the preset natural image statistical model. Color space distortion refers to the Euclidean distance of a boundary pixel relative to its neighboring regions in the Lab or YCrCb color space in terms of color chromaticity components. The physical characteristics of the defect include abnormal thermal gradients, i.e., abrupt changes in temperature gradient detected at the corresponding location in the thermal imaging image.

5. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: In the initial classification module, an initial classification confidence map is generated based on the composite features. This initial classification confidence map represents the initial confidence level that a pixel belongs to a true defect, including: Obtain a training sample set, which includes multiple sets of multi-level resolution image representations and their corresponding composite features, as well as pixel-level labeled category labels, including real defects, HDR imaging artifacts and normal regions; Based on the training sample set, an initial classification model is constructed using a micro neural network. The initial classification model takes the composite feature as input and outputs the confidence level that a pixel belongs to the true defect category. The composite features corresponding to the images at each resolution level are input into the trained initial classification model, and the initial classification confidence map is output.

6. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: In the label propagation module, at the lowest resolution level of the multi-resolution image representation, iterative label propagation is performed on the initial classification confidence map to obtain the lowest resolution level structure label matrix, including: The initial classification confidence map is downsampled to the resolution of the lowest resolution level; Based on the confidence level after downsampling, the initial labels of the pixels are divided into background, weld area and undetermined anomaly area to form the lowest resolution level initial label matrix; Iterative label propagation is performed on the initial label matrix of the lowest resolution level. For a pixel labeled as an undetermined anomaly region, if the number of pixels labeled as weld seam region in its neighborhood is greater than the number of pixels labeled as background, the label of the pixel in the undetermined anomaly region is updated to weld seam region; otherwise, it is updated to background. The iteration continues until convergence, and the label matrix of the lowest resolution level structure is obtained.

7. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: In the stepwise optimization module, the label optimization is achieved by constructing an energy function and iteratively minimizing the energy function; In the energy function, the total cost of assigning a pixel to a certain label category is composed of a weighted sum of data terms, context terms, guidance terms, and feature terms; The data item constrains the label allocation based on the initial classification confidence map of the current level; the context item constrains the label allocation based on the consistency of neighboring pixel labels; the guiding item constrains the label allocation based on the prior structural information passed from the label matrix of the previous layer; and the feature item constrains the label allocation based on the composite feature. The feature term is constructed such that when the pixel is assigned to a defect category, the value of the feature term increases with the increase of the artifact correlation feature intensity of the pixel and decreases with the increase of the physical feature intensity of the defect. When a pixel is assigned to a non-defect category, the value of the feature term increases with the increase of the physical defect feature strength of the pixel and decreases with the increase of the artifact correlation feature strength.

8. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 7, characterized in that: The data item is defined as follows: when the label represents a true defect category, the value of the data item decreases as the confidence that the pixel belongs to a true defect in the initial classification confidence map of the current level increases, and the decreasing trend is logarithmic; when the label represents a non-true defect category, the value of the data item decreases as the confidence that the pixel does not belong to a true defect increases, and the decreasing trend is logarithmic. The context term is defined as follows: the value of the context term is equal to the number of neighboring pixels whose labels are inconsistent when the label assigned to the current pixel is compared with the labels of each pixel in its preset neighborhood. The guiding term is defined as follows: when the label assigned to the current pixel is consistent with the structure category indicated by the label matrix of the previous layer at that pixel position, the value of the guiding term is zero; When the label assigned to the current pixel is inconsistent with the structure category indicated by the label matrix of the upper layer structure at that pixel position, the value of the guiding item is a preset positive number.

9. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: The defect early warning module performs defect pattern recognition, including: Morphological post-processing is performed on the connected regions in the defect mask to obtain each connected region; Extract the morphological features of each connected region, as well as the statistics of the composite features within the corresponding region; The extracted morphological features and statistics are input into a pre-trained classifier, which outputs the defect pattern category corresponding to each connected region; the defect pattern category includes cracks, pores, undercuts and lack of fusion.

10. The intelligent detection and early warning system for automotive welding defects based on multimodal sensor fusion as described in claim 1, characterized in that: In the defect early warning module, the generated early warning signals include: Based on the identified defect patterns, defect sizes, and defect locations, and in conjunction with preset quality standards, an early warning signal is generated. The early warning signal is used to mark the defect location and type. The warning signal is fed back to the production execution system or robot controller.

Images