Automobile parts processing welding spot analysis system based on visual detection

Abstract

This invention belongs to the field of visual inspection technology, specifically relating to a visual inspection-based weld joint analysis system for automotive parts processing. During the welding cooling stage, the system controls a multispectral light source to flash alternately based on a temperature decay curve, acquiring a multi-frame image sequence. The processing server inputs this multi-frame image sequence into a feature fusion network. The spectral attention branch generates a weight matrix based on the reflectivity differences of pixels at different wavelengths, while the spatial gradient branch extracts the weld pool edge contour features. The processing server then multiplies the weight matrix element-wise with these weld pool edge contour features to generate a fused feature map. The defect classifier outputs the weld joint defect type based on the pixel distribution state in the fused feature map, where highly reflective areas are suppressed and edge contours are enhanced. This system suppresses overexposure halo interference, avoids feature extraction distortion, and maintains the accuracy of weld joint defect classification results.

Description

Technical Field

[0001] This invention belongs to the field of visual inspection technology, specifically relating to a visual inspection-based welding point analysis system for automotive parts processing. Background Technology

[0002] Automotive parts often use highly reflective metal materials during manufacturing. Existing visual inspection solutions typically use a single, fixed-wavelength light source for illumination, along with an industrial camera to capture two-dimensional images of the weld joint surface. These images are then input into a conventional convolutional neural network to extract texture features and classify them. In the acquisition stage, the fixed light source struggles to adapt to the complex curvature variations of the highly reflective metal surface, leading to specular reflections in localized areas of the weld joint. In the feature extraction stage, conventional convolutional neural networks use standard convolutional kernels to uniformly process the image, failing to distinguish between overexposed reflective areas and the actual edges of the weld joint texture. This results in the extracted features being contaminated with a large amount of invalid high-frequency signals caused by reflections.

[0003] When using the above-mentioned existing technical solutions to process images of solder joints made of highly reflective metal materials, local overexposure caused by a fixed light source will form a bright halo in the image. Conventional convolutional kernels cannot eliminate this interference signal caused by physical illumination when extracting features, causing the overexposure halo to cover the true edge contour and internal texture of the solder joint molten pool, resulting in distortion in the feature extraction process and consequently causing errors in the subsequent defect classification results. Summary of the Invention

[0004] The purpose of this invention is to provide a vision-based inspection system for analyzing weld points in automotive parts processing, which can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A vision-based automotive parts welding point analysis system includes a multispectral light source array, an image acquisition camera, and a processing server connected to the multispectral light source array and the image acquisition camera. During the welding cooling stage, the processing server controls the alternating flashing frequency of different wavelength light sources in the multispectral light source array according to the temperature decay curve. The image acquisition camera acquires a corresponding multi-frame image sequence, and the processing server inputs the multi-frame image sequence into a feature fusion network, which includes a spectral attention branch and a spatial gradient branch. The spectral attention branch generates a weight matrix based on the reflectance difference of pixels at different wavelengths in the multi-frame image sequence, and the spatial gradient branch extracts the edge contour features of the weld pool. The processing server performs element-wise multiplication and fusion of the weight matrix and the spatial gradient features to generate a fused feature map, and inputs the fused feature map into the defect classifier; The defect classifier outputs the weld point defect type based on the pixel distribution state in the fused feature map where high reflective areas are suppressed and edge contours are enhanced. This is the visual inspection-based automotive parts processing weld point analysis system.

[0006] Preferably, the processing server is connected to an infrared temperature sensor, which collects surface temperature data of the weld joint during the welding cooling stage, and the processing server performs time series fitting on the surface temperature data of the weld joint to generate the temperature decay curve. The multispectral light source array includes infrared light sources, blue light sources, and green light sources distributed around the solder joint area. Each of the infrared light source, the blue light source, and the green light source is connected in series with an independently controlled strobe controller. The processing server sends a pulse width modulation signal to the strobe controller according to the slope change stage of the temperature decay curve. The strobe controller adjusts the lighting duration and interval of the infrared light source, the blue light source, and the green light source according to the pulse width modulation signal.

[0007] Preferably, the hardware trigger input terminal of the image acquisition camera is connected to the synchronization signal output terminal of the strobe controller, and the strobe controller outputs a high-level trigger signal to the image acquisition camera while sending lighting commands to the infrared light source, the blue light source and the green light source; After receiving the high-level trigger signal, the image acquisition camera opens a single exposure window, and the center time of the single exposure window coincides with the peak time of the light intensity of the infrared light source, the blue light source, or the green light source. The image acquisition camera reads the image corresponding to the current wavelength within the single exposure window and caches the image corresponding to the current wavelength in the memory queue of the processing server to form the multi-frame image sequence.

[0008] Preferably, the spectral attention branch includes a channel pooling layer, a fully connected layer, and a Sigmoid activation layer connected in sequence. The channel pooling layer performs global average pooling on the pixel values ​​of the same position in the multi-frame image sequence under different wavelength channels to generate channel feature vectors. The fully connected layer performs nonlinear spatial mapping on the channel feature vectors to generate a channel attention map, and the sigmoid activation layer normalizes the mapping values ​​of each channel in the channel attention map. The normalized mapping values ​​are arranged as weight coefficients to generate the weight matrix, and the number of rows and columns of the weight matrix corresponds to the spatial height dimension and spatial width dimension of the multi-frame image sequence, respectively.

[0009] Preferably, the spatial gradient branch includes three dilated convolutional layers with different dilation rates arranged in parallel and a feature stitching layer. The three dilated convolutional layers with different dilation rates perform convolution operations on the preset reference wavelength images in the multi-frame image sequence to extract texture gradient response maps within different receptive field ranges. The feature splicing layer splices the texture gradient response maps output by the three dilated convolutional layers with different dilation rates end-to-end in the channel dimension to generate a multi-channel gradient tensor. The spatial gradient branch also includes a dimension reduction convolutional layer, which performs channel number compression on the multi-channel gradient tensor to output a single-channel weld pool edge contour feature.

[0010] Preferably, after the processing server performs element-wise multiplication and fusion of the weight matrix and the spatial gradient features, it inputs the multiplication result into a batch normalization layer and a ReLU activation function layer for distribution correction and nonlinear mapping to generate the fused feature map. The defect classifier comprises a sequentially connected global pooling layer, two fully connected hidden layers, and a Softmax output layer. The global pooling layer compresses the spatial dimension of the fused feature map to generate a one-dimensional feature vector. The two fully connected hidden layers perform feature cross operations on the one-dimensional feature vector. The Softmax output layer performs probability distribution mapping on the result of the feature cross operations and outputs a probability vector containing multiple solder joint defect types.

[0011] Preferably, the optical axis of the infrared temperature sensor is arranged at a preset angle with the optical axis of the image acquisition camera, and the center of the field of view of the infrared temperature sensor coincides with the illumination center of the multispectral light source array on the surface of the automotive parts. After acquiring the surface temperature data of the solder joint, the processing server removes abnormal temperature points that exceed the preset temperature threshold range from the surface temperature data of the solder joint, and inputs the remaining temperature points after removing the abnormal temperature points into the sliding window filter. The sliding window filter calculates the mean of the remaining temperature points in chronological order to generate a smoothed continuous temperature sequence, and the processing server performs polynomial fitting on the continuous temperature sequence to generate the temperature decay curve.

[0012] Preferably, the fully connected layer includes a first mapping sublayer and a second mapping sublayer, wherein the number of neurons in the first mapping sublayer is less than the dimension of the channel feature vector; The first mapping sub-layer performs dimensionality reduction and compression on the channel feature vector to generate an intermediate feature vector. The number of neurons in the second mapping sub-layer is the same as the dimension of the channel feature vector. The second mapping sub-layer performs dimensionality recovery on the intermediate feature vector to generate a dimension-recovered feature vector. The fully connected layer also introduces a cross-attention computation mechanism, which uses the dimension recovery feature vector as the query vector and the channel feature vector as the key vector and value vector to perform self-attention operation to generate the channel attention map.

[0013] Preferably, the dimension reduction convolutional layer includes a depthwise separable convolutional kernel with a stride of a preset value. The depthwise separable convolutional kernel performs channel-by-channel two-dimensional convolution operations on the multi-channel gradient tensor in the spatial dimension to generate a spatial dimension reduction gradient tensor. The dimension reduction convolutional layer further includes a pointwise convolutional kernel, which performs a 1×1 convolution operation on the spatial dimension reduction gradient tensor in the channel dimension. The number of channels of the spatial dimension reduction gradient tensor is fused and reduced to a single channel to generate an initial fused gradient map. The spatial gradient branch is also configured with a max pooling layer. The max pooling layer filters out pixels whose gradient response values ​​are lower than a preset response threshold in the initial fusion gradient map, retains gradient extrema, and generates the weld pool edge contour features.

[0014] Preferably, a Dropout layer is embedded between the first fully connected hidden layer and the second fully connected hidden layer in the two fully connected hidden layers. When the first fully connected hidden layer outputs a feature vector, the Dropout layer resets the weights of some neurons in the first fully connected hidden layer to zero according to a preset deactivation ratio. The second fully connected hidden layer performs weight decay calculation on the feature vector after it is set to zero. The Softmax output layer receives the result of the weight decay calculation and maps the result of the weight decay calculation to a numerical range between 0 and 1 according to the exponential function. The defect classifier outputs the solder joint defect type index corresponding to the maximum value in the numerical range as the solder joint defect type.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This system dynamically controls the alternating flashing frequency of light sources of different wavelengths based on the temperature decay curve during the solder joint cooling stage, and ensures that the center of the single exposure window of the image acquisition camera coincides with the peak moment of the light intensity of the light source, thereby acquiring an image sequence containing multi-band information. A weight matrix is ​​generated based on the reflectivity differences of pixels at different wavelengths through a spectral attention branch, and combined with the solder joint molten pool edge contour features extracted by the spatial gradient branch, the two are multiplied and fused element-wise. This processing mechanism suppresses the pixel response of highly reflective areas in the fused feature map by the weight matrix, while preserving and enhancing the solder joint edge contour features. This eliminates the interference of local overexposure halos caused by a fixed light source on the feature extraction process, avoiding feature extraction distortion.

[0016] 2. The spatial gradient branch uses multiple dilated convolutional layers with different dilation rates to extract texture gradient response maps within different receptive fields. After feature concatenation and dimensionality reduction convolution, combined with a max pooling layer to filter low-response pixels, the interference of background noise is eliminated, and edge features reflecting the true physical morphology of the weld pool are obtained. In the classification stage, the fused feature maps are processed by batch normalization and linear rectified function, and then feature cross-operation is performed through two fully connected hidden layers. A random deactivation layer is introduced between the two fully connected hidden layers to reset the weights of some neurons to zero for weight decay calculation. Finally, the classification result is output through a normalized exponential function layer, which reduces the co-adaptation relationship between nodes within the network, suppresses overfitting, and maintains the stability of the weld joint defect classification results. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall workflow of the vision-based inspection-based automotive parts welding point analysis system of the present invention. Figure 2 This is a flowchart of the temperature-adaptive light source flicker and image synchronization acquisition process of the present invention; Figure 3 This is a flowchart of the spectral attention branching process of the present invention; Figure 4 This is a flowchart of the spatial gradient branching process of the present invention; Figure 5 This is a flowchart of the feature fusion and defect classification process of the present invention; Figure 6 This is a flowchart of the multi-frame image sequence acquisition and memory queue management process of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Please refer to Figure 1This embodiment provides a vision-based inspection system for analyzing weld joints in automotive parts processing. The system includes a multispectral light source array, an image acquisition camera, and a processing server. The processing server establishes bidirectional data communication connections with both the multispectral light source array and the image acquisition camera. After the welding process is completed, the automotive parts enter the welding cooling stage. During this stage, the processing server generates a drive control signal for the multispectral light source array based on a pre-acquired temperature decay curve. This drive control signal controls the alternating flashing frequency of different wavelengths of light sources within the multispectral light source array. The multispectral light source array is arranged in a ring around the weld joint area, with different wavelengths of light sources spaced at preset angles. The illumination centers of all light sources coincide with the center of the weld joint area, avoiding blind spots. Each light source in the multispectral light source array is equipped with a collimating lens. This collimating lens adjusts the emitted light into parallel light, preventing blurring of the illumination area edges due to light divergence and ensuring uniform illumination of the weld joint area.

[0020] During the alternating flashing of light sources of different wavelengths, the image acquisition camera synchronously acquires images of the solder joint area illuminated by the corresponding wavelength light source. The optical axis of the image acquisition camera is arranged at a preset angle with the normal direction of the solder joint area, and the imaging plane of the camera completely covers the solder joint area and the surrounding base material area, so that the acquired image can completely present the overall shape of the solder joint molten pool. The processing server preprocesses the raw images output by the image acquisition camera. The preprocessing operations include lens distortion correction and white balance correction. The parameters for lens distortion correction are derived from the pre-completed camera calibration results. The pixel coordinates of the corrected image form a preset mapping relationship with the real-world coordinates to ensure that the spatial position of the subsequent spatial feature extraction and weight matrix is ​​completely matched. White balance correction is used to eliminate the influence of color temperature differences of different wavelength light sources on the image pixel values, ensuring that the pixel values ​​of images corresponding to different wavelengths are comparable. Multiple frames of images corresponding to different wavelengths are arranged in the acquisition time sequence to form a multi-frame image sequence. The processing server transmits the acquired multi-frame image sequence to a pre-constructed and trained feature fusion network.

[0021] The feature fusion network comprises a spectral attention branch and a spatial gradient branch. Multiple image frames are input into these branches for parallel processing. The spectral attention branch receives the image frames, extracts pixels at the same spatial location, calculates the reflectance difference of each pixel at different wavelengths, and generates a weight matrix corresponding to the spatial dimension based on the distribution of reflectance differences. High-reflectivity areas on the metal surface exhibit significant differences in reflectance for different wavelengths, while the reflectance difference between the weld pool edge and the base material region remains stable. Based on this physical characteristic, the spectral attention branch assigns differentiated weight coefficients to pixels with different reflectance differences. The spatial gradient branch receives the image frames, selects the image frame corresponding to a preset reference wavelength, extracts the edge contour features of the weld pool region in that frame, and generates a weld pool edge contour feature map with the same spatial dimension as the multi-frame image sequence. The preset reference wavelength is a wavelength that responds stably to the texture features of the metal surface, ensuring that the extracted edge contour features accurately reflect the true physical morphology of the weld pool.

[0022] The processing server performs element-wise multiplication of the weight matrix output from the spectral attention branch with the weld pool edge contour feature map output from the spatial gradient branch to complete feature fusion processing and generate a fused feature map. During the element-wise multiplication, the weight coefficients corresponding to high-reflectivity regions in the weight matrix are in the low value range, suppressing the edge contour feature response of the corresponding high-reflectivity regions; conversely, the weight coefficients corresponding to the true edge regions of the weld pool are in the high value range, enhancing the contour feature response of the corresponding edge regions. The processing server inputs the generated fused feature map into a defect classifier. The defect classifier analyzes the pixel distribution in the fused feature map, identifies pixel distribution features where high-reflectivity regions are suppressed and edge contours are enhanced, and outputs the corresponding weld defect type based on these pixel distribution features.

[0023] Table 1. Correspondence between wavelength and acquisition timing of multi-frame image sequences. This table is used to standardize the acquisition timing and wavelength correspondence of multi-frame image sequences. The time interval of the acquisition timing nodes in the table is consistent with the flicker interval of the corresponding light source, ensuring that each frame of the image corresponds to the stable lighting stage of a single wavelength light source, avoiding crosstalk between light signals from different wavelength light sources, and ensuring the spatial consistency and numerical accuracy of subsequent reflectivity difference calculations.

[0024] This embodiment fully implements the complete process of multispectral acquisition of solder joint images, dual-branch feature extraction and fusion, and defect classification. By using the weight matrix generated by the spectral attention branch to perform weighted fusion of spatial gradient features, it achieves the suppression of highly reflective areas and the enhancement of edge contour features, eliminates the influence of illumination interference on the feature extraction process, and ensures the accuracy of solder joint defect identification.

[0025] In a preferred embodiment, reference Figure 2 The processing server is connected to an infrared temperature sensor, which is used to collect surface temperature data of the weld joints during the welding cooling stage. The optical axis of the infrared temperature sensor is arranged at a preset angle to the optical axis of the image acquisition camera. The center of the infrared temperature sensor's field of view coincides with the illumination center of the multispectral light source array on the surface of the automotive part, ensuring that the temperature data collected by the infrared temperature sensor corresponds to the weld joint area illuminated by the multispectral light source array, avoiding spatial misalignment between the temperature data and the image acquisition area. The infrared temperature sensor continuously collects temperature data of the weld joint surface at a preset sampling frequency, which matches the maximum flicker frequency of the multispectral light source array, ensuring that corresponding temperature data exists within each flicker cycle of the light source.

[0026] After acquiring the weld joint surface temperature data from the infrared thermometer, the processing server preprocesses the data, removing abnormal temperature points that exceed a preset temperature threshold range. The lower limit of the preset threshold range is the temperature value corresponding to room temperature, and the upper limit is the temperature value corresponding to the melting point of the welding material. Temperature data exceeding this range are abnormal values ​​caused by sampling noise or environmental interference. After removing abnormal temperature points, the processing server inputs the remaining valid temperature points into a sliding window filter. The window size of the sliding window filter is a configurable parameter, and the sliding step size of the window matches the sampling frequency of the infrared thermometer to ensure that each sliding window contains a sufficient number of valid temperature points. The sliding window filter calculates the mean of the valid temperature points within the window in chronological order, generating a smoothed continuous temperature sequence to eliminate random fluctuations during the temperature sampling process. The processing server then performs polynomial fitting on the smoothed continuous temperature sequence, using a third-order polynomial to ensure fitting accuracy while avoiding overfitting, generating a temperature decay curve for the welding cooling stage. This temperature decay curve characterizes the temperature change over time during the weld joint cooling process.

[0027] The multispectral light source array includes infrared, blue, and green light sources distributed around the solder joint area. Each of these light sources is connected in series with an independently controlled strobe controller. Each strobe controller is connected to the control signal output terminal of the processing server to receive control signals sent by the server. The strobe controller has a built-in light intensity feedback loop that collects the output light intensity data of the corresponding light source in real time. When the output light intensity deviates from a preset light intensity threshold, the strobe controller automatically adjusts the drive current to maintain the stability of the light source's output light intensity and prevent light intensity fluctuations from affecting image acquisition quality. The processing server performs first-order derivative calculations on the temperature decay curve to obtain the slope change data. Based on the slope change data, the welding cooling stage is divided into multiple slope change stages. A corresponding pulse width modulation signal is generated for each slope change stage and sent to the strobe controller of the corresponding light source. The strobe controller adjusts the lighting duration and interval of the corresponding light source based on the received pulse width modulation signal, thereby controlling the alternating flashing frequency of the infrared light source, blue light source and green light source, so that the flashing mode of the light source is adaptively matched with the cooling state of the solder joint.

[0028] refer to Figure 6 The hardware trigger input of the image acquisition camera is connected to the synchronization signal output of the strobe controller. While sending a lighting command to the corresponding light source, the strobe controller simultaneously outputs a high-level trigger signal to the image acquisition camera. Upon receiving the high-level trigger signal, the image acquisition camera opens a single-exposure window. The duration of the single-exposure window matches the lighting duration of the corresponding light source, and the center moment of the single-exposure window coincides with the peak moment of the light intensity of the corresponding light source. This ensures that the light intensity of the light source remains stable during image acquisition, avoiding uneven brightness in the image due to light intensity fluctuations. Within the single-exposure window, the image acquisition camera completes the photoelectric conversion and data reading of the image corresponding to the current wavelength. The read image corresponding to the current wavelength is then transmitted to the memory queue of the processing server. Multiple frames corresponding to different wavelengths are arranged in the memory queue according to the acquisition sequence, forming a multi-frame image sequence. The memory queue employs a first-in, first-out (FIFO) read / write mechanism to ensure that the image data reading order is completely consistent with the acquisition sequence, avoiding timing misalignment of image frames.

[0029] Table 2. Correspondence between the slope stage of the temperature decay curve and the flicker control parameters of the light source. This table is used to clarify the light source flicker control strategy corresponding to different slope stages of the temperature decay curve. , , The slope threshold is preset, and , The preset duty cycle parameter, The preset lighting duration parameter, The preset interval period parameter ensures that the alternating flashing frequency of the light source matches the cooling state of the solder joint, guaranteeing that the images acquired at different cooling stages can clearly show the surface morphology of the solder joint.

[0030] This embodiment uses an infrared temperature sensor to collect temperature data during the solder joint cooling process, and fits it to generate a temperature decay curve. Based on the slope change of the temperature decay curve, the strobe parameters of different wavelength light sources are dynamically adjusted to achieve adaptive matching between the light source illumination mode and the solder joint cooling state. By synchronously triggering the strobe controller and the image acquisition camera, the timing of the image exposure window coincides with the peak light intensity of the light source, improving the acquisition quality of multi-frame image sequences and providing stable input data for subsequent feature extraction processing.

[0031] In a preferred embodiment, reference Figure 3 The spectral attention branch consists of sequentially connected channel pooling layers, fully connected layers, and sigmoid activation layers. After a multi-frame image sequence is input into the spectral attention branch, it undergoes dimension normalization, converting the multi-frame image sequence into a multi-channel feature map according to the wavelength dimension. The number of channels in the multi-channel feature map is consistent with the number of different wavelengths in the multi-frame image sequence, and the spatial height and spatial width dimensions of the multi-channel feature map are consistent with the size of a single-frame image. During the dimension normalization process, the processing server performs pixel value normalization on each frame image, mapping the pixel values ​​of all images to a uniform numerical range, eliminating the influence of light intensity differences from different wavelength light sources on pixel values, and ensuring the consistency of numerical distribution of different channel feature maps.

[0032] The channel pooling layer performs global average pooling on the multi-channel feature maps. Specifically, it performs global average pooling on the two-dimensional feature map corresponding to each channel in the multi-channel feature map, along with the spatial height and width dimensions, generating a single feature value for each channel. All feature values ​​are arranged in channel order to generate a channel feature vector. The channel pooling layer also includes a parallel global max pooling branch. This branch extracts the maximum pixel value from each channel's two-dimensional feature map and concatenates it with the feature value generated by global average pooling along the channel dimension to generate the final channel feature vector, further enhancing the expressive power of the channel features. The mathematical expression for global average pooling is: in, For the first The channel feature values ​​corresponding to each channel The spatial height dimension of the multi-channel feature map. The spatial width dimension of the multi-channel feature map. For the first In the feature map of each channel, the coordinates are... The pixel value corresponding to the pixel point. The range of values ​​is , This represents the total number of channels in the multi-channel feature map.

[0033] The fully connected layer consists of a first mapping sublayer and a second mapping sublayer. The number of neurons in the first mapping sublayer is less than the dimension of the channel feature vector. The first mapping sublayer performs dimensionality reduction and compression on the channel feature vector to generate an intermediate feature vector. The mathematical expression for dimensionality reduction and compression is: in, The intermediate feature vectors generated for dimensionality reduction and compression. This is the weight matrix of the first mapping sublayer. For channel feature vectors, The bias vector and weight matrix of the first mapping sublayer. Column number and channel feature vector The dimensions are consistent, and the weight matrix is ​​consistent. The number of rows and the intermediate feature vector The dimensions are consistent.

[0034] The second mapping sublayer has the same number of neurons as the channel feature vector. This second mapping sublayer performs dimensionality upscaling on the intermediate feature vectors to generate dimensionality-restored feature vectors. The mathematical expression for dimensionality upscaling is: in, The dimension recovery feature vector generated for dimension upscaling recovery. This is the weight matrix of the second mapping sublayer. The bias vector and weight matrix of the second mapping sublayer The number of columns and the intermediate feature vector The dimensions are consistent, and the weight matrix is ​​consistent. Row number and channel feature vector The dimensions are consistent.

[0035] The fully connected layer introduces a cross-attention mechanism, using the dimension-recovery feature vector as the query vector and the channel feature vectors as the key and value vectors, respectively, to perform self-attention operations and generate a channel attention map. The cross-attention mechanism employs a multi-head attention structure, mapping the query vector, key vector, and value vector to multiple different subspaces. Self-attention operations are performed in each subspace, and the results from multiple subspaces are concatenated to generate the final channel attention map, strengthening the feature association capabilities between different channels. The mathematical expression for the self-attention operation is: in, The channel attention map generated for self-attention operations. is the dimension of the channel feature vector. For query vector, Let be the transpose of the key vectors, and Softmax be the normalization exponential function used to normalize the attention weight matrix.

[0036] The Sigmoid activation layer normalizes the mapping values ​​of each channel in the channel attention map, using these normalized values ​​as weight coefficients for the corresponding channels. All channel weight coefficients are arranged according to spatial dimensions to generate a weight matrix. The number of rows and columns in the weight matrix correspond to the spatial height and width dimensions of the multi-frame image sequence, respectively. The weight value at each coordinate position in the weight matrix corresponds to the reflectance difference of pixels at the same coordinate position in the multi-frame image sequence at different wavelengths. Pixels with larger reflectance differences have lower weight values, and pixels with smaller reflectance differences have higher weight values. The mathematical expression for Sigmoid activation is: in, For the first The weight coefficients corresponding to each channel For the channel attention map, the first The mapping value corresponding to each channel It is a natural constant.

[0037] Table 3. Correspondence between pixel reflectance differences and weighting coefficients for different wavelength channels. This table clarifies the weighting coefficient allocation rules corresponding to different reflectance differences. , The preset reflectance difference threshold is configured based on the optical properties of the welding material to ensure that the spectral attention branch can achieve differentiated weight allocation based on the reflectance characteristics of the pixels, thereby suppressing high reflectance areas and enhancing effective feature areas.

[0038] This embodiment refines the network structure and computational logic of the spectral attention branch. It extracts global features of different wavelength channels through channel pooling layers, performs dimensionality reduction and dimensionality increase mapping through fully connected layers, and strengthens the feature correlation between different channels by combining cross-attention computing mechanisms. Finally, it generates a weight matrix corresponding to the spatial dimension through a sigmoid activation layer, realizing adaptive weight allocation based on pixel reflectivity differences, and providing an accurate weighting basis for subsequent feature fusion processing.

[0039] In a preferred embodiment, reference Figure 4The spatial gradient branch comprises three dilated convolutional layers with different dilation rates arranged in parallel, and a feature concatenation layer. The input to each of the three dilated convolutional layers with different dilation rates is an image frame corresponding to a preset reference wavelength from a multi-frame image sequence. The preset reference wavelength is the light source corresponding to the wavelength that provides the most stable response to the texture features of the metal surface in the multispectral light source array. The three dilated convolutional layers with different dilation rates perform convolution operations on the input reference wavelength images, extracting texture gradient response maps within different receptive fields. The dilated convolution operation expands the receptive field of the convolution operation by inserting holes between the convolution kernel elements, without increasing the number of convolution kernel parameters, thus achieving multi-scale texture feature extraction. The mathematical expression for the dilated convolution operation is: in, For the expansion rate In the texture gradient response map output by the dilated convolutional layer, the coordinates are... The gradient response value corresponding to the pixel. These are the kernel weights of the dilated convolutional layer. , These are the height and width of the convolution kernel, respectively. The coordinates in the input reference wavelength image are pixel values, The dilation rate of the dilated convolutional layer is given, and the dilation rates of the three dilated convolutional layers are given. The values ​​are 1, 2, and 4 respectively.

[0040] The feature concatenation layer concatenates the texture gradient response maps output by three dilated convolutional layers with different dilation rates, end-to-end along the channel dimension, to generate a multi-channel gradient tensor. The number of channels in the multi-channel gradient tensor is the sum of the number of channels in the feature maps output by the three dilated convolutional layers. During the concatenation process, the spatial height and spatial width dimensions of all texture gradient response maps are kept consistent to ensure that the spatial dimension of the concatenated multi-channel gradient tensor matches the spatial dimension of the input image. The spatial gradient branch also includes a dimensionality reduction convolutional layer, which compresses the number of channels in the multi-channel gradient tensor and outputs a single-channel solder pool edge contour feature.

[0041] The dimensionality reduction convolutional layer includes a depthwise separable convolutional kernel with a preset stride. The stride of the depthwise separable convolutional kernel is 1, and it uses constant-width padding to ensure that the spatial dimensions of the feature map after convolution are consistent with the spatial dimensions of the input feature map, avoiding the positional shift of edge features caused by spatial dimensionality compression. The depthwise separable convolutional kernel performs channel-by-channel two-dimensional convolution operations on the multi-channel gradient tensor in the spatial dimension, generating a spatially reduced gradient tensor. The mathematical expression for the depthwise convolution operation is: in, The first one after depthwise convolution operation The output feature values ​​of each channel For the first The depthwise convolution kernel weights corresponding to each channel. For the multichannel gradient tensor, the first The coordinates of each channel in the feature map are: The pixel value.

[0042] The dimensionality reduction convolutional layer also includes a pointwise convolutional kernel. The pointwise convolutional kernel has a size of 1×1, a stride of 1, and zero padding to ensure that the pointwise convolution operation only fuses features in the channel dimension, without changing the spatial size of the feature map. The pointwise convolutional kernel performs a 1×1 convolution operation on the spatial dimensionality reduction gradient tensor in the channel dimension, fusing and reducing the number of channels of the spatial dimensionality reduction gradient tensor to a single channel, generating an initial fused gradient map. The mathematical expression for the pointwise convolution operation is: in, The coordinates in the initial fusion gradient map are... pixel values, For the pointwise convolution kernel, the first The weight values ​​corresponding to each channel The total number of channels in the spatially reduced gradient tensor.

[0043] The spatial gradient branch also includes a max-pooling layer. This layer filters pixels whose gradient response values ​​in the initial fused gradient map are below a preset response threshold, retaining gradient extrema and generating the weld pool edge contour features. The mathematical expression for max-pooling is: in, The coordinates of the solder pool edge contour feature map output by the maximum pooling layer are: pixel values, For coordinates The set of pixel coordinates corresponding to the pooling window centered on the center. This is for calculating the maximum value.

[0044] refer to Figure 5 The processing server, after element-wise multiplying and fusing the weight matrix and spatial gradient features, inputs the multiplication result into the batch normalization layer and the ReLU activation function layer for distribution correction and nonlinear mapping, generating a fused feature map. The batch normalization layer normalizes the mean and variance of the feature fusion result for each batch, eliminating distribution differences between different batches of feature data and accelerating the network convergence process. The mathematical expression for batch normalization is: in, The feature map after batch normalization. The feature fusion results input in the current batch. This represents the mean of the current batch of feature data. The variance of the feature data in the current batch. To prevent tiny constants with a denominator of zero, This is the scaling factor. This is the offset coefficient.

[0045] The ReLU activation function layer performs a non-linear mapping on the batch-normalized feature map, filtering out negative response values ​​and retaining positive response values, thus enhancing the non-linear expressive power of the features. The mathematical expression for ReLU activation is: in, This is the fused feature map generated after processing with the ReLU activation function. This is for calculating the maximum value.

[0046] The defect classifier consists of sequentially connected global pooling layers, two fully connected hidden layers, and a softmax output layer. The global pooling layer compresses the spatial dimension of the fused feature map, generating a one-dimensional feature vector, eliminating the influence of spatial location information on the classification result, and preserving the global semantic information of the features. The mathematical expression for the global pooling operation is: in, This is a one-dimensional feature vector output by the global pooling layer. , These represent the spatial height and width dimensions of the fused feature map, respectively. To fuse the coordinates in the feature map The pixel value.

[0047] Two fully connected hidden layers perform feature cross operations on the one-dimensional feature vector to uncover the correlation information between features and map it to the feature space for defect classification. A Dropout layer is embedded between the first and second fully connected hidden layers. When the first fully connected hidden layer outputs a feature vector, the Dropout layer resets the weights of some neurons in the first fully connected hidden layer to zero according to a preset deactivation ratio. The preset deactivation ratio of the Dropout layer is configured differently during the network training and inference phases. During training, random deactivation is enabled, and the weights of some neurons are reset to zero according to the preset deactivation ratio. During inference, random deactivation is disabled, and all neurons remain active to ensure stable feature output during inference. The mathematical expression for Dropout processing is: in, This is the feature vector after processing by the Dropout layer. This is the feature vector output by the first fully connected hidden layer. The random mask vector follows a Bernoulli distribution, where the element corresponding to the inactivated neuron has a value of 0, and the element corresponding to the retained neuron has a value of 1.

[0048] The second fully connected hidden layer performs weight decay calculations on the zeroed feature vectors. During this calculation, an L2 regularization term is introduced to constrain the weight values ​​of the neurons, further suppressing overfitting. The Softmax output layer receives the weight decay calculation results and maps them to a numerical range between 0 and 1 using an exponential function, generating a probability vector containing various solder joint defect types. The mathematical expression for the Softmax operation is: in, For the first The probability value corresponding to the type of weld joint defect. The output of the second fully connected hidden layer The characteristic value corresponding to the class, This represents the total number of categories of solder joint defects. It is a natural constant.

[0049] The defect classifier uses the index of the solder joint defect type corresponding to the maximum value in the probability vector as the final output of the solder joint defect type. Solder joint defect types include, but are not limited to, cold solder joints, incomplete solder joints, cracks, porosity, and burn-through. Different defect types correspond to different index values. The processing server matches the corresponding defect type based on the output index value to complete the identification of solder joint defects.

[0050] Table 4. Correspondence between parameters of dilated convolutional layers with different expansion rates and feature extraction methods. This table is used to clarify the feature extraction range and type corresponding to dilated convolutional layers with different dilation rates, ensuring that the spatial gradient branch can cover multi-scale edge features from fine textures to overall contours, eliminating feature omissions caused by a single receptive field, and improving the integrity and accuracy of edge contour features.

[0051] This embodiment refines the network structure and computational logic of the spatial gradient branch. Multi-scale texture gradient features are extracted through dilated convolutional layers with multiple dilation rates. Dimensionality reduction is achieved through depthwise separable convolution and pointwise convolution, and invalid low-response pixels are filtered out by max pooling layers, thus obtaining accurate weld pool edge contour features. Batch normalization and ReLU activation functions are used to complete the distribution correction and nonlinear mapping of fused features. Feature cross-operation is performed by fully connected hidden layers with embedded dropout layers. Finally, a stable defect classification result is generated through the Softmax output layer, suppressing network overfitting and ensuring the stability of the weld defect classification result.

Claims

1. A vision-based inspection-based welding point analysis system for automotive parts, characterized in that, It includes a multispectral light source array, an image acquisition camera, and a processing server connected to the multispectral light source array and the image acquisition camera. During the welding cooling stage, the processing server controls the alternating flashing frequency of different wavelength light sources in the multispectral light source array according to the temperature decay curve. The image acquisition camera acquires a corresponding multi-frame image sequence, and the processing server inputs the multi-frame image sequence into a feature fusion network, which includes a spectral attention branch and a spatial gradient branch. The spectral attention branch generates a weight matrix based on the reflectance difference of pixels at different wavelengths in the multi-frame image sequence, and the spatial gradient branch extracts the edge contour features of the weld pool. The processing server performs element-wise multiplication and fusion of the weight matrix and the spatial gradient features to generate a fused feature map, and inputs the fused feature map into the defect classifier; The defect classifier outputs the weld point defect type based on the pixel distribution state in the fused feature map where high reflective areas are suppressed and edge contours are enhanced. This is the visual inspection-based automotive parts processing weld point analysis system.

2. The automotive parts processing weld point analysis system based on vision inspection according to claim 1, characterized in that, The processing server is connected to an infrared temperature sensor, which collects surface temperature data of the weld joint during the welding cooling stage. The processing server performs time series fitting on the surface temperature data of the weld joint to generate the temperature decay curve. The multispectral light source array includes infrared light sources, blue light sources, and green light sources distributed around the solder joint area. Each of the infrared light source, the blue light source, and the green light source is connected in series with an independently controlled strobe controller. The processing server sends a pulse width modulation signal to the strobe controller according to the slope change stage of the temperature decay curve. The strobe controller adjusts the lighting duration and interval of the infrared light source, the blue light source, and the green light source according to the pulse width modulation signal.

3. The automotive parts processing weld point analysis system based on vision inspection according to claim 2, characterized in that, The hardware trigger input terminal of the image acquisition camera is connected to the synchronization signal output terminal of the strobe controller. The strobe controller sends a lighting command to the infrared light source, the blue light source and the green light source while simultaneously outputting a high-level trigger signal to the image acquisition camera. After receiving the high-level trigger signal, the image acquisition camera opens a single exposure window, and the center time of the single exposure window coincides with the peak time of the light intensity of the infrared light source, the blue light source, or the green light source. The image acquisition camera reads the image corresponding to the current wavelength within the single exposure window and caches the image corresponding to the current wavelength in the memory queue of the processing server to form the multi-frame image sequence.

4. The automotive parts machining weld point analysis system based on vision inspection according to claim 1, characterized in that, The spectral attention branch includes a channel pooling layer, a fully connected layer, and a sigmoid activation layer connected in sequence. The channel pooling layer performs global average pooling on the pixel values ​​of the same position in the multi-frame image sequence under different wavelength channels to generate channel feature vectors. The fully connected layer performs nonlinear spatial mapping on the channel feature vectors to generate a channel attention map, and the sigmoid activation layer normalizes the mapping values ​​of each channel in the channel attention map. The normalized mapping values ​​are arranged as weight coefficients to generate the weight matrix, and the number of rows and columns of the weight matrix corresponds to the spatial height dimension and spatial width dimension of the multi-frame image sequence, respectively.

5. The automotive parts processing weld point analysis system based on vision inspection according to claim 1, characterized in that, The spatial gradient branch includes three dilated convolutional layers with different dilation rates and a feature stitching layer arranged in parallel. The three dilated convolutional layers with different dilation rates perform convolution operations on the preset reference wavelength images in the multi-frame image sequence to extract texture gradient response maps within different receptive field ranges. The feature splicing layer splices the texture gradient response maps output by the three dilated convolutional layers with different dilation rates end-to-end in the channel dimension to generate a multi-channel gradient tensor. The spatial gradient branch also includes a dimension reduction convolutional layer, which performs channel number compression on the multi-channel gradient tensor to output a single-channel weld pool edge contour feature.

6. The automotive parts machining weld point analysis system based on vision inspection according to claim 1, characterized in that, After the processing server performs element-wise multiplication and fusion of the weight matrix and the spatial gradient features, it inputs the multiplication result into the batch normalization layer and the ReLU activation function layer for distribution correction and nonlinear mapping to generate the fused feature map. The defect classifier comprises a sequentially connected global pooling layer, two fully connected hidden layers, and a Softmax output layer. The global pooling layer compresses the spatial dimension of the fused feature map to generate a one-dimensional feature vector. The two fully connected hidden layers perform feature cross operations on the one-dimensional feature vector. The Softmax output layer performs probability distribution mapping on the result of the feature cross operations and outputs a probability vector containing multiple solder joint defect types.

7. The automotive parts machining weld point analysis system based on vision inspection according to claim 2, characterized in that, The optical axis of the infrared temperature sensor is arranged at a preset angle with the optical axis of the image acquisition camera, and the center of the field of view of the infrared temperature sensor coincides with the illumination center of the multispectral light source array on the surface of the automotive parts. After acquiring the surface temperature data of the solder joint, the processing server removes abnormal temperature points that exceed the preset temperature threshold range from the surface temperature data of the solder joint, and inputs the remaining temperature points after removing the abnormal temperature points into the sliding window filter. The sliding window filter calculates the mean of the remaining temperature points in chronological order to generate a smoothed continuous temperature sequence, and the processing server performs polynomial fitting on the continuous temperature sequence to generate the temperature decay curve.

8. The automotive parts machining weld point analysis system based on vision inspection according to claim 4, characterized in that, The fully connected layer includes a first mapping sublayer and a second mapping sublayer, wherein the number of neurons in the first mapping sublayer is less than the dimension of the channel feature vector; The first mapping sub-layer performs dimensionality reduction and compression on the channel feature vector to generate an intermediate feature vector. The number of neurons in the second mapping sub-layer is the same as the dimension of the channel feature vector. The second mapping sub-layer performs dimensionality recovery on the intermediate feature vector to generate a dimension-recovered feature vector. The fully connected layer also introduces a cross-attention computation mechanism, which uses the dimension recovery feature vector as the query vector and the channel feature vector as the key vector and value vector to perform self-attention operation to generate the channel attention map.

9. The vision-based inspection-based weld point analysis system for automotive parts processing according to claim 5, characterized in that, The dimension reduction convolutional layer includes a depth-separable convolutional kernel with a stride of a preset value. The depth-separable convolutional kernel performs channel-by-channel two-dimensional convolution operations on the multi-channel gradient tensor in the spatial dimension to generate a spatial dimension-reduced gradient tensor. The dimension reduction convolutional layer further includes a pointwise convolutional kernel, which performs a 1×1 convolution operation on the spatial dimension reduction gradient tensor in the channel dimension. The number of channels of the spatial dimension reduction gradient tensor is fused and reduced to a single channel to generate an initial fused gradient map. The spatial gradient branch is also configured with a max pooling layer. The max pooling layer filters out pixels whose gradient response values ​​are lower than a preset response threshold in the initial fusion gradient map, retains gradient extrema, and generates the weld pool edge contour features.

10. The vision-based inspection-based weld point analysis system for automotive parts processing according to claim 6, characterized in that, A Dropout layer is embedded between the first fully connected hidden layer and the second fully connected hidden layer in the two fully connected hidden layers. When the first fully connected hidden layer outputs a feature vector, the Dropout layer resets the weights of some neurons in the first fully connected hidden layer to zero according to a preset deactivation ratio. The second fully connected hidden layer performs weight decay calculation on the feature vector after it is set to zero. The Softmax output layer receives the result of the weight decay calculation and maps the result of the weight decay calculation to a numerical range between 0 and 1 according to the exponential function. The defect classifier outputs the solder joint defect type index corresponding to the maximum value in the numerical range as the solder joint defect type.

Images