Defect detection method and system based on multispectral fusion imaging

Abstract

The invention relates to the technical field of industrial vision multispectral imaging detection, in particular to a defect detection method and system based on multispectral fusion imaging, and the method comprises the steps: synchronously collecting a multi-channel spectral image of the surface of an object to be detected, and generating an enhanced feature map fusing multispectral information; identifying and preliminarily marking a suspected defect region by applying a region segmentation algorithm based on anomaly detection; extracting a multi-dimensional spectral response curve of each region in each original spectral channel, and constructing a spectral feature vector; matching the vector with a standard template of a pre-established defect spectral feature database, and classifying and confirming defect types; and carrying out contour refined analysis, calculating the geometric dimension and position of the defect, and generating a structured detection report. According to the method, the sensitivity of defect detection and the classification accuracy are improved through accurate matching of multispectral fusion and spectral features, and automatic and high-precision defect identification is realized.

Description

Technical Field

[0001] This invention relates to the field of industrial vision multispectral imaging inspection technology, and in particular to a defect detection method and system based on multispectral fusion imaging. Background Technology

[0002] In industrial production, especially in precision manufacturing and materials science, the detection of defects on object surfaces is crucial. Current technologies commonly employ visible light visual inspection or single-spectral band imaging for defect identification. These methods primarily rely on differences in grayscale, color, or texture between defects and the background in a single image for segmentation and judgment. However, for defects with low contrast to the background, weak features, or when different types of defects exhibit similar characteristics in a single spectral band, the detection sensitivity and classification accuracy of these methods are significantly insufficient.

[0003] To improve detection capabilities, some technologies attempt to integrate multiple spectral channels. However, common multispectral techniques often remain at the level of simple image overlay or selection of optimal bands, failing to fully explore the correlations and complementary information between multi-channel data. This results in inaccurate initial defect localization, making it susceptible to noise interference and leading to false alarms or missed detections. In the defect classification stage, existing methods largely rely on inspectors' intuitive observation of multispectral images based on experience, or use simple classifiers based on shape and area, lacking quantitative analysis of the spectral properties of the defective material itself. This makes the classification results highly subjective and prone to confusion. Therefore, a technical solution is needed that can more accurately locate suspected defect areas and achieve objective and automated defect classification based on the spectral characteristics of the material. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a defect detection method and system based on multispectral fusion imaging.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a defect detection method based on multispectral fusion imaging, comprising: Simultaneously acquire multi-channel spectral images of the surface of the object under test, and generate an enhanced feature map based on the multi-channel spectral images that fuses multi-spectral information; An anomaly detection-based region segmentation algorithm is applied to the enhanced feature map to identify pixel regions that have statistical differences in fusion features with the surrounding regions, and these pixel regions are initially marked as suspected defect regions. Extract the multi-dimensional spectral response curves of each suspected defect region in the original spectral channel images, and construct the spectral feature vector of the suspected defect region; The matching degree of each suspected defect region's spectral feature vector is calculated with the standard template in the pre-established defect spectral feature database. Based on the matching results, the suspected defect regions are classified to confirm the defect type of the suspected defect regions. For the defect area with confirmed defect type, perform contour refinement analysis, segment the defect boundary, calculate the geometric size and location information of the defect area, and generate a structured inspection report containing defect type, location and size information.

[0006] As a further aspect of the present invention, the generation of the enhanced feature map fused with multispectral information includes: Multiple independent optical sensors are used to simultaneously acquire multi-channel spectral images of the surface of the object under test at different preset wavelengths; Geometric distortion calibration and radiation intensity normalization preprocessing are performed on each acquired multi-channel spectral image to eliminate errors introduced by sensor differences and inconsistent lighting conditions, resulting in a set of aligned multispectral base images with consistent intensity references. A multispectral image fusion model based on deep convolutional feature extraction is constructed. The multispectral base image is input into the multispectral image fusion model. The spatial texture and spectral features of each multispectral base image are extracted through the multispectral image fusion model to generate a set of high-dimensional feature maps. By designing a feature interaction mechanism, the cross-channel correlation weights between feature maps of different spectral channels are calculated. Based on the cross-channel correlation weights, features from different channels are adaptively weighted and fused to generate an enhanced feature map that integrates multispectral information.

[0007] As a further aspect of the present invention, the method of employing multiple independent optical sensors to simultaneously acquire multi-channel spectral images of the surface of the object under test at different preset wavelengths specifically includes: Based on the material properties of the object under test and the spectral absorption characteristics of common defects, multiple discrete and discontinuous specific wavelengths covering visible light, near-infrared, and short-wave infrared are selected as the preset wavelengths. Configure a set of narrowband optical filters and image sensors that correspond one-to-one with the preset wavelength to form multiple independent optical sensors, ensuring that each optical sensor is only sensitive to light signals of a specified wavelength. By using a hardware synchronization triggering device, all independent optical sensors are controlled to start exposure at the same time, capturing raw image data of the surface of the object under test at the same instant and in different spectral dimensions. After acquisition, the wavelength information corresponding to each original image data is marked and combined to form the multi-channel spectral image; The preprocessing of geometric distortion calibration and radiance normalization for each acquired spectral channel image specifically includes: Using a pre-calibrated high-precision checkerboard calibration board, the internal and external parameters of each optical sensor are calibrated separately to establish the mapping relationship between image pixel coordinates and actual physical coordinates; Based on the mapping relationship, perspective transformation and lens distortion correction are performed on the image of each spectral channel to make the same physical point in all channel images perfectly aligned on the pixel coordinates, thus completing geometric distortion calibration. Place a standard whiteboard with known reflectivity next to the object to be tested, and simultaneously acquire images of the standard whiteboard from each channel under the same lighting conditions. The radiation response correction coefficient for each channel is calculated by comparing the gray values ​​of the standard whiteboard area in each channel image with the theoretical reflectance values. Using the aforementioned radiation response correction coefficient, pixel-level grayscale value correction is performed on the images of each channel, unifying the radiation intensity benchmark of all channel images to the same standard, thus completing radiation intensity standardization.

[0008] As a further aspect of the present invention, the construction of a multispectral image fusion model based on deep convolutional feature extraction, wherein the multispectral base image is input into the multispectral image fusion model, specifically includes: Construct a deep convolutional neural network with parallel input branches, each branch corresponding to a spectral channel, and all branches have the same network structure and independent weights; In each branch, multiple convolutional layers and nonlinear activation layers are used to abstract the input multispectral base image layer by layer, extracting multi-scale spatial features from edges, textures to complex patterns; In the middle layer of the network, a cross-channel feature attention module is added, which receives the feature maps output by all branches at the corresponding layer. The cross-channel feature attention module calculates the mutual information or correlation measure between feature maps of any two different spectral channels and generates a correlation matrix between channels. The correlation matrix is ​​normalized and converted into importance weight coefficients of each channel feature map relative to other channels.

[0009] As a further aspect of the present invention, the step of calculating the cross-channel correlation weight between feature maps of different spectral channels through a designed feature interaction mechanism, and adaptively weighting and fusing features from different channels based on the cross-channel correlation weight, specifically includes: The feature map corresponding to each spectral channel is weighted using the importance weight coefficients generated by the cross-channel feature attention module. The weighted feature maps of all channels are concatenated along the channel dimension to form a comprehensive multi-channel feature tensor. A one-to-one convolutional layer is applied to the multi-channel feature tensor to perform cross-channel information integration and dimensionality reduction, compressing the number of channels to a preset value. Through the nonlinear transformation of the one-to-one convolutional layer, the feature information of different spectral channels can fully interact and fuse in the compressed feature space; Output a streamlined and highly condensed enhanced feature map that incorporates complementary information from all input spectral channels.

[0010] As a further aspect of the present invention, the step of applying an anomaly detection-based region segmentation algorithm to the enhanced feature map to identify pixel regions that have statistical differences in fusion features compared to surrounding regions specifically includes: The enhanced feature map is spatially divided into multiple overlapping or non-overlapping image blocks; Calculate the mean and covariance matrix of all pixel feature vectors within each image block to characterize the local feature distribution of the image block; Based on the Gaussian mixture model or the local outlier factor algorithm, the deviation of the feature distribution of each image block from the overall distribution of its surrounding neighboring image blocks is calculated to obtain the outlier score of the image block. A dynamic anomaly score threshold is set, and image blocks whose anomaly scores exceed the anomaly score threshold are marked as abnormal image blocks; Spatially adjacent anomalous image patches are clustered and merged to form one or more connected pixel regions, and each connected region is initially marked as a suspected defect region.

[0011] As a further aspect of the present invention, the step of extracting the multi-dimensional spectral response curve of each suspected defect region in the original spectral channel image specifically includes: Based on the pixel coordinates of the suspected defect area, the corresponding pixel of the suspected defect area in each single-channel image is located from the multispectral base image after geometric distortion calibration and radiometric intensity normalization preprocessing. For each single-channel image, the average gray value of all corresponding pixels within the suspected defect area is calculated as the spectral response intensity of the suspected defect area at the specific wavelength. Arrange the spectral response intensity values ​​obtained from all single-channel images in order of wavelength from shortest to longest; The arranged spectral response intensity values ​​are plotted as a continuous or discrete curve with wavelength as the abscissa and intensity as the ordinate, thus obtaining the multi-dimensional spectral response curve of the suspected defect region. Feature parameters, including peak wavelength, absorption valley position, curve slope, and intensity ratio in a specific band, are extracted from the spectral response curve to form the spectral feature vector of the suspected defect region.

[0012] As a further aspect of the present invention, the step of calculating the matching degree between the spectral feature vector of each suspected defect region and a standard template in a pre-established defect spectral feature database specifically includes: The defect spectral feature database stores standard spectral feature vector templates for various known defect types, with each template associated with a defect type label. Using distance or similarity metrics, calculate the matching distance or similarity score between the spectral feature vector of the suspected defective region and each standard template in the database; Select the top few standard templates that have the smallest matching distance with the spectral feature vector of the suspected defect area or the highest similarity score; Check whether the defect type labels corresponding to the candidate standard templates are consistent. If they are consistent, the defect type is initially determined to be the type of the suspected defect area. If the defect type labels of the candidate templates are inconsistent, the position of the spectral feature vector of the suspected defect region in the feature space and its relative distance to the center point of each candidate type template cluster are further analyzed. The final type confirmation is made by weighted voting or a more complex classifier.

[0013] As a further aspect of the present invention, the step of performing contour refinement analysis on the defect region of the confirmed defect type and segmenting the defect boundary specifically includes: Using the pre-marked suspected defect region as the initial seed region, a region growing algorithm is used to expand the boundary on the enhanced feature map; The growth criterion for region growth is based on the Euclidean distance between the feature vector of a pixel and the average feature vector of the seed region, and a distance threshold is set to control the growth range. After the region growth process is terminated, a more complete defect candidate region is obtained compared to the initial region; Near the boundary of the defect candidate region, an active contour model algorithm is used to iterate the contour curve towards the position with large image gradient, and finally fit the real edge of the defect. The contour curve after evolution and stabilization is smoothed to eliminate jagged edges and obtain a closed defect boundary polygon. The calculation of its geometric dimensions and location information, and the generation of a structured inspection report containing defect type, location, and size information, specifically includes: Based on the defect boundary polygon, calculate its minimum bounding rectangle or fitted ellipse to obtain the defect's length, width, area, and perimeter geometric dimensions. Using the image coordinate system or a preset world coordinate system as a reference, calculate the centroid coordinates of the defect boundary polygon as the location information of the defect; The defect type, calculated geometric dimensions, location information, and corresponding object identification and detection timestamp are integrated. Following the preset report format template, fill the integrated information into the corresponding fields to generate a structured inspection report document containing detailed defect descriptions of all inspection targets.

[0014] As a further aspect of the present invention, the present invention also includes a defect detection system based on multispectral fusion imaging, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements the steps of the defect detection method based on multispectral fusion imaging as described above.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By simultaneously acquiring multi-channel spectral images and generating enhanced feature maps that fuse multispectral information, a region segmentation algorithm based on anomaly detection is applied. This method deeply fuses multi-source spectral information to construct image feature representations that transcend single-channel or simple superposition. In this fused feature space, defect regions, due to their abnormal comprehensive spectral response, exhibit more significant statistical differences from normal background regions. Segmentation based on this difference enhances the ability to locate weak, low-contrast defects while effectively suppressing misjudgments caused by single-channel noise or uniform texture interference, thus improving the accuracy and robustness of initial defect screening.

[0016] Multidimensional spectral response curves for each suspected defect region in the original spectral channels are extracted, and spectral feature vectors are constructed and matched with standard templates in a pre-built database. This process transforms the approach from "image morphology analysis" to "material spectral fingerprinting." By quantifying and comparing the complete spectral response features of defect regions, classification can be directly based on changes in material properties, chemical composition, or physical state. This overcomes the limitations of traditional methods that rely on morphological features and are susceptible to imaging conditions, making it difficult to distinguish defects with similar spectral responses but different types. It achieves objective and refined identification of defect types, reducing reliance on human experience. Attached Figure Description

[0017] Figure 1 This is a flowchart of the defect detection method based on multispectral fusion imaging described in this invention; Figure 2 A radar chart for comprehensive analysis of multi-dimensional defect detection performance; Figure 3 A flowchart for building and running a multispectral image fusion model; Figure 4 A comparison of multi-dimensional spectral response curves between the normal area and the microcrack defect area of ​​a solar panel; Figure 5 The graphs show the multispectral response curves for different defect types. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1 The defect detection method based on multispectral fusion imaging involves simultaneously acquiring multi-channel spectral images of the surface of the object under test and generating an enhanced feature map that fuses multispectral information. On the generated enhanced feature map, an anomaly detection-based region segmentation algorithm is applied. This algorithm identifies pixel regions that exhibit statistical differences in the fused features compared to surrounding areas and initially labels these regions as suspected defect areas. For each labeled suspected defect area, its multi-dimensional spectral response curve in each of the original spectral channels is extracted, and a spectral feature vector representing the characteristics of that region is constructed. The spectral feature vector of each suspected defect area is compared with a standard template in a pre-established defect spectral feature database. Based on the matching results, these regions are classified to confirm their specific defect types. For the defect areas with confirmed defect types, a refined contour analysis is performed to accurately segment the defect boundaries and calculate their geometric dimensions and location information. Finally, a structured inspection report is generated, containing complete information such as the defect type, location, and size.

[0021] In one embodiment of the present invention, generating an enhanced feature map that integrates multispectral information includes simultaneously acquiring multichannel spectral images of the surface of the object under test using multiple independent optical sensors. These optical sensors are configured at different preset wavelengths. For example, in the example scenario of detecting scratches on a metal surface, the preset wavelengths are selected as 450 nm, 550 nm, and 650 nm. The image of each spectral channel corresponds to the reflection characteristics of the object in these specific wavelength bands. Data comparison shows that the 450 nm channel is sensitive to surface oxidation, the 550 nm channel is sensitive to color changes, and the 650 nm channel is sensitive to small indentations. After acquisition, geometric distortion calibration and radiation intensity normalization preprocessing are performed on the image of each spectral channel to eliminate errors introduced by sensor differences and inconsistent illumination, resulting in a set of multispectral base images. In the example, geometric distortion calibration uses a high-precision checkerboard calibration board to establish pixel-to-physical coordinate mapping, and radiation intensity normalization calculates correction coefficients based on the synchronously acquired standard whiteboard image. After preprocessing, the spatial alignment error of the multispectral base image is less than 1 pixel, and the radiation intensity reference variance is reduced to within 5%. A multispectral image fusion model based on deep convolutional feature extraction is constructed. The multispectral base image is input into the model, which includes parallel branches processing images from each channel. Spatial texture and spectral features are extracted through convolutional layers to generate high-dimensional feature maps. In the specific implementation, the feature interaction mechanism calculates the cross-channel correlation weights between feature maps of different spectral channels through a cross-channel feature attention module. The calculation of the cross-channel correlation weights is based on the mutual information between feature maps, for example, using the following formula to calculate the weight coefficients:

[0022] in: Indicates the first The feature map of the first channel is relative to the first... The importance weight of each channel Indicates the first Channel feature map and the Channel feature map Mutual information between them It is the total number of spectral channels. This represents the natural exponential function. This represents the summation operator, which sums all terms from k to N. This represents the high-dimensional feature map obtained after deep convolution extraction of the k-th spectral channel. Features from different channels are adaptively weighted and fused based on cross-channel correlation weights to generate an enhanced feature map. In the example scenario, the enhanced feature map after weighted fusion shows higher contrast in the scratch area compared to the single-channel image. Data comparison shows that the signal-to-noise ratio of the fused feature map in the defect area is improved by approximately 30%. In some embodiments, synchronous acquisition is controlled by hardware triggering to expose all sensors simultaneously, avoiding image misalignment caused by object movement. In specific implementations, the selection of the preset wavelength is based on the spectral characteristics of the material under test; for example, for plastic products, the preset wavelength includes the near-infrared band to detect internal bubbles. Optionally, radiation intensity normalization can be achieved through linear transformation, mapping the grayscale values ​​of each channel image to a uniform range. It can be understood that the deep convolutional structure of the multispectral image fusion model can adjust the number of layers and filters according to the application.

[0023] In one embodiment of the present invention, multiple independent optical sensors are used to simultaneously acquire multi-channel spectral images of the surface of the object under test at different preset wavelengths. Specifically, this includes selecting discrete wavelengths covering specific bands based on the material properties and spectral absorption characteristics of common defects of the object under test. In an example scenario of detecting the soldering quality of a printed circuit board assembly, the material properties of the object under test involve solder, copper foil, and plastic substrate, and common defects include cold solder joints, bridging, and solder balls. Based on this, four discrete and discontinuous specific wavelengths of 450 nm, 700 nm, 900 nm, and 1200 nm are selected as preset wavelengths. Data comparison shows that the 450 nm wavelength is sensitive to flux residue, the 700 nm wavelength is sensitive to surface oxidation, the 900 nm wavelength can penetrate part of the plastic encapsulation to detect the underlying solder joints, and the 1200 nm wavelength is sensitive to temperature distribution and can be used to detect thermal anomalies. A set of narrowband optical filters and image sensors, each corresponding to a preset wavelength, are configured to form multiple independent optical sensors. This ensures that each optical sensor is sensitive only to light signals of a specified wavelength. In the example scenario, a bandpass filter with a center wavelength of 450 nm and a bandwidth of 10 nm and a CCD sensor are configured for the 450 nm wavelength, and a bandpass filter with a center wavelength of 1200 nm and a bandwidth of 20 nm and an InGaAs sensor are configured for the 1200 nm wavelength. A hardware synchronization triggering device controls all independent optical sensors to activate exposure simultaneously, capturing raw image data of the surface of the object under test at the same instant and in different spectral dimensions. After acquisition, the raw image data output by each optical sensor is labeled with its corresponding wavelength information and combined to form a multi-channel spectral image. In some embodiments, the geometric distortion calibration and radiation intensity normalization preprocessing of the acquired images of each spectral channel specifically includes using a pre-calibrated high-precision checkerboard calibration plate to calibrate the internal and external parameters of each optical sensor, establishing a mapping relationship between image pixel coordinates and actual physical coordinates. In this example, the internal parameters include focal length and distortion coefficient, and the external parameters include the sensor's position and orientation relative to the calibration plate. Based on the mapping relationship, perspective transformation and lens distortion correction are performed on the images of each spectral channel to ensure that the same physical point in all channel images is perfectly aligned in pixel coordinates, thus completing geometric distortion calibration. Data comparison shows that the alignment error of corresponding points within the correction plane is reduced from a maximum of 3.2 pixels to less than 0.5 pixels. A standard white board with known reflectivity is placed next to the object under test, and standard white board images of each channel are acquired synchronously under the same illumination conditions. The radiation response correction coefficient of each channel is calculated by comparing the gray value of the standard white board area in each channel image with the theoretical reflectance value. Optionally, the radiation response correction coefficient is calculated. The formula is:

[0024] in: Indicates the first Radiative response correction coefficients for each spectral channel This represents the theoretical reflectance of a standard whiteboard at the corresponding wavelength. Indicates the first The average grayscale value of the standard white board area in each channel image. Pixel-level grayscale value correction is performed on each channel image using a radiometric response correction coefficient, unifying the radiometric intensity benchmark of all channel images to the same standard, thus achieving radiometric intensity standardization. In the example, after applying the correction, the variance of the measured intensity of the same uniform gray board in each channel image decreased from approximately 15% before correction to less than 3%. It can be understood that the hardware synchronization triggering device ensures strict temporal alignment of multispectral images, avoiding relative changes between images caused by object movement or light flicker.

[0025] See Figure 2 This is a radar chart comprehensively analyzing the performance of multi-dimensional defect detection, showcasing a comparison of six key performance indicators between traditional methods and multispectral fusion methods in multispectral defect detection. The multispectral fusion method outperforms the traditional method in all six indicators, forming a more complete and near-perfect radar chart profile, demonstrating the comprehensive superiority of its technical solution. Detection speed is also improved by 10%, indicating that the multispectral fusion method maintains high accuracy without sacrificing efficiency, making it more suitable for industrial online inspection scenarios. This radar chart visually demonstrates that the defect detection method based on multispectral fusion imaging, through technological innovation throughout the entire process of data acquisition, preprocessing, and feature extraction, has achieved a comprehensive surpassing of traditional methods. It not only represents a qualitative leap in the accuracy of defect identification but also improves the stability and efficiency of detection, representing a significant technological breakthrough in the field of industrial visual inspection.

[0026] See Figure 3In one embodiment of the present invention, a multispectral image fusion model based on deep convolutional feature extraction is constructed, and a multispectral base image is input into the multispectral image fusion model. Specifically, this includes constructing a deep convolutional neural network with parallel input branches. Each branch corresponds to a spectral channel, and all branches have the same network structure but independent weights. In an example scenario of detecting apple watercore, the input is a preprocessed four-channel multispectral base image. The deep convolutional neural network is configured with four parallel input branches. The network structure of each branch contains five convolutional layers and uses ReLU as a non-linear activation layer. Each branch performs layer-by-layer abstraction of the input multispectral base image to extract multi-scale spatial features from edges, textures to complex patterns. Data comparison shows that the first layer convolutional output feature map mainly contains edge information, while the fifth layer convolutional output feature map can characterize the complex patterns related to the watercore region. In the middle layer of the deep convolutional neural network, a cross-channel feature attention module is added. The cross-channel feature attention module receives the feature maps output by all branches at the corresponding layer. In a specific implementation, the cross-channel feature attention module is located after the third convolutional layer of the deep convolutional neural network and receives feature maps of the same size from the four branches. The cross-channel feature attention module calculates the mutual information or correlation measure between any two different spectral channel feature maps, generating a channel correlation matrix. In the example scenario, the cross-channel feature attention module uses cosine similarity to calculate the correlation between feature maps of different spectral channels, and the elements of the correlation matrix... The calculation is as follows:

[0027] in: Indicates the first The feature map of the first channel and the first A measure of the correlation between feature maps of each channel. and They represent from the first The branch and the first Input feature maps of each branch, This represents the operation of flattening the feature map into a vector. Represents the vector dot product. The L2 norm of the vector is represented. The correlation matrix is ​​normalized and transformed into importance weight coefficients for each channel feature map relative to other channels. In the example, the weight coefficients are obtained by performing Softmax normalization on each row of the correlation matrix. In some embodiments, the cross-channel correlation weights between feature maps of different spectral channels are calculated through a designed feature interaction mechanism, and the features from different channels are adaptively weighted and fused based on the cross-channel correlation weights. Specifically, the importance weight coefficients generated by the cross-channel feature attention module are used to weight the feature map corresponding to each spectral channel. In the specific implementation, each feature map is multiplied by the corresponding weight coefficient. All the weighted channel feature maps are concatenated along the channel dimension to form a comprehensive multi-channel feature tensor. In the example, the feature maps of the four channels are concatenated after weighting, and the number of channels becomes four times the original. A 1x1 convolutional layer is applied to the multi-channel feature tensor to integrate cross-channel information and reduce dimensionality, compressing the number of channels to a preset value. In the example, the 1x1 convolutional layer compresses the number of channels from 128 to 32. The nonlinear transformation through a 1x1 convolutional layer allows for full interaction and fusion of feature information from different spectral channels in the compressed feature space, ultimately outputting an enhanced feature map with a reduced number of channels and highly condensed information. This enhanced feature map incorporates complementary information from all input spectral channels. Data comparison shows that the fused enhanced feature map increases the feature difference between the apple watercore region and the normal pulp region by an average of approximately 1.8 times compared to the feature map of a single spectral channel. Optionally, a cross-channel feature attention module can be inserted into different layers of the deep convolutional neural network to capture feature interactions at different levels of abstraction.

[0028] In one embodiment of the present invention, an anomaly detection-based region segmentation algorithm is applied to the enhanced feature map to identify pixel regions that have statistical differences in fused features compared to surrounding regions. Specifically, this involves spatially dividing the enhanced feature map into multiple overlapping or non-overlapping image blocks. In an example scenario of detecting microcracks in a solar panel, the enhanced feature map is 512 pixels by 512 pixels, and the image blocks are divided into overlapping blocks of 16 pixels by 16 pixels with a stride of 8 pixels. The mean and covariance matrix of all pixel feature vectors within each image block are calculated to characterize the local feature distribution of the image block. In specific implementations, the feature vectors originate from multiple channels of the enhanced feature map, and the local feature distribution of each image block is jointly characterized by the mean vector μ and the covariance matrix Σ. An anomaly score for each image block is obtained by calculating the deviation of its feature distribution from the overall distribution of its neighboring image blocks based on a Gaussian mixture model or a local outlier algorithm. In the example scenario, a Gaussian mixture model is used to model the feature distribution of the image blocks, and the anomaly score of the image blocks... Calculated using its negative log-likelihood:

[0029] in: This indicates the anomaly score of the current image patch. This indicates the number of Gaussian components in the Gaussian mixture model. Indicates the first The mixing coefficient of the Gaussian components, This represents the sample set consisting of the feature vectors of all pixels within the current image patch. and They represent the first The mean vector and covariance matrix of each Gaussian component. This represents the probability density function of a multivariate Gaussian distribution. A dynamic anomaly score threshold is set to mark image patches with anomaly scores exceeding the threshold as anomalous image patches. The dynamic anomaly score threshold is adaptively determined based on the statistical distribution of anomaly scores across all image patches. Spatially adjacent anomalous image patches are clustered and merged to form one or more connected pixel regions. Each connected region is initially marked as a suspected defect region. In the solar panel example, multiple adjacent high-anomaly-score image patches are clustered into a narrow connected region corresponding to a hidden crack. In some embodiments, extracting the multidimensional spectral response curve of each suspected defect region in the original spectral channel images specifically includes locating the corresponding pixel of the suspected defect region in each single-channel image from the multispectral base image after geometric distortion calibration and radiation intensity normalization preprocessing, based on the pixel coordinate position of the suspected defect region. In the example, the suspected defect region has a corresponding pixel set in the base images of all four spectral channels. For each single-channel image, the average grayscale value of all corresponding pixels within the suspected defect area is calculated as the spectral response intensity of the suspected defect area at a specific wavelength. The spectral response intensity values ​​obtained from all single-channel images are arranged in ascending order of wavelength. The arranged spectral response intensity values ​​are then plotted as a continuous or discrete curve with wavelength as the x-axis and intensity as the y-axis, yielding the multi-dimensional spectral response curve of the suspected defect area. Feature parameters extracted from the spectral response curve include peak wavelength, absorption valley location, curve slope, and intensity ratio at a specific wavelength band, forming the spectral feature vector of the suspected defect area. In the example, the spectral response curves of the microcrack area and the normal area of ​​the solar panel differ at a specific wavelength band; refer to Table 1 for the extracted feature parameters.

[0030] Table 1: Comparison of spectral characteristic parameters between normal and suspected defective areas of solar panels

[0031] Optionally, image patches can be divided using a non-overlapping grid to reduce computation. Anomaly scores reflect the statistical difference between image patch features and surrounding background features. The feature distribution of image patches can be calculated using the local outlier factor algorithm to measure the local density deviation of each image patch relative to its K nearest neighbors. Spectral response curves can directly use discrete intensity value sequences as part of the feature vector. Multidimensional spectral response curves reflect the unique spectral characteristics of suspected defective regions and are an important basis for subsequent classification.

[0032] See Figure 4 This is a comparison of the multi-dimensional spectral response curves of a normal area and a microcrack defect area on a solar panel, representing one of the core visualization results of the multispectral fusion imaging defect detection method. The intensity ratio at 750nm to 900nm is 1.101 in the normal area and 1.303 in the microcrack area. This difference in ratio is a key spectral feature distinguishing the two and can be used in subsequent defect classification algorithms. This figure visually demonstrates that microcrack defects alter the spectral response characteristics of a solar panel, with this change being particularly significant at specific wavelengths. By extracting these spectral features, normal and defective areas can be effectively distinguished, providing a reliable basis for subsequent defect detection and classification. On industrial production lines such as those for solar panels, the detection method represented in this figure enables high-speed, non-contact online quality inspection, significantly improving production efficiency and product yield.

[0033] In one embodiment of the present invention, the matching degree of the spectral feature vector of each suspected defect region is calculated with a standard template in a pre-established defect spectral feature database. Specifically, the defect spectral feature database stores standard spectral feature vector templates for various known defect types, with each template associated with a defect type label. In an example scenario of detecting scratches and stains on a ceramic substrate, the defect spectral feature database contains standard spectral feature vector templates for three types of defects: "scratches," "stains," and "dents." Each template is composed of an average spectral feature vector extracted from a large number of samples. The matching distance or similarity score between the spectral feature vector of the suspected defect region and each standard template in the database is calculated using a distance metric or similarity metric. In a specific implementation, Mahalanobis distance is used to calculate the matching distance. The calculation formula is:

[0034] in: The Mahalanobis distance between the spectral feature vector of a suspected defect region and a standard template of a certain defect type. This represents the spectral feature vector of the suspected defect region. The first in the defect spectral feature database The average feature vector of the standard template for class defects, Indicates the first The covariance matrix of the defect feature vectors, This indicates the transpose operation. This represents the inverse operation of a matrix. The top few standard templates with the smallest matching distance or highest similarity score to the spectral feature vector of the suspected defect region are selected. In the ceramic substrate example, the Mahalanobis distances to the templates for "scratches," "stains," and "dents" for a certain suspected region are calculated to be 1.2, 3.8, and 5.1, respectively. The defect type labels corresponding to the candidate standard templates are checked for consistency. If they are consistent, the defect type is initially determined to be the type of the suspected defect region. In the example, the template label corresponding to the smallest matching distance is "scratches," so the suspected region is initially determined to be a scratch defect. If the defect type labels of the candidate templates are inconsistent, the position of the spectral feature vector of the suspected defect region in the feature space and its relative distance to the center point of each candidate type template cluster are further analyzed. A final type confirmation is made through weighted voting or a more complex classifier.

[0035] In some embodiments, the contour refinement analysis and defect boundary segmentation of regions with confirmed defect types specifically includes using a pre-marked suspected defect region as an initial seed region and employing a region growing algorithm to expand the boundary on the enhanced feature map. The growth criterion for region growing is based on the Euclidean distance between the feature vector of a pixel and the average feature vector of the seed region, and a distance threshold is set to control the growth range. After the region growing process terminates, a more complete defect candidate region is obtained than the initial region. Near the boundary of the defect candidate region, an active contour model algorithm is used to iteratively evolve the contour curve towards the position with the large image gradient, ultimately conforming to the true edge of the defect. In the ceramic substrate scratch example, the initial seed region may only cover a part of the scratch. After region growing and active contour model evolution, the obtained contour more completely outlines the winding shape of the scratch. The evolved and stabilized contour curve is smoothed to eliminate jagged edges, resulting in a closed defect boundary polygon.

[0036] Optionally, calculating the geometric dimensions and location information to generate a structured inspection report containing defect type, location, and size information specifically includes calculating the minimum bounding rectangle or fitting an ellipse based on the defect boundary polygon to obtain the defect's length, width, area, and perimeter geometric dimensions. In the example, the boundary polygon of a scratch defect is fitted to have a length of 2.1 mm, a width of 0.1 mm, and an area of ​​0.22 square millimeters. Using the image coordinate system or a preset world coordinate system as a reference, the centroid coordinates of the defect boundary polygon are calculated as the defect's location information. The defect type, calculated geometric dimensions, location information, and the corresponding object identifier and inspection timestamp are integrated. Following a preset report format template, the integrated information is filled into the corresponding fields to generate a structured inspection report document containing detailed defect descriptions of all inspected targets.

[0037] See Figure 5 This is a multispectral response curve of different defect types, clearly showing the differences in spectral response intensity of different regions on the ceramic substrate surface in the 400nm-1800nm ​​wavelength range. Statistical differences exist in the response trends of different regions across the entire wavelength range, especially in the 400-800nm ​​and 1000-1400nm bands, which are the core characteristic bands for distinguishing defects from normal regions. This provides a data foundation for subsequent template matching using Mahalanobis distance. The high response intensity of scratches stems from changes in light reflection characteristics caused by variations in their surface physical structure; the low response intensity of pits is directly related to increased light absorption caused by regional depressions; the response curve trend of stains is similar to that of normal regions but is generally lower, reflecting the influence of their chemical properties on spectral absorption. The spectral characteristics of different defects are clearly distinguishable, proving that the multispectral image fusion model successfully extracted the complementary information of each channel, achieving the goal of feature enhancement.

[0038] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A defect detection method based on multispectral fusion imaging, characterized in that, Includes the following steps: Simultaneously acquire multi-channel spectral images of the surface of the object under test, and generate an enhanced feature map based on the multi-channel spectral images that fuses multi-spectral information; An anomaly detection-based region segmentation algorithm is applied to the enhanced feature map to identify pixel regions that have statistical differences in fusion features with the surrounding regions, and these pixel regions are initially marked as suspected defect regions. Extract the multi-dimensional spectral response curves of each suspected defect region in the original spectral channel images, and construct the spectral feature vector of the suspected defect region; The matching degree of each suspected defect region's spectral feature vector is calculated with the standard template in the pre-established defect spectral feature database. Based on the matching results, the suspected defect regions are classified to confirm the defect type of the suspected defect regions. For the defect area with confirmed defect type, perform contour refinement analysis, segment the defect boundary, calculate the geometric size and location information of the defect area, and generate a structured inspection report containing defect type, location and size information.

2. The defect detection method based on multispectral fusion imaging according to claim 1, characterized in that, The generation of the enhanced feature map, which incorporates multispectral information, includes: Multiple independent optical sensors are used to simultaneously acquire multi-channel spectral images of the surface of the object under test at different preset wavelengths; Geometric distortion calibration and radiation intensity normalization preprocessing are performed on each acquired multi-channel spectral image to eliminate errors introduced by sensor differences and inconsistent lighting conditions, resulting in a set of aligned multispectral base images with consistent intensity references. A multispectral image fusion model based on deep convolution feature extraction is constructed, and the multispectral base image is input into the multispectral image fusion model; Spatial texture and spectral features of each multispectral base image are extracted using a multispectral image fusion model to generate a set of high-dimensional feature maps. By designing a feature interaction mechanism, the cross-channel correlation weights between feature maps of different spectral channels are calculated. Based on the cross-channel correlation weights, features from different channels are adaptively weighted and fused to generate an enhanced feature map that integrates multispectral information.

3. The defect detection method based on multispectral fusion imaging according to claim 2, characterized in that, The method employs multiple independent optical sensors to simultaneously acquire multi-channel spectral images of the surface of the object under test at different preset wavelengths, specifically including: Based on the material properties of the object under test and the spectral absorption characteristics of common defects, multiple discrete and discontinuous specific wavelengths covering visible light, near-infrared, and short-wave infrared are selected as the preset wavelengths. Configure a set of narrowband optical filters and image sensors that correspond one-to-one with the preset wavelength to form multiple independent optical sensors, ensuring that each optical sensor is only sensitive to light signals of a specified wavelength. By using a hardware synchronization triggering device, all independent optical sensors are controlled to start exposure at the same time, capturing raw image data of the surface of the object under test at the same instant and in different spectral dimensions. After acquisition, the wavelength information corresponding to each original image data is marked and combined to form the multi-channel spectral image; The preprocessing of geometric distortion calibration and radiance normalization for each acquired spectral channel image specifically includes: Using a pre-calibrated high-precision checkerboard calibration board, the internal and external parameters of each optical sensor are calibrated separately to establish the mapping relationship between image pixel coordinates and actual physical coordinates; Based on the mapping relationship, perspective transformation and lens distortion correction are performed on the image of each spectral channel to make the same physical point in all channel images perfectly aligned on the pixel coordinates, thus completing geometric distortion calibration. Place a standard whiteboard with known reflectivity next to the object to be tested, and simultaneously acquire images of the standard whiteboard from each channel under the same lighting conditions. The radiation response correction coefficient for each channel is calculated by comparing the gray values ​​of the standard whiteboard area in each channel image with the theoretical reflectance values. Using the aforementioned radiation response correction coefficient, pixel-level grayscale value correction is performed on the images of each channel, unifying the radiation intensity benchmark of all channel images to the same standard, thus completing radiation intensity standardization.

4. The defect detection method based on multispectral fusion imaging according to claim 3, characterized in that, The construction of a multispectral image fusion model based on deep convolutional feature extraction, specifically including inputting the multispectral base image into the multispectral image fusion model, includes: Construct a deep convolutional neural network with parallel input branches, each branch corresponding to a spectral channel, and all branches have the same network structure and independent weights; In each branch, multiple convolutional layers and nonlinear activation layers are used to abstract the input multispectral base image layer by layer, extracting multi-scale spatial features from edges, textures to complex patterns; In the middle layer of the network, a cross-channel feature attention module is added, which receives the feature maps output by all branches at the corresponding layer. The cross-channel feature attention module calculates the mutual information or correlation measure between feature maps of any two different spectral channels and generates a correlation matrix between channels. The correlation matrix is ​​normalized and converted into importance weight coefficients of each channel feature map relative to other channels.

5. The defect detection method based on multispectral fusion imaging according to claim 4, characterized in that, The aforementioned feature interaction mechanism calculates the cross-channel correlation weights between feature maps of different spectral channels, and adaptively weights and fuses features from different channels based on these cross-channel correlation weights. Specifically, this includes: The feature map corresponding to each spectral channel is weighted using the importance weight coefficients generated by the cross-channel feature attention module. The weighted feature maps of all channels are concatenated along the channel dimension to form a comprehensive multi-channel feature tensor. A one-to-one convolutional layer is applied to the multi-channel feature tensor to perform cross-channel information integration and dimensionality reduction, compressing the number of channels to a preset value. Through the nonlinear transformation of the one-to-one convolutional layer, the feature information of different spectral channels can fully interact and fuse in the compressed feature space; Output a streamlined and highly condensed enhanced feature map that incorporates complementary information from all input spectral channels.

6. The defect detection method based on multispectral fusion imaging according to claim 5, characterized in that, An anomaly detection-based region segmentation algorithm is applied to the enhanced feature map to identify pixel regions that exhibit statistical differences in fused features compared to surrounding regions, specifically including: The enhanced feature map is spatially divided into multiple overlapping or non-overlapping image blocks; Calculate the mean and covariance matrix of all pixel feature vectors within each image block to characterize the local feature distribution of the image block; Based on the Gaussian mixture model or the local outlier factor algorithm, the deviation of the feature distribution of each image block from the overall distribution of its surrounding neighboring image blocks is calculated to obtain the outlier score of the image block. A dynamic anomaly score threshold is set, and image blocks whose anomaly scores exceed the anomaly score threshold are marked as abnormal image blocks; Spatially adjacent anomalous image patches are clustered and merged to form one or more connected pixel regions, and each connected region is initially marked as a suspected defect region.

7. The defect detection method based on multispectral fusion imaging according to claim 6, characterized in that, Extracting the multi-dimensional spectral response curves of each suspected defect region in the original spectral channel images, specifically including: Based on the pixel coordinates of the suspected defect area, the corresponding pixel of the suspected defect area in each single-channel image is located from the multispectral base image after geometric distortion calibration and radiometric intensity normalization preprocessing. For each single-channel image, the average gray value of all corresponding pixels within the suspected defect area is calculated as the spectral response intensity of the suspected defect area at the specific wavelength. Arrange the spectral response intensity values ​​obtained from all single-channel images in order of wavelength from shortest to longest; The arranged spectral response intensity values ​​are plotted as a continuous or discrete curve with wavelength as the abscissa and intensity as the ordinate, thus obtaining the multi-dimensional spectral response curve of the suspected defect region. Feature parameters, including peak wavelength, absorption valley position, curve slope, and intensity ratio in a specific band, are extracted from the spectral response curve to form the spectral feature vector of the suspected defect region.

8. The defect detection method based on multispectral fusion imaging according to claim 7, characterized in that, The step of calculating the matching degree between the spectral feature vector of each suspected defect region and the standard template in the pre-established defect spectral feature database specifically includes: The defect spectral feature database stores standard spectral feature vector templates for various known defect types, with each template associated with a defect type label. Using distance or similarity metrics, calculate the matching distance or similarity score between the spectral feature vector of the suspected defective region and each standard template in the database; Select the top few standard templates that have the smallest matching distance with the spectral feature vector of the suspected defect area or the highest similarity score; Check whether the defect type labels corresponding to the candidate standard templates are consistent. If they are consistent, the defect type is initially determined to be the type of the suspected defect area. If the defect type labels of the candidate templates are inconsistent, further analysis is conducted on the position of the spectral feature vector of the suspected defect region in the feature space, as well as its relative distance to the center point of each candidate type template cluster. The final type confirmation is then made through weighted voting or a more complex classifier.

9. A defect detection method based on multispectral fusion imaging according to claim 8, characterized in that, The step of performing a detailed contour analysis of the defect region for the confirmed defect type and segmenting the defect boundaries specifically includes: Using the pre-marked suspected defect region as the initial seed region, a region growing algorithm is used to expand the boundary on the enhanced feature map; The growth criterion for region growth is based on the Euclidean distance between the feature vector of a pixel and the average feature vector of the seed region, and a distance threshold is set to control the growth range. After the region growth process is terminated, a more complete defect candidate region is obtained compared to the initial region; Near the boundary of the defect candidate region, an active contour model algorithm is used to iterate the contour curve towards the position with large image gradient, and finally fit the real edge of the defect. The contour curve after evolution and stabilization is smoothed to eliminate jagged edges and obtain a closed defect boundary polygon. The calculation of its geometric dimensions and location information, and the generation of a structured inspection report containing defect type, location, and size information, specifically includes: Based on the defect boundary polygon, calculate its minimum bounding rectangle or fitted ellipse to obtain the defect's length, width, area, and perimeter geometric dimensions. Using the image coordinate system or a preset world coordinate system as a reference, calculate the centroid coordinates of the defect boundary polygon as the location information of the defect; The defect type, calculated geometric dimensions, location information, and corresponding object identification and detection timestamp are integrated. Following the preset report format template, fill the integrated information into the corresponding fields to generate a structured inspection report document containing detailed defect descriptions of all inspection targets.

10. A defect detection system based on multispectral fusion imaging, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the defect detection method based on multispectral fusion imaging as described in any one of claims 1 to 9.

Images