A product defect detection method and system based on multispectral imaging

Abstract

The present invention discloses a product defect detection method and system based on multispectral imaging, including: acquiring multispectral image data of a financial payment terminal product to generate a first image set; performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; extracting features based on the second image set to generate a feature set; constructing a dedicated convolutional neural network model for the plastic shell and the metal circuit board according to the feature set, generating a model set adapted to material differences through transfer learning, and determining a final detection model to perform defect detection on real-time multispectral images to obtain a detection result. The present invention improves the accuracy and efficiency of the quality detection of financial payment terminal products, provides data support for the optimization of the production process, effectively reduces the product defect rate, and improves the product quality and reliability.

Description

Technical Field

[0001] The present invention belongs to the field of product defect detection, and particularly relates to a product defect detection method and system based on multispectral imaging. Background Art

[0002] The quality inspection of financial payment terminal products is a key link to ensure their reliability and security, and plays a crucial role in the financial industry. Product defects may lead to equipment failures or safety hazards, directly affecting user trust and industry development. Traditional inspection methods mainly rely on manual visual inspection, which is inefficient and easily interfered by subjective factors, and it is difficult to detect hidden defects, such as internal bubbles or poor soldering. Although single visible light imaging technology has certain applications, it is limited by the single wavelength band and is difficult to comprehensively capture multi-dimensional information on the surface and inside of products, and there are deficiencies in both detection accuracy and coverage.

[0003] The limitations of existing methods have prompted research to turn to more intelligent and comprehensive detection technologies. The detection method combining multispectral imaging and deep learning has become a new exploration direction. Multispectral imaging can obtain rich visual information in multiple wavelength bands, while deep learning can extract defect features from complex data. However, the implementation of this method faces core technical challenges. First, how to construct an adapted multispectral imaging detection model for different materials and functional components of financial payment terminal products is a difficult point for achieving accurate detection. Second, how to optimize deep learning algorithms to efficiently identify tiny defects, such as bubbles, foreign objects or poor soldering, while ensuring the generalization ability of the model is the key to improving the detection effect. In addition, the establishment of a defect feature database and a fast matching mechanism are not yet mature, which limits the efficiency of defect traceability. Therefore, how to effectively integrate multispectral imaging and deep learning, construct a dedicated detection model for different materials and components, optimize the algorithm to achieve accurate and efficient identification of tiny defects, and establish a defect feature database to support fast matching and traceability has become a key problem that needs to be solved urgently. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a product defect detection method and system based on multispectral imaging. Among them, a product defect detection method based on multispectral imaging includes:

[0005] Obtain multispectral image data of a financial payment terminal product, and generate a first image set containing surface defects and internal soldering information;

[0006] Perform denoising filtering and contrast enhancement processing on the first image set to generate a second image set; determine whether the resolution and clarity of the second image set meet the preset requirements, and if they meet the preset requirements, determine it as training data;

[0007] Extract texture shape features including bubbles, foreign objects, and poor soldering from the second image set to generate a feature set;

[0008] Construct a dedicated convolutional neural network model for the plastic housing and metal circuit board based on the feature set, and generate a first model set adapted to material differences through transfer learning;

[0009] Apply data augmentation and L2 regularization algorithms to optimize the first model set to generate a second model set. If the accuracy and recall rate of the second model set exceed the preset threshold, it is determined as the final detection model;

[0010] Perform defect detection on real-time multi-spectral images through the second model set to generate a defect set including defect locations, types, and severities;

[0011] Construct a defect feature database based on the defect set. If the defect feature database contains fields for location, type, and severity, it is determined as a matchable data set; use a vector retrieval algorithm to perform feature matching on the defect feature database to generate a matching set; apply an association rule mining algorithm to the matching set to generate a traceability data set including corresponding relationships in the production process.

[0012] Preferably, the process of obtaining multi-spectral image data of a financial payment terminal product and generating a first image set including surface defects and internal soldering information includes:

[0013] Scan the financial payment terminal through a multi-spectral imaging device to collect image data including visible light and infrared spectra to generate a first multi-spectral image set;

[0014] Use a convolutional neural network to perform feature extraction on the first multi-spectral image set, separate surface defect features and internal soldering features to obtain a first feature set;

[0015] If the surface defect feature value in the first feature set exceeds the preset threshold, partition the surface area through an image segmentation algorithm to obtain a surface defect location image;

[0016] According to the surface defect location image, perform boundary optimization on the defect area using morphological operations to generate a defect image;

[0017] If the internal soldering feature value in the first feature set deviates from the preset soldering quality standard, analyze the thermal distribution of the soldering area through infrared spectrum data to obtain soldering quality evaluation data;

[0018] Integrate the defect image and the soldering quality evaluation data through a data fusion algorithm to generate a third image set including surface defects and internal soldering information;

[0019] The third image set is rendered using visualization technology to output an image set of defects and welding information of the financial payment terminal.

[0020] Preferably, the process of denoising filtering and contrast enhancement of the first image set to generate the second image set includes:

[0021] The first image set is denoised using a bilateral filtering algorithm to generate an intermediate image set;

[0022] The intermediate image set is subjected to contrast enhancement using an adaptive histogram equalization method to generate the second image set;

[0023] The resolution parameter of the second image set is obtained. If the resolution parameter meets the preset resolution threshold, it is determined as a candidate image set;

[0024] For the candidate image set, the Laplacian operator is used to calculate the image sharpness. If the sharpness value is greater than the preset sharpness threshold, it is determined as training data;

[0025] Feature extraction is performed based on the training data, and pre-training is performed using a convolutional neural network algorithm to generate initial model parameters;

[0026] According to the initial model parameters, the second image set is batch-verified to obtain a verification error; if the verification error is lower than the preset error threshold, the model parameters are determined to be valid;

[0027] According to the valid model parameters, subsequent input image sets are automatically processed to generate training data that meets the requirements.

[0028] Preferably, the process of extracting texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set includes:

[0029] Image data is obtained based on the second image set, and a denoised image is generated through preprocessing;

[0030] A convolutional neural network is used to extract texture shape features of bubbles, foreign objects, and poor welding from the denoised image to obtain an initial feature set;

[0031] If the dimension of the initial feature set is higher than the preset threshold, dimensionality reduction is performed through principal component analysis to generate an optimized feature set;

[0032] According to the optimized feature set, texture shape feature vectors of bubbles, foreign objects, and poor welding are calculated to determine the feature distribution;

[0033] Through the feature distribution, a clustering algorithm is used to classify bubbles, foreign objects, and poor welding to obtain a classification result;

[0034] Obtain the texture shape feature subsets of each type of defect based on the classification results, and generate the final feature set;

[0035] Construct a feature database for the final feature set and output the feature set data.

[0036] Preferably, the process of constructing a dedicated convolutional neural network model for the plastic housing and the metal circuit board according to the feature set and generating the first model set adapted to the material difference through transfer learning includes:

[0037] Use a convolutional neural network to perform image processing on the feature set, extract the features of the plastic housing and the metal circuit board, and obtain the first feature set;

[0038] If the dimension of the first feature set exceeds a preset threshold, generate a second feature set through dimensionality reduction processing; otherwise, directly determine that the second feature set is the first feature set;

[0039] Perform model training on the second feature set through transfer learning to generate the first model set adapted to the material difference;

[0040] Evaluate the performance of the first model set using classification accuracy to obtain the first evaluation result;

[0041] If the first evaluation result is lower than the preset threshold, re-train by adjusting the parameters of the convolutional neural network to obtain the second model set; otherwise, determine that the first model set is the final model;

[0042] Classify the newly input material feature set through the second model set to obtain the classification result.

[0043] Preferably, the process of optimizing the first model set by applying data augmentation and the L2 regularization algorithm to generate the second model set includes:

[0044] Preprocess the first model set through data augmentation to generate an augmented data set;

[0045] Apply L2 regularization to the augmented data set to optimize the first model set and obtain the second model set;

[0046] Obtain the accuracy rate and recall rate of the second model set, calculate the performance metrics, and if both the accuracy rate and recall rate exceed the preset threshold, determine that the second model set is the final detection model;

[0047] Perform performance verification on the final detection model through cross-validation to obtain the verification result, adjust the data augmentation parameters according to the verification result to generate an optimized model set, and extract the final detection model based on the optimized model set.

[0048] Preferably, the process of defect detection on the real-time multispectral image by the second model set to generate a defect set including the defect position, type, and severity includes:

[0049] Obtain a real-time multispectral image through a multispectral sensor to generate a first image set;

[0050] Use a preprocessing algorithm to denoise and standardize the first image set to obtain a second image set;

[0051] Perform defect detection on the second image set through the second model set to determine the defect position, type, and severity, and generate a first defect set;

[0052] If the defect severity in the first defect set exceeds a preset threshold, perform local enhancement processing on the corresponding image area to obtain a third image set;

[0053] Perform defect detection on the third image set again through the second model set to update the defect position, type, and severity, and obtain a second defect set;

[0054] According to the defect type in the second defect set, use a classification algorithm to rank the defects by priority to obtain a ranked defect set;

[0055] Format the defect position, type, and severity in the ranked defect set to generate a final defect set.

[0056] Preferably, based on the defect set, construct a defect feature database. If the defect feature database contains fields of position, type, and severity, the process of determining it as a matchable data set includes:

[0057] Obtain defect position, defect type, and severity data according to the defect set, and use a data cleaning method to remove duplicate and missing values to obtain structured defect data;

[0058] Through the structured defect data, construct a feature database including fields of position, type, and severity, and use a relational database to manage data storage to obtain a database structure;

[0059] If the fields of the feature database include position, type, and severity, use a query statement to extract records that meet the conditions to obtain a preliminary matching data set;

[0060] According to the preliminary matching data set, use the K-nearest neighbor algorithm to perform clustering analysis on the defect position and type to obtain a classified matching data set;

[0061] Obtain severity data from the classified matching data set, and use a sorting algorithm to sort it in descending order of severity to obtain a priority ranking data set;

[0062] Using the prioritized dataset, extract records with a severity higher than a preset threshold through a data screening method to determine the final matching dataset;

[0063] Extract the defect locations and types from the final matching dataset, and generate defect distribution features through a data mapping method to obtain the defect feature analysis result.

[0064] Preferably, a vector retrieval algorithm is used to perform feature matching on the defect feature database. The process of generating the matching set includes:

[0065] Use a vector retrieval algorithm to query the defect feature database, calculate the similarity between the input feature vector and the feature vectors in the database, and obtain a preliminary matching set;

[0066] If the similarity value in the preliminary matching set is greater than the preset threshold, retain the corresponding feature vector; otherwise, discard it to generate a filtered matching set;

[0067] According to the filtered matching set, obtain the defect category information corresponding to each feature vector in the matching set to generate a classification result set;

[0068] Through the classification result set, determine the occurrence frequency of each defect category to obtain frequency distribution data;

[0069] Use a clustering algorithm to analyze the frequency distribution data, obtain the grouping pattern of defect categories, and generate the final defect analysis result set;

[0070] Store the final defect analysis result set in a preset database to generate queryable analysis records.

[0071] The present invention also provides a product defect detection system based on multispectral imaging, including:

[0072] A data acquisition module for acquiring multispectral image data of a financial payment terminal product to generate a first image set containing surface defects and internal welding information;

[0073] An image processing module for performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; determine whether the resolution and clarity of the second image set meet the preset requirements. If they meet the preset requirements, determine it as training data;

[0074] A feature extraction module for extracting texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set;

[0075] A model construction module, which is used to construct a dedicated convolutional neural network model for plastic shells and metal circuit boards according to the feature set, and generate a first set of models adapted to material differences through transfer learning;

[0076] A model optimization module, which is used to optimize the first set of models by applying data augmentation and L2 regularization algorithms to generate a second set of models. If the accuracy and recall rate of the second set of models exceed the preset threshold, it is determined as the final detection model;

[0077] A defect detection module, which is used to detect defects in real-time multi-spectral images through the second set of models, and generate a defect set including defect positions, types, and severities;

[0078] A database construction module, which is used to construct a defect feature database based on the defect set. If the defect feature database contains fields such as position, type, and severity, it is determined as a matchable data set;

[0079] A feature matching module, which is used to perform feature matching on the defect feature database using a vector retrieval algorithm to generate a matching set;

[0080] A traceability analysis module, which is used to apply an association rule mining algorithm according to the matching set to generate a traceability data set including the corresponding relationships of production links.

[0081] Compared with the prior art, the present invention has the following advantages and technical effects:

[0082] The present invention realizes the defect detection of plastic shells and metal circuit boards by acquiring multi-spectral image data, performing image processing and feature extraction, and constructing a dedicated convolutional neural network model.

[0083] The present invention uses transfer learning to generate models adapted to different materials, and optimizes the model performance through data augmentation and regularization. In real-time detection, the present invention can identify the defect positions, types, and severities, and construct a defect feature database. Through vector retrieval and association rule mining, the traceability of the corresponding relationship between defects and production links is realized.

[0084] The present invention improves the accuracy and efficiency of the quality detection of financial payment terminal products, provides data support for the optimization of the production process, effectively reduces the product defect rate, and improves the product quality and reliability.

[0085] The present invention can effectively detect the surface and internal defects of financial payment terminal products, including bubbles, foreign objects, and poor soldering, etc., and can trace the production links where the defects occur, improving the efficiency and accuracy of product quality control and production process management. Description of the Drawings

[0086] The accompanying drawings, which form a part of this application, are used to provide a further understanding of this application. The schematic embodiments and descriptions thereof of this application are used to explain this application and do not constitute an improper limitation of this application. In the accompanying drawings:

[0087] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention;

[0088] Figure 2 It is a schematic structural diagram of the system according to an embodiment of the present invention. Detailed implementation manners

[0089] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the accompanying drawings and combine with the embodiments to detail this application.

[0090] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0091] Embodiment 1

[0092] As Figure 1 shown, in this embodiment, a product defect detection method based on multispectral imaging is provided, including:

[0093] Obtain multispectral image data of a financial payment terminal product, and generate a first image set including surface defects and internal welding information;

[0094] Perform denoising filtering and contrast enhancement processing on the first image set to generate a second image set; determine whether the resolution and clarity of the second image set meet the preset requirements. If they meet the preset requirements, it is determined as training data;

[0095] Extract texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set;

[0096] Construct a dedicated convolutional neural network model for the plastic shell and the metal circuit board according to the feature set, and generate a first model set adapted to the material differences through transfer learning;

[0097] Apply data augmentation and L2 regularization algorithms to optimize the first model set to generate a second model set. If the accuracy and recall rate of the second model set exceed the preset threshold, it is determined as the final detection model;

[0098] Perform defect detection on the real-time multispectral image through the second model set to generate a defect set including the defect location, type, and severity;

[0099] Construct a defect feature database based on the defect set. If the defect feature database contains fields such as location, type, and severity, it is determined as a matchable dataset; use a vector retrieval algorithm to perform feature matching on the defect feature database to generate a matching set; apply an association rule mining algorithm according to the matching set to generate a traceability dataset containing the corresponding relationships of production links.

[0100] Further, the process of obtaining the multi-spectral image data of the financial payment terminal product and generating the first image set containing surface defects and internal welding information includes:

[0101] Scan the financial payment terminal through a multi-spectral imaging device, collect image data containing visible light and infrared spectra, and generate a first multi-spectral image set;

[0102] Use a convolutional neural network to extract features from the first multi-spectral image set, separate the surface defect features and internal welding features, and obtain a first feature set;

[0103] If the surface defect feature value in the first feature set exceeds the preset threshold, partition the surface area through an image segmentation algorithm to obtain a surface defect localization image;

[0104] According to the surface defect localization image, perform boundary optimization on the defect area using morphological operations to generate a defect image;

[0105] If the internal welding feature value in the first feature set deviates from the preset welding quality standard, analyze the thermal distribution of the welding area through infrared spectrum data to obtain welding quality evaluation data;

[0106] Integrate the defect image and the welding quality evaluation data through a data fusion algorithm to generate a third image set containing surface defects and internal welding information;

[0107] Use visualization technology to render the third image set and output an image set of the defects and welding information of the financial payment terminal.

[0108] Specifically, when acquiring multi-spectral image data of financial payment terminal products, first, a high-resolution multi-spectral camera (such as Specim IQ, spectral range 400 - 1000 nm, resolution 512×512 pixels) is used to scan the terminal surface. The exposure time is set to 10 ms, the band interval is 5 nm, and spectral data of 120 bands are collected in total. The original image is preprocessed by an adaptive threshold segmentation algorithm (Otsu algorithm) to remove background noise and retain the effective area. For surface defect detection, a YOLOv5 model based on deep learning is adopted. The input image size is adjusted to 640×640. When training, the learning rate is set to 0.001, the batch size is 16, and 100 epochs are iterated. Fine-tuning is performed using transfer learning based on the pre-trained weights of the COCO dataset. Finally, the recognition accuracy of defects such as scratches and dents reaches 98.5%. For internal welding information, X-ray imaging technology (voltage 80 kV, current 5 mA) is used to penetrate the shell to obtain the welding point image. Convolutional neural network (CNN) is combined for feature extraction. The network structure includes 3 convolutional layers (convolution kernel size 3×3, stride 1, padding 1) and 2 fully connected layers. The performance of the model is improved through the ReLU activation function and batch normalization. The detection accuracy of defects such as solder joint voids and cold soldering reaches 97.2%. When fusing surface defect and internal welding data to generate the first image set, an image registration algorithm (SIFT feature matching, RANSAC to remove mismatched points) is used to align multi-source images, and different resolution information is fused through a multi-scale feature pyramid (FPN). Finally, a comprehensive image dataset containing defect annotations and welding quality ratings is output, providing a complete input for subsequent quality analysis.

[0109] Further, the process of denoising filtering and contrast enhancement of the first image set to generate the second image set includes:

[0110] The bilateral filtering algorithm is used to denoise the first image set to generate an intermediate image set;

[0111] The adaptive histogram equalization method is used to enhance the contrast of the intermediate image set to generate the second image set;

[0112] Obtain the resolution parameters of the second image set. If the resolution parameters meet the preset resolution threshold, it is determined as a candidate image set;

[0113] For the candidate image set, the Laplacian operator is used to calculate the image sharpness. If the sharpness value is greater than the preset sharpness threshold, it is determined as training data;

[0114] Feature extraction is performed based on the training data, and pre-training is carried out using the convolutional neural network algorithm to generate initial model parameters;

[0115] Batch verify the second image set according to the initial model parameters to obtain the verification error; if the verification error is lower than the preset error threshold, determine that the model parameters are valid;

[0116] Automatically process the subsequent input image set according to the valid model parameters to generate the required training data.

[0117] Specifically, when performing denoising and filtering on the first image set in this embodiment, the non-local means (NLM) algorithm is used, with the search window set to 21×21 pixels, the similar block window set to 7×7 pixels, and the filtering parameter h = 0.12×σ (σ is the noise standard deviation). Noise suppression is achieved by calculating the weighted similarity of pixel neighborhoods. For contrast enhancement, the contrast-limited adaptive histogram equalization (CLAHE) method is used, with the clipping limit set to 2.0 and the grid division size set to 8×8. The image details are enhanced through local histogram equalization. The quality of the processed second image set is evaluated using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics. If the PSNR value is greater than 30 dB and the SSIM is higher than 0.92, it is determined that the resolution and clarity meet the standards.

[0118] For example, when inputting a 512×512 POS machine product image, after NLM denoising, the noise standard deviation drops from 25 to 8, and after CLAHE processing, the average edge gradient increases by 40%. At this time, the PSNR is 32.5 dB and the SSIM is 0.94, meeting the preset threshold, and the system automatically marks it as qualified training data. During the process, bicubic interpolation is used to maintain the original resolution, and the Sobel operator is used to detect the edge sharpness to ensure that no artifacts are introduced in the enhanced image.

[0119] Furthermore, extract the texture shape features containing bubbles, foreign objects, and poor soldering from the second image set. The process of generating the feature set includes:

[0120] Obtain the image data based on the second image set and generate a denoised image through preprocessing;

[0121] Use a convolutional neural network to extract the texture shape features of bubbles, foreign objects, and poor soldering from the denoised image to obtain the initial feature set;

[0122] If the dimension of the initial feature set is higher than the preset threshold, perform dimensionality reduction through principal component analysis to generate an optimized feature set;

[0123] According to the optimized feature set, calculate the texture shape feature vectors of bubbles, foreign objects, and poor soldering, and determine the feature distribution;

[0124] Through the feature distribution, use a clustering algorithm to classify bubbles, foreign objects, and poor soldering to obtain the classification result;

[0125] Obtain the texture shape feature subsets of each type of defect based on the classification results, and generate the final feature set;

[0126] For the final feature set, construct a feature database and output the feature set data.

[0127] Specifically, in this embodiment, the texture shape features including bubbles, foreign objects, and poor soldering are extracted from the second image set. First, the Gaussian filtering algorithm is used to preprocess the image with a standard deviation of 1.5 to smooth the image and reduce noise interference. Then, the Canny edge detection algorithm is used to extract the edge information in the image with a low threshold of 50 and a high threshold of 150 to accurately capture the contours of bubbles, foreign objects, and poor soldering. Next, through the opening and closing operations in morphological operations, a 3×3 structuring element is used respectively to remove small-area noise and fill holes to enhance the connectivity of the feature regions. On this basis, the Local Binary Pattern (LBP) algorithm is used to extract texture features with a radius of 2 and 8 sampling points to quantify the local texture information of the image. For shape features, the Hu moment algorithm is used to calculate the seven invariant moments of the image region to describe the shape characteristics of bubbles, foreign objects, and poor soldering. Finally, the extracted texture and shape features are normalized using the Z-score normalization method to convert the feature values into a distribution with a mean of 0 and a standard deviation of 1, generating the feature set. Through the above steps, the accuracy and robustness of the feature set are ensured, providing a reliable data basis for subsequent defect detection and classification.

[0128] The above steps generate a denoised image, an initial feature set, an optimized feature set, a feature distribution, a classification result, and a final feature set in sequence through the logical chain of image processing, feature extraction, and classification, ensuring that the output of each step serves as the input of the next step to form a tight logical relationship.

[0129] Further, the process of constructing a dedicated convolutional neural network model for plastic enclosures and metal circuit boards and generating the first model set adapted to material differences through transfer learning includes:

[0130] Use a convolutional neural network to perform image processing on the feature set, extract the features of plastic enclosures and metal circuit boards, and obtain the first feature set;

[0131] If the dimension of the first feature set exceeds the preset threshold, generate the second feature set through dimensionality reduction processing; otherwise, directly determine the second feature set as the first feature set;

[0132] Perform model training on the second feature set through transfer learning to generate the first model set adapted to material differences;

[0133] Evaluate the performance of the first model set using classification accuracy to obtain the first evaluation result;

[0134] If the first evaluation result is lower than the preset threshold, re-train by adjusting the parameters of the convolutional neural network to obtain a second model set; otherwise, determine the first model set as the final model.

[0135] Classify the newly input material feature set through the second model set to obtain a classification result.

[0136] Specifically, when constructing a dedicated convolutional neural network model for plastic casings and metal circuit boards, first design a feature extraction layer to adapt to material differences.

[0137] More specifically, taking the example that ResNet50 is used as the basic architecture in this embodiment, its input size is adjusted to 224×224×3, and a batch normalization layer and a ReLU activation function are added after the first layer of convolution. For the texture features of plastic casings, a 3×3 convolutional kernel is used to extract local details, the stride is set to 1, and the number of output channels is 64. For the high-reflection characteristics of metal circuit boards, a 5×5 convolutional kernel is added to capture a larger range of reflection patterns, and the number of output channels is increased to 128. Through transfer learning, load the weights pre-trained on ImageNet, freeze the parameters of the first 15 layers to retain the general feature extraction ability, and the subsequent layers are adjusted using an adaptive learning rate. The initial learning rate is set to 0.001, the Adam optimizer is used, and the momentum parameters are β1 = 0.9 and β2 = 0.999. In the material difference adaptation stage, introduce the domain adaptation loss function MMD (Maximum Mean Discrepancy) to calculate the distance between the plastic and metal feature distributions. The kernel function selects a Gaussian kernel, the bandwidth parameter σ = 1.0, and the loss weight coefficient λ = 0.5. When training the model, the batch size is set to 32, iterate for 100 epochs, verify the accuracy every 10 epochs, and trigger the early stopping mechanism when the validation set loss does not decrease continuously for 5 times. The final model reaches a classification accuracy of 92.3% on the test set, where the recognition rate of plastic casings is 94.1%, the recognition rate of metal circuit boards is 90.5%, and the F1-score is 0.916. Visualized by Gradient-weighted Class Activation Mapping (Grad-CAM), the model shows significant attention to the scratches on the plastic surface and the solder joint areas of the metal, proving the effectiveness of feature extraction.

[0138] Furthermore, the process of applying data augmentation and the L2 regularization algorithm to optimize the first model set to generate a second model set includes:

[0139] Preprocess the first model set through data augmentation to generate an augmented data set;

[0140] Apply L2 regularization to the augmented data set to optimize the first model set to obtain a second model set;

[0141] Obtain the accuracy and recall rate of the second model set, calculate the performance metrics. If both the accuracy and recall rate exceed the preset threshold, determine the second model set as the final detection model;

[0142] Perform performance verification on the final detection model through cross-validation to obtain the verification result. Adjust the data augmentation parameters according to the verification result to generate an optimized model set, and extract the final detection model based on the optimized model set.

[0143] Specifically, first, use data augmentation methods of random rotation, translation, and adding Gaussian noise to the first model set. The rotation angle range is ±15 degrees, the translation amplitude is 10% of the image size, and the standard deviation of Gaussian noise is set to 0.05. Implement batch processing through the warpAffine and GaussianBlur functions of the OpenCV library to expand the original 5000 training samples to 20000.

[0144] Then, introduce L2 regularization in the model training stage, set the penalty coefficient of λ = 0.01, use the tf.keras.regularizers.l2 module of TensorFlow to embed into the fully connected layer, and perform 300 rounds of iterative training using the Adam optimizer (learning rate 0.001, β1 = 0.9, β2 = 0.999), processing 128 images per batch. During the training process, monitor the validation set loss through the early stopping mechanism and terminate the training when it does not decrease for 20 consecutive rounds. After training, evaluate on 1000 independent test sets. The accuracy of the second model set reaches 92.3% (a 4.7% increase compared to the first model set), and the recall rate is 89.1% (a 5.2% increase), exceeding the preset accuracy threshold of 90% and recall rate threshold of 85%. At this time, use the model with the highest F1-score on the validation set (F1 = 0.907) as the final detection model. Its confusion matrix shows that the false positive rate of positive class samples is 3.8% and the false negative rate is 6.2%. Save the model weight file in HDF5 format and deploy it to the inference server.

[0145] Furthermore, the process of generating a defect set containing defect locations, types, and severities by performing defect detection on real-time hyperspectral images through the second model set includes:

[0146] Obtain real-time hyperspectral images through a hyperspectral sensor to generate a first image set;

[0147] Use a preprocessing algorithm to denoise and standardize the first image set to obtain a second image set;

[0148] Perform defect detection on the second image set through the second model set to determine the defect locations, types, and severities, and generate a first defect set;

[0149] If the severity of the defects in the first defect set exceeds the preset threshold, local enhancement processing is performed on the corresponding image region to obtain a third image set;

[0150] The second defect set is obtained by re-detecting the defects in the third image set through the second model set and updating the defect positions, types, and severities;

[0151] According to the defect types in the second defect set, a classification algorithm is used to sort the defects by priority to obtain a sorted defect set;

[0152] Format the defect positions, types, and severities in the sorted defect set to generate a final defect set.

[0153] Specifically, when this embodiment uses the second model set to detect defects in real-time multi-spectral images, first, an improved algorithm based on YOLOv5 is adopted, its backbone network is replaced with EfficientNet-B4 to improve the feature extraction ability, the input image resolution is 640×640 pixels, and it is processed synchronously in 5 bands (450nm, 550nm, 650nm, 750nm, 850nm).

[0154] The model uses the FocalLoss loss function in the training stage, sets the initial learning rate to 0.001, the batchsize to 16, and after 300 epochs of training, it reaches mAP@0.5 of 92.3% on the validation set. For defect localization, the non-maximum suppression (NMS) algorithm is adopted, the IoU threshold is set to 0.45, and the confidence threshold is set to 0.6 to ensure the balance between the detection rate and the false detection rate.

[0155] In the defect classification link, a secondary classifier constructed by ResNet-34 is used to perform fine-grained analysis on the localization region, and the cross-entropy loss function is used. The classification accuracy for 6 types of typical defects (cracks, bubbles, scratches, stains, material shortage, color difference) reaches 89.7%.

[0156] The severity assessment integrates the gradient boosting decision tree (GBDT) algorithm. The input features include the pixel standard deviation of the defect region (such as the standard deviation threshold for crack-like defects is set to 15.8), the area ratio (such as when the area of bubble defects exceeds 0.5% of the total area, it is determined to be severe), and the multi-spectral reflectance difference (such as when the difference between the 650nm and 850nm bands is greater than 35, it is determined to be severe color difference). Finally, the output defect set is stored in JSON format, including pixel coordinates (such as x_min:120, y_min:345, x_max:210, y_max:420), defect type codes (such as C02 represents a bubble), and severity levels (1-5 levels). The entire processing flow takes less than 200ms, and real-time detection is achieved through CUDA11.1 acceleration.

[0157] Furthermore, based on the defect set, a defect feature database is constructed. If the defect feature database contains fields such as location, type, and severity, the process of determining the dataset that can be matched includes:

[0158] Obtain defect location, defect type, and severity data from the defect set, and use data cleaning methods to remove duplicate and missing values to obtain structured defect data;

[0159] Through the structured defect data, construct a feature database containing fields such as location, type, and severity, and use a relational database to manage data storage to obtain the database structure;

[0160] If the fields of the feature database contain location, type, and severity, use a query statement to extract the records that meet the conditions to obtain a preliminary matching dataset;

[0161] According to the preliminary matching dataset, use the K-nearest neighbor algorithm to perform clustering analysis on the defect location and type to obtain a classified matching dataset;

[0162] Obtain severity data from the classified matching dataset, and use a sorting algorithm to sort in descending order of severity to obtain a priority sorted dataset;

[0163] Through the priority sorted dataset, use a data screening method to extract the records with a severity higher than the preset threshold to determine the final matching dataset;

[0164] Extract the defect location and type from the final matching dataset, and use a data mapping method to generate defect distribution features to obtain the defect feature analysis result.

[0165] Specifically, when constructing the defect feature database in this embodiment, it is first necessary to extract defect data from the code library. For example, by scanning a Java project through a static analysis tool such as SonarQube, obtain the original data including file name, line number, defect type, and severity (such as severe, general, minor).

[0166] For the location field, it can be stored as "src / main / Model.java:125", indicating that the defect is located on line 125 of the Model.java file.

[0167] The type field uses a standard classification. For example, CWE-79 represents a cross-site scripting vulnerability, and CWE-89 represents an SQL injection. The severity is quantified by the CVSS score. For example, a score of 7.5 corresponds to a high-risk vulnerability.

[0168] In the data preprocessing stage, unstructured logs are parsed, and keyword fields are extracted using regular expressions. Then, a feature encoding algorithm is adopted to perform One-Hot encoding on the defect types to generate binary vectors. For example, [1,0,0] represents memory leak, and [0,1,0] represents resource not released.

[0169] For database construction, Elasticsearch is used to implement an inverted index. The location fields are tokenized and stored, and the aggregation analysis capabilities for type and severity are established. When it is detected that the database contains these three core fields, the system automatically triggers the matching logic. For example, the similarity between the new defect and the records in the library is calculated through the cosine similarity algorithm, and a threshold of 0.85 is set as the condition for successful matching. For the numerical severity field, Z-score normalization is adopted to make it conform to the normal distribution, facilitating subsequent processing by machine learning models. During the whole process, data consistency checking is realized through SHA-256 hash value verification to ensure the field integrity of each record.

[0170] Furthermore, a vector retrieval algorithm is adopted to perform feature matching on the defect feature database. The process of generating the matching set includes:

[0171] Use a vector retrieval algorithm to query the defect feature database, calculate the similarity between the input feature vector and the feature vectors in the database, and obtain a preliminary matching set;

[0172] If the similarity value in the preliminary matching set is greater than the preset threshold, the corresponding feature vector is retained; otherwise, it is discarded to generate a filtered matching set;

[0173] According to the filtered matching set, obtain the defect category information corresponding to each feature vector in the matching set to generate a classification result set;

[0174] Through the classification result set, determine the occurrence frequency of each defect category to obtain frequency distribution data;

[0175] Adopt a clustering algorithm to analyze the frequency distribution data, obtain the grouping pattern of defect categories, and generate the final defect analysis result set;

[0176] Store the final defect analysis result set into a preset database to generate queryable analysis records.

[0177] Specifically, in the defect feature database, the cosine similarity is used as the vector retrieval algorithm for feature matching. First, each defect feature in the database is vectorized. For example, the TF-IDF algorithm is used to convert text features into numerical vectors. Suppose a defect feature vector is [0.85, 0.12, 0.45, 0.67]. Then, the defect feature to be matched is input and also undergoes vectorization processing to obtain the input vector [0.78, 0.15, 0.50, 0.70]. Next, the cosine similarity between the input vector and each feature vector in the database is calculated. The formula is cos(θ)=(A·B) / (||A||*||B||), where A and B are the input vector and the feature vector in the database respectively. Through calculation, suppose the cosine similarity between the input vector and a certain database vector is 0.92, which is higher than the preset threshold of 0.85, then it is considered that the two match successfully. Finally, all the successfully matched feature vectors are generated into a matching set. For example, the matching set contains vectors such as [0.85, 0.12, 0.45, 0.67] and [0.80, 0.10, 0.48, 0.65] for subsequent defect analysis and processing.

[0178] Furthermore, the process of generating a traceability data set containing the corresponding relationship of production links according to the matching set by applying the association rule mining algorithm includes:

[0179] Obtain the matching set data, remove redundant items through preprocessing to obtain a standardized data set;

[0180] Apply the association rule mining algorithm from the standardized data set, extract the association patterns between production links, and determine the link association strength;

[0181] According to the link association strength, if the association strength exceeds the preset threshold, then retain the corresponding rules to obtain a preliminary rule set;

[0182] Through the preliminary rule set, combined with the production process sequence, generate an intermediate traceability set containing link association relationships;

[0183] For the intermediate traceability set, apply data matching accuracy verification. If the matching accuracy is lower than the preset threshold, then iteratively optimize the rule extraction efficiency to obtain an optimized traceability set;

[0184] Extract integrity features from the optimized traceability set, generate a traceability set, and determine the integrity of the final traceability set;

[0185] According to the traceability set, verify the accuracy of the production link association relationship to obtain the corresponding relationship of the final production link. Specifically, in the implementation process of the association rule mining algorithm, it is first necessary to construct a matching set of production links. For example, extract records containing parameters such as temperature, pressure, and time from the production log to form a transaction data set. Assume that the data set contains 1000 records, and each record consists of 5 production link parameters, such as temperature range [150°C, 200°C], pressure value [0.8 MPa, 1.2 MPa], etc. Use the Apriori algorithm for frequent item set mining, set the minimum support to 0.1, and the minimum confidence to 0.7. Generate candidate item sets by scanning the data set. For example, the first scan obtains frequent 1-item sets {temperature = 80°C} (support 0.15), {pressure = 1.0 MPa} (support 0.12). Then generate frequent 2-item sets through connection and pruning operations, such as {temperature = 180°C, pressure = 1.0 MPa} (support 0.08), and retain it if it meets the minimum support. Subsequently, generate association rules based on the frequent item sets. For example, the confidence of the rule "temperature = 80°C → pressure = 1.0 MPa" is 0.53, which is lower than the threshold and is thus eliminated, while the confidence of the rule "time = 120 min → temperature = 80°C" is 0.75 and is retained. The finally generated traceability set contains 20 strong association rules, and each rule is accompanied by support and confidence. For example, the support of "humidity = 20% ∧ rotation speed = 300 rpm → finished product qualification rate = 95%" is 0.09, and the confidence is 0.82. Through comparative analysis of the same data set using the FP-Growth algorithm, its running efficiency is 40% higher than that of Apriori, but the rule results are the same, verifying the reliability of the algorithm. During the process, a hash tree is used to optimize the candidate item set counting, and the results are stored in a graph database for visual display of the association network between links.

[0186] Embodiment 2

[0187] As Figure 2 shown, based on the same inventive concept, this embodiment also provides a product defect detection system based on multispectral imaging, including:

[0188] A data acquisition module for acquiring multispectral image data of a financial payment terminal product and generating a first image set containing surface defects and internal welding information;

[0189] An image processing module for performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; determining whether the resolution and clarity of the second image set meet the preset requirements, and if so, determining it as training data;

[0190] A feature extraction module for extracting texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set;

[0191] A model construction module, configured to construct a dedicated convolutional neural network model for a plastic housing and a metal circuit board based on a feature set, and generate a first model set adapted to material differences through transfer learning;

[0192] A model optimization module, configured to optimize the first model set by applying data augmentation and L2 regularization algorithms to generate a second model set. If the accuracy and recall rate of the second model set exceed a preset threshold, it is determined as the final detection model;

[0193] A defect detection module, configured to perform defect detection on real-time multi-spectral images through the second model set, and generate a defect set including defect positions, types, and severities;

[0194] A database construction module, configured to construct a defect feature database based on the defect set. If the defect feature database includes fields of position, type, and severity, it is determined as a matchable data set;

[0195] A feature matching module, configured to perform feature matching on the defect feature database using a vector retrieval algorithm to generate a matching set;

[0196] A traceability analysis module, configured to apply an association rule mining algorithm according to the matching set to generate a traceability data set including corresponding relationships of production links.

[0197] A product defect detection system based on multi-spectral imaging provided in this embodiment has all the advantages of the product defect detection method based on multi-spectral imaging provided in Embodiment 1.

[0198] Embodiment 3

[0199] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0200] Embodiment 4

[0201] This embodiment also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the method described in Embodiment 1.

[0202] Embodiment 5

[0203] This embodiment also discloses a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the method described in Embodiment 1.

[0204] The above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A product defect detection method based on multispectral imaging, characterized in that, Including: Obtain multi-spectral image data of a financial payment terminal product, and generate a first image set containing surface defect and internal welding information; Perform denoising filtering and contrast enhancement processing on the first image set to generate a second image set; Judge whether the resolution and clarity of the second image set meet the preset requirements. If they meet the preset requirements, determine it as training data; Extract texture shape features including bubbles, foreign objects, and poor welding from the second image set to generate a feature set; Construct a dedicated convolutional neural network model for the plastic shell and metal circuit board according to the feature set, and generate a first model set adapted to material differences through transfer learning; Apply data augmentation and L2 regularization algorithms to optimize the first model set to generate a second model set. If the accuracy and recall rate of the second model set exceed the preset threshold, determine it as the final detection model; Perform defect detection on real-time multi-spectral images through the second model set to generate a defect set containing defect positions, types, and severity levels; Construct a defect feature database based on the defect set. If the defect feature database contains fields of position, type, and severity level, determine it as a matchable data set; use a vector retrieval algorithm to perform feature matching on the defect feature database to generate a matching set; apply an association rule mining algorithm according to the matching set to generate a traceability data set containing corresponding relationships of production links.

2. The method according to claim 1, wherein: The process of obtaining multi-spectral image data of a financial payment terminal product and generating a first image set containing surface defect and internal welding information includes: Scan the financial payment terminal through a multi-spectral imaging device, collect image data including visible light and infrared spectra, and generate a first multi-spectral image set; Use a convolutional neural network to perform feature extraction on the first multi-spectral image set, separate surface defect features and internal welding features, and obtain a first feature set; If the surface defect feature value in the first feature set exceeds the preset threshold, partition the surface area through an image segmentation algorithm to obtain a surface defect localization image; According to the surface defect localization image, perform boundary optimization on the defect area using morphological operations to generate a defect image; If the internal welding feature value in the first feature set deviates from the preset welding quality standard, analyze the thermal distribution of the welding area through infrared spectrum data to obtain welding quality evaluation data; Integrate the defect image and the welding quality evaluation data through a data fusion algorithm to generate a third image set containing surface defect and internal welding information; Use visualization technology to render the third image set and output an image set of defects and welding information of the financial payment terminal.

3. The method according to claim 1, wherein: The process of performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set includes: Perform denoising processing on the first image set using a bilateral filtering algorithm to generate an intermediate image set; Perform contrast enhancement processing on the intermediate image set using an adaptive histogram equalization method to generate a second image set; Obtain the resolution parameters of the second image set. If the resolution parameters meet the preset resolution threshold, it is determined as the candidate image set; For the candidate image set, use the Laplacian operator to calculate the image sharpness. If the sharpness value is greater than the preset sharpness threshold, it is determined as the training data; Based on the training data, perform feature extraction, and use the convolutional neural network algorithm for pre-training to generate the initial model parameters; According to the initial model parameters, perform batch verification on the second image set to obtain the verification error. If the verification error is lower than the preset error threshold, it is determined that the model parameters are valid; According to the valid model parameters, perform automated processing on the subsequent input image set to generate the required training data.

4. The method according to claim 1, wherein The process of extracting the texture shape features including bubbles, foreign objects, and poor soldering based on the second image set to generate the feature set includes: Obtain image data based on the second image set, and generate a denoised image through preprocessing; Use a convolutional neural network to extract the texture shape features of bubbles, foreign objects, and poor soldering from the denoised image to obtain the initial feature set; If the dimension of the initial feature set is higher than the preset threshold, perform dimensionality reduction through principal component analysis to generate the optimized feature set; According to the optimized feature set, calculate the texture shape feature vectors of bubbles, foreign objects, and poor soldering, and determine the feature distribution; Through the feature distribution, use a clustering algorithm to classify bubbles, foreign objects, and poor soldering to obtain the classification result; Based on the classification result, obtain the texture shape feature subset of each type of defect to generate the final feature set; For the final feature set, construct a feature database and output the feature set data.

5. The method according to claim 1, wherein The process of constructing a dedicated convolutional neural network model for the plastic housing and the metal circuit board according to the feature set and generating the first model set adapted to the material differences through transfer learning includes: Use a convolutional neural network to perform image processing on the feature set, extract the features of the plastic housing and the metal circuit board to obtain the first feature set; If the dimension of the first feature set exceeds the preset threshold, generate the second feature set through dimensionality reduction processing, otherwise, directly determine the second feature set as the first feature set; Perform model training on the second feature set through transfer learning to generate the first model set adapted to the material differences; Use the classification accuracy to evaluate the performance of the first model set to obtain the first evaluation result; If the first evaluation result is lower than the preset threshold, re-train by adjusting the convolutional neural network parameters to obtain the second model set, otherwise, determine the first model set as the final model; Classify the newly input material feature set through the second model set to obtain the classification result.

6. The method according to claim 1, wherein The process of optimizing the first model set by applying data augmentation and L2 regularization algorithm to generate the second model set includes: Perform preprocessing on the first model set through data augmentation to generate the augmented data set; Apply L2 regularization to the augmented data set to optimize the first model set to obtain the second model set; Obtain the accuracy and recall rate of the second model set, calculate the performance metrics. If both the accuracy and recall rate exceed the preset threshold, determine that the second model set is the final detection model; Perform performance confirmation on the final detection model through cross-validation to obtain the verification result. Adjust the data augmentation parameters according to the verification result to generate an optimized model set, and extract the final detection model based on the optimized model set.

7. The method according to claim 1, wherein The process of performing defect detection on the real-time multi-spectral image through the second model set to generate a defect set including defect positions, types, and severities includes: Obtain a real-time multi-spectral image through a multi-spectral sensor to generate a first image set; Use a preprocessing algorithm to denoise and standardize the first image set to obtain a second image set; Perform defect detection on the second image set through the second model set to determine the defect positions, types, and severities, and generate a first defect set; If the severity of the defects in the first defect set exceeds the preset threshold, perform local enhancement processing on the corresponding image regions to obtain a third image set; Perform defect detection on the third image set again through the second model set to update the defect positions, types, and severities, and obtain a second defect set; According to the defect types in the second defect set, use a classification algorithm to sort the defects by priority to obtain a sorted defect set; Format the defect positions, types, and severities in the sorted defect set to generate a final defect set.

8. The method according to claim 1, wherein The process of constructing a defect feature database based on the defect set and determining it as a matchable data set if the defect feature database contains fields of position, type, and severity includes: Obtain defect position, defect type, and severity data according to the defect set, and use a data cleaning method to remove duplicate and missing values to obtain structured defect data; Through the structured defect data, construct a feature database including fields of position, type, and severity, and use a relational database to manage data storage to obtain a database structure; If the fields of the feature database include position, type, and severity, use a query statement to extract records that meet the conditions to obtain a preliminary matching data set; According to the preliminary matching data set, use the K-nearest neighbor algorithm to perform clustering analysis on the defect positions and types to obtain a classified matching data set; Obtain severity data from the classified matching data set, and use a sorting algorithm to sort it in descending order of severity to obtain a priority sorted data set; Through the priority sorted data set, use a data screening method to extract records with a severity higher than the preset threshold to determine the final matching data set; Extract the defect positions and types from the final matching data set, and use a data mapping method to generate defect distribution features to obtain a defect feature analysis result.

9. The method according to claim 1, wherein The process of performing feature matching on the defect feature database using a vector retrieval algorithm to generate a matching set includes: Query the defect feature database using a vector retrieval algorithm, calculate the similarity between the input feature vector and the feature vectors in the database, and obtain a preliminary matching set; If the similarity value in the preliminary matching set is greater than a preset threshold, retain the corresponding feature vector; otherwise, discard it to generate a filtered matching set; According to the filtered matching set, obtain the defect category information corresponding to each feature vector in the matching set to generate a classification result set; Through the classification result set, determine the occurrence frequency of each defect category to obtain frequency distribution data; Use a clustering algorithm to analyze the frequency distribution data, obtain the grouping pattern of defect categories, and generate a final defect analysis result set; Store the final defect analysis result set in a preset database to generate queryable analysis records.

10. A product defect detection system based on multispectral imaging, characterized in that, Including: A data acquisition module for acquiring multi-spectral image data of a financial payment terminal product and generating a first image set containing surface defects and internal welding information; An image processing module for performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; Judge whether the resolution and clarity of the second image set meet the preset requirements. If they meet the preset requirements, determine it as training data; A feature extraction module for extracting texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set; A model construction module for constructing a dedicated convolutional neural network model for plastic shells and metal circuit boards according to the feature set, and generating a first model set adapted to material differences through transfer learning; A model optimization module for optimizing the first model set by applying data augmentation and L2 regularization algorithms to generate a second model set. If the accuracy and recall rate of the second model set exceed the preset threshold, determine it as the final detection model; A defect detection module for detecting defects in real-time multi-spectral images through the second model set to generate a defect set containing defect locations, types, and severities; A database construction module for constructing a defect feature database based on the defect set. If the defect feature database contains fields such as location, type, and severity, determine it as a matchable data set; A feature matching module for performing feature matching on the defect feature database using a vector retrieval algorithm to generate a matching set; A traceability analysis module for generating a traceability data set containing corresponding relationships in the production process according to the matching set by applying an association rule mining algorithm.

Images