Product defect detection method and system based on multispectral imaging

Abstract

The invention discloses a product defect detection method and system based on multispectral imaging, and the method comprises the steps: obtaining multispectral image data of a financial payment terminal product, and generating a first image set; performing de-noising filtering and contrast enhancement processing on the first image set to generate a second image set; performing feature extraction based on the second image set to generate a feature set; and constructing a special convolutional neural network model of the plastic shell and the metal circuit board according to the feature set, generating a model set adaptive to material difference through transfer learning, and determining a final detection model to perform defect detection on the real-time multispectral image to obtain a detection result. The accuracy and efficiency of financial payment terminal product quality detection are improved, data support is provided for production process optimization, the product defect rate is effectively reduced, and the product quality and reliability are improved.

Description

Technical Field

[0001] The present invention belongs to the field of product defect detection, and particularly relates to a product defect detection method and system based on multispectral imaging. Background Art

[0002] The quality inspection of financial payment terminal products is a key link to ensure their reliability and security, and plays a crucial role in the financial industry. Product defects may lead to equipment failures or safety hazards, directly affecting user trust and industry development. Traditional inspection methods mainly rely on manual visual inspection, which is inefficient and easily interfered by subjective factors, and it is difficult to detect hidden defects such as internal bubbles or poor soldering. Although single visible light imaging technology has certain applications, it is limited by a single wavelength band and is difficult to comprehensively capture multi-dimensional information on the surface and inside of products, and there are deficiencies in both detection accuracy and coverage.

[0003] The limitations of existing methods have prompted research to turn to more intelligent and comprehensive detection technologies. The detection method combining multispectral imaging and deep learning has become a new exploration direction. Multispectral imaging can obtain rich visual information in multiple wavelength bands, while deep learning can extract defect features from complex data. However, the implementation of this method faces core technical challenges. First, how to construct an adapted multispectral imaging detection model for different materials and functional components of financial payment terminal products is a difficult point for achieving accurate detection. Second, how to optimize deep learning algorithms to efficiently identify tiny defects such as bubbles, foreign objects or poor soldering, while ensuring the generalization ability of the model, is the key to improving the detection effect. In addition, the establishment of a defect feature database and a fast matching mechanism are not yet mature, which limits the efficiency of defect traceability. Therefore, how to effectively integrate multispectral imaging and deep learning, construct a dedicated detection model for different materials and components, optimize the algorithm to achieve accurate and efficient identification of tiny defects, and establish a defect feature database to support fast matching and traceability has become a key problem that needs to be solved urgently. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a product defect detection method and system based on multispectral imaging. Among them, a product defect detection method based on multispectral imaging includes:

[0005] Obtain multispectral image data of a financial payment terminal product, and generate a first image set containing surface defects and internal soldering information;

[0006] Perform denoising filtering and contrast enhancement processing on the first image set to generate a second image set; determine whether the resolution and clarity of the second image set meet the preset requirements, and if they meet the preset requirements, determine it as training data;

[0007] Extract texture shape features including bubbles, foreign objects, and soldering defects from the second image set to generate a feature set;

[0008] Construct a dedicated convolutional neural network model for the plastic housing and metal circuit board based on the feature set, and generate a first model set adapted to material differences through transfer learning;

[0009] Apply data augmentation and L2 regularization algorithms to optimize the first model set to generate a second model set. If the accuracy and recall rate of the second model set exceed the preset threshold, it is determined as the final detection model;

[0010] Perform defect detection on the real-time multi-spectral image through the second model set to generate a defect set containing defect locations, types, and severities;

[0011] Construct a defect feature database based on the defect set. If the defect feature database contains fields for location, type, and severity, it is determined as a matchable data set; use a vector retrieval algorithm to perform feature matching on the defect feature database to generate a matching set; apply an association rule mining algorithm to the matching set to generate a traceability data set containing the corresponding relationships of production links.

[0012] Preferably, the process of obtaining multi-spectral image data of the financial payment terminal product and generating a first image set containing surface defects and internal soldering information includes:

[0013] Scan the financial payment terminal through a multi-spectral imaging device to collect image data containing visible light and infrared spectra, and generate a first multi-spectral image set;

[0014] Use a convolutional neural network to perform feature extraction on the first multi-spectral image set, separate surface defect features and internal soldering features, and obtain a first feature set;

[0015] If the surface defect feature value in the first feature set exceeds the preset threshold, partition the surface area through an image segmentation algorithm to obtain a surface defect localization image;

[0016] According to the surface defect localization image, perform boundary optimization on the defect area using morphological operations to generate a defect image;

[0017] If the internal soldering feature value in the first feature set deviates from the preset soldering quality standard, analyze the thermal distribution of the soldering area through infrared spectrum data to obtain soldering quality evaluation data;

[0018] Integrate the defect image and the soldering quality evaluation data through a data fusion algorithm to generate a third image set containing surface defects and internal soldering information;

[0019] Render the third image set using visualization technology to output an image set of defects and welding information of the financial payment terminal.

[0020] Preferably, the process of denoising filtering and contrast enhancement processing on the first image set to generate the second image set includes:

[0021] Perform denoising processing on the first image set using a bilateral filtering algorithm to generate an intermediate image set;

[0022] Perform contrast enhancement processing on the intermediate image set using an adaptive histogram equalization method to generate the second image set;

[0023] Obtain the resolution parameter of the second image set. If the resolution parameter meets the preset resolution threshold, it is determined as a candidate image set;

[0024] For the candidate image set, calculate the image sharpness using a Laplacian operator. If the sharpness value is greater than the preset sharpness threshold, it is determined as training data;

[0025] Extract features based on the training data and perform pre-training using a convolutional neural network algorithm to generate initial model parameters;

[0026] According to the initial model parameters, perform batch verification on the second image set to obtain a verification error; if the verification error is lower than the preset error threshold, it is determined that the model parameters are valid;

[0027] According to the valid model parameters, perform automated processing on the subsequent input image set to generate training data that meets the requirements.

[0028] Preferably, the process of extracting texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set includes:

[0029] Obtain image data based on the second image set and generate a denoised image through preprocessing;

[0030] Use a convolutional neural network to extract texture shape features of bubbles, foreign objects, and poor welding from the denoised image to obtain an initial feature set;

[0031] If the dimension of the initial feature set is higher than the preset threshold, perform dimensionality reduction through principal component analysis to generate an optimized feature set;

[0032] According to the optimized feature set, calculate the texture shape feature vectors of bubbles, foreign objects, and poor welding, and determine the feature distribution;

[0033] Through the feature distribution, use a clustering algorithm to classify bubbles, foreign objects, and poor welding to obtain a classification result;

[0034] Obtain the texture shape feature subset of each type of defect based on the classification result, and generate the final feature set;

[0035] Construct a feature database for the final feature set and output the feature set data.

[0036] Preferably, the process of constructing a dedicated convolutional neural network model for the plastic housing and the metal circuit board according to the feature set and generating the first model set adapted to the material difference through transfer learning includes:

[0037] Perform image processing on the feature set using a convolutional neural network, extract the features of the plastic housing and the metal circuit board, and obtain the first feature set;

[0038] If the dimension of the first feature set exceeds a preset threshold, generate a second feature set through dimensionality reduction processing; otherwise, directly determine that the second feature set is the first feature set;

[0039] Perform model training on the second feature set through transfer learning to generate the first model set adapted to the material difference;

[0040] Evaluate the performance of the first model set using classification accuracy to obtain the first evaluation result;

[0041] If the first evaluation result is lower than the preset threshold, retrain by adjusting the parameters of the convolutional neural network to obtain the second model set; otherwise, determine that the first model set is the final model;

[0042] Classify the newly input material feature set through the second model set to obtain the classification result.

[0043] Preferably, the process of optimizing the first model set by applying data augmentation and L2 regularization algorithm to generate the second model set includes:

[0044] Perform preprocessing on the first model set through data augmentation to generate an augmented data set;

[0045] Apply L2 regularization to the augmented data set to optimize the first model set and obtain the second model set;

[0046] Obtain the accuracy rate and recall rate of the second model set, calculate the performance metrics. If both the accuracy rate and recall rate exceed the preset threshold, determine that the second model set is the final detection model;

[0047] Perform performance confirmation on the final detection model through cross-validation to obtain the verification result, adjust the data augmentation parameters according to the verification result to generate an optimized model set, and extract the final detection model based on the optimized model set.

[0048] Preferably, the process of defect detection on the real-time multispectral image by the second model set to generate a defect set including defect positions, types, and severities includes:

[0049] Obtain a real-time multispectral image through a multispectral sensor to generate a first image set;

[0050] Use a preprocessing algorithm to denoise and standardize the first image set to obtain a second image set;

[0051] Perform defect detection on the second image set through the second model set to determine defect positions, types, and severities, and generate a first defect set;

[0052] If the severity of the defects in the first defect set exceeds a preset threshold, perform local enhancement processing on the corresponding image regions to obtain a third image set;

[0053] Perform defect detection on the third image set again through the second model set to update defect positions, types, and severities, and obtain a second defect set;

[0054] According to the defect types in the second defect set, use a classification algorithm to rank the defects by priority to obtain a ranked defect set;

[0055] Format the defect positions, types, and severities in the ranked defect set to generate a final defect set.

[0056] Preferably, based on the defect set, construct a defect feature database. If the defect feature database contains fields of position, type, and severity, the process of determining it as a matchable data set includes:

[0057] Obtain defect position, defect type, and severity data according to the defect set, and use a data cleaning method to remove duplicate and missing values to obtain structured defect data;

[0058] Through the structured defect data, construct a feature database including fields of position, type, and severity, and use a relational database to manage data storage to obtain a database structure;

[0059] If the fields of the feature database include position, type, and severity, use a query statement to extract records that meet the conditions to obtain a preliminary matching data set;

[0060] According to the preliminary matching data set, use the K-nearest neighbor algorithm to perform clustering analysis on defect positions and types to obtain a classified matching data set;

[0061] Obtain severity data from the classified matching data set, and use a sorting algorithm to sort it in descending order of severity to obtain a priority ranking data set;

[0062] Through the priority sorted data set, a data screening method is used to extract records with severity higher than a preset threshold to determine the final matching data set;

[0063] Extract the defect location and type from the final matching data set, and use a data mapping method to generate defect distribution characteristics to obtain the defect feature analysis result.

[0064] Preferably, a vector retrieval algorithm is used to perform feature matching on the defect feature database. The process of generating the matching set includes:

[0065] Use a vector retrieval algorithm to query the defect feature database, calculate the similarity between the input feature vector and the feature vectors in the database to obtain a preliminary matching set;

[0066] If the similarity value in the preliminary matching set is greater than the preset threshold, retain the corresponding feature vector, otherwise discard it to generate a filtered matching set;

[0067] According to the filtered matching set, obtain the defect category information corresponding to each feature vector in the matching set to generate a classification result set;

[0068] Through the classification result set, determine the occurrence frequency of each defect category to obtain frequency distribution data;

[0069] Use a clustering algorithm to analyze the frequency distribution data, obtain the grouping pattern of defect categories, and generate the final defect analysis result set;

[0070] Store the final defect analysis result set in a preset database to generate queryable analysis records.

[0071] The present invention also provides a product defect detection system based on multispectral imaging, including:

[0072] A data acquisition module for acquiring multispectral image data of a financial payment terminal product to generate a first image set containing surface defects and internal welding information;

[0073] An image processing module for performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; judge whether the resolution and clarity of the second image set meet the preset requirements. If they meet the preset requirements, determine it as training data;

[0074] A feature extraction module for extracting texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set;

[0075] A model construction module, configured to construct a dedicated convolutional neural network model for plastic shells and metal circuit boards based on the feature set, and generate a first model set adapted to material differences through transfer learning;

[0076] A model optimization module, configured to optimize the first model set by applying data augmentation and L2 regularization algorithms, generate a second model set, and if the accuracy and recall rate of the second model set exceed a preset threshold, determine it as the final detection model;

[0077] A defect detection module, configured to perform defect detection on real-time multi-spectral images through the second model set, and generate a defect set including defect positions, types, and severities;

[0078] A database construction module, configured to construct a defect feature database based on the defect set, and if the defect feature database includes fields of position, type, and severity, determine it as a matchable data set;

[0079] A feature matching module, configured to perform feature matching on the defect feature database by using a vector retrieval algorithm, and generate a matching set;

[0080] A traceability analysis module, configured to apply an association rule mining algorithm according to the matching set, and generate a traceability data set including corresponding relationships of production links.

[0081] Compared with the prior art, the present invention has the following advantages and technical effects:

[0082] The present invention realizes defect detection of plastic shells and metal circuit boards by acquiring multi-spectral image data, performing image processing and feature extraction, and constructing a dedicated convolutional neural network model.

[0083] The present invention uses transfer learning to generate models adapted to different materials, and optimizes the model performance through data augmentation and regularization. In real-time detection, the present invention can identify defect positions, types, and severities, and construct a defect feature database. Through vector retrieval and association rule mining, the traceability of the corresponding relationship between defects and production links is realized.

[0084] The present invention improves the accuracy and efficiency of quality detection of financial payment terminal products, provides data support for production process optimization, effectively reduces the product defect rate, and improves the product quality and reliability.

[0085] The present invention can effectively detect surface and internal defects of financial payment terminal products, including bubbles, foreign objects, and poor soldering, etc., and can trace the production links where the defects occur, improving the efficiency and accuracy of product quality control and production process management. Description of the Drawings

[0086] The accompanying drawings, which form a part of this application, are used to provide a further understanding of this application. The schematic embodiments and descriptions thereof of this application are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0087] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention;

[0088] Figure 2 It is a schematic structural diagram of the system according to an embodiment of the present invention. Detailed Embodiments

[0089] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the accompanying drawings and combine with the embodiments to detail this application.

[0090] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0091] Embodiment 1

[0092] As Figure 1 shown, in this embodiment, a product defect detection method based on multispectral imaging is provided, including:

[0093] Obtain multispectral image data of a financial payment terminal product, and generate a first image set including surface defects and internal welding information;

[0094] Perform denoising filtering and contrast enhancement processing on the first image set to generate a second image set; judge whether the resolution and clarity of the second image set meet the preset requirements. If they meet the preset requirements, it is determined as training data;

[0095] Extract texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set;

[0096] Construct a dedicated convolutional neural network model for the plastic shell and the metal circuit board according to the feature set, and generate a first model set adapted to the material differences through transfer learning;

[0097] Apply data augmentation and L2 regularization algorithms to optimize the first model set to generate a second model set. If the accuracy and recall rate of the second model set exceed the preset threshold, it is determined as the final detection model;

[0098] Perform defect detection on the real-time multispectral image through the second model set to generate a defect set including the defect location, type, and severity;

[0099] Construct a defect feature database based on the defect set. If the defect feature database contains fields such as location, type, and severity, it is determined as a matchable data set. Use a vector retrieval algorithm to perform feature matching on the defect feature database to generate a matching set. Apply an association rule mining algorithm based on the matching set to generate a traceability data set containing the corresponding relationships of production links.

[0100] Further, the process of obtaining multi-spectral image data of a financial payment terminal product and generating a first image set containing surface defects and internal welding information includes:

[0101] Scan the financial payment terminal through a multi-spectral imaging device to collect image data containing visible light and infrared spectra, and generate a first multi-spectral image set;

[0102] Use a convolutional neural network to extract features from the first multi-spectral image set, separate the surface defect features and internal welding features, and obtain a first feature set;

[0103] If the surface defect feature value in the first feature set exceeds a preset threshold, partition the surface area through an image segmentation algorithm to obtain a surface defect localization image;

[0104] According to the surface defect localization image, perform boundary optimization on the defect area using morphological operations to generate a defect image;

[0105] If the internal welding feature value in the first feature set deviates from the preset welding quality standard, analyze the thermal distribution of the welding area through infrared spectrum data to obtain welding quality evaluation data;

[0106] Integrate the defect image and the welding quality evaluation data through a data fusion algorithm to generate a third image set containing surface defects and internal welding information;

[0107] Use visualization technology to render the third image set and output an image set of defects and welding information of the financial payment terminal.

[0108] Specifically, when acquiring the multispectral image data of financial payment terminal products, first, a high-resolution multispectral camera (such as Specim IQ, spectral range 400 - 1000 nm, resolution 512×512 pixels) is used to scan the terminal surface. The exposure time is set to 10 ms, the band interval is 5 nm, and spectral data of 120 bands are collected in total. The original image is preprocessed by an adaptive threshold segmentation algorithm (Otsu algorithm) to remove background noise and retain the valid region. For surface defect detection, a YOLOv5 model based on deep learning is adopted. The input image size is adjusted to 640×640. During training, the learning rate is set to 0.001, the batch size is 16, and 100 epochs are iterated. Fine-tuning is performed using transfer learning based on the pre-trained weights of the COCO dataset. Finally, the recognition accuracy of defects such as scratches and dents reaches 98.5%. For internal welding information, X-ray imaging technology (voltage 80 kV, current 5 mA) is used to penetrate the shell to obtain the welding point image. Convolutional neural network (CNN) is combined for feature extraction. The network structure includes 3 convolutional layers (convolution kernel size 3×3, stride 1, padding 1) and 2 fully connected layers. The model performance is improved through the ReLU activation function and batch normalization. The detection accuracy of defects such as solder joint voids and cold soldering reaches 97.2%. When fusing the surface defect and internal welding data to generate the first image set, an image registration algorithm (SIFT feature matching, RANSAC to remove mismatched points) is used to align the multi-source images, and different resolution information is fused through a multi-scale feature pyramid (FPN). Finally, a comprehensive image dataset containing defect annotations and welding quality ratings is output, providing a complete input for subsequent quality analysis.

[0109] Further, the process of denoising filtering and contrast enhancement for the first image set to generate the second image set includes:

[0110] The bilateral filtering algorithm is used to denoise the first image set to generate an intermediate image set;

[0111] The adaptive histogram equalization method is used to enhance the contrast of the intermediate image set to generate the second image set;

[0112] The resolution parameters of the second image set are obtained. If the resolution parameters meet the preset resolution threshold, it is determined as a candidate image set;

[0113] For the candidate image set, the Laplacian operator is used to calculate the image sharpness. If the sharpness value is greater than the preset sharpness threshold, it is determined as training data;

[0114] Feature extraction is performed based on the training data, and pre-training is carried out using the convolutional neural network algorithm to generate initial model parameters;

[0115] According to the initial model parameters, batch verification is performed on the second image set to obtain the verification error. If the verification error is lower than the preset error threshold, it is determined that the model parameters are valid.

[0116] According to the valid model parameters, automated processing is performed on the subsequent input image set to generate qualified training data.

[0117] Specifically, when performing denoising and filtering on the first image set in this embodiment, the non-local means (NLM) algorithm is used. The search window is set to 21×21 pixels, the similar block window is 7×7 pixels, and the filtering parameter h = 0.12×σ (σ is the noise standard deviation). Noise suppression is achieved by calculating the weighted similarity of pixel neighborhoods. For contrast enhancement, the contrast-limited adaptive histogram equalization (CLAHE) method is used. The clipping limit is set to 2.0, and the grid division size is 8×8. The image details are enhanced through local histogram equalization. The quality of the processed second image set is evaluated by the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics. If the PSNR value is greater than 30 dB and the SSIM is higher than 0.92, it is determined that the resolution and clarity meet the standards.

[0118] For example, when inputting a 512×512 POS machine product image, after NLM denoising, the noise standard deviation drops from 25 to 8. After CLAHE processing, the average edge gradient increases by 40%. At this time, the PSNR is 32.5 dB and the SSIM is 0.94, meeting the preset threshold. The system automatically marks it as qualified training data. During the process, bicubic interpolation is used to maintain the original resolution, and the Sobel operator is used to detect the edge sharpness to ensure that no artifacts are introduced in the enhanced image.

[0119] Furthermore, texture shape features including bubbles, foreign objects, and poor soldering are extracted based on the second image set. The process of generating the feature set includes:

[0120] Obtain image data based on the second image set and generate a denoised image through preprocessing.

[0121] Use a convolutional neural network to extract texture shape features of bubbles, foreign objects, and poor soldering from the denoised image to obtain an initial feature set.

[0122] If the dimension of the initial feature set is higher than the preset threshold, dimensionality reduction is performed through principal component analysis to generate an optimized feature set.

[0123] According to the optimized feature set, calculate the texture shape feature vectors of bubbles, foreign objects, and poor soldering to determine the feature distribution.

[0124] Through the feature distribution, use a clustering algorithm to classify bubbles, foreign objects, and poor soldering to obtain the classification result.

[0125] Obtain the texture shape feature subset of each type of defect based on the classification result, and generate the final feature set;

[0126] For the final feature set, construct a feature database and output the feature set data.

[0127] Specifically, in this embodiment, the texture shape features including bubbles, foreign objects, and poor soldering are extracted from the second image set. First, the Gaussian filtering algorithm is used to preprocess the image, with a standard deviation set to 1.5 to smooth the image and reduce noise interference. Then, the Canny edge detection algorithm is used to extract the edge information in the image, with a low threshold set to 50 and a high threshold set to 150 to accurately capture the contours of bubbles, foreign objects, and poor soldering. Next, through the opening and closing operations in morphological operations, a 3×3 structuring element is used respectively to remove small-area noise and fill holes to enhance the connectivity of the feature regions. On this basis, the Local Binary Pattern (LBP) algorithm is used to extract texture features, with a radius set to 2 and the number of sampling points set to 8 to quantify the local texture information of the image. For the shape features, the Hu moment algorithm is used to calculate the seven invariant moments of the image region to describe the shape characteristics of bubbles, foreign objects, and poor soldering. Finally, the extracted texture and shape features are normalized, and the Z-score normalization method is used to convert the feature values into a distribution with a mean of 0 and a standard deviation of 1 to generate the feature set. Through the above steps, the accuracy and robustness of the feature set are ensured, providing a reliable data basis for subsequent defect detection and classification.

[0128] The above steps generate a denoised image, an initial feature set, an optimized feature set, a feature distribution, a classification result, and a final feature set in sequence through the logical chain of image processing, feature extraction, and classification, ensuring that the output of each step serves as the input of the next step to form a tight logical relationship.

[0129] Furthermore, the process of constructing a dedicated convolutional neural network model for plastic casings and metal circuit boards and generating the first model set adapted to material differences through transfer learning includes:

[0130] Use a convolutional neural network to perform image processing on the feature set, extract the features of plastic casings and metal circuit boards, and obtain the first feature set;

[0131] If the dimension of the first feature set exceeds the preset threshold, generate the second feature set through dimensionality reduction processing; otherwise, directly determine the second feature set as the first feature set;

[0132] Perform model training on the second feature set through transfer learning to generate the first model set adapted to material differences;

[0133] Use classification accuracy to evaluate the performance of the first model set and obtain the first evaluation result;

[0134] If the first evaluation result is lower than the preset threshold, retrain by adjusting the parameters of the convolutional neural network to obtain a second model set; otherwise, determine the first model set as the final model.

[0135] Classify the newly input material feature set through the second model set to obtain a classification result.

[0136] Specifically, when constructing a dedicated convolutional neural network model for plastic casings and metal circuit boards, a feature extraction layer is first designed to adapt to material differences.

[0137] More specifically, taking the example that ResNet50 is used as the basic architecture in this embodiment, its input size is adjusted to 224×224×3, and a batch normalization layer and a ReLU activation function are added after the first layer of convolution. For the texture features of plastic casings, a 3×3 convolutional kernel is used to extract local details, the stride is set to 1, and the number of output channels is 64. For the high-reflection characteristics of metal circuit boards, a 5×5 convolutional kernel is added to capture a larger range of reflection patterns, and the number of output channels is increased to 128. Through transfer learning, the weights pre-trained on ImageNet are loaded, the parameters of the first 15 layers are frozen to retain the general feature extraction ability, and the subsequent layers are adjusted using an adaptive learning rate. The initial learning rate is set to 0.001, and the Adam optimizer is used, with the momentum parameter β 1 = 0.9, β 2 = 0.999. In the material difference adaptation stage, the domain adaptation loss function MMD (Maximum Mean Discrepancy) is introduced to calculate the distance between the plastic and metal feature distributions. The kernel function selects a Gaussian kernel, the bandwidth parameter σ = 1.0, and the loss weight coefficient λ = 0.5. When training the model, the batch size is set to 32, and it iterates for 100 epochs. The accuracy is verified every 10 epochs. When the loss of the validation set does not decrease for 5 consecutive times, the early stopping mechanism is triggered. The final model achieves a classification accuracy of 92.3% on the test set, where the recognition rate of plastic casings is 94.1%, the recognition rate of metal circuit boards is 90.5%, and the F1-score is 0.916. Visualized by Gradient-weighted Class Activation Mapping (Grad-CAM), the model shows significant attention to the scratches on the plastic surface and the solder joint areas of the metal, proving the effectiveness of feature extraction.

[0138] Furthermore, the process of applying data augmentation and the L2 regularization algorithm to optimize the first model set to generate the second model set includes:

[0139] Preprocess the first model set through data augmentation to generate an augmented data set;

[0140] Apply L2 regularization to the augmented data set to optimize the first model set and obtain the second model set;

[0141] Obtain the accuracy and recall rate of the second model set, calculate the performance metrics. If both the accuracy and recall rate exceed the preset threshold, determine the second model set as the final detection model;

[0142] Perform performance verification on the final detection model through cross-validation to obtain the verification result. Adjust the data augmentation parameters according to the verification result to generate an optimized model set, and extract the final detection model based on the optimized model set.

[0143] Specifically, first, use data augmentation methods of random rotation, translation, and adding Gaussian noise to the first model set. The rotation angle range is ±15 degrees, the translation amplitude is 10% of the image size, and the standard deviation of Gaussian noise is set to 0.05. Implement batch processing through the warpAffine and GaussianBlur functions of the OpenCV library to expand the original 5000 training samples to 20000.

[0144] Then, introduce L2 regularization in the model training stage, set the penalty coefficient of λ = 0.01, use the tf.keras.regularizers.l2 module of TensorFlow to embed into the fully connected layer, and adopt the Adam optimizer (learning rate 0.001, β 1 = 0.9, β 2 = 0.999) for 300 rounds of iterative training, with 128 images processed in each batch. During the training process, monitor the validation set loss through the early stopping mechanism and terminate the training when it does not decrease for 20 consecutive rounds. After training is completed, evaluate on 1000 independent test sets. The accuracy of the second model set reaches 92.3% (a 4.7% increase compared to the first model set), and the recall rate is 89.1% (a 5.2% increase), exceeding the preset accuracy threshold of 90% and recall rate threshold of 85%. At this time, use the model with the highest F1-score on the validation set (F1 = 0.907) as the final detection model. Its confusion matrix shows that the false positive rate of positive class samples is 3.8%, and the false negative rate is 6.2%. Save the model weight file in HDF5 format and deploy it to the inference server.

[0145] Furthermore, the process of generating a defect set containing defect positions, types, and severity levels by performing defect detection on real-time multi-spectral images through the second model set includes:

[0146] Obtain real-time multi-spectral images through a multi-spectral sensor to generate a first image set;

[0147] Adopt a preprocessing algorithm to denoise and standardize the first image set to obtain a second image set;

[0148] Perform defect detection on the second image set through the second model set to determine the defect positions, types, and severity levels, and generate a first defect set;

[0149] If the severity of the defects in the first defect set exceeds a preset threshold, local enhancement processing is performed on the corresponding image region to obtain a third image set;

[0150] Defect detection is performed on the third image set again through the second model set to update the defect positions, types, and severities, obtaining a second defect set;

[0151] According to the defect types in the second defect set, a classification algorithm is used to sort the defects by priority, obtaining a sorted defect set;

[0152] Format the defect positions, types, and severities in the sorted defect set to generate a final defect set.

[0153] Specifically, when performing defect detection on real-time multi-spectral images through the second model set in this embodiment, an improved algorithm based on YOLOv5 is first adopted. Its backbone network is replaced with EfficientNet-B4 to improve the feature extraction ability. The input image resolution is 640×640 pixels, and it is processed synchronously in 5 bands (450nm, 550nm, 650nm, 750nm, 850nm).

[0154] The model uses the FocalLoss loss function during the training phase, sets the initial learning rate to 0.001, the batchsize to 16, and after 300 epochs of training, it reaches mAP@0.5 of 92.3% on the validation set. For defect localization, the non-maximum suppression (NMS) algorithm is adopted, with the IoU threshold set to 0.45 and the confidence threshold set to 0.6 to ensure the balance between the detection rate and the false detection rate.

[0155] In the defect classification link, a secondary classifier constructed by ResNet-34 performs fine-grained analysis on the localization region, uses the cross-entropy loss function, and the classification accuracy for 6 types of typical defects (cracks, bubbles, scratches, stains, material shortage, color difference) reaches 89.7%.

[0156] Severity assessment integrated gradient boosting decision tree (GBDT) algorithm. The input features include the standard deviation of the pixels in the defect area (e.g., the standard deviation threshold for crack-like defects is set to 15.8), the area ratio (e.g., if the area of a bubble defect exceeds 0.5% of the total area, it is determined to be severe), and the difference in multispectral reflectance (e.g., if the difference between the 650nm and 850nm bands is greater than 35, it is determined to be severe color difference). The final output defect set is stored in JSON format, including pixel coordinates (e.g., x_min: 120, y_min: 345, x_max: 210, y_max: 420), defect type code (e.g., C02 represents a bubble), and severity level (1 - 5 levels). The entire processing process is controlled within 200ms and real-time detection is achieved through CUDA11.1 acceleration.

[0157] Furthermore, based on the defect set, a defect feature database is constructed. If the defect feature database contains fields such as location, type, and severity, the process of determining it as a matchable data set includes:

[0158] Obtain defect location, defect type, and severity data from the defect set, and use data cleaning methods to remove duplicate and missing values to obtain structured defect data;

[0159] Based on the structured defect data, construct a feature database containing fields such as location, type, and severity, and use a relational database to manage data storage to obtain the database structure;

[0160] If the fields of the feature database include location, type, and severity, use query statements to extract records that meet the conditions to obtain a preliminary matching data set;

[0161] Based on the preliminary matching data set, use the K-nearest neighbor algorithm to perform clustering analysis on the defect location and type to obtain a classified matching data set;

[0162] Obtain severity data from the classified matching data set, and use a sorting algorithm to sort it in descending order of severity to obtain a priority sorted data set;

[0163] Based on the priority sorted data set, use data screening methods to extract records with severity higher than the preset threshold to determine the final matching data set;

[0164] Extract the defect location and type from the final matching data set, and use data mapping methods to generate defect distribution features to obtain the defect feature analysis result.

[0165] Specifically, when constructing the defect feature database in this embodiment, it is first necessary to extract defect data from the code library. For example, by scanning a Java project using a static analysis tool such as SonarQube, obtain the original data including file name, line number, defect type, and severity (such as severe, general, minor).

[0166] For the location field, it can be stored as "src / main / Model.java:125", indicating that the defect is located on line 125 of the Model.java file.

[0167] The type field adopts standard classification. For example, CWE-79 represents cross-site scripting vulnerability, and CWE-89 represents SQL injection. The severity is quantified by CVSS score. For example, a score of 7.5 corresponds to a high-risk vulnerability.

[0168] In the data preprocessing stage, unstructured logs are parsed, and keyword fields are extracted using regular expressions. Then, a feature encoding algorithm is adopted to perform One-Hot encoding on the defect types to generate binary vectors. For example, [1,0,0] represents memory leak, and [0,1,0] represents resource not released.

[0169] The database is built using Elasticsearch to implement an inverted index. The location field is tokenized and stored, and the aggregation analysis capabilities of type and severity are established. When it is detected that the database contains these three core fields, the system automatically triggers the matching logic. For example, the similarity between the new defect and the records in the library is calculated through the cosine similarity algorithm, and a threshold of 0.85 is set as the condition for successful matching. For the numerical severity field, Z-score standardization is adopted to make it conform to the normal distribution, facilitating subsequent processing by machine learning models. During the whole process, data consistency check is realized through SHA-256 hash value verification to ensure the field integrity of each record.

[0170] Furthermore, a vector retrieval algorithm is adopted to perform feature matching on the defect feature database. The process of generating the matching set includes:

[0171] Query the defect feature database using a vector retrieval algorithm, calculate the similarity between the input feature vector and the feature vectors in the database, and obtain a preliminary matching set;

[0172] If the similarity value in the preliminary matching set is greater than the preset threshold, the corresponding feature vector is retained; otherwise, it is discarded to generate a filtered matching set;

[0173] According to the filtered matching set, obtain the defect category information corresponding to each feature vector in the matching set to generate a classification result set;

[0174] Through the classification result set, determine the occurrence frequency of each defect category to obtain frequency distribution data;

[0175] Adopt a clustering algorithm to analyze the frequency distribution data, obtain the grouping pattern of defect categories, and generate a final defect analysis result set;

[0176] Store the final defect analysis result set in a preset database to generate queryable analysis records.

[0177] Specifically, in the defect feature database, the cosine similarity is used as the vector retrieval algorithm for feature matching. First, each defect feature in the database is vectorized. For example, the TF-IDF algorithm is used to convert text features into numerical vectors. Suppose a certain defect feature vector is [0.85, 0.12, 0.45, 0.67]. Then, the defect feature to be matched is input and vectorized in the same way to obtain the input vector [0.78, 0.15, 0.50, 0.70]. Next, calculate the cosine similarity between the input vector and each feature vector in the database. The formula is cos(θ)=(A·B) / (||A||*||B||), where A and B are the input vector and the feature vector in the database respectively. Through calculation, suppose the cosine similarity between the input vector and a certain database vector is 0.92, which is higher than the preset threshold of 0.85, then it is considered that the two match successfully. Finally, generate a matching set for all successfully matched feature vectors. For example, the matching set contains vectors such as [0.85, 0.12, 0.45, 0.67] and [0.80, 0.10, 0.48, 0.65] for subsequent defect analysis and processing.

[0178] Furthermore, the process of generating a traceability data set containing the corresponding relationship of production links according to the matching set by applying the association rule mining algorithm includes:

[0179] Obtain the matching set data, remove redundant items through preprocessing to obtain a standardized data set;

[0180] Apply the association rule mining algorithm from the standardized data set to extract the association patterns between production links and determine the link association strength;

[0181] According to the link association strength, if the association strength exceeds the preset threshold, then retain the corresponding rules to obtain a preliminary rule set;

[0182] Through the preliminary rule set, combined with the production process sequence, generate an intermediate traceability set containing link association relationships;

[0183] For the intermediate traceability set, apply data matching accuracy verification. If the matching accuracy is lower than the preset threshold, then iteratively optimize the rule extraction efficiency to obtain an optimized traceability set;

[0184] Extract integrity features from the optimized traceability set to generate a traceability set and determine the integrity of the final traceability set;

[0185] According to the traceability set, verify the accuracy of the production link association relationship to obtain the final corresponding relationship of the production link. Specifically, in the implementation process of the association rule mining algorithm, it is first necessary to construct a matching set of production links. For example, extract records containing parameters such as temperature, pressure, and time from the production log to form a transaction data set. Assume that the data set contains 1000 records, and each record consists of 5 production link parameters, such as temperature range [150°C, 200°C], pressure value [0.8 MPa, 1.2 MPa], etc. Use the Apriori algorithm for frequent item set mining, set the minimum support to 0.1 and the minimum confidence to 0.7, and generate candidate item sets by scanning the data set. For example, the first scan obtains the frequent 1-item set {temperature = 80°C} (support 0.15), {pressure = 1.0 MPa} (support 0.12). Then generate frequent 2-item sets through connection and pruning operations, such as {temperature = 180°C, pressure = 1.0 MPa} (support 0.08), and retain it if it meets the minimum support. Subsequently, generate association rules based on the frequent item sets. For example, the confidence of the rule "temperature = 80°C → pressure = 1.0 MPa" is 0.53, which is lower than the threshold and is therefore excluded, while the confidence of the rule "time = 120 min → temperature = 80°C" is 0.75 and is retained. The finally generated traceability set contains 20 strong association rules, and each rule is accompanied by support and confidence. For example, the support of "humidity = 20% ∧ rotation speed = 300 rpm → finished product qualification rate = 95%" is 0.09 and the confidence is 0.82. Through the FP-Growth algorithm, a comparative analysis is carried out on the same data set, and its operation efficiency is 40% higher than that of Apriori, but the rule results are the same, verifying the reliability of the algorithm. During the process, a hash tree is used to optimize the candidate item set counting, and the results are stored in a graph database for visual display of the association network between links.

[0186] Embodiment 2

[0187] As Figure 2 shown, based on the same inventive concept, this embodiment also provides a product defect detection system based on multispectral imaging, including:

[0188] A data acquisition module, configured to acquire multispectral image data of a financial payment terminal product and generate a first image set including surface defects and internal welding information;

[0189] An image processing module, configured to perform denoising filtering and contrast enhancement processing on the first image set to generate a second image set; determine whether the resolution and clarity of the second image set meet the preset requirements, and if they meet the preset requirements, determine it as training data;

[0190] A feature extraction module, configured to extract texture shape features including bubbles, foreign objects, and poor welding based on the second image set to generate a feature set;

[0191] A model construction module, configured to construct a dedicated convolutional neural network model for a plastic housing and a metal circuit board according to a feature set, and generate a first model set adapted to material differences through transfer learning;

[0192] A model optimization module, configured to optimize the first model set by applying data augmentation and L2 regularization algorithms to generate a second model set. If the accuracy and recall rate of the second model set exceed a preset threshold, it is determined as the final detection model;

[0193] A defect detection module, configured to perform defect detection on a real-time multi-spectral image through the second model set, and generate a defect set including defect positions, types, and severities;

[0194] A database construction module, configured to construct a defect feature database based on the defect set. If the defect feature database includes fields of positions, types, and severities, it is determined as a matchable data set;

[0195] A feature matching module, configured to perform feature matching on the defect feature database by using a vector retrieval algorithm to generate a matching set;

[0196] A traceability analysis module, configured to apply an association rule mining algorithm according to the matching set to generate a traceability data set including corresponding relationships of production links.

[0197] A product defect detection system based on multi-spectral imaging provided in this embodiment has all the advantages of the product defect detection method based on multi-spectral imaging provided in Embodiment 1.

[0198] Embodiment 3

[0199] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0200] Embodiment 4

[0201] This embodiment also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the method described in Embodiment 1.

[0202] Embodiment 5

[0203] This embodiment also discloses a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the method described in Embodiment 1.

[0204] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A product defect detection method based on multispectral imaging, characterized in that: include: Acquire multispectral image data of a financial payment terminal product and generate a first image set including surface defects and internal welding information; Performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; Determining whether the resolution and clarity of the second image set meet preset requirements, and if so, determining the second image set as training data; Extracting texture shape features including bubbles, foreign matter and poor welding based on the second image set to generate a feature set; Constructing a dedicated convolutional neural network model for the plastic housing and the metal circuit board according to the feature set, and generating a first model set adapted to the material difference through transfer learning; Applying data enhancement and L2 regularization algorithms to optimize the first model set to generate a second model set, and if the accuracy and recall rate of the second model set exceed a preset threshold, determining the second model set as the final detection model; Performing defect detection on the real-time multispectral image by using the second model set to generate a defect set including defect location, type and severity; A defect feature database is constructed based on the defect set. If the defect feature database contains location, type and severity fields, it is determined to be a matchable data set; a vector retrieval algorithm is used to perform feature matching on the defect feature database to generate a matching set; an association rule mining algorithm is applied according to the matching set to generate a traceability data set containing the corresponding relationship of the production links.

2. The method according to claim 1, characterized in that The process of acquiring multispectral image data of a financial payment terminal product and generating a first image set containing surface defects and internal welding information includes: Scanning a financial payment terminal with a multispectral imaging device to collect image data including visible light and infrared spectra to generate a first multispectral image set; Using a convolutional neural network to extract features from the first multispectral image set, separating surface defect features and internal welding features, and obtaining a first feature set; If the surface defect feature value in the first feature set exceeds a preset threshold, the surface area is partitioned by an image segmentation algorithm to obtain a surface defect positioning image; According to the surface defect positioning image, a morphological operation is used to optimize the boundary of the defect area to generate a defect image; If the internal welding characteristic value in the first characteristic set deviates from the preset welding quality standard, analyzing the thermal distribution of the welding area through infrared spectrum data to obtain welding quality assessment data; The defect image and the welding quality assessment data are integrated by a data fusion algorithm to generate a third image set including surface defects and internal welding information; The third image set is rendered using visualization technology to output a defect and welding information image set of the financial payment terminal.

3. The method according to claim 1, characterized in that The process of performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set includes: Using a bilateral filtering algorithm to perform denoising on the first image set to generate an intermediate image set; Performing contrast enhancement processing on the intermediate image set by using an adaptive histogram equalization method to generate a second image set; Acquiring a resolution parameter of the second image set, and determining the second image set as a candidate image set if the resolution parameter meets a preset resolution threshold; For the candidate image set, the Laplace operator is used to calculate the image clarity, and if the clarity value is greater than a preset clarity threshold, it is determined as training data; Extracting features based on the training data, pre-training using a convolutional neural network algorithm, and generating initial model parameters; According to the initial model parameters, batch verification is performed on the second image set to obtain a verification error; if the verification error is lower than a preset error threshold, it is determined that the model parameters are valid; According to the effective model parameters, the subsequent input image sets are automatically processed to generate training data that meets the requirements.

4. The method according to claim 1, characterized in that The texture shape features including bubbles, foreign matter and poor welding are extracted based on the second image set, and the process of generating the feature set includes: Acquire image data based on the second image set, and generate a denoised image through preprocessing; A convolutional neural network is used to extract texture and shape features of bubbles, foreign matter and poor welding from the denoised image to obtain an initial feature set; If the dimension of the initial feature set is higher than a preset threshold, the dimension is reduced by principal component analysis to generate an optimized feature set; According to the optimized feature set, the texture shape feature vectors of bubbles, foreign matter and poor welding are calculated to determine the feature distribution; Based on the characteristic distribution, a clustering algorithm is used to classify bubbles, foreign matter and poor welding to obtain a classification result; Based on the classification results, a texture shape feature subset of each type of defect is obtained to generate a final feature set; For the final feature set, a feature database is constructed and feature set data is output.

5. The method according to claim 1, characterized in that The process of constructing a dedicated convolutional neural network model for the plastic housing and the metal circuit board according to the feature set and generating a first model set adapted to the material difference through transfer learning includes: Using a convolutional neural network to perform image processing on the feature set, extracting features of the plastic housing and the metal circuit board, and obtaining a first feature set; If the dimension of the first feature set exceeds a preset threshold, a second feature set is generated by dimensionality reduction processing; otherwise, the second feature set is directly determined to be the first feature set; Performing model training on the second feature set through transfer learning to generate a first model set adapted to material differences; Evaluate the performance of the first model set using classification accuracy to obtain a first evaluation result; If the first evaluation result is lower than a preset threshold, retraining is performed by adjusting the convolutional neural network parameters to obtain a second model set; otherwise, the first model set is determined as the final model; The newly input material feature set is classified using the second model set to obtain a classification result.

6. The method according to claim 1, characterized in that The process of applying data enhancement and L2 regularization algorithm to optimize the first model set to generate the second model set includes: Preprocessing the first model set by data enhancement to generate an enhanced data set; Applying L2 regularization to the enhanced data set to optimize the first model set to obtain a second model set; Obtaining the accuracy and recall of the second model set, calculating the performance index, and if both the accuracy and the recall exceed a preset threshold, determining that the second model set is the final detection model; The performance of the final detection model is confirmed by cross-validation to obtain a verification result, and the data enhancement parameters are adjusted according to the verification result to generate an optimized model set, and the final detection model is extracted based on the optimized model set.

7. The method according to claim 1, characterized in that The process of performing defect detection on the real-time multispectral image by using the second model set to generate a defect set including defect location, type and severity includes: Acquire real-time multispectral images through a multispectral sensor to generate a first image set; Using a preprocessing algorithm to denoise and standardize the first image set to obtain a second image set; Perform defect detection on the second image set by using a second model set to determine the defect location, type and severity, and generate a first defect set; If the severity of defects in the first defect set exceeds a preset threshold, local enhancement processing is performed on the corresponding image area to obtain a third image set; Re-detect defects on the third image set using the second model set, update defect locations, types, and severity, and obtain a second defect set; According to the defect types in the second defect set, a classification algorithm is used to prioritize the defects to obtain a sorted defect set; The defect locations, types and severity in the sorted defect set are formatted to generate a final defect set.

8. The method according to claim 1, characterized in that A defect feature database is constructed based on the defect set. If the defect feature database contains location, type and severity fields, the process of determining a matching data set includes: Obtaining defect location, defect type and severity data according to the defect set, removing duplicate and missing values ​​using a data cleaning method, and obtaining structured defect data; Using the structured defect data, a feature database including location, type and severity fields is constructed, and a relational database is used to manage data storage to obtain a database structure; If the fields of the feature database include location, type and severity, a query statement is used to extract records that meet the conditions to obtain a preliminary matching data set; Based on the preliminary matching data set, a K-nearest neighbor algorithm is used to perform cluster analysis on defect locations and types to obtain a classified matching data set; Acquire severity data from the classified matching data set, and arrange them in descending order of severity using a sorting algorithm to obtain a priority sorted data set; By using the priority sorting data set, extracting records with severity levels higher than a preset threshold using a data screening method, and determining a final matching data set; The defect location and type are extracted from the final matching data set, and the defect distribution characteristics are generated by using a data mapping method to obtain a defect characteristic analysis result.

9. The method according to claim 1, characterized in that: The process of performing feature matching on the defect feature database using a vector retrieval algorithm and generating a matching set includes: Using a vector retrieval algorithm to query the defect feature database, calculating the similarity between the input feature vector and the feature vector in the database, and obtaining a preliminary matching set; If the similarity value in the preliminary matching set is greater than a preset threshold, the corresponding feature vector is retained, otherwise it is discarded to generate a filtered matching set; According to the filtered matching set, defect category information corresponding to each feature vector in the matching set is obtained to generate a classification result set; Through the classification result set, determine the occurrence frequency of each defect category and obtain frequency distribution data; Using a clustering algorithm to analyze the frequency distribution data, obtain a grouping pattern of defect categories, and generate a final defect analysis result set; The final defect analysis result set is stored in a preset database to generate a queryable analysis record.

10. A product defect detection system based on multispectral imaging, characterized in that: include: A data acquisition module, used to acquire multispectral image data of a financial payment terminal product and generate a first image set including surface defects and internal welding information; An image processing module, used for performing denoising filtering and contrast enhancement processing on the first image set to generate a second image set; Determining whether the resolution and clarity of the second image set meet preset requirements, and if so, determining the second image set as training data; A feature extraction module, used to extract texture shape features including bubbles, foreign matter and poor welding based on the second image set to generate a feature set; A model building module, used to build a dedicated convolutional neural network model for the plastic housing and the metal circuit board according to the feature set, and generate a first model set adapted to the material difference through transfer learning; A model optimization module, used to apply data enhancement and L2 regularization algorithm to optimize the first model set to generate a second model set, and if the accuracy and recall rate of the second model set exceed a preset threshold, it is determined as the final detection model; A defect detection module, used to perform defect detection on the real-time multispectral image by using the second model set, and generate a defect set including defect location, type and severity; A database construction module, used to construct a defect feature database based on the defect set, and if the defect feature database contains location, type and severity fields, it is determined to be a matching data set; A feature matching module, used to perform feature matching on the defect feature database using a vector retrieval algorithm to generate a matching set; The traceability analysis module is used to apply an association rule mining algorithm according to the matching set to generate a traceability data set containing the corresponding relationship of the production links.

Images