Multi-modal defect automatic classification method and device, equipment and storage medium

Abstract

The invention discloses a multi-mode defect automatic classification method, device and equipment and a storage medium. The method comprises the following steps: acquiring a microscope image, an X-ray image and defect feature information data of a to-be-detected product; first classification processing is preferentially carried out based on the microscope image, when the first classification processing result meets a first preset reliability condition, the first classification processing result is taken as a final classification result, and when the first classification processing result does not meet the first preset reliability condition, the final classification result is taken as a final classification result. Performing feature fusion based on the X-ray image and the defect feature information data, and executing second classification processing; when the second classification processing result meets a second preset reliability condition, the second classification processing result serves as a final classification result, and when the second classification processing result does not meet the second preset reliability condition, defect type judgment is conducted based on key parameters in the defect feature information data, and the final classification result is output; the classification accuracy and efficiency can be improved, and the error recognition rate is reduced.

Description

Technical Field

[0001] This invention relates to the field of industrial automation inspection technology, and in particular to a method, apparatus, equipment and storage medium for automatic classification of multimodal defects. Background Technology

[0002] With the rapid development of intelligent manufacturing, Automatic Defect Classification (ADC) systems are being used more and more widely in industrial production.

[0003] It is applied to visual inspection in the process, providing a high-quality data foundation for data analysis and mining. Compared with traditional manual judgment or re-judgment of product surface defects, ADC can more accurately judge a wide variety of defect types. Its powerful manual re-judgment capability complements automatic image judgment, and it covers 100% of manual, semi-manual, and fully automatic image judgment scenarios.

[0004] In recent years, machine learning-based ADC systems have gradually become mainstream in the industry. By integrating high-resolution industrial cameras and advanced image processing algorithms, they achieve automated identification and classification of product defects. Currently, mainstream ADC systems mainly adopt a single-modality data processing architecture, that is, independently analyzing data collected by specific inspection equipment. Although this has achieved certain results in specific scenarios, it still faces many challenges when facing complex and ever-changing industrial production environments.

[0005] In existing technologies, deep learning-based methods have shown certain advantages in image feature extraction, but they are highly dependent on training data and have limited generalization ability when facing novel defects or small sample scenarios. While traditional rule-based classification methods have good interpretability, they are difficult to cope with increasingly complex defect morphologies and changing process conditions.

[0006] More importantly, most current ADC systems lack an effective fusion mechanism for multi-source heterogeneous data, and cannot make full use of the complementary advantages of information obtained by different detection methods. This results in a significant lack of accuracy in identifying boundary cases or low-contrast defects, and still requires a large amount of manual review.

[0007] In addition, existing ADC systems also have significant shortcomings in terms of data utilization depth.

[0008] Although the system can generate a large amount of defect data, due to the lack of effective data correlation analysis and knowledge mining capabilities, most of this data is only used for judging the results of the current test and has failed to form a closed-loop quality improvement mechanism.

[0009] When the system classification reliability is insufficient, the workload of manual review will increase significantly, which contradicts the original intention of intelligent manufacturing to reduce costs and increase efficiency. In advanced manufacturing processes, the types of defects are increasing, and traditional ADC systems can no longer meet the needs of high-precision and high-efficiency detection. Summary of the Invention

[0010] The main objective of this invention is to provide a method, apparatus, device, and storage medium for automatic classification of multimodal defects, aiming to solve the technical problems in existing automatic defect classification systems, such as the lack of a multimodal data collaborative processing mechanism, insufficient reliability assessment of classification results, and excessive reliance on manual review in boundary case scenarios.

[0011] In a first aspect, the present invention provides an automatic classification method for multimodal defects, the automatic classification method for multimodal defects comprising the following steps: Acquire microscopic images, X-ray images, and defect feature information data of the product to be inspected; The first classification process is performed based on the microscope image. When the result of the first classification process meets the first preset reliability condition, the first classification process result is directly used as the final classification result. When the result of the first classification process does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and the second classification process is executed. When the second classification result meets the second preset reliability condition, the second classification result is taken as the final classification result. When the second classification result does not meet the second preset reliability condition, the defect type is judged based only on the key parameters in the defect feature information data and the final classification result is output.

[0012] Optionally, acquiring the microscope image, X-ray image, and defect feature information data of the product to be inspected includes: Microscopic images of the product to be inspected are acquired using an automated optical inspection (AOI) device. These microscopic images are used to characterize the morphological features of surface defects in the product to be inspected. An X-ray image of the product to be inspected is acquired using an X-ray imaging system. The X-ray image is used to characterize the distribution of internal defects in the product to be inspected. Structured quantitative feature data is extracted simultaneously during the acquisition of the microscope images and the X-ray images, and the structured quantitative feature data is used as defect feature information data.

[0013] Optionally, the first classification processing is performed based on the microscope image. When the result of the first classification processing meets a first preset reliability condition, the first classification processing result is directly used as the final classification result. When the result of the first classification processing does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification processing is executed, including: The microscope image is normalized to obtain a normalized target microscope image; The target microscope image is input into a pre-trained deep learning classification model to obtain a first classification result and a first confidence value corresponding to the first classification result; When the first confidence value is not less than the first preset threshold, it is determined that the first preset reliability condition is met, and the first classification result is directly used as the final classification result. When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met, and feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed.

[0014] Optionally, the step of normalizing the microscope image to obtain a normalized target microscope image includes: The microscope image is read from an image processing library, and the light intensity value of each pixel in the microscope image is converted into an image matrix; The image matrix is ​​normalized using the following formula to obtain the normalized target microscope image:

[0015]

[0016] in, This is the normalized target microscope image. This refers to the light intensity value of a single pixel in a microscope image. This is the mean value of all pixel values ​​in the microscope image. This is the adjusted standard deviation. The standard deviation of the microscope image. This represents the total number of pixels in the microscope image.

[0017] Optionally, the step of determining that the first preset reliability condition is not met when the first confidence value is less than the first preset threshold, and performing feature fusion and second classification processing based on the X-ray image and the defect feature information data, includes: When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met. The SIFT algorithm is used to extract features from the X-ray image to obtain the gradient features and orientation features of key points in the X-ray image. The gradient features, orientation features and original grayscale features are then stitched together to form a three-channel image. Principal component analysis (PCA) is used to reduce the dimensionality of the three-channel image, retaining the eigenvector with the largest eigenvalue to form X-ray image features. Structured quantitative feature parameters related to the current detection are extracted from the defect feature information database, and the X-ray image features and the structured quantitative feature parameters are sequentially stitched together to form multidimensional features; The multidimensional features are subjected to Softmax normalization to obtain the fused and normalized target feature vector; The target feature vector is input into a pre-trained XGBoost machine learning classification model to perform a second classification process.

[0018] Optionally, the step of performing Softmax normalization on the multidimensional features to obtain the fused and normalized target feature vector includes: Analyze the statistical distribution characteristics of each dimension in the multidimensional features, and calculate the central tendency and dispersion of each feature dimension; Based on the central tendency and the degree of dispersion, the feature values ​​of each dimension are standardized to obtain standardized feature values; The standardized feature values ​​are converted into a probability distribution form by using the Softmax normalization function, so that the sum of the weight values ​​of each feature dimension is 1 and all are non-negative, thus obtaining the normalized weight coefficients. The original feature vectors corresponding to the X-ray image features and the structured quantitative feature parameters are weighted and fused according to the weight coefficients to obtain the fused and normalized target feature vector.

[0019] Optionally, when the second classification processing result meets the second preset reliability condition, the second classification processing result is taken as the final classification result; when the second classification processing result does not meet the second preset reliability condition, the defect type is determined and the final classification result is output based solely on the key parameters in the defect feature information data, including: When the second confidence level of the second classification processing result is not less than the second preset threshold, it is determined that the second preset reliability condition is met, and the second classification processing result is taken as the final classification result. When the second confidence level is less than the second preset threshold, it is determined that the second preset reliability condition is not met, and the verified key parameters in the defect feature information data are obtained. Based solely on the key parameters, a deterministic matching algorithm is executed using a preset threshold rule set defined in a preset expert knowledge base to determine the defect type and output the final classification result.

[0020] Secondly, to achieve the above objectives, the present invention also proposes an automatic multimodal defect classification device, the automatic multimodal defect classification device comprising: The data acquisition module is used to acquire microscope images, X-ray images, and defect feature information data of the product to be inspected; The initial classification module is used to perform a first classification process based on the microscope image. When the result of the first classification process meets the first preset reliability condition, the first classification process result is directly used as the final classification result. When the result of the first classification process does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed. The final classification module is used to take the second classification result as the final classification result when the second classification result meets the second preset reliability condition, and to determine the defect type and output the final classification result based only on the key parameters in the defect feature information data when the second classification result does not meet the second preset reliability condition.

[0021] Thirdly, to achieve the above objectives, the present invention also proposes a multimodal defect automatic classification device, the multimodal defect automatic classification device comprising: a memory, a processor, and a multimodal defect automatic classification program stored in the memory and executable on the processor, the multimodal defect automatic classification program being configured to implement the steps of the multimodal defect automatic classification method as described above.

[0022] Fourthly, to achieve the above objectives, the present invention also proposes a storage medium storing a multimodal defect automatic classification program, wherein the multimodal defect automatic classification program, when executed by a processor, implements the steps of the multimodal defect automatic classification method as described above.

[0023] The multimodal defect automatic classification method proposed in this invention acquires microscope images, X-ray images, and defect feature information data of the product to be inspected. It prioritizes a first classification process based on the microscope images. When the result of the first classification process meets a first preset reliability condition, it is directly used as the final classification result. When the result does not meet the first preset reliability condition, feature fusion is performed based on the X-ray images and the defect feature information data, and a second classification process is executed. When the result of the second classification process meets a second preset reliability condition, it is used as the final classification result. When the result does not meet the second preset reliability condition, defect type judgment is performed only based on key parameters in the defect feature information data, and the final classification result is output. This method optimizes the technical performance of the defect classification system, effectively improves the overall classification accuracy, reduces the false positive rate, enhances the system's adaptability to various defect features, and optimizes the allocation efficiency of computing resources. It enables the classification system to maintain high accuracy while possessing stronger robustness and fault tolerance, enhancing the system's adaptability to complex situations. This provides more reliable technical support for industrial automation inspection, reduces the false recognition rate, and improves the speed and efficiency of multimodal defect automatic classification. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the multimodal defect automatic classification method of the present invention; Figure 3 This is a flowchart illustrating the second embodiment of the multimodal defect automatic classification method of the present invention; Figure 4 This is a flowchart illustrating the third embodiment of the multimodal defect automatic classification method of the present invention; Figure 5 This is a flowchart illustrating the fourth embodiment of the multimodal defect automatic classification method of the present invention; Figure 6 This is a schematic diagram of the training process in the multimodal defect automatic classification method of the present invention; Figure 7 This is a schematic diagram of the prediction process in the multimodal defect automatic classification method of the present invention; Figure 8 This is a functional block diagram of the first embodiment of the multimodal defect automatic classification device of the present invention.

[0025] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] The solution of this invention mainly involves: acquiring microscope images, X-ray images, and defect feature information data of the product to be inspected; preferentially performing a first classification process based on the microscope images; when the result of the first classification process meets a first preset reliability condition, directly using the first classification process result as the final classification result; when the first classification process result does not meet the first preset reliability condition, performing feature fusion based on the X-ray images and the defect feature information data and performing a second classification process; when the result of the second classification process meets a second preset reliability condition, using the second classification process result as the final classification result; when the second classification process result does not meet the second preset reliability condition, only based on the defect feature information data... The system uses key parameters to determine the defect type and outputs the final classification result. This optimizes the technical performance of the defect classification system, effectively improving overall classification accuracy, reducing misclassification rate, enhancing the system's adaptability to various defect features, and optimizing the allocation efficiency of computing resources. This allows the classification system to maintain high accuracy while possessing stronger robustness and fault tolerance, enhancing its ability to cope with complex situations. It provides more reliable technical support for industrial automation inspection, reduces the false recognition rate, and improves the speed and efficiency of automatic multimodal defect classification. This solves the technical problems of existing automatic defect classification systems, such as the lack of multimodal data collaborative processing mechanisms, insufficient reliability assessment of classification results, and excessive reliance on manual review in boundary case scenarios.

[0028] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0029] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0030] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0031] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating device, a network communication module, a user interface module, and a multimodal defect automatic classification program.

[0032] The device of the present invention calls the multimodal defect automatic classification program stored in the memory 1005 through the processor 1001 and performs the following operations: Acquire microscopic images, X-ray images, and defect feature information data of the product to be inspected; The first classification process is performed based on the microscope image. When the result of the first classification process meets the first preset reliability condition, the first classification process result is directly used as the final classification result. When the result of the first classification process does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and the second classification process is executed. When the second classification result meets the second preset reliability condition, the second classification result is taken as the final classification result. When the second classification result does not meet the second preset reliability condition, the defect type is judged based only on the key parameters in the defect feature information data and the final classification result is output.

[0033] The device of the present invention, through processor 1001 calling the multimodal defect automatic classification program stored in memory 1005, also performs the following operations: Microscopic images of the product to be inspected are acquired using an automated optical inspection (AOI) device. These microscopic images are used to characterize the morphological features of surface defects in the product to be inspected. An X-ray image of the product to be inspected is acquired using an X-ray imaging system. The X-ray image is used to characterize the distribution of internal defects in the product to be inspected. Structured quantitative feature data is extracted simultaneously during the acquisition of the microscope images and the X-ray images, and the structured quantitative feature data is used as defect feature information data.

[0034] The device of the present invention, through processor 1001 calling the multimodal defect automatic classification program stored in memory 1005, also performs the following operations: The microscope image is normalized to obtain a normalized target microscope image; The target microscope image is input into a pre-trained deep learning classification model to obtain a first classification result and a first confidence value corresponding to the first classification result; When the first confidence value is not less than the first preset threshold, it is determined that the first preset reliability condition is met, and the first classification result is directly used as the final classification result. When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met, and feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed.

[0035] The device of the present invention, through processor 1001 calling the multimodal defect automatic classification program stored in memory 1005, also performs the following operations: The microscope image is read from an image processing library, and the light intensity value of each pixel in the microscope image is converted into an image matrix; The image matrix is ​​normalized using the following formula to obtain the normalized target microscope image:

[0036]

[0037] in, This is the normalized target microscope image. This refers to the light intensity value of a single pixel in a microscope image. This is the mean value of all pixel values ​​in the microscope image. This is the adjusted standard deviation. The standard deviation of the microscope image. This represents the total number of pixels in the microscope image.

[0038] The device of the present invention, through processor 1001 calling the multimodal defect automatic classification program stored in memory 1005, also performs the following operations: When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met. The SIFT algorithm is used to extract features from the X-ray image to obtain the gradient features and orientation features of key points in the X-ray image. The gradient features, orientation features and original grayscale features are then stitched together to form a three-channel image. Principal component analysis (PCA) is used to reduce the dimensionality of the three-channel image, retaining the eigenvector with the largest eigenvalue to form X-ray image features. Structured quantitative feature parameters related to the current detection are extracted from the defect feature information database, and the X-ray image features and the structured quantitative feature parameters are sequentially stitched together to form multidimensional features; The multidimensional features are subjected to Softmax normalization to obtain the fused and normalized target feature vector; The target feature vector is input into a pre-trained XGBoost machine learning classification model to perform a second classification process.

[0039] The device of the present invention, through processor 1001 calling the multimodal defect automatic classification program stored in memory 1005, also performs the following operations: Analyze the statistical distribution characteristics of each dimension in the multidimensional features, and calculate the central tendency and dispersion of each feature dimension; Based on the central tendency and the degree of dispersion, the feature values ​​of each dimension are standardized to obtain standardized feature values; The standardized feature values ​​are converted into a probability distribution form by using the Softmax normalization function, so that the sum of the weight values ​​of each feature dimension is 1 and all are non-negative, thus obtaining the normalized weight coefficients. The original feature vectors corresponding to the X-ray image features and the structured quantitative feature parameters are weighted and fused according to the weight coefficients to obtain the fused and normalized target feature vector.

[0040] The device of the present invention, through processor 1001 calling the multimodal defect automatic classification program stored in memory 1005, also performs the following operations: When the second confidence level of the second classification processing result is not less than the second preset threshold, it is determined that the second preset reliability condition is met, and the second classification processing result is taken as the final classification result. When the second confidence level is less than the second preset threshold, it is determined that the second preset reliability condition is not met, and the verified key parameters in the defect feature information data are obtained. Based solely on the key parameters, a deterministic matching algorithm is executed using a preset threshold rule set defined in a preset expert knowledge base to determine the defect type and output the final classification result.

[0041] This embodiment, through the above-described scheme, acquires microscope images, X-ray images, and defect feature information data of the product to be inspected; it prioritizes performing a first classification process based on the microscope images; when the result of the first classification process meets a first preset reliability condition, the first classification process result is directly used as the final classification result; when the first classification process result does not meet the first preset reliability condition, feature fusion is performed based on the X-ray images and the defect feature information data, and a second classification process is executed; when the result of the second classification process meets a second preset reliability condition, the second classification process result is used as the final classification result; when the second classification process result does not meet the second preset reliability condition, defect type judgment is performed only based on key parameters in the defect feature information data, and the final classification result is output. This enables the optimization of the technical performance of the defect classification system, effectively improving the overall classification accuracy, reducing the false judgment rate, enhancing the system's adaptability to various defect features, and optimizing the allocation efficiency of computing resources. This allows the classification system to maintain high accuracy while possessing stronger robustness and fault tolerance, enhancing the system's adaptability to complex situations, providing more reliable technical support for industrial automated inspection, reducing the false recognition rate, and improving the speed and efficiency of automatic multimodal defect classification.

[0042] Based on the above hardware structure, an embodiment of the multimodal defect automatic classification method of the present invention is proposed.

[0043] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the multimodal defect automatic classification method of the present invention.

[0044] In the first embodiment, the automatic multimodal defect classification method includes the following steps: Step S10: Obtain microscope images, X-ray images, and defect feature information data of the product to be inspected.

[0045] It should be noted that by acquiring microscope images, X-ray images, and defect feature information of the product under inspection through different devices, this multimodal data acquisition strategy breaks through the limitations of a single detection method. It not only covers the complete spatial dimension of the product surface and interior, but also integrates the complementary advantages of visual information and quantitative parameters, providing a comprehensive and reliable data foundation for subsequent defect identification and classification.

[0046] Step S20: Prioritize the first classification processing based on the microscope image. When the first classification processing result meets the first preset reliability condition, directly use the first classification processing result as the final classification result. When the first classification processing result does not meet the first preset reliability condition, perform feature fusion based on the X-ray image and the defect feature information data and execute the second classification processing.

[0047] It should be understood that using microscopic images of product surfaces for rapid classification and evaluation fully leverages the advantages of intuitive and easily identifiable surface morphological features. When the preliminary classification result is deemed sufficiently reliable, it is directly adopted as the final determination, avoiding unnecessary additional calculations and significantly improving processing efficiency. However, when the preliminary classification result is uncertain, a more comprehensive analysis process is automatically initiated, improving classification accuracy through complementary verification of multi-dimensional data.

[0048] Step S30: When the second classification processing result meets the second preset reliability condition, the second classification processing result is taken as the final classification result. When the second classification processing result does not meet the second preset reliability condition, the defect type is judged based only on the key parameters in the defect feature information data and the final classification result is output.

[0049] Understandably, when the second-level classification result obtained from the fusion analysis of X-ray images and structured feature parameters meets the preset reliability standard, the result is directly used as the final judgment, making full use of the complementary advantages of multi-dimensional information and improving the accuracy of identifying complex defects. However, when the fusion analysis result still has uncertainty, it automatically switches to a more robust deterministic rule judgment mode based on key parameters to ensure that the classification result remains reliable in boundary cases or ambiguous situations.

[0050] This embodiment, through the above-described scheme, acquires microscope images, X-ray images, and defect feature information data of the product to be inspected; it prioritizes performing a first classification process based on the microscope images; when the result of the first classification process meets a first preset reliability condition, the first classification process result is directly used as the final classification result; when the first classification process result does not meet the first preset reliability condition, feature fusion is performed based on the X-ray images and the defect feature information data, and a second classification process is executed; when the result of the second classification process meets a second preset reliability condition, the second classification process result is used as the final classification result; when the second classification process result does not meet the second preset reliability condition, defect type judgment is performed only based on key parameters in the defect feature information data, and the final classification result is output. This enables the optimization of the technical performance of the defect classification system, effectively improving the overall classification accuracy, reducing the false judgment rate, enhancing the system's adaptability to various defect features, and optimizing the allocation efficiency of computing resources. This allows the classification system to maintain high accuracy while possessing stronger robustness and fault tolerance, enhancing the system's adaptability to complex situations, providing more reliable technical support for industrial automated inspection, reducing the false recognition rate, and improving the speed and efficiency of automatic multimodal defect classification.

[0051] Furthermore, Figure 3 This is a flowchart illustrating the second embodiment of the automatic multimodal defect classification method of the present invention, as shown below. Figure 3 As shown, based on the first embodiment, a second embodiment of the automatic multimodal defect classification method of the present invention is proposed. In this embodiment, step S10 specifically includes the following steps: Step S11: Acquire a microscope image of the product to be inspected using an automated optical inspection (AOI) device. The microscope image is used to characterize the morphological features of the surface defects of the product to be inspected.

[0052] It should be noted that high-precision automated optical inspection (AOI) equipment can perform non-contact surface scanning of products under inspection, using a high-resolution industrial camera and an adjustable light source system to acquire microscopic image data of the product surface. These microscopic images can clearly present surface details at the micrometer or even nanometer level, accurately capturing the morphological characteristics of defects such as geometry, size, edge contour, distribution density, and spatial location. Through the digital characterization of these morphological characteristics, the system can objectively quantify subtle defects that are difficult to accurately assess using traditional manual visual inspection, providing an intuitive and reliable data foundation for subsequent automatic classification and quality analysis.

[0053] Step S12: Obtain an X-ray image of the product to be inspected using an X-ray imaging system. The X-ray image is used to characterize the distribution of internal defects in the product to be inspected.

[0054] It is understandable that when X-rays penetrate materials, the differences in absorption characteristics caused by density differences generate high-contrast images that reflect the internal structure of the product. These X-ray images can clearly show internal defects that are not visible to the naked eye, such as pores, cracks, inclusions, welding defects and structural discontinuities inside the material, and accurately characterize the spatial distribution, geometric shape and severity of defects.

[0055] Step S13: Simultaneously extract structured quantitative feature data during the acquisition of the microscope image and the X-ray image, and use the structured quantitative feature data as defect feature information data.

[0056] It should be understood that real-time image analysis algorithms synchronously extract quantitative feature parameters with clear physical meaning, including structured data such as the geometric dimensions, shape complexity, location distribution, grayscale features, and texture characteristics of the defect area. After standardization, these feature parameters form a structured quantitative feature dataset, which can accurately characterize the essential attributes and severity of defects, making up for the shortcomings of pure image data in terms of interpretability and quantitative analysis.

[0057] This embodiment utilizes the above-described scheme to acquire microscopic images of the product under inspection using an automated optical inspection (AOI) device. These microscopic images characterize the morphological features of surface defects on the product. An X-ray imaging system is then used to acquire X-ray images of the product, which characterize the distribution of internal defects. Structured quantitative feature data is extracted simultaneously during the acquisition of both the microscopic and X-ray images. Using this structured quantitative feature data as defect feature information significantly improves the timeliness and consistency of the feature data. The collaborative acquisition of multi-source heterogeneous data not only enriches the dimensions of defect characterization but also provides a solid foundation for subsequent feature fusion and reliability assessment, enabling the system to more accurately distinguish similar defect types and identify complex composite defects.

[0058] Furthermore, Figure 4 This is a flowchart illustrating the third embodiment of the multimodal defect automatic classification method of the present invention, as shown below. Figure 4 As shown, based on the first embodiment, a third embodiment of the automatic multimodal defect classification method of the present invention is proposed. In this embodiment, step S20 specifically includes the following steps: Step S21: Normalize the microscope image to obtain a normalized target microscope image.

[0059] It should be noted that by normalizing the microscope image, a normalized target microscope image can be obtained.

[0060] Furthermore, step S21 specifically includes the following steps: The microscope image is read from an image processing library, and the light intensity value of each pixel in the microscope image is converted into an image matrix; The image matrix is ​​normalized using the following formula to obtain the normalized target microscope image:

[0061]

[0062] in, This is the normalized target microscope image. This refers to the light intensity value of a single pixel in a microscope image. This is the mean value of all pixel values ​​in the microscope image. This is the adjusted standard deviation. The standard deviation of the microscope image. This represents the total number of pixels in the microscope image.

[0063] Understandably, after converting the original microscopic images into numerical matrices using an image processing library, an improved standardization algorithm was implemented. This algorithm not only calculates the mean and standard deviation of the image pixel values ​​for conventional normalization, but also introduces an adjusted standard deviation calculation mechanism to ensure numerical stability even when the image contrast is extremely low or close to uniform regions. This normalization process effectively eliminates image differences caused by factors such as light intensity and equipment sensitivity under different acquisition conditions, placing all image data within a uniform numerical distribution range. This significantly improves the ability of subsequent deep learning models to extract defect features, avoids the numerical explosion problem caused by the standard deviation approaching zero in low-contrast regions, and ensures the robustness of the preprocessing process.

[0064] Step S22: Input the target microscope image into a pre-trained deep learning classification model to obtain a first classification result and a first confidence value corresponding to the first classification result.

[0065] Understandably, the standardized target microscope image is input into a pre-trained deep learning classification model. This model, based on a convolutional neural network architecture, has been optimized and trained using a large amount of historical defect sample data. It can automatically extract multi-level features from the image and perform pattern recognition. After the system performs forward propagation calculations, the model outputs the first classification result for the current product to be inspected, i.e., the defect type determined by the system (such as predefined categories like scratches, particulate contamination, and cracks). At the same time, it generates a first confidence value corresponding to this classification result. This value quantitatively reflects the degree of certainty of the model's current classification decision. The calculation of this confidence value is based on the probability distribution of the model's output layer, which is usually expressed as the relative probability of the target category among all possible categories, providing a key quantitative basis for subsequent reliability assessment.

[0066] Step S23: When the first confidence value is not less than the first preset threshold, it is determined that the first preset reliability condition is met, and the first classification processing result is directly used as the final classification result.

[0067] It should be understood that when the confidence value of the first classification result output by the deep learning model reaches or exceeds the preset reliability threshold, the result is deemed to have sufficient certainty and accuracy, and can be directly adopted as the final classification conclusion without additional complex analysis. This judgment logic is based on the quantitative assessment of the reliability of the classification result by the confidence value. The higher the confidence value, the more certain the model is in identifying the current defect type. When it exceeds the threshold set by the system (usually determined based on historical data statistics and optimization in actual application scenarios), it means that the probability of misclassification of the classification result has dropped to below an acceptable range. Through this intelligent judgment mechanism, a large number of obvious and clearly characterized defect cases can be processed quickly, avoiding unnecessary secondary analysis of highly reliable results, significantly improving detection efficiency and reducing computational resource consumption.

[0068] Step S24: When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met, and feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed.

[0069] It should be noted that when the first confidence value is less than the first preset threshold, it can be determined that the first preset reliability condition is not met. Then, feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed. This effectively avoids the risk of misjudgment caused by low confidence results, and enables the system to maintain a high classification accuracy when facing complex, ambiguous or novel defects.

[0070] Furthermore, step S24 specifically includes the following steps: When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met. The SIFT algorithm is used to extract features from the X-ray image to obtain the gradient features and orientation features of key points in the X-ray image. The gradient features, orientation features and original grayscale features are then stitched together to form a three-channel image. Principal component analysis (PCA) is used to reduce the dimensionality of the three-channel image, retaining the eigenvector with the largest eigenvalue to form X-ray image features. Structured quantitative feature parameters related to the current detection are extracted from the defect feature information database, and the X-ray image features and the structured quantitative feature parameters are sequentially stitched together to form multidimensional features; The multidimensional features are subjected to Softmax normalization to obtain the fused and normalized target feature vector; The target feature vector is input into a pre-trained XGBoost machine learning classification model to perform a second classification process.

[0071] Understandably, applying the Scale-Invariant Feature Transform (SIFT) algorithm for robust feature extraction from X-ray images accurately captures structured information such as gradient magnitude, direction, and spatial distribution of key points in the image. These features are highly invariant to illumination changes and image deformation, making them particularly suitable for identifying low-contrast defects in X-ray images. Subsequently, the extracted gradient and direction features are innovatively combined with the original grayscale features to form a three-channel image representation, preserving the complete structural information of the X-ray image while providing a standardized input format for subsequent processing. Then, principal component analysis (PCA) is used... PCA (Programmable Dimensional Analysis) intelligently reduces the dimensionality of the three-channel image, selecting the most discriminative feature vectors. This effectively compresses data dimensionality while retaining key information, significantly improving computational efficiency. Simultaneously, it extracts highly relevant structured quantitative feature parameters (such as defect size and shape coefficient) from the database and sequentially concatenates them with the dimensionality-reduced X-ray image features to construct a high-dimensional feature vector that integrates multi-source information. To optimize feature contribution, Softmax normalization is further implemented, dynamically adjusting the weight distribution of each feature dimension to highlight the contribution of key discriminative features. Finally, the normalized target feature vector is input into a specially trained XGBoost classification model. This model, based on the gradient boosting tree algorithm, excels at processing structured feature data and can extract deep-seated discriminative patterns from fused features. It is particularly suitable for handling boundary cases and ambiguous situations, thus providing a more accurate and robust second-level classification when the first-level classification result is unreliable. This significantly improves the system's ability to identify complex defects and its overall robustness.

[0072] Furthermore, the multidimensional features are subjected to Softmax normalization to obtain the fused and normalized target feature vector, specifically including the following steps: Analyze the statistical distribution characteristics of each dimension in the multidimensional features, and calculate the central tendency and dispersion of each feature dimension; Based on the central tendency and the degree of dispersion, the feature values ​​of each dimension are standardized to obtain standardized feature values; The standardized feature values ​​are converted into a probability distribution form by using the Softmax normalization function, so that the sum of the weight values ​​of each feature dimension is 1 and all are non-negative, thus obtaining the normalized weight coefficients. The original feature vectors corresponding to the X-ray image features and the structured quantitative feature parameters are weighted and fused according to the weight coefficients to obtain the fused and normalized target feature vector.

[0073] It should be understood that, firstly, a detailed statistical analysis is performed on the multidimensional feature vector formed by splicing, accurately calculating the mean, median, and other central tendency indicators, as well as the standard deviation, interquartile range, and other dispersion indicators for each feature dimension, to comprehensively grasp the distribution characteristics of each dimension's features. Based on these statistics, the system performs standardization transformation on the feature values ​​of each dimension, eliminating the dimensional differences between different features and ensuring that all features are on a comparable, uniform scale. Subsequently, the standardized feature values ​​are transformed into a probability distribution form through the Softmax normalization function, ensuring that the sum of the weight coefficients of each dimension is 1 and all are positive. This process essentially assigns a weight to each feature dimension that reflects its value. The dynamic weights of its discriminative power are then applied to the fusion process of the original feature sources. By weighting and combining the X-ray image feature vector and the structured quantitative feature parameter vector, a fused and normalized target feature vector is generated that retains key information while suppressing noise interference. This intelligent weighting mechanism based on statistical properties effectively highlights the feature dimensions that contribute significantly to classification, weakens the influence of redundant or noisy features, and makes the fused feature representation more focused on the key information for defect discrimination. This significantly improves the discriminative power and robustness of the subsequent XGBoost classification model, making it particularly suitable for handling complex and ever-changing defect identification scenarios in industrial inspection. This embodiment, through the above-described scheme, normalizes the microscope image to obtain a normalized target microscope image; inputs the target microscope image into a pre-trained deep learning classification model to obtain a first classification result and a first confidence value corresponding to the first classification result; when the first confidence value is not less than a first preset threshold, it is determined that the first preset reliability condition is met, and the first classification result is directly used as the final classification result; when the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met, and feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed. This significantly improves the quality consistency of image data, effectively eliminates image variations caused by factors such as lighting conditions and equipment differences, avoids unnecessary complex calculations, greatly improves the system processing speed and throughput, and significantly enhances the ability to identify complex defects.

[0074] Furthermore, Figure 5 This is a flowchart illustrating the fourth embodiment of the multimodal defect automatic classification method of the present invention, as shown below. Figure 5 As shown, based on the first embodiment, a fourth embodiment of the automatic multimodal defect classification method of the present invention is proposed. In this embodiment, step S30 specifically includes the following steps: Step S31: When the second confidence level of the second classification processing result is not less than the second preset threshold, it is determined that the second preset reliability condition is met, and the second classification processing result is taken as the final classification result.

[0075] It should be noted that when the second classification process based on multimodal data fusion (usually performed by an ensemble learning model such as XGBoost) outputs results, the second confidence value of the result is evaluated (reflecting the model's certainty about the current classification decision). When the confidence value reaches or exceeds the second threshold preset by the system, it indicates that the fusion analysis result has sufficient reliability and accuracy and can effectively distinguish boundary cases and complex defects. The result of the second classification process is then used as the final classification result.

[0076] Step S32: When the second confidence level is less than the second preset threshold, it is determined that the second preset reliability condition is not met, and the verified key parameters in the defect feature information data are obtained.

[0077] Understandably, when the confidence level of the second-level classification result based on multimodal data fusion is lower than the system's preset second threshold, the system intelligently determines that the result is still insufficient to ensure classification reliability and cannot be directly used as the final judgment basis. At this point, instead of relying on complex machine learning models, it switches to a more robust deterministic analysis mode based on key parameters. It actively extracts key parameters (such as the proportion of defect area, edge irregularity, grayscale contrast, and other quantitative indicators with clear physical meaning) that have been verified by historical data from the defect feature information database. These key parameters are obtained through the analysis of a large number of historical cases and the screening of expert experience, and have high stability and interpretability. They can provide a reliable classification basis in the boundary cases where machine learning models fail.

[0078] Step S33: Based solely on the key parameters, execute a deterministic matching algorithm using a preset threshold rule set defined in a preset expert knowledge base to determine the defect type and output the final classification result.

[0079] It should be understood that when the system still cannot obtain a high-confidence result after two levels of machine learning classification, a deterministic judgment process based on expert knowledge is initiated: only key parameters extracted from defect feature information and verified by a large number of historical cases are used to avoid introducing additional information that may bring uncertainty.

[0080] In the specific implementation, see Figure 6 , Figure 6 This is a schematic diagram of the training process in the multimodal defect automatic classification method of the present invention, as shown below. Figure 6As shown, a large number of labeled microscopic images (characterizing the morphology of surface defects), X-ray images (revealing the distribution of internal defects), and structured defect feature information data (including quantitative parameters such as size and shape) are first collected as the training basis. Then, the original image data undergoes standardized preprocessing, including illumination correction, noise suppression, and normalization, to ensure the consistency and quality of the input data. For X-ray images, robust feature extraction is performed using traditional image algorithms such as SIFT to accurately capture the gradient features, orientation features, and spatial distribution information of key points. Principal component analysis (PCA) is used to optimize the dimensionality of the extracted image features, while key parameters are selected from the defect feature information to construct a multi-source heterogeneous... The system constructs a feature set; it innovatively adopts a weighted fusion strategy to organically integrate image features with structured parameters to form a highly discriminative fusion feature vector; based on this, two complementary models are trained in stages—a deep learning model is trained using a convolutional neural network architecture to specifically process microscope images and output the first classification result and confidence score; simultaneously, an ensemble learning model such as XGBoost is trained to perform a second-level classification task based on the fusion features; through iterative optimization and hyperparameter tuning of a large number of historical defect samples, a high-precision combination of deep learning and machine learning models is finally obtained, enabling the system to maintain high efficiency while possessing the ability to accurately identify various complex defects, providing a solid technical foundation for an automatic defect classification system.

[0081] In the specific implementation, see Figure 7 , Figure 7 This is a schematic diagram of the prediction process in the multimodal defect automatic classification method of the present invention, as shown below. Figure 7As shown, after training, two models will be obtained: a deep learning model and a machine learning model, which will predict defects in images not yet trained. The system first receives microscopic images (representing surface defect morphology), X-ray images (revealing internal structural features), and structured defect feature information as input data. The pre-trained detection model performs preliminary analysis on the microscopic images, while traditional image extraction algorithms such as SIFT are applied to accurately identify potential defect areas and obtain key information such as defect size and shape. The system first uses a confidence threshold to determine if a valid defect has been detected; if the threshold is not reached, it is directly classified as defect-free. Once a defect is confirmed, the system further analyzes the defect size features. If the defect size exceeds the large defect threshold but the classification score is lower than the confidence threshold, it outputs "Others" to indicate an abnormal situation where classification is unclear. For medium-sized defects, the system performs feature fusion, integrating X-ray image features and structured parameters, and accurately identifies the defect using a pre-trained classification model, outputting the predicted defect type and the corresponding 128×128 pixel region of interest. The system provides an area for review (ROI) verification; for small defects (smaller than the small defect threshold), the system directly outputs a predefined small defect type; for particulate defects, the system specifically outputs the particle type and its precise center coordinates for subsequent location processing; the entire process achieves intelligent decision-making and diversion through a multi-level threshold judgment mechanism, which not only ensures the rapid identification of obvious defects, but also enables refined analysis of boundary cases, ultimately achieving accurate classification and location of various defects, significantly improving the automation level and reliability of industrial inspection.

[0082] It should be noted that the specific implementation process of this embodiment is as follows: 1. Training Phase 1) AOI equipment acquires images (microscope images, X-ray images, feature information data); 2) Image annotation: Annotate the location and category information of defects in microscope images, and perform feature extraction and type annotation on X-ray images; 3) Label a large amount of microscope image data, normalize and augment all microscope data, and then feed them into the detection model for training. After training, a microscope image classification model will be obtained. 4) Label a large amount of X-ray image data, extract features from all X-ray data, fuse them with feature information data, normalize the fused features, and then feed them into a machine learning model for training. After training, a machine learning classification model will be obtained.

[0083] 2. Reasoning Stage The reasoning process primarily utilizes a trained model to accurately classify images of unknown defect types. For example... Input: A set of data of unknown defect type (microscope image, X-ray image, feature information), where the microscope image is 3-channel, 1024*1024 pixels; the X-ray image is 1-channel, 64*64 pixels; the feature information includes 10 dimensions such as average gray level, defect size, intensity, sensitivity, etc.

[0084] Output: Defect type (Type A, Type B, Type C, ..., Minor defect type, Other types) 1) First, normalize the microscope images. 2) Feed the normalized image from the previous step into the trained deep learning model for inference. When the maximum score Smax in the score vector predicted by the model is not less than the confidence threshold Th of the model, output the defect type and the corresponding ROI coordinates. 3) When the maximum score Smax in the predicted score vector is less than the model's confidence threshold Th, the gradient and orientation features of any pixel in the X-ray image are extracted using the SIFT algorithm. These features, along with the original grayscale features, are then concatenated into a 3-channel image (1, 3, 64, 64). PCA is then used to reduce the dimensionality of the 3-channel image, retaining only the feature vector N (N=14) with the largest eigenvalue. 4) Concatenate the N (N=14) dimensional features extracted in the previous step with the M dimensional feature information of the defect itself, and then perform Softmax normalization on the concatenated K (K=N+M) dimensional features; 5) Load the normalized features from the previous step into the trained machine learning model for inference. When the maximum score Jmax in the score vector predicted by the machine learning model is not less than the confidence threshold Jh of the machine model, output the defect type. 6) When the maximum score Jmax in the score vector predicted by the machine learning model is less than the confidence threshold Jh of the model, it will be determined whether it is a small defect type based on the defect size in the defect feature information data. If the defect size Size is less than the set defect size threshold St, the small defect type will be output; otherwise, other types will be output.

[0085] It should be noted that Automated Optical Inspection (AOI) equipment is an automated system that uses optical imaging and image processing technology to perform non-contact defect detection. It is widely used in semiconductor manufacturing for defect identification in wafer, chip, and packaging processes.

[0086] SIFT is used to obtain the gradient m(x, y) and direction thta(x, y) of any pixel L(x, y) in the image.

[0087]

[0088]

[0089] in, coordinates gradient at point, The gradient component is in the horizontal direction. The gradient component is in the vertical direction. coordinates The gradient direction at that point.

[0090] The gradient image obtained by SIFT, the orientation image, and the original image are concatenated into a 3-channel image, which is then subjected to PCA dimensionality reduction, and finally K feature vectors are retained.

[0091] The features and feature information extracted in the previous step are sequentially concatenated, and then the concatenated features are normalized using Softmax.

[0092] The formula for calculating Softmax normalization is as follows:

[0093] in, The value is the normalized value. For the original data points, The mean of the data. Standard deviation It is the base of the natural logarithm (Euler's number).

[0094] Deep learning models can be improved by using the commonly used and faster region-based convolutional neural network (Faster-RCNN), and Resnext50 with added exponential moving average (EMA) attention can be used as the backbone network.

[0095] The core of the EMA attention mechanism lies in its ability to learn across space; it divides the feature map into multiple sub-feature groups and captures spatial information at different scales through parallel sub-network structures; within each sub-feature group, EMA recalibrates channel weights and captures pixel-level pairwise relationships through global information encoding and cross-dimensional interaction; this multi-scale attention design enables the model to significantly improve feature representation capabilities without increasing too many parameters and computational costs.

[0096] Advantages of EMA: High efficiency: The EMA attention mechanism avoids complex sequential processing and improves computational efficiency through parallel sub-network design.

[0097] Multi-scale attention: It can capture channel and spatial information simultaneously, providing a more comprehensive feature representation for the model.

[0098] Low computational overhead: Through feature grouping and parallel processing, the amount of computation is reduced, making it suitable for deployment in resource-constrained environments.

[0099] Wide range of applications: It performs well in various computer vision tasks such as image classification and object detection, and has broad application prospects.

[0100] In a specific implementation, the pre-trained deep learning classification model is a ResNet-50 architecture, which may include an input layer, a 7×7 convolutional layer, a max pooling layer, a first residual block group, a second residual block group, a third residual block group, a fourth residual block group, a global average pooling layer, and a fully connected output layer connected in sequence. The first residual block group contains 3 residual blocks, the second residual block group contains 4 residual blocks, the third residual block group contains 6 residual blocks, and the fourth residual block group contains 3 residual blocks. The number of neurons in the fully connected output layer is equal to the number of defect types.

[0101] The parameters of the pre-trained XGBoost machine learning classification model can be configured as follows: maximum tree depth of 8-12, learning rate of 0.05-0.1, subsample ratio of 0.7-0.9, column sampling ratio of 0.6-0.8, minimum leaf node sample weight sum of 1-3, regularization parameter λ of 0.5-1.5, γ of 0-0.2, objective function of multi-class softmax, and evaluation metric of multi-class log loss mlogloss.

[0102] The first preset threshold has a value range of 0.80-0.95, the second preset threshold has a value range of 0.75-0.90, and the first preset threshold is greater than the second preset threshold.

[0103] The key parameters include: the ratio of the defect area to the total product area, the ratio of the perimeter of the defect edge to the area, the mean and standard deviation of the grayscale of the defect area, the aspect ratio of the defect area, the roundness of the defect area, and the minimum distance between the defect area and the product edge. The key parameters are converted into actual physical dimensions through industrial camera calibration coefficients.

[0104] The pre-defined expert knowledge base can be constructed through the following steps: Obtain a historical defect sample dataset, which contains samples of confirmed defect types and their corresponding structured quantitative features; Cluster analysis was performed on the historical defect sample dataset to identify key feature patterns for each defect type; Based on the key feature patterns, domain experts define threshold rule sets for each defect type, including single-parameter threshold rules and multi-parameter combination rules. The threshold rule set is optimized through cross-validation to ensure that the classification accuracy of the rule set on the validation set is not less than 90%. A manual re-inspection step can also be added. That is, when the first classification result does not meet the first preset reliability condition and the second classification result does not meet the second preset reliability condition, a defect re-inspection request is generated. The defect re-inspection request includes the identification information of the product to be inspected, the microscope image, the X-ray image, and the defect feature information data. The defect re-inspection request is sent to the manual re-inspection terminal, and the re-inspection result from the manual re-inspection terminal is received. The re-inspection result is used as the final classification result, and the training datasets of the pre-trained deep learning classification model and the pre-trained XGBoost machine learning classification model are updated.

[0105] The feature fusion also includes any of the following fusion methods: X-ray image features and structured quantitative feature parameters are weighted and fused using an attention mechanism, which calculates the importance weight of each feature. X-ray image features and structured quantitative feature parameters are input into a multilayer perceptron for nonlinear fusion; X-ray image features and structured quantitative feature parameters are dynamically selected and fused using a gating mechanism.

[0106] This embodiment, through the above-described scheme, determines that the second preset reliability condition is met when the second confidence level of the second classification processing result is not less than the second preset threshold, and uses the second classification processing result as the final classification result; when the second confidence level is less than the second preset threshold, it is determined that the second preset reliability condition is not met, and the verified key parameters in the defect feature information data are obtained; based solely on the key parameters, a deterministic matching algorithm is executed using a preset threshold rule set defined in a preset expert knowledge base to determine the defect type and output the final classification result. This enables the technical performance optimization of the defect classification system, effectively improving the overall classification accuracy, reducing the false positive rate, enhancing the system's adaptability to various defect features, and optimizing the allocation efficiency of computing resources. This allows the classification system to maintain high accuracy while possessing stronger robustness and fault tolerance, enhancing the system's adaptability to complex situations, providing more reliable technical support for industrial automation inspection, reducing the false recognition rate, and improving the speed and efficiency of automatic multimodal defect classification.

[0107] Accordingly, the present invention further provides an automatic multimodal defect classification device.

[0108] Reference Figure 8 , Figure 8 This is a functional block diagram of the first embodiment of the multimodal defect automatic classification device of the present invention.

[0109] In a first embodiment of the automatic multimodal defect classification device of the present invention, the automatic multimodal defect classification device includes: The data acquisition module 10 is used to acquire microscope images, X-ray images and defect feature information data of the product to be inspected.

[0110] The initial classification module 20 is used to perform a first classification process based on the microscope image. When the result of the first classification process meets the first preset reliability condition, the first classification process result is directly used as the final classification result. When the result of the first classification process does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed.

[0111] The final classification module 30 is used to take the second classification processing result as the final classification result when the second classification processing result meets the second preset reliability condition, and to determine the defect type and output the final classification result based only on the key parameters in the defect feature information data when the second classification processing result does not meet the second preset reliability condition.

[0112] The steps for implementing each functional module of the multimodal defect automatic classification device can be referred to in the various embodiments of the multimodal defect automatic classification method of the present invention, and will not be repeated here.

[0113] Furthermore, this embodiment of the invention also proposes a storage medium storing a multimodal defect automatic classification program, which, when executed by a processor, implements the operations described in the above embodiments of the multimodal defect automatic classification method.

[0114] Those skilled in the art will understand that all or part of the steps in the methods described above can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium is a computer-readable storage medium, including: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program code.

[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0116] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0117] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An automatic classification method for multimodal defects, characterized in that, The automatic classification method for multimodal defects includes: Acquire microscopic images, X-ray images, and defect feature information data of the product to be inspected; The first classification process is performed based on the microscope image. When the result of the first classification process meets the first preset reliability condition, the first classification process result is directly used as the final classification result. When the result of the first classification process does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and the second classification process is executed. When the second classification result meets the second preset reliability condition, the second classification result is taken as the final classification result. When the second classification result does not meet the second preset reliability condition, the defect type is judged based only on the key parameters in the defect feature information data and the final classification result is output.

2. The automatic multimodal defect classification method as described in claim 1, characterized in that, The acquisition of microscope images, X-ray images, and defect feature information data of the product to be inspected includes: Microscopic images of the product to be inspected are acquired using an automated optical inspection (AOI) device. These microscopic images are used to characterize the morphological features of surface defects in the product to be inspected. An X-ray image of the product to be inspected is acquired using an X-ray imaging system. The X-ray image is used to characterize the distribution of internal defects in the product to be inspected. Structured quantitative feature data is extracted simultaneously during the acquisition of the microscope images and the X-ray images, and the structured quantitative feature data is used as defect feature information data.

3. The automatic multimodal defect classification method as described in claim 1, characterized in that, The process prioritizes performing a first classification based on the microscope image. When the result of the first classification meets a first preset reliability condition, the first classification result is directly used as the final classification result. When the result of the first classification does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed, including: The microscope image is normalized to obtain a normalized target microscope image; The target microscope image is input into a pre-trained deep learning classification model to obtain a first classification result and a first confidence value corresponding to the first classification result; When the first confidence value is not less than the first preset threshold, it is determined that the first preset reliability condition is met, and the first classification result is directly used as the final classification result. When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met, and feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed.

4. The automatic multimodal defect classification method as described in claim 3, characterized in that, The step of normalizing the microscope image to obtain a normalized target microscope image includes: The microscope image is read from an image processing library, and the light intensity value of each pixel in the microscope image is converted into an image matrix; The image matrix is ​​normalized using the following formula to obtain the normalized target microscope image: in, This is the normalized target microscope image. This refers to the light intensity value of a single pixel in a microscope image. This is the mean value of all pixel values ​​in the microscope image. This is the adjusted standard deviation. The standard deviation of the microscope image. This represents the total number of pixels in the microscope image.

5. The automatic multimodal defect classification method as described in claim 3, characterized in that, When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met. Based on the X-ray image and the defect feature information data, feature fusion is performed and a second classification process is executed, including: When the first confidence value is less than the first preset threshold, it is determined that the first preset reliability condition is not met. The SIFT algorithm is used to extract features from the X-ray image to obtain the gradient features and orientation features of key points in the X-ray image. The gradient features, orientation features and original grayscale features are then stitched together to form a three-channel image. Principal component analysis (PCA) is used to reduce the dimensionality of the three-channel image, retaining the eigenvector with the largest eigenvalue to form X-ray image features. Structured quantitative feature parameters related to the current detection are extracted from the defect feature information database, and the X-ray image features and the structured quantitative feature parameters are sequentially stitched together to form multidimensional features; The multidimensional features are subjected to Softmax normalization to obtain the fused and normalized target feature vector; The target feature vector is input into a pre-trained XGBoost machine learning classification model to perform a second classification process.

6. The automatic multimodal defect classification method as described in claim 5, characterized in that, The step of performing Softmax normalization on the multidimensional features to obtain the fused and normalized target feature vector includes: Analyze the statistical distribution characteristics of each dimension in the multidimensional features, and calculate the central tendency and dispersion of each feature dimension; Based on the central tendency and the degree of dispersion, the feature values ​​of each dimension are standardized to obtain standardized feature values; The standardized feature values ​​are converted into a probability distribution form by using the Softmax normalization function, so that the sum of the weight values ​​of each feature dimension is 1 and all are non-negative, thus obtaining the normalized weight coefficients. The original feature vectors corresponding to the X-ray image features and the structured quantitative feature parameters are weighted and fused according to the weight coefficients to obtain the fused and normalized target feature vector.

7. The automatic multimodal defect classification method as described in claim 1, characterized in that, When the second classification processing result meets the second preset reliability condition, the second classification processing result is taken as the final classification result; when the second classification processing result does not meet the second preset reliability condition, the defect type is determined and the final classification result is output based solely on the key parameters in the defect feature information data, including: When the second confidence level of the second classification processing result is not less than the second preset threshold, it is determined that the second preset reliability condition is met, and the second classification processing result is taken as the final classification result. When the second confidence level is less than the second preset threshold, it is determined that the second preset reliability condition is not met, and the verified key parameters in the defect feature information data are obtained. Based solely on the key parameters, a deterministic matching algorithm is executed using a preset threshold rule set defined in a preset expert knowledge base to determine the defect type and output the final classification result.

8. An automatic multimodal defect classification device, characterized in that, The multimodal defect automatic classification device includes: The data acquisition module is used to acquire microscope images, X-ray images, and defect feature information data of the product to be inspected; The initial classification module is used to perform a first classification process based on the microscope image. When the result of the first classification process meets the first preset reliability condition, the first classification process result is directly used as the final classification result. When the result of the first classification process does not meet the first preset reliability condition, feature fusion is performed based on the X-ray image and the defect feature information data, and a second classification process is executed. The final classification module is used to take the second classification result as the final classification result when the second classification result meets the second preset reliability condition, and to determine the defect type and output the final classification result based only on the key parameters in the defect feature information data when the second classification result does not meet the second preset reliability condition.

9. An automatic multimodal defect classification device, characterized in that, The multimodal defect automatic classification device includes: a memory, a processor, and a multimodal defect automatic classification program stored in the memory and executable on the processor, the multimodal defect automatic classification program being configured to implement the steps of the multimodal defect automatic classification method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a multimodal defect automatic classification program, which, when executed by a processor, implements the steps of the multimodal defect automatic classification method as described in any one of claims 1 to 7.

Images