Classification method of SMT 3D image data based on deep learning framework

Abstract

The present invention relates to the technical field of image data classification, and discloses a classification method for SMT 3D image data based on a deep learning framework. The deep learning technology is effectively used to reduce the dependence on a large amount of labeled data, and the transfer learning technology is introduced into the SMT 3D image classification. By using a model pre-trained on a large-scale data set and fine-tuning it based on production line data, feature learning of a specific production line is achieved, the training efficiency of the model is improved, and efficient and accurate classification of SMT 3D images is achieved, thereby further improving the quality inspection level in the SMT manufacturing process.

Description

Technical Field

[0001] The present invention relates to the technical field of image data classification, and in particular to a classification method for SMT 3D image data based on a deep learning framework. Background Art

[0002] Surface Mount Technology (SMT) is a widely used technology in modern electronic manufacturing. It achieves miniaturization and high density of electronic products by directly mounting electronic components on the surface of printed circuit boards (PCBs). In the SMT manufacturing process, the quality inspection of components and solder joints is crucial to ensure the reliability and performance of the final product. Traditional SMT inspection methods mainly rely on 2D image processing technology, which has some limitations, such as lack of depth information and difficulty in detecting complex defects. To overcome these problems, 3D imaging technology has gradually been introduced into SMT inspection. SMT 3D images provide three-dimensional information of components and solder joints, enabling the inspection system to more accurately identify and classify various defects, such as too much solder, too little solder, bridging, open circuits, etc. However, with the continuous advancement and complexity of SMT manufacturing processes, the amount of 3D image data is large and complex, and traditional image processing methods have been unable to meet the requirements of efficient and accurate inspection.

[0003] In recent years, deep learning technology, especially convolutional neural networks (CNN), has made significant progress in the field of image recognition and classification. Deep learning models achieve efficient and accurate image classification by automatically learning image features. The introduction of this technology provides a new solution for SMT3D image classification. However, training a high-performance deep learning model usually requires a large amount of labeled data and computing resources. The acquisition and labeling process of SMT3D image data is complex and expensive. Especially in the electronics manufacturing industry, different types of defects require professional knowledge for accurate labeling. Therefore, the cost of data acquisition and labeling is too high and the demand for computing resources is too large, which further affects the efficiency and accuracy of SMT3D image classification. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a classification method for SMT 3D image data based on a deep learning framework. By utilizing migration technology, the pre-trained model and 3D image feature extraction are effectively combined to achieve efficient and accurate classification of SMT 3D images, thereby improving the quality inspection level in the SMT manufacturing process.

[0005] The present invention discloses a classification method for SMT 3D image data based on a deep learning framework, the method comprising the following steps:

[0006] Acquiring production line data, the production line data including SMT 3D image data, equipment configuration data, operating condition data, and product feature data;

[0007] Performing data preprocessing on the production line data to obtain first processed data;

[0008] Associating the equipment configuration data, the operating condition data, the product feature data with the corresponding SMT 3D image data based on the first processed data to obtain first associated data;

[0009] Extracting features of the SMT 3D image data based on the first associated data, and analyzing the impact of equipment configuration, operating conditions, and product features on the SMT 3D image features to obtain a first analysis result;

[0010] Selecting a pre-trained deep learning model, wherein the pre-trained deep learning model is trained based on a large-scale SMT 3D image dataset;

[0011] Based on the first associated data and the first analysis result, fine-tuning the pre-trained deep learning model by using a migration technology to obtain an SMT 3D image data classification model;

[0012] The target SMT 3D image data is classified by the SMT 3D image data classification model.

[0013] Furthermore, SMT 3D image data is acquired through a 3D scanning device, the SMT 3D image data includes point cloud data of components and solder joints, and the pre-trained deep learning model includes a point cloud data processing module.

[0014] Further, the device configuration data, the operating condition data, the product feature data and the corresponding SMT 3D image data are associated based on the first processed data to obtain the first associated data, specifically including:

[0015] Based on the first processed data, the device configuration data, the operating condition data, the product feature data, and the timestamp of the SMT 3D image data are obtained, all the data are sorted and aligned according to the timestamps, and a linear interpolation algorithm is used to perform alignment processing to obtain first aligned data;

[0016] The physical position of each SMT device, component and solder joint is marked using a unified coordinate system to obtain three-dimensional space coordinate data;

[0017] A position alignment operation is performed on the first alignment data based on the three-dimensional space coordinate data to obtain first associated data.

[0018] Furthermore, the feature of extracting the SMT 3D image data based on the first associated data specifically includes:

[0019] Extracting geometric features from the SMT 3D image point cloud data in the first associated data by using a 3D shape descriptor;

[0020] Extracting texture features from the SMT 3D image point cloud data in the first associated data by using a 3D texture descriptor;

[0021] Color features are extracted from the SMT 3D image point cloud data in the first associated data.

[0022] Further, the analysis operation in analyzing the influence of equipment configuration, operating conditions and product characteristics on SMT 3D image characteristics includes multivariate regression analysis and principal component analysis.

[0023] Furthermore, the multivariate regression analysis operation specifically includes:

[0024] Combine the extracted geometry, texture, and color features with equipment configuration data, operating condition data, and product feature data to form a comprehensive input data set;

[0025] A multivariate regression model was used to evaluate and quantify the degree of influence of equipment configuration, operating conditions, and product characteristics on image features to obtain the first analysis sub-result.

[0026] Furthermore, the principal component analysis operation specifically includes:

[0027] performing a standardization operation on the integrated input data set;

[0028] The principal component analysis algorithm is used to reduce the dimension of the data after the standardization operation and extract the main features to obtain the second sub-analysis results;

[0029] The first analysis result includes a first sub-analysis result and a second sub-analysis result.

[0030] Furthermore, the pre-trained deep learning model is a joint architecture based on convolutional neural network and graph convolutional network.

[0031] Furthermore, the training process of the pre-trained deep model specifically includes:

[0032] Collecting a large-scale SMT 3D image dataset, wherein the large-scale SMT 3D image dataset includes SMT 3D image data in a production environment consisting of different equipment configurations, operating conditions, and product characteristics;

[0033] Performing a data preprocessing operation on the collected large-scale SMT 3D image data set to obtain second processed data;

[0034] Construct a convolutional neural network and extract the geometric features of components and solder joints in large-scale SMT 3D image data through multi-layer convolution and pooling layers of the convolutional neural network;

[0035] Construct a graph convolutional network and extract the topological structure features of large-scale SMT 3D image point cloud data through multiple layers of graph convolutional layers and aggregation layers in the graph convolutional network;

[0036] The output features of the convolutional neural network and the graph convolutional network are fused to obtain a joint architecture model;

[0037] The second processed data is divided into a training set and a validation set, and the joint architecture model is pre-trained by a supervised learning algorithm based on the divided training set and the validation set to obtain a pre-trained deep learning model;

[0038] Among them, in the process of extracting SMT 3D image data features through convolutional neural networks and graph convolutional networks and fusing the output features of convolutional neural networks and graph convolutional networks, the features are dynamically adjusted through the attention mechanism.

[0039] Furthermore, the method of fine-tuning the pre-trained deep learning model by using the migration technology based on the first associated data and the first analysis result to obtain the SMT 3D image data classification model specifically includes:

[0040] Determine a training data set for a pre-trained deep learning model based on the first associated data and the first analysis result;

[0041] Initialize the pre-trained deep learning model and determine the hierarchical structure of the model;

[0042] Based on the characteristics of the training dataset and the initial training progress, select some layers to freeze and only train higher-level layers;

[0043] After each training cycle, the model performance is evaluated, and the frozen layers and training layers are dynamically adjusted. Based on the evaluation results, the lower-level layers are gradually unfrozen and the higher-level layers are frozen, and optimization is performed layer by layer.

[0044] After the model converges, all layers are jointly trained to obtain the SMT 3D image data classification model.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention reduces the dependence on a large amount of labeled data by effectively utilizing deep learning technology, and introduces transfer learning technology into SMT 3D image classification. By using a model pre-trained on a large-scale data set and fine-tuning it based on production line data, feature learning for a specific production line is achieved, thereby improving the training efficiency of the model and achieving efficient and accurate classification of SMT 3D images, thereby further improving the quality inspection level in the SMT manufacturing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0048] Figure 1 A schematic flow chart of a method for classifying SMT 3D image data based on a deep learning framework disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0050] Embodiment 1

[0051] See also Figure 1 , Figure 1 : is a flow chart of a method for classifying SMT 3D image data based on a deep learning framework disclosed in an embodiment of the present invention, the method comprising the following steps:

[0052] Acquire production line data, which includes SMT 3D image data, equipment configuration data, operating condition data, and product feature data.

[0053] Specifically, in the embodiments of the present invention, including but not limited to obtaining SMT 3D image data through a 3D scanning device, for example, SMT 3D image data can also be collected by configuring sensors and data recording systems. SMT 3D image data includes point cloud data of components and solder joints, and the point cloud data of components and solder joints is the three-dimensional coordinates of components and solder joints. In addition, SMT 3D image data can also include high-resolution 3D images showing the shapes and positions of components and solder joints, etc., which are not limited in the embodiments of the present invention.

[0054] Equipment configuration data includes but is not limited to equipment types such as placement machines, reflow soldering machines, and 3D AOI equipment; equipment models and specifications such as the specific model of the equipment, manufacturer, and performance parameters; equipment calibration data, including camera calibration and laser calibration; current equipment operating status (such as online, offline, faulty, etc.); and equipment maintenance and servicing records, including replacement parts and calibration records.

[0055] The operating condition data includes but is not limited to environmental conditions such as the temperature, humidity, air pressure and other data of the production environment; the operating speed, mounting speed and welding speed of the production line; the specific operating parameters of each process, such as mounting pressure, welding temperature and welding time; the ID, operating shift and operating records of the operators involved in the production operation; and the ID, start time and end time of each production batch.

[0056] Product characteristic data includes but is not limited to specific product models and specifications; model, specification, supplier and other information of each component; model, number of layers, material and size of circuit boards; circuit board design drawings, component layout diagrams, welding point diagrams; and product inspection records, defect records, qualified rates, etc.

[0057] The production line data is preprocessed to obtain first processed data.

[0058] Specifically, the operations of data preprocessing on the production line data include but are not limited to data cleaning, format unification, data standardization and normalization, with the purpose of ensuring data quality.

[0059] Based on the first processed data, the equipment configuration data, the operating condition data, the product feature data and the corresponding SMT 3D image data are associated to obtain first associated data.

[0060] Specifically, based on the first processed data, the timestamps of the device configuration data, the operating condition data, the product feature data, and the SMT 3D image data are obtained, all the data are sorted and aligned according to the timestamps, and the linear interpolation algorithm is used to perform alignment processing to obtain the first aligned data. For each time point, the value of the missing data is interpolated. For example, for timestamp t, if a data source has no data at t, the data of the two closest time points before and after t can be used for linear interpolation to calculate the value at time t.

[0061] The physical position of each SMT device, component and solder joint is marked using a unified coordinate system to obtain three-dimensional space coordinate data.

[0062] First, a unified three-dimensional space coordinate system is defined to mark the physical location of each SMT device, component, and solder joint. Specifically, the coordinate system is defined based on the layout of the production line, for example, the starting point of the production line is used as the origin, the X-axis is along the length of the production line, the Y-axis is along the width of the production line, and the Z-axis is the vertical direction.

[0063] Then, the physical location of each SMT device, component, and solder joint is marked, and its three-dimensional coordinates in a unified coordinate system are recorded. For example, for a solder joint, its position can be expressed as (x, y, z), where x, y, and z represent the X, Y, and Z axis coordinates of the solder joint in the unified coordinate system, respectively.

[0064] Performing a position alignment operation on the first alignment data based on the three-dimensional spatial coordinate data to obtain first associated data. Aligning the device configuration data, operating condition data, product feature data and SMT 3D image data according to the same spatial position to ensure that the spatial positions of different data sources are consistent.

[0065] After all data are aligned to the same spatial position, the data at each time point contains equipment configuration, operating conditions, product characteristics and SMT 3D image data.

[0066] Through the above steps, the time and space alignment of the production line data can be achieved, ensuring the consistency and integrity of the data, and providing a high-quality data foundation for subsequent feature extraction and deep learning model training.

[0067] The features of the SMT 3D image data are extracted based on the first associated data, and the influence of the equipment configuration, operating conditions and product features on the SMT 3D image features is analyzed to obtain a first analysis result.

[0068] Furthermore, geometric features are extracted from the SMT 3D image point cloud data in the first associated data through a 3D shape descriptor; texture features are extracted from the SMT 3D image point cloud data in the first associated data through a 3D texture descriptor; and color features are extracted from the SMT 3D image point cloud data in the first associated data.

[0069] In an embodiment of the present invention, a 3D shape descriptor including surface normal, curvature, and point feature histogram is used to extract geometric features of SMT 3D image point cloud data. Specifically, surface normal calculation mainly includes calculating the surface normal of each point in its local neighborhood. The surface normal provides local geometric information of each point, and the normal direction of the point is calculated using the principal component analysis method; curvature calculation mainly includes calculating the curvature of each point to describe the surface change of the point. The curvature calculation can be based on the normal change rate or the spatial distribution of the neighborhood points; point feature histogram calculation mainly includes calculating the point feature histogram or fast point feature histogram of each point. These features describe the local geometric relationship of the points. The point feature histogram generates a feature vector by calculating the angle and distance information between the point pairs in the neighborhood of each point. The fast point feature histogram is an accelerated version of the point feature histogram, which improves the feature extraction efficiency by reducing the computational complexity.

[0070] The texture features of the SMT 3D image point cloud data are extracted using 3D texture descriptors including grayscale co-occurrence matrix and local binary pattern. Specifically, in the grayscale co-occurrence matrix, the point cloud data is converted into a grayscale image by projecting it, and the grayscale co-occurrence matrix is ​​calculated to describe the spatial relationship between the grayscale values ​​of pixels, and texture features are extracted from the grayscale co-occurrence matrix, including but not limited to contrast, energy, and entropy. In the local binary pattern, the point cloud data is also converted into a grayscale image by projecting it, and the local binary pattern of each pixel is calculated to generate a binary pattern map describing the texture, and texture features such as the histogram distribution of the pattern are extracted from the binary pattern map.

[0071] When extracting color features from the SMT 3D image point cloud data in the first associated data, specifically, the RGB color value of each point in the point cloud data is extracted, and a color histogram is calculated and normalized. In addition, the average value and standard deviation of each color channel are calculated as a color feature vector.

[0072] Furthermore, the analysis operations in analyzing the impact of equipment configuration, operating conditions and product characteristics on SMT 3D image characteristics include multivariate regression analysis and principal component analysis.

[0073] The multivariate regression analysis operation specifically includes:

[0074] The extracted geometry, texture, and color features are combined with equipment configuration data, operating condition data, and product characteristic data to form a comprehensive input data set.

[0075] The multivariate regression model is used to evaluate and quantify the influence of equipment configuration, operating conditions, and product characteristics on image features to obtain the first analysis sub-result. The regression coefficients or feature importance obtained by the multivariate regression model are analyzed to determine which equipment configuration, operating conditions, and product characteristics have the most significant influence on image features.

[0076] The principal component analysis operations include:

[0077] A standardization operation is performed on the comprehensive input data set; the dimension of the data after the standardization operation is reduced by a principal component analysis algorithm, and the main features are extracted to obtain the second sub-analysis result.

[0078] Specifically, the covariance matrix of the data set after the standardization operation is calculated, and the relationship between different features is reflected by the covariance matrix. After that, the covariance matrix is ​​subjected to eigenvalue decomposition to obtain eigenvalues ​​and corresponding eigenvectors, where the eigenvalues ​​represent the variance in the data and the eigenvectors represent the direction of the principal components. The first few principal components with higher cumulative variance contribution rates are selected according to the eigenvalue size. The original data is projected onto the selected principal component to obtain the reduced-dimensional data set, i.e., the second sub-analysis result.

[0079] The first analysis result includes a first sub-analysis result and a second sub-analysis result. The first analysis sub-result is generated by multivariate regression analysis and specifically includes:

[0080] Impact assessment: The multivariate regression model assesses and quantifies the impact of equipment configuration, operating conditions, and product characteristics on SMT 3D image features. Therefore, the first analysis sub-result includes the specific numerical impact of each factor on the image features, such as the regression coefficient or weight of each feature.

[0081] Identification of important features: Determine which equipment configurations, operating conditions and product features have the most significant impact on SMT 3D image data under the current production environment, thereby assisting in identifying key production parameters and providing guidance for subsequent optimization and adjustment of the production line.

[0082] The second analysis sub-results are generated by principal component analysis, including:

[0083] Main feature extraction: The main features extracted after principal component analysis and dimensionality reduction can reflect the main change patterns in the original data. These principal components can be regarded as new representations that integrate multiple input features, with better interpretability and lower dimension.

[0084] Dimensionality reduction representation of data: It provides a compact representation of the original data set, reduces the complexity and redundancy of the feature space, helps identify patterns and associations in the data, and provides more effective input for subsequent pre-trained deep learning models.

[0085] Furthermore, a pre-trained deep learning model is selected and trained based on a large-scale SMT 3D image dataset, wherein the pre-trained deep learning model is a joint architecture based on a convolutional neural network and a graph convolutional network.

[0086] Specifically, the training process of the pre-trained deep model includes:

[0087] Collect a large-scale SMT 3D image dataset, which includes SMT 3D image data in production environments with different equipment configurations, operating conditions, and product features.

[0088] The collected large-scale SMT 3D image data set is subjected to data preprocessing to obtain second processed data. Specifically, a filter is used to perform denoising on the point cloud data to remove abnormal points and noise. The geometric features of the point cloud data are then normalized, and data enhancement is performed through operations such as rotation, scaling, and translation. Similarly, the large-scale SMT 3D image data is subjected to standardization and feature extraction of geometric features, texture features, and color features to form second processed data.

[0089] Construct a convolutional neural network and extract the geometric features of components and solder joints in large-scale SMT 3D image data through multi-layer convolution and pooling layers of the convolutional neural network.

[0090] In an embodiment of the present invention, the convolutional neural network is multi-layered, including convolutional layers, pooling layers, and fully connected layers. The geometric features of components and solder joints in the SMT 3D image data are extracted through the convolutional layer, and the features are reduced and compressed using maximum pooling or average pooling to retain important features. It should be noted that the geometric features extracted by the convolutional neural network are different from the geometric features extracted in the preprocessing operation. The geometric feature extraction in the data preprocessing is preliminary and basic, such as the three-dimensional coordinates and distance of the point cloud data. The main purpose is to clean and standardize the data to provide clean and standardized data for subsequent model input. The geometric features extracted by the convolutional neural network are deep and abstract, such as the edges and textures of the local area. Through multi-layer convolution and pooling operations, high-level features are gradually extracted. The purpose is to extract high-level features in the data through automatic learning for accurate classification.

[0091] Then, a graph convolutional network is constructed to extract the topological structure features of large-scale SMT 3D image point cloud data through multiple layers of graph convolutional layers and aggregation layers in the graph convolutional network.

[0092] The output features of the convolutional neural network and the graph convolutional network are fused to obtain a joint architecture model.

[0093] The second processed data is divided into a training set and a validation set, and the joint architecture model is pre-trained through a supervised learning algorithm based on the divided training set and validation set to obtain a pre-trained deep learning model.

[0094] Among them, in the process of extracting SMT 3D image data features through convolutional neural networks and graph convolutional networks and fusing the output features of convolutional neural networks and graph convolutional networks, the features are dynamically adjusted through the attention mechanism.

[0095] After obtaining the pre-trained deep learning model, the pre-trained deep learning model is fine-tuned by using a migration technology based on the first associated data and the first analysis result to obtain an SMT 3D image data classification model.

[0096] In an embodiment of the present invention, the first associated data is the result of associating the equipment configuration data, operating condition data, product feature data with the corresponding SMT 3D image data. This data set is highly associated and consistent, ensuring that the model can learn the correlation between various types of data during the training process. In addition, adjusting the structure and parameters of the pre-trained deep learning model according to the first analysis result can make it better adapt to current production conditions and data characteristics. By training using associated data and analysis results, the model can learn the characteristics of SMT 3D image data under different conditions and configurations, thereby improving its generalization ability. Training based on associated data and analysis results can more efficiently adjust the pre-trained model so that the model can achieve better classification results.

[0097] After the SMT 3D image data classification model is obtained, the target SMT 3D image data is classified by using the SMT 3D image data classification model.

[0098] Furthermore, the pre-trained deep learning model includes a point cloud data processing module for processing SMT 3D image point cloud data.

[0099] Furthermore, the method of fine-tuning the pre-trained deep learning model by using the migration technology based on the first associated data and the first analysis result to obtain the SMT 3D image data classification model specifically includes:

[0100] Determine a training data set for a pre-trained deep learning model based on the first associated data and the first analysis result;

[0101] Initialize the pre-trained deep learning model and determine the hierarchical structure of the model;

[0102] Based on the characteristics of the training dataset and the initial training progress, select some layers to freeze and only train higher-level layers;

[0103] After each training cycle, the model performance is evaluated, and the frozen layers and training layers are dynamically adjusted. Based on the evaluation results, the lower-level layers are gradually unfrozen and the higher-level layers are frozen, and optimization is performed layer by layer.

[0104] After the model converges, all layers are jointly trained to obtain the SMT 3D image data classification model.

[0105] Furthermore, as a preferred implementation of Example 1 of the present invention, an attention mechanism module is introduced in the high-level feature extraction layer and the classification layer. The module includes a self-attention layer and a multi-head attention layer. During the training process, the weight of the attention mechanism is dynamically adjusted according to the characteristics of the input data, so that the model can focus on information with important features. After each training cycle, the parameters of the attention mechanism are adjusted using the loss function and the evaluation index to ensure that the attention mechanism can adapt to the data characteristics in real time.

[0106] Furthermore, as another preferred implementation of the embodiment of the present invention, a meta-learning mechanism is introduced during model fine-tuning. Specifically, a small amount of data is selected from the first associated data and the first analysis result as a data set for meta-learning. The meta-learning model is initialized, and trained on a small amount of fine-tuning data through a meta-learning algorithm so that the model can quickly adapt to the new production environment. After the meta-learning model converges, its parameters are applied to the pre-trained deep learning model as initial parameters for fine-tuning.

[0107] Through the above operations, transfer learning technology can not only adapt to the new production environment, but also significantly improve the adaptability, efficiency and accuracy of the model, and further improve the classification accuracy of the SMT 3D image data classification model.

[0108] Finally, it should be noted that the classification method for SMT 3D image data based on a deep learning framework disclosed in the embodiment of the present invention only discloses a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A classification method for SMT 3D image data based on a deep learning framework, characterized in that: The method comprises the following steps: Acquiring production line data, the production line data including SMT 3D image data, equipment configuration data, operating condition data, and product feature data; Performing data preprocessing on the production line data to obtain first processed data; Associating the device configuration data, the operating condition data, the product feature data with the corresponding SMT 3D image data based on the first processed data to obtain first associated data; Extracting features of the SMT 3D image data based on the first associated data, and analyzing the impact of equipment configuration, operating conditions, and product features on the SMT 3D image features to obtain a first analysis result; Selecting a pre-trained deep learning model, wherein the pre-trained deep learning model is trained based on a large-scale SMT 3D image dataset; Based on the first associated data and the first analysis result, fine-tuning the pre-trained deep learning model by using a migration technology to obtain an SMT 3D image data classification model; Classify the target SMT 3D image data through the SMT 3D image data classification model; The method of associating the equipment configuration data, the operating condition data, the product feature data and the corresponding SMT 3D image data based on the first processed data to obtain the first associated data specifically includes: Based on the first processed data, the device configuration data, the operating condition data, the product feature data, and the timestamp of the SMT 3D image data are obtained, all the data are sorted and aligned according to the timestamps, and a linear interpolation algorithm is used to perform alignment processing to obtain first aligned data; The physical position of each SMT device, component and solder joint is marked using a unified coordinate system to obtain three-dimensional space coordinate data; Performing a position alignment operation on the first alignment data based on the three-dimensional space coordinate data to obtain first associated data; The analysis operation in analyzing the influence of equipment configuration, operating conditions and product characteristics on SMT 3D image characteristics includes multivariate regression analysis and principal component analysis; The multivariate regression analysis operation specifically includes: Combine the extracted geometry, texture, and color features with equipment configuration data, operating condition data, and product feature data to form a comprehensive input data set; Use a multivariate regression model to evaluate and quantify the degree of influence of equipment configuration, operating conditions, and product characteristics on image features to obtain the first analysis sub-result; The principal component analysis operation specifically includes: performing a standardization operation on the integrated input data set; The principal component analysis algorithm is used to reduce the dimension of the data after the standardization operation and extract the main features to obtain the second sub-analysis result; The first analysis result includes a first sub-analysis result and a second sub-analysis result.

2. The classification method of SMT 3D image data based on deep learning framework according to claim 1, characterized in that: The SMT 3D image data is acquired by a 3D scanning device, the SMT 3D image data includes point cloud data of components and solder joints, and the pre-trained deep learning model includes a point cloud data processing module.

3. The classification method of SMT 3D image data based on deep learning framework according to claim 2, characterized in that: The feature extraction of the SMT 3D image data based on the first associated data specifically includes: Extracting geometric features from the SMT 3D image point cloud data in the first associated data by using a 3D shape descriptor; Extracting texture features from the SMT 3D image point cloud data in the first associated data by using a 3D texture descriptor; Color features are extracted from the SMT 3D image point cloud data in the first associated data.

4. The classification method of SMT 3D image data based on a deep learning framework according to any one of claims 1 to 3, characterized in that: The pre-trained deep learning model is a joint architecture based on convolutional neural network and graph convolutional network.

5. The classification method of SMT 3D image data based on deep learning framework according to claim 4, characterized in that: The training process of the pre-trained deep learning model specifically includes: Collecting a large-scale SMT 3D image dataset, wherein the large-scale SMT 3D image dataset includes SMT 3D image data in a production environment consisting of different equipment configurations, operating conditions, and product characteristics; Performing a data preprocessing operation on the collected large-scale SMT 3D image data set to obtain second processed data; Construct a convolutional neural network and extract the geometric features of components and solder joints in large-scale SMT 3D image data through multi-layer convolution and pooling layers of the convolutional neural network; Construct a graph convolutional network and extract the topological structure features of large-scale SMT 3D image point cloud data through multiple layers of graph convolutional layers and aggregation layers in the graph convolutional network; The output features of the convolutional neural network and the graph convolutional network are fused to obtain a joint architecture model; The second processed data is divided into a training set and a validation set, and the joint architecture model is pre-trained by a supervised learning algorithm based on the divided training set and the validation set to obtain a pre-trained deep learning model; Among them, in the process of extracting SMT 3D image data features through convolutional neural networks and graph convolutional networks and fusing the output features of convolutional neural networks and graph convolutional networks, the features are dynamically adjusted through the attention mechanism.

6. The classification method of SMT 3D image data based on deep learning framework according to claim 5, characterized in that: The method of fine-tuning the pre-trained deep learning model by using the migration technology based on the first associated data and the first analysis result to obtain the SMT 3D image data classification model specifically includes: Determine a training data set for a pre-trained deep learning model based on the first associated data and the first analysis result; Initialize the pre-trained deep learning model and determine the hierarchical structure of the model; Based on the characteristics of the training dataset and the initial training progress, select some layers to freeze and only train higher-level layers; After each training cycle, the model performance is evaluated, and the frozen layers and training layers are dynamically adjusted. Based on the evaluation results, the lower-level layers are gradually unfrozen and the higher-level layers are frozen, and optimization is performed layer by layer. After the model converges, all layers are jointly trained to obtain the SMT 3D image data classification model.

Images