TP lamination process production process defect diagnosis method and system based on image analysis

Abstract

The embodiments of the present invention relate to the field of image processing technology, and specifically to a TP bonding process production process defect diagnosis method and system based on image analysis. The embodiments of the present invention first collect a real-time image sequence corresponding to the bonding components to be inspected that are continuously transmitted on the TP bonding process production line, and the sequence includes component surface images and edge area images at different bonding stages; secondly, the real-time image sequence is subjected to defect-sensitive feature enhancement processing to obtain a defect-sensitive feature set including surface texture features, edge contour features, and regional grayscale features; then, the defect-sensitive feature set is subjected to feature association analysis processing through a trained TP bonding defect diagnosis model to generate a preliminary defect diagnosis result; finally, the defect type and location distribution information are determined based on the preliminary diagnosis result, and then a production process defect diagnosis report containing defect diagnosis content is generated, thereby realizing accurate diagnosis of defects in the TP bonding process production process.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method and system for diagnosing defects in a TP lamination process production process based on image analysis. Background Art

[0002] In the field of TP bonding process production, with the continuous improvement of product quality requirements, accurate diagnosis of defects in the production process has become crucial. Existing TP bonding process defect diagnosis technologies mainly rely on manual inspection or simple image analysis methods. However, manual inspection methods are inefficient and highly subjective, making them difficult to adapt to the needs of large-scale production and prone to missed detections and false detections. Simple image analysis methods are often only able to extract and analyze single features from images, such as focusing only on the partial texture of the component surface or the rough outline of the edge, and are unable to fully capture defect information at different bonding stages. At the same time, these methods lack in-depth exploration of the correlation between features, resulting in low accuracy and reliability of diagnostic results. Therefore, how to achieve more efficient, accurate, and comprehensive TP bonding process defect diagnosis is a technical problem that needs to be solved. Summary of the Invention

[0003] In view of this, the present invention provides a method and system for diagnosing defects in a TP lamination process production process based on image analysis.

[0004] An embodiment of the present invention provides a method for diagnosing defects in a TP laminating process production process based on image analysis, which is applied to a system for diagnosing defects in a TP laminating process production process based on image analysis. The method includes:

[0005] Collecting real-time image sequences corresponding to the components to be inspected that are continuously transmitted on the TP bonding process production line. The real-time image sequences include component surface images and edge area images at different bonding stages.

[0006] Performing defect-sensitive feature enhancement processing on the real-time image sequence to obtain a defect-sensitive feature set of the bonded component to be inspected, wherein the defect-sensitive feature set includes surface texture features, edge contour features, and regional grayscale features;

[0007] Performing feature correlation analysis on the defect-sensitive feature set using a trained TP bonding defect diagnosis model to generate a preliminary defect diagnosis result for the bonding component to be inspected;

[0008] Based on the preliminary defect diagnosis results, determine the defect type of the assembly to be inspected and the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and generate a production process defect diagnosis report containing defect diagnosis content based on the defect type and the position distribution information.

[0009] The present invention also provides a TP lamination process production process defect diagnosis system based on image analysis, including: a memory for storing program instructions and data; a processor for coupling with the memory and executing the instructions in the memory to implement the above method.

[0010] The present invention also provides a computer storage medium comprising instructions, which implement the above method when executed on a processor.

[0011] The embodiments of the present invention significantly improve the accuracy, comprehensiveness, and efficiency of defect diagnosis during the TP lamination process. First, a real-time image sequence containing component surface images and edge region images at different lamination stages is collected, providing a rich and dynamic data source for defect diagnosis, enabling the capture of potential defect information at each stage of the lamination process. Second, defect-sensitive feature enhancement processing is performed on the real-time image sequence to obtain a defect-sensitive feature set containing surface texture features, edge contour features, and regional grayscale features. This effectively highlights key defect-related features and improves the sensitivity of subsequent diagnosis. Then, feature correlation analysis is performed on the defect-sensitive feature set using a trained TP lamination defect diagnosis model, which can explore the inherent connections between different features and generate more reliable preliminary defect diagnosis results. Finally, based on the preliminary diagnosis results, the defect type and location distribution information are determined, and a production process defect diagnosis report containing the defect diagnosis content is generated. This achieves accurate defect location and comprehensive diagnosis, providing strong support for production process optimization and quality control. As a result, the embodiments of the present invention can promptly detect defects during the TP lamination process, reduce the defective rate, and improve production efficiency and product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 A schematic flow chart of the steps of a method for diagnosing defects in a TP lamination process production process based on image analysis provided by an embodiment of the present invention.

[0014] Figure 2 This is a structural block diagram of a TP lamination process production process defect diagnosis system based on image analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The technical solutions of the present invention will be described below in conjunction with the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention. It should be noted that the terms "first", "second", etc. in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0016] See also Figure 1 , Figure 1 This is a flow chart of a method for diagnosing defects in a TP laminating process production process based on image analysis provided by an embodiment of the present invention. The method is applied to a system for diagnosing defects in a TP laminating process production process based on image analysis, and may further include steps 110 to 140.

[0017] Step 110: Collecting a real-time image sequence corresponding to the bonded components to be inspected that are continuously transmitted on the TP bonding process production line. The real-time image sequence includes component surface images and edge area images at different bonding stages.

[0018] In the TP lamination process production environment, high-definition industrial cameras are installed at key locations on the production line to comprehensively and accurately capture the status of the components being inspected. These cameras are capable of high-speed imaging, capturing the continuously transmitted components at a suitable frame rate. The components undergo different lamination stages on the production line, such as initial lamination, intermediate lamination, and final lamination. At each lamination stage, the surface and edges of the components exhibit different characteristics and conditions. As the components pass through, the high-definition industrial cameras capture their surfaces and edge areas, producing a real-time image sequence containing images of the component surface and edge areas at different lamination stages.

[0019] For example, during the initial bonding phase, the component surface may exhibit slight bubbles or uneven bonding, details that can be clearly captured in the surface image. The edge area image, on the other hand, reveals the alignment of the component edges and any warping. As the bonding process progresses, towards the intermediate bonding phase, the surface texture and edge contours further change, and the camera continuously records these changes. By the final bonding phase, the image sequence demonstrates the final bonded state of the component.

[0020] Step 120: performing defect-sensitive feature enhancement processing on the real-time image sequence to obtain a defect-sensitive feature set of the bonded component to be inspected, wherein the defect-sensitive feature set includes surface texture features, edge contour features, and regional grayscale features.

[0021] Because the raw real-time image sequences captured may be affected by factors such as lighting and noise, defect-related features may not be obvious. To more accurately identify defects in the assembled components being inspected, defect-sensitive feature enhancement is performed on the real-time image sequences. This process aims to highlight features closely related to the defect, making them easier to detect and identify in subsequent analysis.

[0022] Specifically, surface texture features can reflect the microstructure and quality of a component's surface, such as the presence of scratches or irregular lines. Edge profile features can reveal the shape and fit of component edges, such as whether they are neat and have gaps. Regional grayscale features can reveal variations in brightness and contrast across different regions of a component, aiding in the identification of surface grayscale anomalies that may be related to internal fitting defects. By performing a series of processing on real-time image sequences, a defect-sensitive feature set encompassing these three features is ultimately generated.

[0023] In an optional embodiment, performing defect-sensitive feature enhancement processing on the real-time image sequence to obtain a defect-sensitive feature set of the assembly to be inspected includes:

[0024] Step 121: performing adaptive illumination compensation processing on the component surface image in the real-time image sequence by multi-scale Gaussian filtering to eliminate non-uniform illumination interference and generate an illumination normalized surface image.

[0025] In actual production environments, lighting conditions are often uneven, which can cause component surface images to appear locally too bright or too dark, affecting the accurate extraction of surface texture features. To address this issue, a multi-scale Gaussian filter can be used to perform adaptive illumination compensation on component surface images.

[0026] In detail, multi-scale Gaussian filtering is a filtering method based on Gaussian functions that can smooth images at different scales. When processing component surface images, the method first adaptively selects an appropriate scale for filtering based on local image characteristics, such as brightness variations and texture complexity. For areas with strong illumination, the filtering scale may be relatively large to reduce the effects of illumination; for areas with weak illumination, the filtering scale may be smaller to preserve more detailed information.

[0027] This adaptive filtering method effectively eliminates the effects of non-uniform illumination, making the image's illumination distribution more uniform. The resulting illumination-normalized surface image provides high-quality material for extracting accurate surface texture features, allowing surface texture details to be more clearly displayed.

[0028] Step 122: performing texture enhancement processing on the illumination normalized surface image, extracting texture detail features at different scales using Laplacian pyramid decomposition, and enhancing texture edge information through contrast-limited adaptive histogram equalization to obtain enhanced surface texture features.

[0029] After obtaining the illumination-normalized surface image, in order to capture the surface texture features more clearly, it needs to be texture enhanced. This process mainly includes two key steps: Laplacian pyramid decomposition and contrast-limited adaptive histogram equalization.

[0030] Laplacian pyramid decomposition is a method for multi-scale image analysis that decomposes an image into sub-images of varying scales. When processing illumination-normalized surface images, the process begins with the original image and gradually downsamples and filters it, producing a series of sub-images of varying resolutions. Each sub-image represents texture details at a different scale. For example, larger sub-images can reflect the overall texture structure of the image, while smaller sub-images can capture more subtle texture variations.

[0031] Contrast-limited adaptive histogram equalization is a method used to enhance local contrast in images. When processing the sub-images obtained from Laplacian pyramid decomposition, each sub-image is divided into multiple small regions, and the histogram of each region is then equalized. Furthermore, to avoid noise amplification caused by over-enhancement, the histogram contrast is limited. This method can enhance texture edge information and make surface texture features more prominent.

[0032] Combining these two steps, we can finally obtain enhanced surface texture features, which can more accurately reflect the microstructure and quality status of the component surface.

[0033] Step 123: performing edge detection preprocessing on the edge region image in the real-time image sequence, extracting edge contour point sets using the Canny operator, and connecting broken contour lines using a morphological closing operation to generate continuous edge contour features.

[0034] For edge area images in real-time image sequences, a series of edge detection preprocessing is required in order to accurately extract edge contour features.

[0035] First, the Canny operator is used to process the edge region image. As an edge detection algorithm, the Canny operator has high edge detection accuracy and noise immunity. When processing edge region images, the Canny operator first performs Gaussian smoothing on the image to reduce the impact of noise. It then calculates the image's gradient magnitude and direction, using non-maximum suppression to refine the edges. Finally, a double thresholding method is used to determine the true edge points, thereby extracting the edge contour point set.

[0036] However, due to factors such as image noise and lighting variations, the extracted edge contours may be broken. To address this issue, a morphological closing operation is used to process the edge contour point set. Morphological closing is a method based on morphological operations that first performs a dilation operation and then an erosion operation to fill in the broken parts of the edge contour and connect the broken contour lines.

[0037] After Canny operator extraction and morphological closing operation processing, the generated continuous edge contour features can more accurately reflect the shape and fit of the component edge. For example, it can determine whether the edge is neat and whether there are defects such as gaps.

[0038] Step 124: performing region division processing on the illumination normalized surface image, uniformly dividing the image into a plurality of non-overlapping rectangular area units, calculating the grayscale mean and grayscale variance within each area unit, and constructing a regional grayscale feature matrix as the regional grayscale feature.

[0039] In order to analyze the grayscale features of the illumination normalized surface image in more detail, it is necessary to perform region division processing on it. This process mainly includes evenly dividing the image into multiple non-overlapping rectangular area units and calculating the grayscale mean and grayscale variance of each area unit.

[0040] First, the illumination-normalized surface image is evenly divided into multiple rectangular area units of equal size based on the image size and analysis requirements. Each area unit contains a certain number of pixels, and the grayscale values ​​of these pixels reflect the brightness information of the area.

[0041] Then, for each rectangular area unit, the grayscale mean and grayscale variance of all pixels within it are calculated. The grayscale mean represents the average brightness of the area, which can reflect the overall lighting conditions of the area; while the grayscale variance reflects the distribution of grayscale values ​​within the area, which can reflect the degree of brightness variation in the area.

[0042] The grayscale mean and grayscale variance of each regional unit are combined to construct a regional grayscale feature matrix. This matrix quantifies and integrates the grayscale characteristics of each regional unit to form a regional grayscale signature. By analyzing the regional grayscale feature matrix, we can discover grayscale variations in different areas of the component surface and identify possible grayscale anomalies, such as localized brightness changes, which may be related to internal bonding defects.

[0043] Step 125: Perform feature dimension alignment processing on the surface texture features, edge contour features, and regional grayscale features, unify the spatial resolution and number of channels of each feature, and generate a defect sensitive feature set containing multi-dimensional defect sensitive information.

[0044] After obtaining surface texture features, edge contour features, and regional grayscale features, since these features are obtained from different processing steps, their spatial resolution and number of channels may be inconsistent, which will affect subsequent feature analysis and processing. Therefore, it is necessary to align the feature dimensions of these features.

[0045] First, to unify the spatial resolution, an appropriate target resolution is selected based on the importance of the feature and the needs of subsequent analysis. For features with higher spatial resolution, downsampling may be used to reduce the feature size; for features with lower spatial resolution, interpolation is used to increase the feature size to match the target resolution.

[0046] Secondly, to unify the number of channels, we will merge or split the channels of features based on their nature and purpose. For example, if some features have redundant channel information, these channels can be merged; if some features require more channels to represent different information, the channels can be split.

[0047] Through feature dimension alignment processing, the surface texture features, edge contour features and regional grayscale features are effectively fused to generate a defect-sensitive feature set containing multi-dimensional defect-sensitive information. This set integrates information from multiple aspects such as surface texture, edge contour and regional grayscale, and can more comprehensively reflect the status of the bonded components to be inspected.

[0048] Step 130: Perform feature association analysis on the defect-sensitive feature set using the trained TP bonding defect diagnosis model to generate a preliminary defect diagnosis result for the bonding component to be inspected.

[0049] After obtaining the defect-sensitive feature set, the trained TP bonding defect diagnosis model is used to perform feature correlation analysis to generate a preliminary defect diagnosis result for the bonding component to be inspected.

[0050] The TP bonding defect diagnosis model is a deep learning model trained on a large amount of sample data. It learns the relationships between different features and the mapping between these features and defect types. When processing a set of defect-sensitive features, the model first comprehensively analyzes surface texture features, edge contour features, and regional grayscale features to identify their interrelationships and dependencies.

[0051] For example, certain texture anomalies in surface texture features may be associated with specific deformations in edge contour features, or with grayscale changes in regional grayscale features. The model learns these associations and combines different features to form a more representative feature vector.

[0052] The model then classifies and predicts different defect types based on these feature vectors. By learning from a large amount of sample data, the model is able to establish a mapping relationship between feature vectors and defect types, thereby determining the possible defect type of the assembly being inspected based on the input feature vectors.

[0053] Ultimately, the model outputs a probability distribution vector containing multiple defect types, where each element represents the probability of a corresponding defect type. Based on these probabilities, the most likely defect types can be screened and a preliminary defect diagnosis result for the bonded component under inspection can be generated.

[0054] As an implementation method, the defect-sensitive feature set is subjected to feature correlation analysis by the trained TP bonding defect diagnosis model to generate a preliminary defect diagnosis result of the bonding component to be inspected, including:

[0055] Step 131: Input the surface texture features, edge contour features and regional grayscale features in the defect sensitive feature set into the feature fusion layer of the TP fitting defect diagnosis model, and calculate the correlation weight coefficients between different features through the cross-feature attention mechanism.

[0056] In the TP bonding defect diagnosis model, the feature fusion layer is a key module. Its main function is to effectively fuse surface texture features, edge contour features, and regional grayscale features. To achieve this goal, a cross-feature attention mechanism is used to calculate the correlation weight coefficients between different features.

[0057] The cross-feature attention mechanism is a method that dynamically assigns weights to different features. It can adaptively adjust the importance of each feature in the fusion process based on the correlation between the features. In the feature fusion layer, surface texture features, edge contour features, and regional grayscale features are first input into an attention weight calculation unit.

[0058] In this unit, a series of calculations are performed to determine the degree of correlation between different features. Specifically, the relationships between features are considered from multiple perspectives, such as spatial position correlation and grayscale correlation. By calculating these correlations, we can determine the correlation weight coefficients between different features. These correlation weight coefficients represent the importance of each feature in the fusion process. The larger the weight coefficient, the closer the correlation with other features, and the higher the weight should be given in the fusion process. In this way, the correlation between different features can be more effectively explored, improving the model's ability to identify defects.

[0059] Preferably, step 131 further includes:

[0060] Step 1311: Determine the spatial position correlation by calculating the cosine similarity of the pixel values ​​at corresponding positions in the feature map of the surface texture feature and the feature map of the edge contour feature, and construct a first correlation matrix of the surface texture feature and the edge contour feature in the feature fusion layer.

[0061] In order to determine the spatial position correlation between surface texture features and edge contour features, the cosine similarity method of the pixel values ​​at corresponding positions in the feature map is used.

[0062] In the feature fusion layer, surface texture features and edge contour features are stored as feature maps. For each pixel value at a corresponding position in these feature maps, the cosine similarity is calculated. Cosine similarity measures the cosine of the angle between two vectors. In image feature analysis, cosine similarity can be used to measure the similarity between pixel values ​​at corresponding positions.

[0063] Specifically, the pixel values ​​at corresponding locations in the surface texture feature map and the edge contour feature map are treated as two vectors, and their cosine similarity is calculated. The closer the directions of the two vectors, the closer the cosine similarity value is to 1, indicating that the pixel values ​​at the two locations are more similar. Conversely, the greater the difference in the directions of the two vectors, the closer the cosine similarity value is to -1, indicating that the pixel values ​​at the two locations are more different.

[0064] The cosine similarity values ​​of all corresponding positions calculated are combined together to construct the first correlation matrix of surface texture features and edge contour features. Each element in the matrix represents the spatial position correlation between the surface texture features and edge contour features at the corresponding position, which reflects the spatial correlation between the two features.

[0065] Step 1312: perform Euclidean distance calculation based on the row vectors of the regional grayscale feature matrix and the coordinates of the contour point set of the edge contour feature to obtain the region-contour space distance correlation, and construct a second correlation matrix of the edge contour feature and the regional grayscale feature.

[0066] In order to determine the spatial distance correlation between regional grayscale features and edge contour features, the Euclidean distance between the row vector of the regional grayscale feature matrix and the coordinates of the contour point set of the edge contour feature is calculated.

[0067] In this embodiment of the present invention, each row vector in the regional grayscale feature matrix represents the grayscale characteristics of a rectangular area unit, while the contour point set coordinates of the edge contour feature represent the location information of the component edge. When calculating the spatial distance correlation, each row vector of the regional grayscale feature matrix is ​​treated as a multidimensional vector and the Euclidean distance is calculated with the contour point set coordinates of the edge contour feature.

[0068] Euclidean distance is a distance metric that measures the distance between two points in space. The Euclidean distance is calculated between the grayscale feature vector of each area unit and the coordinates of each contour point. The closer the distance, the closer the area unit is to the edge contour, and the higher the degree of association between them.

[0069] The calculated Euclidean distance values ​​between all regional units and contour points are combined together to construct a second correlation matrix of edge contour features and regional grayscale features. Each element in this matrix represents the correlation between regional grayscale features and edge contour features in spatial distance, which reflects the relative position relationship between the two features in space and helps to further analyze the relationship between them.

[0070] Step 1313: Determine the texture-region grayscale correlation by calculating the Pearson correlation coefficient of the grayscale co-occurrence matrix of the texture detail feature and the grayscale mean of the regional grayscale feature, and construct a third correlation matrix of the surface texture feature and the regional grayscale feature.

[0071] In order to determine the correlation between surface texture features and regional grayscale features, the Pearson correlation coefficient method of calculating the grayscale co-occurrence matrix of texture detail features and the grayscale mean of regional grayscale features is adopted.

[0072] The gray level co-occurrence matrix of texture detail features is a matrix used to describe image texture features, which can reflect the spatial distribution of gray values ​​in the image. The gray mean of regional gray features represents the average brightness of each rectangular area unit.

[0073] When calculating the Pearson correlation coefficient, each element in the grayscale co-occurrence matrix of the texture detail feature is compared with the grayscale mean of the regional grayscale feature. The Pearson correlation coefficient is a statistic that measures the linear correlation between two variables, and its value ranges from -1 to 1. If the correlation coefficient value is close to 1, it indicates a positive linear correlation between the two variables; if the correlation coefficient value is close to -1, it indicates a negative linear correlation between the two variables; if the correlation coefficient value is close to 0, it indicates no linear correlation between the two variables.

[0074] All the calculated correlation coefficient values ​​are combined together to construct the third correlation matrix of surface texture features and regional grayscale features. Each element in this matrix represents the linear correlation between surface texture features and regional grayscale features, which reflects the correlation between these two features in grayscale changes and provides more information for feature fusion.

[0075] Step 1314: Input the first correlation matrix, the second correlation matrix and the third correlation matrix into the attention weight calculation unit, normalize the matrix elements through the softmax function, and generate cross-feature correlation weight coefficients between each feature.

[0076] After obtaining the first correlation matrix of surface texture features and edge contour features, the second correlation matrix of edge contour features and regional grayscale features, and the third correlation matrix of surface texture features and regional grayscale features, these matrices are input into the attention weight calculation unit for processing.

[0077] The main function of the attention weight calculation unit is to generate cross-feature correlation weight coefficients between features based on these correlation matrices. To make these weight coefficients comparable and interpretable, the softmax function is used to normalize the matrix elements.

[0078] The softmax function can convert matrix elements into probability distributions so that the sum of all elements is 1. When processing these three association matrices, each element in the matrix is ​​used as input and calculated through the softmax function to obtain the normalized weight coefficient.

[0079] These cross-feature correlation weights represent the strength of the association between different features. A larger weight indicates a closer association between the two features. This normalization ensures that the correlation weights between features are within a reasonable range.

[0080] Step 132: performing weighted fusion processing on the surface texture features, edge contour features, and regional grayscale features based on the associated weight coefficients to generate a joint feature vector that fuses multi-dimensional defect information.

[0081] After obtaining the cross-feature correlation weight coefficients between each feature, the surface texture features, edge contour features and regional grayscale features will be weighted fused based on these weight coefficients.

[0082] The weighted fusion process multiplies each feature by its associated weight coefficient and then adds or concatenates them to form a joint feature vector that integrates multi-dimensional defect information. Specifically, surface texture features are weighted based on their associated weight coefficients with other features; the same process is performed for edge contour features and regional grayscale features.

[0083] For example, if the surface texture feature has a large correlation weight with other features, the surface texture feature will be given a higher weight during the fusion process, and its contribution to the joint feature vector will also be greater. This weighted fusion method can effectively integrate the information of different features, so that the joint feature vector can more comprehensively reflect the status of the component to be inspected.

[0084] The generated joint feature vector contains defect information in multiple aspects such as surface texture, edge contour and regional grayscale, which provides a richer and more representative data basis for subsequent defect diagnosis, thereby improving the model's accuracy in defect recognition.

[0085] Step 133: performing principal component analysis on the joint feature vector through the feature dimension reduction layer of the TP bonding defect diagnosis model, retaining the principal component components whose cumulative contribution rate exceeds a preset ratio, and obtaining a dimension-reduced core defect feature vector.

[0086] In order to reduce the data dimension of the joint feature vector and improve the computational efficiency and generalization ability of the model, the joint feature vector will be subjected to principal component analysis through the feature dimensionality reduction layer of the TP fitting defect diagnosis model.

[0087] Principal component analysis (PCA) is a data dimensionality reduction method that converts high-dimensional data into low-dimensional data while preserving the data's key information. When processing a joint eigenvector, the covariance matrix of the joint eigenvector is first calculated. The covariance matrix reflects the correlations between the dimensions in the joint eigenvector.

[0088] Then, by performing eigenvalue decomposition on the covariance matrix, we obtain a series of eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​represent the importance of each principal component, while the eigenvectors represent the direction of the principal component. The principal components are sorted according to the size of the eigenvalues, with larger eigenvalues ​​indicating higher importance.

[0089] Next, the contribution rate of each principal component is calculated. This rate represents the explanatory power of that principal component for the original data. The cumulative contribution rate is calculated by adding the contribution rates of the previous principal components. Based on a preset ratio, such as a ratio selected appropriately for practical purposes, principal components with cumulative contribution rates exceeding that ratio are retained.

[0090] For example, if the preset ratio is 80%, the principal components that cumulatively explain 80% of the original data will be selected. After principal component analysis, the retained principal components are combined to form a dimensionality-reduced core defect feature vector. This vector retains the main defect information while reducing the data dimension, lowering the model's computational complexity and improving the model's training and prediction efficiency.

[0091] Step 134: Input the core defect feature vector into the defect classification subnetwork of the TP fitting defect diagnosis model, and output a probability distribution vector containing multiple defect types through a combination of a fully connected layer and an activation function.

[0092] The reduced-dimensional core defect feature vector is input into the defect classification subnetwork of the TP fitting defect diagnosis model. The main function of this subnetwork is to classify different defect types according to the core defect feature vector and output the probability of each defect type.

[0093] The defect classification subnetwork primarily consists of fully connected layers and activation functions, which process the core defect feature vector through their combined operations. A fully connected layer is a neural network layer that connects every neuron in the input layer to every neuron in the output layer. It linearly transforms the input core defect feature vector, mapping it to a different feature space.

[0094] When processing the core defect feature vector, it is first input into the first fully connected layer. In this fully connected layer, the core defect feature vector is linearly transformed using the weight matrix and bias vector, converting it into a new feature vector. This new feature vector has a different representation in the new feature space, which helps extract more abstract features.

[0095] The feature vector processed by the first fully connected layer is then input into the activation function. The activation function introduces nonlinearity to enhance the model's expressiveness. Common activation functions, such as the ReLU function, set values ​​less than 0 to 0 and values ​​greater than 0 unchanged. This increases the model's nonlinearity, enabling it to learn more complex patterns.

[0096] After the activation function, the feature vector is fed into the next fully connected layer, repeating the linear transformation and nonlinear activation process. Typically, multiple fully connected layers and activation functions are applied alternately, gradually mapping the core defect feature vector to a higher-level feature space.

[0097] Finally, after a series of fully connected layers and activation functions, the output feature vector is fed into the output layer. In the output layer, a softmax activation function is used to convert the output feature vector into a probability distribution vector containing various defect types. The softmax activation function converts the input vector into a probability distribution where the sum of all elements is 1, and each element represents the probability of the corresponding defect type. This probability distribution vector provides an understanding of the likelihood of various defect types in the tested assembly.

[0098] Preferably, step 134 further includes:

[0099] Step 1341: Input the core defect feature vector into the first fully connected layer of the defect classification subnetwork, map the core defect feature vector to the first hidden feature space through linear transformation of the weight matrix and the bias vector, and generate a first hidden layer feature vector.

[0100] In the defect classification subnetwork, when the core defect feature vector is input into the first fully connected layer, the first fully connected layer will perform a linear transformation on it. The linear transformation process is achieved through the weight matrix and bias vector.

[0101] The weight matrix is ​​a two-dimensional matrix whose number of rows and columns are related to the dimension of the input core defect feature vector and the number of neurons in the first hidden layer, respectively. The bias vector is a one-dimensional vector whose length is the same as the number of neurons in the first hidden layer.

[0102] During the linear transformation, the core defect feature vector is multiplied by the weight matrix and then added with the bias vector. This method maps the core defect feature vector from the original feature space to the first hidden feature space, generating a first hidden layer feature vector. This first hidden layer feature vector has a different representation in the first hidden feature space, which can extract some more abstract features from the core defect feature vector.

[0103] Step 1342: Perform nonlinear activation processing on the first hidden layer feature vector, use the ReLU activation function to enhance feature expression capability, and generate an activated hidden feature vector.

[0104] After obtaining the first hidden layer feature vector, it will be subjected to nonlinear activation processing, using the ReLU activation function. The ReLU activation function can set values ​​less than 0 to 0 and values ​​greater than 0 to remain unchanged.

[0105] When processing the first hidden layer feature vector, for each element in the vector, if the value of the element is less than 0, then after processing by the ReLU activation function, the value of the element will become 0; if the value of the element is greater than 0, then the value of the element remains unchanged.

[0106] This nonlinear activation process introduces nonlinear factors and enhances the model's feature representation capabilities. Because the relationship between defect type and features in actual defect classification problems is often nonlinear, the ReLU activation function enables the model to learn more complex patterns. The resulting activated hidden feature vector better represents defect characteristics, improving the model's classification performance and enabling it to more accurately identify different defect types.

[0107] Step 1343: Input the activated hidden feature vector into the second fully connected layer of the defect classification subnetwork, map the activated hidden feature vector to the second hidden feature space through a linear transformation of another set of weight matrices and bias vectors, and generate a second hidden layer feature vector.

[0108] After the activated hidden feature vector is input into the second fully connected layer of the defect classification subnetwork, it undergoes a linear transformation. Similar to the first fully connected layer, this linear transformation is also implemented using another set of weight matrices and bias vectors.

[0109] The number of rows and columns of this weight matrix corresponds to the dimension of the activated hidden feature vector and the number of neurons in the second hidden layer, respectively. The length of the bias vector is the same as the number of neurons in the second hidden layer. During the linear transformation, the activated hidden feature vector is multiplied by this weight matrix and then the bias vector is added.

[0110] Through this linear transformation, the activated hidden feature vector is mapped from the first hidden feature space to the second hidden feature space to generate a second hidden layer feature vector. The second hidden layer feature vector has a higher-level feature representation in the second hidden feature space, which can further extract more abstract and complex features from the activated hidden feature vector.

[0111] Step 1344: performing batch normalization processing on the second hidden layer feature vector to eliminate distribution differences between features of different batches and generate a normalized hidden feature vector.

[0112] In order to eliminate the distribution differences between features in different batches and improve the training efficiency and generalization ability of the model, the feature vectors of the second hidden layer will be batch normalized.

[0113] The batch normalization process primarily involves calculating the mean and variance of the feature vectors, and then normalizing them. When processing the second hidden layer feature vectors, the mean and variance of the second hidden layer feature vectors are first calculated for a batch of data. The mean represents the average level of the feature vectors in the batch, while the variance indicates the degree of dispersion of the feature vectors.

[0114] Then, the second hidden layer feature vectors are normalized by subtracting the mean from each feature vector and dividing it by the square root of the variance, so that the mean of the feature vector is 0 and the variance is 1. This normalization operation can make the feature vectors of different batches have similar distributions, reducing the impact of batch differences.

[0115] Finally, to preserve the expressive power of the feature vector, the normalized feature vector is scaled and translated, and adjusted using learnable parameters. After batch normalization, a normalized hidden feature vector is generated, which has a more stable distribution. This helps improve the training efficiency and generalization ability of the model, allowing the model to better adapt to different batches of data.

[0116] Step 1345: Input the normalized hidden feature vector into the output layer of the defect classification subnetwork, calculate the probability value corresponding to each defect type through the softmax activation function, and generate a probability distribution vector containing multiple defect types.

[0117] After the normalized hidden feature vector is input into the output layer of the defect classification subnetwork, the output layer calculates the probability value corresponding to each defect type through the softmax activation function.

[0118] When processing the normalized hidden feature vector, the softmax activation function performs an exponential operation on each element in the vector and then divides the exponential value of each element by the sum of the exponential values ​​of all elements.

[0119] After processing each element through the softmax activation function, it generates a probability value between 0 and 1, representing the likelihood of the corresponding defect type. Combining the probability values ​​corresponding to all defect types generates a probability distribution vector encompassing multiple defect types. This probability distribution vector provides a basis for subsequent defect type screening. By analyzing the probability values ​​in the probability distribution vector, we can determine the possible defect types of the assembly being inspected and the likelihood of each defect type.

[0120] Step 135: Based on the probability values ​​corresponding to the defect types in the probability distribution vector, select defect types whose probability values ​​exceed a set threshold as candidate defect types, and generate a preliminary defect diagnosis result including the candidate defect types and the corresponding probability values.

[0121] After obtaining a probability distribution vector containing multiple defect types, the probability values ​​corresponding to each defect type are used for screening. First, an appropriate threshold is set, which is determined based on actual defect diagnosis needs and experience.

[0122] Each element in the probability distribution vector, i.e., the probability value corresponding to each defect type, is compared with a set threshold. If the probability value of a defect type exceeds the set threshold, the defect type is selected as a candidate defect type.

[0123] For example, if the threshold is set to an appropriate value, when the probability value of a certain defect type is greater than the threshold, it means that the possibility of the existence of the defect type is high, and it is regarded as a candidate defect type.

[0124] The screened candidate defect types and their corresponding probability values ​​are combined together to generate a preliminary defect diagnosis result containing the candidate defect types and corresponding probability values. This result provides a preliminary screening range for the subsequent determination of the final defect type, reduces the workload of subsequent verification, and improves the efficiency of defect diagnosis.

[0125] Step 140: Determine the defect type existing in the assembly to be inspected and the position distribution information of the defect corresponding to the defect type in the real-time image sequence based on the preliminary defect diagnosis result, and generate a production process defect diagnosis report containing defect diagnosis content based on the defect type and the position distribution information.

[0126] Based on the generated preliminary defect diagnosis results, it is necessary to further determine the actual defect types present in the assembly being inspected and the location and distribution of these defects in the real-time image sequence. Since the preliminary defect diagnosis results are only a preliminary screening, there may be misjudgments, so further verification and confirmation is required.

[0127] To determine the defect type, the candidate defect type will be secondary verified to ensure its accuracy. Determining the location and distribution of defects in the real-time image sequence is very important for subsequent repairs and improvements, because only by knowing the specific location of the defect can targeted measures be taken.

[0128] Finally, based on the determined defect type and location distribution information, a production process defect diagnosis report containing defect diagnosis content is generated. This report can provide an important reference basis for the optimization of the production process, helping production personnel to promptly identify problems and take corresponding measures to improve production quality and efficiency.

[0129] In an optional embodiment, the determining, based on the preliminary defect diagnosis result, the defect type of the to-be-inspected assembly and the position distribution information of the defect corresponding to the defect type in the real-time image sequence includes:

[0130] Step 141: parse the candidate defect types and corresponding probability values ​​in the preliminary defect diagnosis result, sort the candidate defect types in descending order of probability value, and select a preset number of candidate defect types with the highest order as the defect types to be verified.

[0131] First, the preliminary defect diagnosis results are parsed to extract candidate defect types and their corresponding probability values. These candidate defect types are then sorted from highest to lowest probability. The purpose of this sorting is to prioritize defect types with higher probability, thereby improving verification efficiency.

[0132] Next, the top candidate defect types are selected as the defect types to be verified, based on a preset number. The preset number is determined based on actual needs and resource availability. For example, if the preset number is an appropriate value, the top candidate defect types with the highest probability values ​​are selected as the defect types to be verified.

[0133] In this way, the workload of subsequent verification can be reduced, and efforts can be focused on verifying the more likely defect types, thereby improving the accuracy and efficiency of defect diagnosis.

[0134] Step 142: For each defect type to be verified, extract feature components related to the defect type to be verified from the defect-sensitive feature set to construct a type-specific feature subset.

[0135] For each defect type to be verified, the relevant feature components are extracted from the defect-sensitive feature set to construct a type-specific feature subset. Because the defect-sensitive feature set includes multiple features such as surface texture, edge contour, and regional grayscale, different defect types may be associated with different feature components.

[0136] For example, a specific defect type to be verified may be associated with certain texture patterns in surface texture features, specific contour shapes in edge contour features, and certain grayscale variations in regional grayscale features. Based on these correlations, feature components relevant to the defect type to be verified are extracted from the defect-sensitive feature set.

[0137] These relevant feature components are combined to construct a type-specific feature subset. This process is repeated for each defect type to be verified, resulting in a specific type-specific feature subset. These subsets provide targeted data for subsequent secondary verification, making the verification process more accurate and efficient.

[0138] Step 143: Perform secondary verification processing on the type-specific feature subset using the trained defect type verification model, and output the verification confidence of the defect type to be verified.

[0139] The constructed type-specific feature subset is input into the trained defect type verification model for secondary verification. The defect type verification model is a model specifically used to verify the accuracy of defect types, which is trained with a large amount of sample data.

[0140] When processing a subset of type-specific features, the model determines the type of defect to be verified based on the feature components in the subset. The model internally includes a series of processing layers and neurons that perform complex calculations and analysis on the input feature components.

[0141] First, a subset of type-specific features is fed into the model's input layer. This subset is then processed by multiple hidden layers, each of which abstracts and transforms the input features to varying degrees, extracting higher-level features. Finally, after processing at the output layer, the model outputs a verification confidence score, which indicates the likelihood of the defect type being verified being accurate. A higher verification confidence score indicates a greater likelihood that the defect type being verified is a true defect. This secondary verification process improves the accuracy of defect type determination and reduces misjudgments.

[0142] Step 144: Compare the verification confidence with a preset verification threshold. When the verification confidence exceeds the verification threshold, determine that the defect type to be verified is the final defect type.

[0143] After obtaining the verification confidence of the defect type to be verified, it will be compared with the preset verification threshold. The preset verification threshold is determined based on actual needs and experience, and is the standard for judging whether a defect type to be verified is the final defect type.

[0144] If the verification confidence exceeds the verification threshold, it indicates that the defect type to be verified is likely to be a true defect type and will be determined as the final defect type. For example, if the preset verification threshold is an appropriate value, when the verification confidence of a defect type to be verified exceeds the threshold, it will be determined as the final defect type.

[0145] Through this comparison and judgment method, the true defect type can be further screened out and the accuracy of defect diagnosis can be improved.

[0146] Step 145: Based on the spatial position coordinates of the feature components corresponding to the final defect type in the defect sensitive feature set, determine the minimum circumscribed rectangular area of ​​the defect corresponding to the final defect type in the real-time image sequence, and use the vertex coordinate set of the minimum circumscribed rectangular area as the position distribution information of the defect corresponding to the defect type in the real-time image sequence.

[0147] After determining the final defect type, it is necessary to determine the position distribution information of these defects in the real-time image sequence. This is achieved by the spatial position coordinates of the feature components corresponding to the final defect type in the defect sensitive feature set.

[0148] First, the feature components corresponding to the final defect type are found from the defect-sensitive feature set, and their spatial position coordinates are recorded. These spatial position coordinates represent the approximate position of the defect in the image.

[0149] Then, based on these spatial position coordinates, the minimum bounding rectangle of the defect corresponding to the final defect type in the real-time image sequence is determined. The minimum bounding rectangle is the smallest rectangular area that can completely contain all the defect position coordinates, which can accurately define the scope of the defect.

[0150] Finally, the vertex coordinate set of the minimum circumscribed rectangular area is used as the position distribution information of the defect corresponding to the final defect type in the real-time image sequence. These vertex coordinate sets provide a specific position reference for subsequent defect repair and production process optimization, allowing staff to accurately locate the location of the defect.

[0151] In a preferred embodiment, step 145 includes:

[0152] Step 1451: extracting a texture abnormality region coordinate point set of a feature component corresponding to the final defect type in the surface texture feature, each coordinate point including a horizontal and vertical coordinate value of a two-dimensional image plane.

[0153] In order to more accurately determine the position of the defect corresponding to the final defect type in the real-time image sequence, the coordinate point set of the texture abnormality area of ​​the feature component corresponding to the final defect type in the surface texture feature is first extracted.

[0154] In the surface texture features, the characteristic components related to the final defect type may show texture anomalies, such as texture discontinuity, line deformation, etc. These texture anomaly areas can be found by analyzing the surface texture features.

[0155] For each texture anomaly region, its coordinate point set is recorded. Each coordinate point contains the horizontal and vertical coordinate values ​​of the two-dimensional image plane. These coordinate points accurately represent the position of the texture anomaly region in the image. By extracting these texture anomaly region coordinate point sets, the location of the defect can be more accurately located.

[0156] Step 1452: performing coordinate normalization processing on the coordinate point set of the texture abnormality region, converting the coordinate values ​​into proportional coordinates relative to the image resolution, and eliminating coordinate differences caused by different image sizes.

[0157] After obtaining the coordinate point set of the texture abnormality area, in order to eliminate the coordinate differences caused by different image sizes, the coordinate point set will be normalized.

[0158] Different images may have different resolutions, which may cause the coordinate values ​​of the same location to be represented differently in different images. To solve this problem, the coordinate values ​​are converted to scaled coordinates relative to the image resolution.

[0159] Specifically, for each coordinate point, the horizontal and vertical coordinate values ​​will be divided by the width and height of the image respectively to obtain the corresponding proportional coordinates. Therefore, regardless of the size of the image, the coordinates at the same position will have the same proportional representation after normalization.

[0160] Through coordinate normalization processing, the coordinate point set has better comparability and universality, providing a more stable data basis for the subsequent minimum enclosing rectangle calculation and avoiding calculation errors caused by image size differences.

[0161] Step 1453: Use the rotating caliper algorithm to calculate the minimum bounding rectangle of the normalized coordinate point set to determine the rectangular area with the smallest area that completely contains all the coordinate points; extract the coordinates of the four vertices of the minimum bounding rectangle area, and sort the vertex coordinates in the clockwise direction of the image coordinate system to generate an ordered vertex coordinate sequence.

[0162] After obtaining the normalized coordinate point set, the minimum bounding rectangle (MBR) of the point set is calculated using the rotating calculus algorithm. The rotating calculus algorithm is an efficient algorithm for calculating the minimum bounding rectangle of a point set.

[0163] When processing the normalized set of coordinate points, the rotating caliper algorithm will find the smallest rectangular area that can completely contain all the coordinate points by continuously rotating. This minimum circumscribed rectangular area can most accurately define the scope of the defect.

[0164] Next, the coordinates of the four vertices of the minimum bounding rectangle are extracted. To facilitate subsequent processing and use, these four vertex coordinates are sorted clockwise in the image coordinate system. This sorted sequence of vertex coordinates creates a regular arrangement.

[0165] Through the rotating caliper algorithm and vertex coordinate sorting, the minimum circumscribed rectangular area of ​​the defect corresponding to the final defect type in the real-time image sequence can be accurately determined, and an ordered vertex coordinate sequence can be obtained, providing accurate data for subsequent position distribution information determination.

[0166] Step 1454: converting the scale coordinates in the ordered vertex coordinate sequence into pixel coordinates at the original image resolution to obtain position distribution information of the defect corresponding to the defect type including four pixel vertex coordinates in the real-time image sequence.

[0167] After obtaining the ordered vertex coordinate sequence, since the coordinates in the sequence are scaled coordinates, in order to obtain the accurate position of the defect at the original image resolution, the scaled coordinates need to be converted into pixel coordinates at the original image resolution.

[0168] This is achieved by multiplying the scaled coordinates by the width and height of the original image. For each coordinate point in the ordered vertex coordinate sequence, its horizontal coordinate scale value is multiplied by the width of the original image, and its vertical coordinate scale value is multiplied by the height of the original image to obtain the corresponding pixel coordinate.

[0169] By combining the converted four pixel vertex coordinates, we can obtain the position distribution information of the defect corresponding to the final defect type containing the four pixel vertex coordinates in the real-time image sequence. These pixel vertex coordinates accurately represent the position of the defect in the original image, providing a specific position reference for subsequent defect repair and production process optimization.

[0170] As another optional embodiment, the generating of the production process defect diagnosis report containing defect diagnosis content based on the defect type and the location distribution information includes:

[0171] Step 1461: querying a preset defect severity assessment rule library according to the defect type to obtain a severity assessment index system corresponding to the defect type, wherein the assessment index system includes defect area ratio, defect number density, and defect location importance coefficient.

[0172] After determining the final defect type and its location distribution information in the real-time image sequence, a production process defect diagnosis report containing defect diagnosis content needs to be generated. First, the preset defect severity assessment rule library is queried based on the defect type.

[0173] The defect severity assessment rule base is a pre-established database that contains severity assessment indicators for various defect types. For each defect type, there is a corresponding set of assessment indicators used to assess the severity of that defect type.

[0174] These evaluation metrics include defect area ratio, defect density, and defect location importance coefficient. The defect area ratio represents the proportion of the defect area on the surface of the bonded component, reflecting the severity of the defect; the defect density represents the number of defects within a certain range, reflecting the distribution of defects; and the defect location importance coefficient indicates the importance of the defect location, such as whether the defect is located in a critical bonding area.

[0175] By querying the defect severity assessment rule library, the severity assessment index system corresponding to the final defect type is obtained, which can be used as a specific indicator and standard for evaluating the severity of the defect.

[0176] Step 1462: Calculate the defect area based on the minimum circumscribed rectangular area in the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and use the ratio of the defect area to the total surface area of ​​the bonding component as the defect area ratio.

[0177] After obtaining the defect severity assessment index system, the defect area ratio index is calculated. First, the defect area is calculated based on the minimum enclosing rectangular area of ​​the defect position distribution information corresponding to the final defect type in the real-time image sequence.

[0178] The coordinates of the four vertices of the minimum circumscribed rectangular area accurately define the defect's extent. By calculating the area of ​​this rectangular area, we can determine the approximate area of ​​the defect. The calculated defect area is then compared to the total surface area of ​​the bonded component, and their ratio is calculated. This ratio is the defect area ratio, which reflects the proportion of the surface area occupied by the defect on the bonded component. A larger defect area ratio indicates a wider impact range and potentially greater impact on the performance and quality of the bonded component. Calculating the defect area ratio provides an important quantitative indicator for assessing defect severity.

[0179] Step 1463: Count the number of defects of the same defect type in the real-time image sequence, and use the ratio of the number of defects to the total number of frames in the image sequence as the defect number density.

[0180] In addition to the defect area percentage, the defect number density is also calculated. This is achieved by counting the number of defects of the same defect type in a real-time image sequence.

[0181] In a real-time image sequence, each frame is analyzed, and the number of defects of the same defect type is counted based on defect type and location distribution information. The resulting defect count is then compared with the total number of frames in the image sequence, and their ratio is calculated. This ratio is the defect density, which reflects the distribution of defects within a certain range. A higher defect density indicates a higher frequency of defects in the production process, potentially causing a greater impact on production efficiency and product quality. Calculating defect density provides another important quantitative indicator for assessing defect severity.

[0182] Step 1464: Determine whether the area where the defect is located is a key fitting area based on the position distribution information of the defect corresponding to the defect type in the real-time image sequence. If so, set the defect position importance coefficient to a first preset value, otherwise set it to a second preset value.

[0183] Next, based on the location distribution of the defect corresponding to the final defect type in the real-time image sequence, it is determined whether the defect area is a critical bonding area. Critical bonding areas are areas that have a significant impact on the performance and quality of the bonded component, such as the core bonding area and key connection points.

[0184] If the defect is located in a critical bonding area, it means that the defect may have a significant impact on the performance and quality of the bonded component, and the defect location importance coefficient will be set to the first preset value. The first preset value is usually a relatively large value to highlight the importance of the defect.

[0185] If the defect area is not a critical bonding area, it means that the defect has a relatively small impact on the performance and quality of the bonded component, and the defect location importance coefficient will be set to a second preset value. The second preset value is usually a relatively small value.

[0186] In this way, the defect location importance coefficient is reasonably set according to the importance of the defect area, which provides an indicator that comprehensively considers location factors for evaluating the severity of the defect.

[0187] Step 1465: Input the defect area ratio, defect number density and defect location importance coefficient into the severity calculation function to generate the defect severity level.

[0188] After obtaining the defect area ratio, defect density, and defect location importance coefficient, these indicators are input into the severity calculation function. The severity calculation function is a predefined function that comprehensively calculates the severity level of the defect based on these input indicators.

[0189] During the calculation process, the severity calculation function weights each metric differently based on its importance. For example, the defect area percentage may carry a larger weight in the calculation because it directly reflects the size of the defect. The defect location importance coefficient also affects the final severity level based on its value.

[0190] This comprehensive calculation method generates a defect severity level. The defect severity level can be divided into different levels, such as mild, moderate, and severe, which intuitively reflects the impact of the defect on the assembly.

[0191] Step 1466: Fusion the defect type, the position distribution information of the defect corresponding to the defect type in the real-time image sequence, the defect severity level and the corresponding real-time image sequence timestamp to generate a production process defect diagnosis report containing text and image combined with diagnostic annotations.

[0192] Finally, the defect type, the location distribution information of the defect corresponding to the defect type in the real-time image sequence, the defect severity level and the corresponding real-time image sequence timestamp will be fused to generate a production process defect diagnosis report containing text and image combined with diagnostic annotations.

[0193] During the fusion process, textual information such as defect type, location distribution, and severity level is associated with relevant images in the real-time image sequence. For each image in the real-time image sequence with a defect, the location and type of the defect are annotated on the image, along with the corresponding severity level information.

[0194] At the same time, these graphic and text information will be arranged in the time stamp order of the real-time image sequence to form a complete report. The report not only contains detailed text diagnostic information, but also intuitively displays the location and situation of the defects through images, allowing production personnel to more intuitively understand the defects in the production process.

[0195] The generated production process defect diagnosis report provides comprehensive information for the optimization of the production process. Production personnel can take corresponding measures in a timely manner according to the content of the report, such as adjusting the production process, repairing defects, etc., to improve production quality and efficiency.

[0196] As an alternative technical solution, before step 1466, it also includes: verifying the temporal continuity of the position distribution information of the defect corresponding to the defect type in the real-time image sequence, extracting the position distribution information of the same defect type in adjacent frame images, and calculating the position offset; when the position offset is less than a preset offset threshold, determining that the defect is a real defect; when the position offset is greater than or equal to the preset offset threshold, determining that the defect is a suspected falsely detected defect; performing secondary feature extraction on the defect-sensitive feature set of the suspected falsely detected defect, detecting and analyzing the continuity of the edge contour feature and the mutation of the regional grayscale feature; when the edge contour continuity index in the secondary feature extraction result exceeds the contour threshold and the regional grayscale mutation index exceeds the grayscale threshold, re-determining the suspected falsely detected defect as a real defect; otherwise, eliminating the suspected falsely detected defect from the defect type; retaining the defect types corresponding to all real defects, the position distribution information of the defects corresponding to the defect types in the real-time image sequence, and the defect severity level.

[0197] Before generating a production process defect diagnosis report, in order to ensure the accuracy of defect diagnosis, it is necessary to verify the temporal continuity of the position distribution information of the defect corresponding to the defect type in the real-time image sequence.

[0198] First, the position distribution information of the same defect type in adjacent frames is extracted. Since real-time image sequences are continuous, the position of the same defect in adjacent frames should normally have a certain degree of continuity. By comparing the position distribution information of the same defect type in adjacent frames, the position offset between them is calculated.

[0199] The position offset represents the degree of change in the defect's position between adjacent frames. If the position offset is less than the preset offset threshold, it indicates that the defect's position has changed little and has good temporal continuity, and the defect is considered a real defect.

[0200] If the position offset is greater than or equal to the preset offset threshold, it means that the position of the defect has changed significantly, which may be caused by noise, false detection, etc. In this case, the defect is determined to be a suspected false detection defect.

[0201] For suspected falsely detected defects, secondary feature extraction is performed on their defect-sensitive feature set. This primarily examines and analyzes the continuity of edge contour features and the sudden change in regional grayscale features. The continuity of edge contour features can indicate whether the defect boundary is clear and stable; the sudden change in regional grayscale features can indicate whether the grayscale changes in the defect area are abnormal.

[0202] Furthermore, a contour threshold and a grayscale threshold can be set. When the edge contour continuity index in the secondary feature extraction result exceeds the contour threshold and the regional grayscale mutation index exceeds the grayscale threshold, it indicates that the suspected falsely detected defect has a certain degree of authenticity and is re-determined as a real defect.

[0203] Otherwise, the suspected falsely detected defect is removed from the defect type list. Finally, the defect types corresponding to all true defects, their corresponding location distribution information in the real-time image sequence, and the defect severity level are retained. This temporal continuity verification and secondary feature extraction method can further improve the accuracy of defect diagnosis and provide a guarantee for the generation of more reliable production process defect diagnosis reports.

[0204] As another alternative technical solution, step 1466 includes:

[0205] Step 14661: Create an independent defect diagnosis sub-report for each defect type, wherein the defect diagnosis sub-report includes the defect type name, the defect occurrence timestamp, the vertex coordinate sequence in the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and the defect severity level.

[0206] When generating a production process defect diagnosis report, a separate defect diagnosis sub-report will be created for each defect type. This is to make the report clearer and more organized, so that production personnel can conduct targeted analysis and processing of different types of defects.

[0207] Each defect diagnosis sub-report includes the defect type name, defect occurrence timestamp, vertex coordinate sequence of the defect's corresponding position distribution in the real-time image sequence, and defect severity level. The defect type name clarifies the specific defect type, making it easier for production personnel to identify; the defect occurrence timestamp records the time of defect occurrence, which helps analyze the pattern of defect occurrence; the vertex coordinate sequence accurately indicates the defect's location in the image, providing a specific location reference for defect repair; and the defect severity level intuitively reflects the severity of the defect, providing a basis for decision-making.

[0208] By creating independent defect diagnosis sub-reports, the relevant information of each defect type can be centrally organized and displayed, improving the readability and practicality of the report.

[0209] Step 14662: intercept an image segment containing the area corresponding to the position distribution information of the defect corresponding to the defect type in the real-time image sequence from the real-time image sequence, perform defect labeling on the defect area in the image segment, and generate a labeled defect image.

[0210] Next, image segments corresponding to the area of ​​the position distribution information of the defect corresponding to each defect type in the real-time image sequence are captured from the real-time image sequence. These image segments can intuitively show the actual situation of the defect.

[0211] For captured image segments, defective areas are annotated. This annotation includes information such as the defect type and severity level. This annotation allows production personnel to more intuitively understand the specific details of the defect and quickly locate its location and characteristics.

[0212] The generated annotated defect images provide intuitive image data for subsequent report presentation and analysis, helping production personnel to better understand the actual situation of the defects and take corresponding measures to repair and improve them.

[0213] Step 14663: associate and store the annotated defect image with the text description information in the defect diagnosis sub-report to establish a one-to-one correspondence between the image and the text.

[0214] In order to make the graphic and text information cooperate with each other and better display the defect diagnosis results, the annotated defect image will be associated with the text description information in the defect diagnosis sub-report and stored.

[0215] During the storage process, a one-to-one correspondence between images and text will be established, that is, each annotated defect image corresponds to the text description information in the corresponding defect diagnosis sub-report. Therefore, when production personnel view the report, they can quickly find the corresponding annotated defect image through the text description information and intuitively understand the defect situation; at the same time, they can also understand the content of the text description more deeply through the image.

[0216] By establishing this associative storage and the correspondence between images and text, the efficiency of information transmission in reports is improved, enabling production personnel to grasp the defect diagnosis results more comprehensively and accurately.

[0217] Step 14664: Sort all defect diagnosis sub-reports in the order of the timestamps of the real-time image sequence to generate a defect diagnosis sequence arranged in chronological order.

[0218] In order to make the report more consistent with the chronological order of the production process and facilitate production personnel to analyze the patterns of defect occurrence, all defect diagnosis sub-reports will be sorted according to the timestamp order of the real-time image sequence.

[0219] Timestamps record the time each defect occurred. Sorting the defect diagnosis subreports in chronological order clearly demonstrates the occurrence of defects during the production process. This chronological sequence makes the report more logical and coherent. Production personnel can use this sequence to analyze defect frequency and trends, providing more targeted recommendations for optimizing the production process.

[0220] Step 14665: Integrate all defect diagnosis sub-reports and corresponding annotated defect images in the defect diagnosis sequence into a structured document, add the report generation time, production line number and bonding component batch information, and generate a complete production process defect diagnosis report.

[0221] Finally, all defect diagnosis sub-reports and their corresponding annotated defect images in the chronologically ordered defect diagnosis sequence are integrated into a structured document. During the integration process, the text and image information are properly formatted and laid out to make the report more aesthetically pleasing and easier to read.

[0222] The report generation time, production line number, and assembly batch information are also included. The report generation time records the moment the report was generated, making it easier for production personnel to understand the timeliness of the report. The production line number identifies the specific production line where the defect occurred, helping to locate the source of the problem. The assembly batch information can help production personnel analyze the quality of components from the same batch.

[0223] By integrating and adding this information, a complete production process defect diagnosis report is generated. The report contains detailed defect diagnosis information, intuitive image data and related production information, providing a comprehensive and accurate basis for production process optimization and quality control.

[0224] In a non-limiting embodiment, after generating a production process defect diagnosis report containing defect diagnosis content based on the defect type and the location distribution information, the method further includes: extracting defect sample data containing the defect type and corresponding location distribution information from the production process defect diagnosis report, the defect sample data containing feature components of each defect type in a real-time image sequence and corresponding diagnostic labels; performing feature space alignment processing on the defect sample data with sample data in a historical defect sample library, unifying the spatial resolution and number of channels of a defect-sensitive feature set, and generating a model optimization feedback data set; calculating the contribution coefficient of each feature component in the defect-sensitive feature set to the defect diagnosis result through a feature contribution analysis algorithm, and determining inferior contribution feature components whose contribution coefficients are lower than a preset contribution threshold; reducing the weight ratio of the inferior contribution feature components in the cross-feature attention mechanism based on the associated weight coefficients of the inferior contribution feature components in the feature fusion layer of the TP fitting defect diagnosis model, and obtaining a TP fitting defect diagnosis model after weight adjustment; inputting the model optimization feedback data set into the TP fitting defect diagnosis model after weight adjustment for iterative training, updating the model parameters of each layer through a back propagation algorithm, and generating a TP fitting defect diagnosis model after parameter optimization.

[0225] After generating a production process defect diagnosis report containing defect diagnosis content, the TP fitting defect diagnosis model needs to be optimized to further improve its performance. First, defect sample data containing defect type and corresponding location distribution information is extracted from the production process defect diagnosis report. This defect sample data includes the characteristic components of each defect type in the real-time image sequence, such as surface texture features, edge contour features, regional grayscale features, etc., as well as the corresponding diagnostic label. The diagnostic label clearly defines the defect type corresponding to the sample.

[0226] Next, the extracted defect sample data is aligned with the sample data in the historical defect sample library through feature space alignment. Since data collected at different times may differ in spatial resolution and number of channels, in order to ensure data consistency and comparability, it is necessary to unify the spatial resolution and number of channels of the defect-sensitive feature set. For example, for feature data with different spatial resolutions, interpolation or downsampling methods can be used to achieve a uniform resolution; for inconsistent channel numbers, adjustments can be made through operations such as channel merging or splitting. After feature space alignment, a model optimization feedback dataset is generated, which combines defect samples from the current production process and historical defect samples.

[0227] Next, a feature contribution analysis algorithm is used to calculate the contribution coefficient of each feature component in the defect-sensitive feature set to the defect diagnosis results. This algorithm comprehensively considers factors such as the correlation between the feature and the defect diagnosis results and the role of the feature in the model. It then calculates a contribution coefficient for each feature component, which reflects its importance to the defect diagnosis results. By calculating the contribution coefficient, it is possible to determine which feature components contribute significantly to the diagnosis results and which contribute less. Next, a preset contribution threshold is set to screen out feature components with contribution coefficients below this threshold and identify them as inferior contributing feature components.

[0228] Based on the associated weight coefficients of the inferior contributing feature components in the feature fusion layer of the TP fitting defect diagnosis model, their weight in the cross-feature attention mechanism is adjusted. In the feature fusion layer, the cross-feature attention mechanism assigns weights to each feature based on the degree of correlation between the features to achieve effective feature fusion. For inferior contributing feature components, since they contribute less to the defect diagnosis results, their weight in the cross-feature attention mechanism is reduced, reducing the model's reliance on these features when fusing features. After weight adjustment, the TP fitting defect diagnosis model with adjusted weights is obtained.

[0229] Finally, the model optimization feedback dataset is fed into the weighted TP fit defect diagnosis model for iterative training. During training, the backpropagation algorithm is used to update the model parameters at each layer. Based on the error between the model's output and the true label, the backpropagation algorithm calculates gradients layer by layer, starting from the output layer. The algorithm then adjusts the model parameters based on the gradients, gradually aligning the model's output with the true label. Through multiple iterative training cycles, the model parameters are continuously optimized, ultimately generating a parameter-optimized TP fit defect diagnosis model. This optimized model demonstrates greater accuracy and reliability in defect diagnosis.

[0230] In a non-limiting embodiment, after generating a production process defect diagnosis report containing defect diagnosis content based on the defect type and the location distribution information, the method further includes: identifying suspected diagnosis deviation defect cases whose diagnosis confidence is lower than a preset confidence threshold from the production process defect diagnosis report, and extracting a defect sensitive feature set corresponding to the suspected diagnosis deviation defect case; performing feature distribution difference analysis on the defect sensitive feature set of the suspected diagnosis deviation defect case and the defect sensitive feature set of the historical normal bonding component, and determining the distribution deviation index of the two types of feature sets in surface texture features, edge contour features and regional grayscale features; based on the distribution deviation index, screening out key feature dimensions whose deviation index exceeds a preset deviation threshold, the key feature dimension being the main feature component that causes diagnostic deviation; adjusting the principal component analysis parameters of the feature dimensionality reduction layer of the TP bonding defect diagnosis model according to the key feature dimension, and increasing the retention ratio of the principal component component corresponding to the key feature dimension in the core defect feature vector; embedding the feature dimensionality reduction layer after adjusting the parameters into the TP bonding defect diagnosis model, performing model verification through the defect sensitive feature set of the suspected diagnosis deviation defect case, and generating a TP bonding defect diagnosis model after optimizing the feature dimensionality reduction strategy.

[0231] After generating the production process defect diagnosis report, in order to further optimize the TP fitting defect diagnosis model, it is necessary to deal with possible diagnostic deviations. First, identify suspected diagnostic deviation defect cases whose diagnostic confidence is lower than the preset confidence threshold from the production process defect diagnosis report. The diagnostic confidence reflects the degree of certainty of the model's diagnosis result for a certain defect. When the confidence is lower than the preset confidence threshold, it indicates that the diagnostic result may be biased. For these suspected diagnostic deviation defect cases, the corresponding defect sensitive feature set is extracted. This set contains defect-related feature information such as surface texture features, edge contour features, and regional grayscale features.

[0232] Next, we analyze the difference in feature distributions between the defect-sensitive feature sets of suspected diagnostic deviation defects and those of historically properly bonded components. By comparing the distribution of surface texture, edge contour, and regional grayscale features between the two feature sets, we determine a distribution deviation index. This index measures the degree of difference between the two feature sets across various feature dimensions. For example, we can determine this by calculating the difference in statistical quantities such as the mean, variance, and probability density function of the feature distribution.

[0233] Based on the calculated distribution deviation index, a preset deviation threshold is set to screen out key feature dimensions whose deviation index exceeds this threshold. These key feature dimensions are the main characteristic components that cause diagnostic deviation because their distribution differs significantly between suspected diagnostic deviation defect cases and normal fitting components. For example, within a specific frequency range of surface texture features, the feature distribution of suspected diagnostic deviation defect cases may be significantly different from that of normal components.

[0234] Based on the selected key feature dimensions, adjust the principal component analysis parameters of the feature dimensionality reduction layer of the TP fitting defect diagnosis model. Principal component analysis is a method used by the feature dimensionality reduction layer to reduce data dimensionality. When adjusting the parameters, increase the proportion of principal components corresponding to key feature dimensions retained in the core defect feature vector. This aims to make the model place greater emphasis on these key feature dimensions during the dimensionality reduction process, thereby avoiding the loss of important diagnostic information.

[0235] The parameter-adjusted feature dimensionality reduction layer is embedded in the TP bonding defect diagnosis model. The model is then validated using a set of defect-sensitive features from cases suspected of diagnostic bias. During the validation process, the model's diagnostic results for these cases are observed to check whether diagnostic confidence is improved and diagnostic bias is reduced. Through continuous adjustment and validation, a TP bonding defect diagnosis model with an optimized feature dimensionality reduction strategy is ultimately generated. This optimized model exhibits improved diagnostic accuracy and stability when handling similar cases of suspected diagnostic bias, enabling more effective identification and diagnosis of defects in the TP bonding process.

[0236] In summary, the embodiments of the present invention significantly improve the accuracy, comprehensiveness, and efficiency of defect diagnosis during the TP lamination process. First, a real-time image sequence containing component surface images and edge region images at different lamination stages is collected, providing a rich and dynamic data source for defect diagnosis, enabling the capture of potential defect information at each stage of the lamination process. Second, defect-sensitive feature enhancement processing is performed on the real-time image sequence to obtain a defect-sensitive feature set containing surface texture features, edge contour features, and regional grayscale features. This effectively highlights key defect-related features and improves the sensitivity of subsequent diagnosis. Then, feature correlation analysis is performed on the defect-sensitive feature set using a trained TP lamination defect diagnosis model, which can explore the inherent connections between different features and generate more reliable preliminary defect diagnosis results. Finally, based on the preliminary diagnosis results, defect type and location distribution information are determined, and a production process defect diagnosis report containing the defect diagnosis content is generated, achieving precise defect location and comprehensive diagnosis, providing strong support for production process optimization and quality control. As a result, the embodiments of the present invention can promptly detect defects during the TP lamination process, reduce the defective rate, and improve production efficiency and product quality.

[0237] Further, Figure 2 The structural block diagram of the TP bonding process production process defect diagnosis system 300 based on image analysis is shown, including: a memory 310 for storing program instructions and data; a processor 320 for coupling with the memory 310 and executing the instructions in the memory 310 to implement the above method.

[0238] Furthermore, a computer storage medium is provided, comprising instructions, which implement the above method when executed on a processor.

[0239] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for diagnosing defects in a TP lamination process based on image analysis, characterized in that: include: Collecting real-time image sequences corresponding to the components to be inspected that are continuously transmitted on the TP bonding process production line. The real-time image sequences include component surface images and edge area images at different bonding stages. Performing defect-sensitive feature enhancement processing on the real-time image sequence to obtain a defect-sensitive feature set of the bonded component to be inspected, wherein the defect-sensitive feature set includes surface texture features, edge contour features, and regional grayscale features; Performing feature correlation analysis on the defect-sensitive feature set using a trained TP bonding defect diagnosis model to generate a preliminary defect diagnosis result for the bonding component to be inspected; Determining, based on the preliminary defect diagnosis result, the defect type of the assembly to be inspected and the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and generating a production process defect diagnosis report containing defect diagnosis content based on the defect type and the position distribution information; The determining, based on the preliminary defect diagnosis result, of the defect type of the to-be-detected assembly and the position distribution information of the defect corresponding to the defect type in the real-time image sequence includes: Analyzing the candidate defect types and corresponding probability values ​​in the preliminary defect diagnosis result, sorting the candidate defect types in descending order of probability value, and selecting a preset number of candidate defect types with the highest order as the defect types to be verified; For each defect type to be verified, extract feature components related to the defect type to be verified from the defect-sensitive feature set to construct a type-specific feature subset; Performing secondary verification processing on the type-specific feature subset using the trained defect type verification model, and outputting the verification confidence of the defect type to be verified; Comparing the verification confidence with a preset verification threshold, and when the verification confidence exceeds the verification threshold, determining the defect type to be verified as the final defect type; Based on the spatial position coordinates of the characteristic components corresponding to the final defect type in the defect sensitive feature set, the minimum circumscribed rectangular area of ​​the defect corresponding to the final defect type in the real-time image sequence is determined, and the vertex coordinate set of the minimum circumscribed rectangular area is used as the position distribution information of the defect corresponding to the defect type in the real-time image sequence.

2. The method according to claim 1, characterized in that The performing defect-sensitive feature enhancement processing on the real-time image sequence to obtain a defect-sensitive feature set of the assembly to be inspected includes: Performing adaptive illumination compensation processing on the component surface image in the real-time image sequence by multi-scale Gaussian filtering to eliminate non-uniform illumination interference and generate an illumination normalized surface image; performing texture enhancement processing on the illumination normalized surface image, extracting texture detail features at different scales using Laplace pyramid decomposition, and enhancing texture edge information through contrast-limited adaptive histogram equalization to obtain enhanced surface texture features; Performing edge detection preprocessing on edge area images in the real-time image sequence, extracting edge contour point sets using the Canny operator, and connecting broken contour lines using a morphological closing operation to generate continuous edge contour features; Performing region division processing on the illumination normalized surface image, evenly dividing the image into a plurality of non-overlapping rectangular area units, calculating the grayscale mean and grayscale variance within each area unit, and constructing a regional grayscale feature matrix as a regional grayscale feature; The surface texture features, edge contour features and regional grayscale features are aligned in feature dimensions to unify the spatial resolution and number of channels of each feature, thereby generating a defect-sensitive feature set containing multi-dimensional defect-sensitive information.

3. The method according to claim 1, characterized in that The trained TP bonding defect diagnosis model is used to perform feature correlation analysis on the defect sensitive feature set to generate a preliminary defect diagnosis result of the bonding component to be inspected, including: Input the surface texture features, edge contour features, and regional grayscale features in the defect-sensitive feature set into the feature fusion layer of the TP fitting defect diagnosis model, and calculate the correlation weight coefficients between different features through the cross-feature attention mechanism; Performing weighted fusion processing on the surface texture features, edge contour features, and regional grayscale features based on the association weight coefficient to generate a joint feature vector that fuses multi-dimensional defect information; Performing principal component analysis on the joint feature vector through the feature dimension reduction layer of the TP bonding defect diagnosis model, retaining the principal component components whose cumulative contribution rate exceeds a preset ratio, and obtaining a reduced-dimensional core defect feature vector; Input the core defect feature vector into the defect classification subnetwork of the TP fitting defect diagnosis model, and output a probability distribution vector containing multiple defect types through a combination of a fully connected layer and an activation function; According to the probability value corresponding to each defect type in the probability distribution vector, defect types with probability values ​​exceeding a set threshold are screened out as candidate defect types, and a preliminary defect diagnosis result including the candidate defect types and the corresponding probability values ​​is generated.

4. The method according to claim 3, characterized in that The surface texture features, edge contour features, and regional grayscale features in the defect sensitive feature set are input into the feature fusion layer of the TP fitting defect diagnosis model, and the correlation weight coefficients between different features are calculated through the cross-feature attention mechanism, including: Determine the spatial position correlation by calculating the cosine similarity of pixel values ​​at corresponding positions in the feature map of the surface texture feature and the feature map of the edge contour feature, and construct a first correlation matrix of the surface texture feature and the edge contour feature in the feature fusion layer; Based on the Euclidean distance calculation between the row vector of the regional grayscale feature matrix and the coordinates of the contour point set of the edge contour feature, the region-contour space distance correlation is obtained, and a second correlation matrix between the edge contour feature and the regional grayscale feature is constructed; By calculating the Pearson correlation coefficient of the grayscale co-occurrence matrix of the texture detail features and the grayscale mean of the regional grayscale features, the texture-region grayscale correlation is determined, and the third correlation matrix of the surface texture features and the regional grayscale features is constructed; The first correlation matrix, the second correlation matrix and the third correlation matrix are input into the attention weight calculation unit, and the matrix elements are normalized by the softmax function to generate cross-feature correlation weight coefficients between each feature.

5. The method according to claim 3, characterized in that The core defect feature vector is input into the defect classification sub-network of the TP fitting defect diagnosis model, and a probability distribution vector containing multiple defect types is output through a combination of a fully connected layer and an activation function, including: Inputting the core defect feature vector into the first fully connected layer of the defect classification subnetwork, mapping the core defect feature vector to the first hidden feature space through linear transformation of the weight matrix and the bias vector, and generating a first hidden layer feature vector; Performing nonlinear activation processing on the first hidden layer feature vector, using a ReLU activation function to enhance feature expression capability, and generating an activated hidden feature vector; Inputting the activated hidden feature vector into the second fully connected layer of the defect classification subnetwork, mapping the activated hidden feature vector to a second hidden feature space through a linear transformation of another set of weight matrices and bias vectors, and generating a second hidden layer feature vector; Performing batch normalization on the second hidden layer feature vector to eliminate distribution differences between features of different batches and generate a normalized hidden feature vector; The normalized hidden feature vector is input into the output layer of the defect classification subnetwork, and the probability value corresponding to each defect type is calculated through the softmax activation function to generate a probability distribution vector containing multiple defect types.

6. The method according to claim 1, characterized in that The determining, based on the spatial position coordinates of the feature components corresponding to the final defect type in the defect sensitive feature set, of a minimum bounding rectangular area of ​​the defect corresponding to the final defect type in the real-time image sequence, and using a vertex coordinate set of the minimum bounding rectangular area as position distribution information of the defect corresponding to the defect type in the real-time image sequence, includes: Extracting a texture abnormality region coordinate point set of a characteristic component corresponding to the final defect type in the surface texture feature, each coordinate point including a horizontal and vertical coordinate value of a two-dimensional image plane; Performing coordinate normalization processing on the coordinate point set of the texture abnormality area, converting the coordinate values ​​into proportional coordinates relative to the image resolution, and eliminating coordinate differences caused by different image sizes; The rotating caliper algorithm is used to calculate the minimum bounding rectangle of the normalized coordinate point set to determine the rectangular area with the smallest area that completely contains all the coordinate points. Extracting the coordinates of four vertices of the minimum circumscribed rectangular area, and sorting the vertex coordinates in a clockwise direction of the image coordinate system to generate an ordered vertex coordinate sequence; The scale coordinates in the ordered vertex coordinate sequence are converted into pixel coordinates at the original image resolution to obtain position distribution information of defects corresponding to the defect type in the real-time image sequence, which includes four pixel vertex coordinates.

7. The method according to claim 1, characterized in that The generating of a production process defect diagnosis report containing defect diagnosis content based on the defect type and the location distribution information includes: Querying a preset defect severity assessment rule library according to the defect type to obtain a severity assessment index system corresponding to the defect type, wherein the assessment index system includes defect area ratio, defect number density, and defect location importance coefficient; Calculating the defect area based on the minimum circumscribed rectangular area of ​​the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and taking the ratio of the defect area to the total surface area of ​​the bonded component as the defect area ratio; Counting the number of defects of the same defect type in the real-time image sequence, and taking the ratio of the number of defects to the total number of frames in the image sequence as the defect number density; determining whether the defect area is a critical lamination area based on the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and if so, setting the defect position importance coefficient to a first preset value; otherwise, setting the defect position importance coefficient to a second preset value; Inputting the defect area ratio, defect number density and defect location importance coefficient into a severity calculation function to generate a defect severity level; Fusion of the defect type, location distribution information of the defect corresponding to the defect type in the real-time image sequence, the defect severity level, and the corresponding real-time image sequence timestamp to generate a production process defect diagnosis report containing text and image combined with diagnostic annotations; Before fusing the defect type, the position distribution information of the defect corresponding to the defect type in the real-time image sequence, the defect severity level and the corresponding real-time image sequence timestamp to generate a production process defect diagnosis report including graphic and text combined with diagnostic annotations, the method further includes: Performing temporal continuity verification on position distribution information of defects corresponding to the defect type in the real-time image sequence, extracting position distribution information of the same defect type in adjacent frame images, and calculating a position offset; When the position offset is less than the preset offset threshold, the defect is determined to be a real defect; when the position offset is greater than or equal to the preset offset threshold, the defect is determined to be a suspected false detection defect; Performing secondary feature extraction on the defect-sensitive feature set of the suspected falsely detected defect to detect and analyze the continuity of edge contour features and the mutation of regional grayscale features; When the edge contour continuity index in the secondary feature extraction result exceeds the contour threshold and the regional grayscale mutation index exceeds the grayscale threshold, the suspected falsely detected defect is re-determined as a real defect; otherwise, the suspected falsely detected defect is removed from the defect type; retaining defect types corresponding to all real defects, location distribution information of defects corresponding to the defect types in the real-time image sequence, and defect severity levels; The method of fusing the defect type, the position distribution information of the defect corresponding to the defect type in the real-time image sequence, the defect severity level and the corresponding real-time image sequence timestamp to generate a production process defect diagnosis report containing graphic and text combined with diagnostic annotations includes: Creating an independent defect diagnosis sub-report for each defect type, the defect diagnosis sub-report including the defect type name, the defect occurrence timestamp, the vertex coordinate sequence of the position distribution information of the defect corresponding to the defect type in the real-time image sequence, and the defect severity level; intercepting an image segment containing an area corresponding to position distribution information of a defect corresponding to the defect type in the real-time image sequence from the real-time image sequence, performing defect labeling processing on the defect area in the image segment, and generating a labeled defect image; The annotated defect image is associated with the text description information in the defect diagnosis sub-report and stored to establish a one-to-one correspondence between the image and the text; Sort all defect diagnosis sub-reports according to the time stamp order of the real-time image sequence to generate a defect diagnosis sequence arranged in chronological order; All defect diagnosis sub-reports and corresponding annotated defect images in the defect diagnosis sequence are integrated into a structured document, and the report generation time, production line number and bonding component batch information are added to generate a complete production process defect diagnosis report.

8. A TP lamination process production process defect diagnosis system based on image analysis, characterized in that: include: Memory, used to store program instructions and data; A processor, coupled to a memory, and configured to execute instructions in the memory to implement the method according to any one of claims 1 to 7.

9. A computer storage medium, characterized in that The method comprises instructions which, when executed on a processor, implement the method according to any one of claims 1 to 7.

Images