Computer vision processing method and system for industrial defect real-time detection

Abstract

The invention relates to a computer vision processing method and system for industrial defect real-time detection. The method comprises the following steps: extracting geometric features and textural features of predefined defect types, and generating a structured descriptor set; generating a synthetic defect image set based on the defect-free image set and the structured descriptor set; inputting the synthesized defect image set into a double-flow feature extraction network to obtain a fusion feature vector; generating a defect category threshold set based on the vector and the structured descriptor set; and inputting the to-be-detected image and the corresponding defect-free reference image into the double-flow feature extraction network, calculating defect probability distribution in combination with the structured descriptor set and the dynamic classifier, and outputting a defect category decision result based on the defect category threshold set. According to the method, the precision, robustness and adaptability of defect detection are improved by means of fusing the global features of the defect-free reference image and the local features of the defect image and expanding training samples by using the synthetic defect image set.

Description

Technical Field

[0001] This invention belongs to the field of industrial inspection technology, and in particular relates to a computer vision processing method and system for real-time detection of industrial defects. Background Technology

[0002] In industrial production, product defect detection is a crucial step in ensuring product quality. Computer vision technology, with its advantages of being non-contact, highly efficient, and automated, has become the mainstream technique for industrial defect detection. Traditional computer vision defect detection methods are mostly based on manually designed features, such as extracting geometric or textural features of defects through edge detection, threshold segmentation, and texture analysis, and then combining these with classifiers to complete defect identification. While these methods offer good real-time performance in structured scenarios, they are poorly adaptable to the diversity of defect types, changes in ambient lighting, and differences in product surface texture, lacking robustness and failing to meet the high-precision detection requirements of complex industrial scenarios.

[0003] With the development of deep learning technology, defect detection methods based on deep neural networks have been widely used. For example, models such as Convolutional Neural Networks (CNN), YOLO, and Faster R-CNN are used to directly extract features and classify defect images. These methods rely on a large number of labeled real defect samples for model training, and can achieve high detection accuracy in scenarios with sufficient samples. However, in actual industrial production, defect samples are usually scarce, especially novel defect samples. Manually labeling defect samples is not only time-consuming and labor-intensive, but also suffers from strong subjectivity and poor consistency, making it difficult to fully train supervised learning models and severely limiting detection accuracy and generalization ability.

[0004] To alleviate the problem of sample scarcity, some technologies have proposed sample synthesis methods based on data augmentation or generative adversarial networks (GANs). However, traditional data augmentation can only perform simple transformations on existing samples, making it difficult to generate defect samples with new shapes and textures, thus limiting the improvement in sample diversity. Furthermore, defect samples generated by conventional generative adversarial networks often suffer from blurred textures and significant deviations in defect shapes from real-world scenes, which can easily lead to overfitting during model training and fail to effectively improve detection performance. Moreover, existing deep learning detection models mostly employ single-stream network architectures, extracting only a single feature from defect images. This fails to fully utilize the correlation information between the global features of the defect-free reference image and the local features of the defect image, making them prone to missed or false detections in complex backgrounds or scenarios involving small defects.

[0005] Furthermore, existing defect detection methods often rely on fixed thresholds for classification decisions, making it difficult to adapt to the varying confidence distributions of different defect types. In boundary scenarios where the probabilities of multiple defect categories are close to the threshold, they lack effective decision-making mechanisms, leading to classification errors. Additionally, when new defect types are added to the production line, existing models require retraining to adapt, exhibiting poor scalability and failing to meet the demands for rapid detection of new defects in industrial settings. Summary of the Invention

[0006] Therefore, it is necessary to provide a computer vision processing method and system for real-time industrial defect detection to address the above-mentioned technical problems, aiming to improve the accuracy and robustness of defect detection, enhance the adaptability of the defect detection model to defect types, and improve the detection performance in scenarios with scarce samples.

[0007] In a first aspect, this application provides a computer vision processing method for real-time detection of industrial defects, including:

[0008] Extract geometric and texture features of predefined defect types, and generate a set of structured descriptors based on the geometric and texture features; generate a synthetic defect image set based on the set of defect-free images and the set of structured descriptors through a condition generator;

[0009] A dual-stream feature extraction network is invoked, and the synthesized defect image set is input into the dual-stream feature extraction network to obtain a fused feature vector. Based on the fused feature vector and the set of structured descriptors, a dynamic classifier is used to perform classification calculations to generate a set of defect category thresholds.

[0010] The test image and the defect-free reference image matched with the test image are acquired in real time. The test image and the defect-free reference image are input into a two-stream feature extraction network for feature extraction to obtain the test feature vector.

[0011] Based on the test feature vector, the set of structured descriptors, and the dynamic classifier, the defect probability distribution is calculated; based on the defect probability distribution and the set of defect category thresholds, the defect category decision result is output.

[0012] In one embodiment, geometric and texture features of predefined defect types are extracted; a set of structured descriptors is generated based on the geometric and texture features; and a synthetic defect image set is generated using a condition generator based on the defect-free image set and the set of structured descriptors, including:

[0013] Extract the geometric features of each type of defect from the predefined defect types, and construct a geometric feature vector based on the geometric features; the geometric features include aspect ratio and curvature;

[0014] Extract texture features for each type of defect from the predefined defect types, and construct a texture feature vector based on the texture features; the texture features include texture entropy values;

[0015] Geometric feature vectors and texture feature vectors are concatenated to generate structured descriptors, and a set of structured descriptors is formed by multiple structured descriptors.

[0016] Defect-free images are selected from a set of defect-free images as input images for the conditional generator; the conditional generator is constructed based on the U-Net architecture.

[0017] Construct a multilayer perceptron, input the structured descriptor into the multilayer perceptron for encoding processing, and obtain the encoded descriptor;

[0018] Based on the geometric feature range of a predefined defect type, a random defect region mask is generated by random sampling; the size and shape range of the random defect region mask are matched with the aspect ratio and curvature of the geometric feature, respectively.

[0019] The input image, the encoded descriptor, and the random defect region mask are input into the condition generator, and the input image is processed to synthesize defects to obtain a synthesized defect image.

[0020] Associate the synthesized defect image, the corresponding structured descriptor, and the corresponding random defect region mask to construct multiple triplet data, and combine the triplet data to form a synthetic defect image set.

[0021] In one embodiment, the dual-stream feature extraction network includes a defect feature stream, a reference feature stream, a fully connected layer, and a fusion layer; the defect feature stream adopts a depthwise separable convolutional network architecture, and the reference feature stream adopts a preset lightweight Transformer network architecture.

[0022] The two-stream feature extraction network is invoked, and the synthesized defect image set is input into the two-stream feature extraction network to obtain the fused feature vector, including:

[0023] Synthetic defect images are selected from the set of synthetic defect images. The synthetic defect images are input into the defect feature stream. The feature stream is used to extract features from the synthetic defect images to obtain defect feature maps. The defect feature maps are two-dimensional spatial features with a preset channel dimension.

[0024] A standard defect-free reference image paired with the synthetic defect image is obtained. The standard defect-free reference image is input into a reference feature stream. Feature extraction processing is performed on the standard defect-free reference image through the reference feature stream to obtain a global reference feature vector. The global reference feature vector is a one-dimensional global feature.

[0025] The global reference feature vector is input into the fully connected layer of the dual-stream feature extraction network for dimension adaptation processing to obtain the feature map statistical parameters. The feature map statistical parameters include mean parameters and variance parameters, and the dimensions of the mean parameters and variance parameters are matched with the preset channel dimensions of the defect feature map.

[0026] Based on the feature map statistical parameters, the defect feature map is adaptively normalized using a preset formula to obtain a spatially aligned feature map. The spatially aligned feature map and the global reference feature vector are associated through the feature map statistical parameters to satisfy the spatial alignment relationship.

[0027] Through the fusion layer, the spatially aligned feature map is weighted based on the channel attention mechanism to obtain the fused feature map. The fused feature map is then subjected to global average pooling to obtain the fused feature vector.

[0028] In one embodiment, based on the fused feature vector and the set of structured descriptors, a dynamic classifier is used to perform classification calculations to generate a set of defect category thresholds, including:

[0029] The synthetic defect image set is divided into a training set and a validation set; the dynamic classifier is pre-trained using the training set to obtain the pre-trained dynamic classifier; the dynamic classifier is a classification network containing fully connected layers.

[0030] The fused feature vector is concatenated with the structured descriptors in the set of structured descriptors to obtain the concatenated feature vector.

[0031] The concatenated feature vector is input into the pre-trained dynamic classifier for classification calculation to obtain the classification result.

[0032] Based on the classification results and validation set, calculate the precision and recall of each defect type in the predefined defect types, and statistically analyze the confidence distribution of each defect type.

[0033] Based on the principle of maximizing the harmonic mean of precision and recall, a threshold is calculated for the confidence distribution of each type of defect to obtain the optimal decision threshold for each type of defect.

[0034] The defect category threshold set consists of multiple optimal decision thresholds.

[0035] In one embodiment, a real-time captured image to be tested and a defect-free reference image matched with the image to be tested are acquired. The image to be tested and the defect-free reference image are input into a two-stream feature extraction network for feature extraction to obtain a test feature vector. Based on the test feature vector, a set of structured descriptors, and a dynamic classifier, a defect probability distribution is calculated, including:

[0036] Acquire the real-time captured image to be tested, and select a defect-free image with the same model as the image to be tested as a defect-free reference image;

[0037] Calculate the first gray-level mean of the image under test and the second gray-level mean of the defect-free reference image; calculate the gray-level mean ratio based on the first and second gray-level mean; perform illumination correction processing on the image under test based on the gray-level mean ratio to obtain the corrected image under test.

[0038] The corrected image to be tested is input into the defect feature stream of the dual-stream feature extraction network, and feature extraction is performed through the defect feature stream to obtain the test feature map.

[0039] A defect-free reference image is input into the reference feature stream of a dual-stream feature extraction network, and feature extraction is performed through the reference feature stream to obtain a test reference feature vector.

[0040] Adaptive instance normalization is applied to the test feature map to obtain a test space aligned feature map; the test space aligned feature map is weighted based on the channel attention mechanism to obtain a fused test feature map; global average pooling is applied to the fused test feature map to obtain a test feature vector.

[0041] The test feature vector is concatenated with the structured descriptors in the structured descriptor set to obtain the test concatenated vector; the test concatenated vector is then input into the dynamic classifier, which performs probability calculations to obtain the defect probability distribution.

[0042] In one embodiment, based on the defect probability distribution and the defect category threshold set, a defect category decision result is output, including:

[0043] Analyze the defect probability distribution and extract the highest probability value and the corresponding defect category index from the defect probability distribution;

[0044] The defect probability distribution is analyzed, and the highest probability value and its corresponding first defect category index are extracted from the defect probability distribution. The second high probability value and its corresponding second defect category index are also extracted, and the difference between the highest probability value and the highest probability value is less than a preset probability difference threshold.

[0045] The first target decision threshold is obtained from the defect category threshold set based on the first defect category index; if a second defect category index exists, the second target decision threshold is obtained from the defect category threshold set based on the second defect category index.

[0046] Compare the highest probability value with the first target decision threshold. If a second target decision threshold exists, compare the second highest probability value with the corresponding second target decision threshold to obtain the comparison result.

[0047] If the comparison result is that the highest probability value is greater than or equal to the first target decision threshold, and the second highest probability value is less than the corresponding second target decision threshold, the first defect category identifier corresponding to the first defect category index is output, and the defect category decision result is obtained by combining the first defect category identifier and the highest probability value.

[0048] If the comparison result is that the highest probability value is greater than or equal to the first target decision threshold, and there is at least one second high probability value greater than or equal to the corresponding second target decision threshold, calculate the feature similarity between the structured descriptor and the test feature vector corresponding to the first defect category index and the second defect category index, respectively. Take the defect category identifier corresponding to the defect category index with the highest feature similarity as the final defect category identifier. Combine the final defect category identifier and the corresponding probability value to obtain the defect category decision result.

[0049] If the comparison result shows that the highest probability value is less than the first target decision threshold, output a defect-free label and use the defect-free label as the defect category decision result.

[0050] In one embodiment, the preset formula is:

[0051]

[0052]

[0053]

[0054] in, For adaptive instance normalization operators, For spatial alignment feature maps, This is a defect feature map with dimensions of [dimensional value missing]. , The height of the defect feature map. The width of the defect feature map. The number of feature channels, This represents the channel-level mean of the defect feature map in the spatial dimension. The channel-level standard deviation of the defect feature map in the spatial dimension. The mean parameter of the global reference feature vector after dimensionality adaptation by the fully connected layer, and Matching the number of channels, The variance parameter is the result of dimensionality adaptation of the global reference feature vector through a fully connected layer. This is a preset minimum value used to avoid numerical instability caused by a denominator of 0. This represents the pixel values ​​of each channel in the i-th row and j-th column of the feature map.

[0055] Secondly, this application also provides a computer vision processing system for real-time detection of industrial defects, comprising:

[0056] The feature extraction and synthesis module is used to extract geometric and texture features of predefined defect types, generate a set of structured descriptors based on the geometric and texture features, and generate a set of synthesized defect images based on the set of defect-free images and the set of structured descriptors through a condition generator.

[0057] The feature fusion and threshold generation module is used to call the dual-stream feature extraction network, input the synthesized defect image set into the dual-stream feature extraction network, and obtain the fused feature vector; based on the fused feature vector and the set of structured descriptors, the dynamic classifier performs classification calculations to generate a set of defect category thresholds;

[0058] The real-time image feature extraction module is used to acquire the real-time captured image to be tested and the defect-free reference image matched with the image to be tested. The image to be tested and the defect-free reference image are input into the dual-stream feature extraction network for feature extraction to obtain the test feature vector.

[0059] The defect probability calculation and decision module is used to calculate the defect probability distribution based on the test feature vector, the set of structured descriptors and the dynamic classifier; and output the defect category decision result based on the defect probability distribution and the defect category threshold set.

[0060] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the first aspect.

[0061] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the first aspect.

[0062] The aforementioned computer vision processing method and system for real-time industrial defect detection first generates a structured descriptor by extracting the geometric and texture features of predefined defects. Then, it combines a set of defect-free images with a conditional generator to synthesize a defect image set, providing a rich data foundation for model training and addressing the core pain point of scarce real defect samples in traditional detection. Secondly, a dual-stream feature extraction network is used to process the synthesized image to obtain a fused feature vector, fully combining local defect features with global reference features of defect-free images. This solves the problem of one-sided feature representation in single-stream networks and improves feature discrimination capability. Furthermore, based on the fused features and structured descriptors, a dynamic classifier generates a set of category thresholds, replacing traditional fixed thresholds, adapting to the different confidence distributions of various defects, and improving classification accuracy. Finally, dual-stream feature extraction is performed on the real-time test image and the matching reference image to obtain test features. The association information between the two images enhances the detection robustness in complex environments and avoids false negatives and missed detections caused by interference such as lighting. Finally, based on the probability distribution and threshold set, the decision results are output. Through multi-scenario adapted decision logic, the problem of fuzzy classification of boundary scenarios is solved, ensuring the reliability of real-time detection in industrial scenarios and providing strong support for efficient quality inspection. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A flowchart of a computer vision processing method for real-time detection of industrial defects is provided as an exemplary embodiment of the present invention.

[0065] Figure 2 A flowchart illustrating a method for generating a set of defect category thresholds, provided as an exemplary embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of a computer vision processing system for real-time detection of industrial defects, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] In one embodiment, such as Figure 1As shown, a computer vision processing method for real-time detection of industrial defects is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0069] S101: Extract the geometric and texture features of predefined defect types, and generate a set of structured descriptors based on the geometric and texture features; generate a set of synthetic defect images based on the set of defect-free images and the set of structured descriptors through a condition generator.

[0070] Specifically, in industrial production, the geometric and textural features of defects are key attributes for defect identification. Geometric features can include information such as the shape, size, and boundaries of the defect, while textural features involve the texture pattern and grayscale distribution of the defect surface. These features can be extracted from labeled defect samples using image processing algorithms such as edge detection and texture analysis, and further generated into a set of structured descriptors. This set is a data structure that can accurately represent defect features, quantifying and encoding geometric and textural features to provide a foundation for subsequent image synthesis and classification. Subsequently, based on the set of defect-free images and the set of structured descriptors, a conditional generator can be used to generate a set of synthetic defect images. The conditional generator is a deep learning-based generative model that uses defect-free images as a background and combines them with the defect features defined in the set of structured descriptors to generate synthetic defect images with specific geometric and textural features. This process not only increases the number of training samples but also improves the model's adaptability to different defect types by generating diverse defect morphologies and textures.

[0071] S102: Call the dual-stream feature extraction network, input the synthesized defect image set into the dual-stream feature extraction network to obtain the fused feature vector; based on the fused feature vector and the set of structured descriptors, perform classification calculation through a dynamic classifier to generate a set of defect category thresholds.

[0072] Specifically, the two-stream feature extraction network (DBN) is a special neural network architecture containing two parallel feature extraction paths: one for extracting global features from the defect-free reference image, and the other for extracting local features from the defective image. This two-stream architecture can fully utilize the background information of the defect-free reference image and the detailed information of the defective image, thus providing a more comprehensive representation of defect features. Through the DBN, the synthesized defective image set can be processed into a fused feature vector, which integrates global and local features, providing rich feature information for subsequent classification calculations. Based on the fused feature vector and the structured descriptor set, classification can also be performed using a dynamic classifier. A dynamic classifier is a model that can dynamically adjust its classification strategy based on input features, combining prior knowledge from the fused feature vector and the structured descriptor set to classify different types of defects. During the classification process, the dynamic classifier can consider not only the numerical information of the feature vector but also the defect features defined in the structured descriptor set, thereby improving the accuracy and robustness of the classification. The final set of defect category thresholds provides a basis for decision-making in subsequent defect detection. It can dynamically adjust the classification threshold according to the feature distribution of different defect types, thereby effectively reducing classification errors in boundary scenarios where the probabilities of multiple defect categories are close to the threshold.

[0073] S103: Acquire the real-time captured image to be tested and the defect-free reference image matched with the image to be tested. Input the image to be tested and the defect-free reference image into a dual-stream feature extraction network for feature extraction to obtain the test feature vector.

[0074] Specifically, in actual industrial production, the image to be tested is usually acquired in real time from the production line, while the defect-free reference image is a defect-free image that matches the image to be tested in terms of background and lighting conditions. Inputting these two images into a dual-stream feature extraction network (DVR) allows for feature extraction. The DVR's dual-path architecture extracts local features from the image to be tested and global features from the defect-free reference image separately, then fuses them into a test feature vector. This feature extraction method not only preserves the defect details of the image to be tested but also provides background information through the global features of the defect-free reference image, thus effectively reducing the possibility of missed or false detections in complex backgrounds or scenarios involving the detection of minute defects.

[0075] S104: Calculate the defect probability distribution based on the test feature vector, the set of structured descriptors, and the dynamic classifier; output the defect category decision result based on the defect probability distribution and the set of defect category thresholds.

[0076] Specifically, by combining real-time feature information from the test feature vector, prior knowledge from the structured descriptor set, and the classification ability of the dynamic classifier, the probability of each defect category can be calculated. The resulting defect probability distribution provides an intuitive understanding of the likelihood of each defect category in the image under test, thus providing a basis for subsequent decision-making. Subsequently, based on the defect probability distribution and the defect category threshold set, the defect category decision result is output. Through this decision-making mechanism based on probability distribution and dynamic thresholds, this method can effectively adapt to the differences in confidence distribution among different types of defects, reducing classification errors and improving the accuracy and reliability of decisions in boundary scenarios where the probabilities of multiple defect categories are close to the threshold.

[0077] The above method first extracts geometric and texture features of predefined defect types and generates a set of structured descriptors. Then, it uses a set of defect-free images and the descriptor set to generate a synthetic defect image set through a conditional generator, effectively solving the problem of insufficient model training due to scarce samples in traditional methods. Secondly, a two-stream feature extraction network is used to process the synthetic defect image set to obtain a fused feature vector. Based on the fused feature vector and the set of structured descriptors, a dynamic classifier is used to calculate classification, generating a set of defect category thresholds, enhancing the model's ability to extract defect features and improve classification accuracy. Further, real-time test images and their matching defect-free reference images are acquired and input into the two-stream feature extraction network to obtain test feature vectors. The test feature vectors, the set of structured descriptors, and the dynamic classifier are combined to calculate the defect probability distribution. Finally, based on the probability distribution and the threshold set, the defect category decision result is output, improving the real-time performance and reliability of defect detection. Overall, this method improves the accuracy, robustness, and adaptability of defect detection, enhances the model's detection performance in scenarios with scarce samples, and improves its ability to quickly respond to new defect types, providing a more efficient and accurate solution for defect detection in industrial production.

[0078] In one embodiment, geometric and texture features of predefined defect types are extracted; a set of structured descriptors is generated based on the geometric and texture features; and a synthetic defect image set is generated using a condition generator based on the defect-free image set and the set of structured descriptors, including:

[0079] Extract the geometric features of each type of defect from the predefined defect types, and construct a geometric feature vector based on the geometric features; the geometric features include aspect ratio and curvature;

[0080] Extract texture features for each type of defect from the predefined defect types, and construct a texture feature vector based on the texture features; the texture features include texture entropy values;

[0081] Geometric feature vectors and texture feature vectors are concatenated to generate structured descriptors, and a set of structured descriptors is formed. Defect-free images are selected from a set of defect-free images as input images for the conditional generator. The conditional generator is built based on the U-Net architecture.

[0082] A multilayer perceptron is constructed, and the structured descriptor is input into the multilayer perceptron for encoding to obtain the encoded descriptor. Based on the geometric feature range of the predefined defect type, a random defect region mask is generated by random sampling. The size and shape range of the random defect region mask are matched with the aspect ratio and curvature in the geometric features, respectively.

[0083] The input image, encoded descriptor, and random defect region mask are input into the condition generator to perform defect synthesis processing on the input image, resulting in a synthesized defect image. The synthesized defect image, the corresponding structured descriptor, and the corresponding random defect region mask are associated to construct multiple triplet data, and the synthesized defect image set is formed by combining the triplet data.

[0084] Specifically, sample images of manually labeled defect areas can be obtained first, and the Canny edge detection algorithm can be used to extract the contour curve of the defect area. This algorithm uses a dual-threshold method to determine edge pixels: a high threshold T1 and a low threshold T2 are set, pixels with a gray-level gradient greater than T1 are marked as strong edges, and pixels with a gray-level gradient between T1 and T2 and connected to strong edges are marked as weak edges. Finally, strong and weak edges are merged to form a complete defect contour. For the extracted contour curve, the minimum bounding rectangle algorithm can be used to determine its minimum bounding rectangle. The width and height of this rectangle are measured, and the aspect ratio is calculated. For curvature calculation, a B-spline curve can be used to fit the defect contour, obtaining N uniformly distributed sampling points on the contour. For each sampling point (x, y), the first and second derivatives of that point are calculated using numerical differentiation, and then the curvature of that point is calculated using the curvature formula. Finally, the average curvature of the N sampling points can be taken as the curvature feature value of this type of defect. By using aspect ratio and mean curvature as geometric feature components and arranging them in a fixed order of aspect ratio-curvature, a geometric feature vector with dimension 2 can be constructed.

[0085] Specifically, texture entropy can be calculated using the gray-level co-occurrence matrix (GLCM). For example, the defect sample image is first quantized to gray levels, mapping the original image's 256 gray levels to 8 gray levels, reducing computational complexity. A 5×5 sliding window is selected within the defect annotation area. Using the center pixel of the window as a reference, the number of co-occurrences of pixels with gray value i and gray value j at a distance d (d = 1 pixel) in the 0° direction is counted, constructing an 8×8 GLCM, where the matrix element P(i,j) represents the number of co-occurrences of gray values ​​i and j at a given direction and distance. The GLCM is then normalized to obtain the probability matrix. The texture entropy value H can then be calculated using the entropy formula:

[0086]

[0087] in The elements of the normalized gray-level co-occurrence matrix represent the probability of gray-level values ​​i and j co-occurring. A larger calculated entropy value H indicates a more complex texture in the defective region. Illustratively, the texture entropy value H can be used as a texture feature component to construct a texture feature vector with dimension 1.

[0088] Subsequently, when concatenating the geometric feature vector and the texture feature vector, the geometric feature vector (dimension 2) and the texture feature vector (dimension 1) can be concatenated in the order of "geometric feature vector first, texture feature vector second" to form an initial feature vector of dimension 3. Furthermore, to eliminate the dimensional differences between different feature components, the initial feature vector can be normalized using a min-max method, mapping the values ​​of each component to the [0,1] interval. The normalized feature vector is the structured descriptor for a single type of defect. Following the predefined defect type numbering order, the structured descriptors for all categories can be organized into a structured descriptor set. Further, the selection of the defect-free image set must be consistent with the acquisition scene of the defect sample images, ensuring that parameters such as shooting angle, light intensity, and resolution are the same. During the selection process, the defect-free images need to undergo quality screening to remove images with noise, blur, or shooting deviations. Subsequently, images can be randomly selected from the filtered set of defect-free images as input images for the conditional generator. Before input, the image size must be adjusted to match the defective sample images, and normalization processing (mapping pixel values ​​to the [0,1] interval) must be performed. The conditional generator is built on the U-Net architecture, which includes an encoder and a decoder. The encoder consists of four convolutional blocks, each containing two 3×3 convolutional layers, one batch normalization layer, and one ReLU activation function layer. Adjacent convolutional blocks are downsampled through 2×2 max pooling layers. The decoder is symmetrical to the encoder and consists of four deconvolutional blocks, each containing one 2×2 deconvolutional layer, two 3×3 convolutional layers, one batch normalization layer, and one ReLU activation function layer. Corresponding layers in the encoder and decoder achieve feature fusion through skip connections to preserve image detail information. Furthermore, the generator's output layer uses the Sigmoid activation function, which ensures that the output image pixel values ​​are in the [0,1] interval.

[0089] Specifically, a multilayer perceptron can employ a 3-layer fully connected structure. The input layer has 3 neurons (consistent with the dimension of the structured descriptor), the first hidden layer has 64 neurons, the second hidden layer has 128 neurons, and the output layer has 256 neurons (matching the number of channels in a feature map at the conditional generator's decoding end). Each hidden layer uses the ReLU activation function, and the output layer uses a linear activation function. During encoding, the structured descriptor is first input to the input layer, then processed by the first hidden layer... ,in This is the weight matrix from the input layer to the first hidden layer. For structured descriptors, This is the bias vector of the first hidden layer. This is the output of the first hidden layer. It is then calculated through the second hidden layer. ,in This is the weight matrix from the first hidden layer to the second hidden layer. This is the bias vector for the second hidden layer. This is the output of the second hidden layer. Finally, it is calculated by the output layer. ,in This is the weight matrix from the second hidden layer to the output layer. This is the bias vector for the output layer. This refers to the encoded descriptor.

[0090] Furthermore, in the process of generating the random defect region mask, the geometric feature range of a predefined defect type can be used as a constraint. First, the aspect ratio range [Rmin, Rmax] and curvature range [kmin, kmax] of the defect samples are statistically analyzed. Then, the aspect ratio r of the mask, r ∈ [Rmin, Rmax], is generated through uniform random sampling, and the width w and height h of the mask are generated, satisfying r = w / h. Illustratively, the mean curvature k of the mask contour, k ∈ [kmin, kmax], can also be generated through uniform random sampling. A closed contour conforming to the mean curvature k is generated based on a parametric curve (such as a B-spline curve). By setting the pixel values ​​inside the contour to 1 and the pixel values ​​outside the contour to 0, a binarized random defect region mask can be obtained, with the mask resolution consistent with the input image.

[0091] Specifically, the input image, encoded descriptor, and random defect region mask can be input into the conditional generator. During input, the input image is downsampled at the encoder to obtain multi-scale feature maps. The encoded descriptor is then converted into modulation parameters matching the number of channels in each scale feature map through a feature modulation module, adaptively modulating each scale feature map to carry the texture feature information of the defects. The random defect region mask, after being processed by a convolutional layer, can be concatenated with the deepest feature map output from the encoder and used as input to the decoder. The spatial constraints of the mask guide the generator to generate defects in specified regions. The decoder can achieve upsampling through deconvolution, combining the detailed features from the encoder passed through skip connections to gradually restore the image resolution, ultimately outputting a synthesized defect image with the same size as the input image. Furthermore, during the synthesis process, a loss function combining adversarial loss and L1 loss can be used to optimize the conditional generator. The adversarial loss ensures the realism of the synthesized image, while the L1 loss ensures the consistency between the synthesized defects and the structured descriptor. After synthesis, a single synthesized defect image, the structured descriptor used to generate that image, and a random defect region mask are associated to construct a triplet dataset of "synthetic defect image - structured descriptor - random defect region mask". Repeating the above process generates a preset number of triplet datasets, which are then aggregated to obtain a set of synthesized defect images for subsequent network training.

[0092] In one embodiment, the dual-stream feature extraction network includes a defect feature stream, a reference feature stream, a fully connected layer, and a fusion layer; the defect feature stream adopts a depthwise separable convolutional network architecture, and the reference feature stream adopts a preset lightweight Transformer network architecture.

[0093] Specifically, a two-stream feature extraction network is invoked, and the synthesized defect image set is input into the two-stream feature extraction network to obtain a fused feature vector, including:

[0094] Synthetic defect images are selected from the set of synthetic defect images. The synthetic defect images are input into the defect feature stream. The feature stream is used to extract features from the synthetic defect images to obtain defect feature maps. The defect feature maps are two-dimensional spatial features with a preset channel dimension.

[0095] A standard defect-free reference image paired with the synthetic defect image is obtained. The standard defect-free reference image is input into a reference feature stream. Feature extraction processing is performed on the standard defect-free reference image through the reference feature stream to obtain a global reference feature vector. The global reference feature vector is a one-dimensional global feature.

[0096] The global reference feature vector is input into the fully connected layer of the dual-stream feature extraction network for dimension adaptation processing to obtain the feature map statistical parameters. The feature map statistical parameters include mean parameters and variance parameters, and the dimensions of the mean parameters and variance parameters are matched with the preset channel dimensions of the defect feature map.

[0097] Based on the feature map statistical parameters, the defect feature map is adaptively normalized using a preset formula to obtain a spatially aligned feature map. The spatially aligned feature map and the global reference feature vector are associated through the feature map statistical parameters to satisfy the spatial alignment relationship.

[0098] Through the fusion layer, the spatially aligned feature map is weighted based on the channel attention mechanism to obtain the fused feature map. The fused feature map is then subjected to global average pooling to obtain the fused feature vector.

[0099] Specifically, the defect feature stream employs a depthwise separable convolutional network architecture, which consists of multiple depthwise separable convolutional blocks connected in series. Each convolutional block contains a depthwise convolutional layer, a pointwise convolutional layer, a batch normalization layer, and a ReLU activation function layer. The depthwise convolutional layers use single-channel convolutional kernels to traverse each channel of the input feature, extracting spatial features within each channel. The pointwise convolutional layers use 1×1 convolutional kernels to achieve feature fusion between channels. After the synthesized defect image is input, it can be processed sequentially through each convolutional block, gradually reducing the spatial size of the feature map and increasing the channel dimension. The final output is a defect feature map with dimensions H×W×C, where H is the feature map height, W is the feature map width, and C is the preset channel dimension. This feature map preserves the local texture and morphological information of the defect region in the synthesized defect image in a two-dimensional spatial form.

[0100] Specifically, the reference feature stream employs a lightweight Transformer network architecture, which includes multiple Transformer encoding layers. Each encoding layer consists of a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The multi-head self-attention sub-layer calculates the global association weights of features through multiple attention heads, enabling global information modeling of the defect-free reference image. The feedforward neural network sub-layer performs non-linear transformations on the attention output features. After preprocessing (sizing and normalization), the standard defect-free reference image can be converted into a feature sequence. After being input into the reference feature stream, each encoding layer sequentially performs global association and transformation on the feature sequence, ultimately outputting a one-dimensional global reference feature vector of dimension D. This vector integrates the background texture and global structural information of the defect-free reference image.

[0101] Subsequently, the global reference feature vector can be input into the fully connected layer of the dual-stream feature extraction network for dimension adaptation. The input dimension of the fully connected layer is the dimension D of the global reference feature vector, and the output dimension is the preset channel dimension C of the defect feature map. A linear transformation maps the one-dimensional global reference feature vector to a mean parameter of dimension C. With variance parameter The mean parameter The target mean and variance parameters for each channel of the corresponding defect feature map. The target variance of each channel in the corresponding defect feature map has the same dimension as the number of channels in the defect feature map, which can provide a suitable statistical basis for subsequent feature normalization.

[0102] Specifically, based on the statistical parameters of the feature map, adaptive instance normalization can be performed on the defect feature map using a preset formula to obtain a spatially aligned feature map. The preset formula is the calculation formula for adaptive instance normalization, specifically:

[0103]

[0104] in, For adaptive instance normalization operators, For spatial alignment feature maps, This is a defect feature map. The channel-level mean of the defect feature map in the spatial dimension is calculated as follows:

[0105]

[0106] In the formula, The height of the defect feature map. The width of the defect feature map. This represents the channel-level mean of the defect feature map in the spatial dimension. The channel-level standard deviation of the defect feature map in the spatial dimension is calculated as follows:

[0107]

[0108] In the formula, This is a preset minimum value used to avoid numerical instability caused by a denominator of 0. This represents the pixel values ​​of each channel in the i-th row and j-th column of the feature map. The mean parameter of the global reference feature vector after dimensionality adaptation by the fully connected layer, and Matching the number of channels, The variance parameter is the global reference feature vector after dimensionality adaptation by a fully connected layer.

[0109] As an illustration, during implementation, the defect feature map can be calculated first. Channel-level mean with standard deviation Then combine it with the mean parameter variance parameter Substituting into the above formula, the distribution of the defect feature map is adjusted to match the global reference feature vector, thereby obtaining a spatially aligned feature map and achieving spatial alignment between the local features of the defect and the reference global features.

[0110] Specifically, the fusion layer can incorporate a channel attention mechanism module. This module first performs global average pooling on the spatially aligned feature map to obtain a channel feature vector of dimension C. This vector is then input into a transformation network consisting of two fully connected layers (the first fully connected layer reduces the dimension, and the second fully connected layer restores it to dimension C). After passing through the sigmoid activation function, it outputs a channel weight vector of dimension C. Each element of this vector corresponds to the importance weight of a channel in the spatially aligned feature map. Multiplying the channel weight vector with the spatially aligned feature map by channel achieves weighted enhancement of each channel of the feature map, resulting in a fused feature map. Finally, global average pooling is performed on the fused feature map to calculate the mean of each channel, yielding a fused feature vector of dimension C. This vector integrates key information from local defect features and reference global features, providing a feature foundation for subsequent classification calculations.

[0111] In one embodiment, such as Figure 2 As shown, based on the fused feature vector and structured descriptor set, a dynamic classifier is used to perform classification calculations, generating a set of defect category thresholds, including:

[0112] S201: Divide the synthetic defect image set into a training set and a validation set; pre-train the dynamic classifier using the training set to obtain the pre-trained dynamic classifier; wherein the dynamic classifier is a classification network containing fully connected layers.

[0113] S202: Concatenate the fused feature vector with the structured descriptors in the set of structured descriptors to obtain a concatenated feature vector; input the concatenated feature vector into the pre-trained dynamic classifier for classification calculation to obtain the classification result;

[0114] S203: Based on the classification results and validation set, calculate the precision and recall of each type of defect in the predefined defect types, and statistically analyze the confidence distribution of each type of defect;

[0115] S204: Based on the principle of maximizing the harmonic mean of precision and recall, a threshold is calculated for the confidence distribution of each type of defect to obtain the optimal decision threshold for each type of defect; a defect category threshold set is composed of multiple optimal decision thresholds.

[0116] Specifically, the synthetic defect image set can be randomly divided into training and validation sets according to a preset ratio. The training set can then be used to enable the classifier to learn defect features, while the validation set is used to evaluate the classifier's performance and avoid overfitting. The dynamic classifier's network structure uses fully connected layers as its core component. Its input layer dimension is consistent with the dimension of the subsequently concatenated feature vectors. The hidden layers contain at least one fully connected layer, and non-linear activation functions such as ReLU can be used to achieve non-linear transformation of features. The output layer dimension is consistent with the number of predefined defect types, and the Softmax activation function can be used to output the probability distribution of each category. During pre-training, the concatenated feature vectors corresponding to the samples in the training set are input into the dynamic classifier. The difference between the probability distribution obtained from the output layer and the true defect category label of the sample (represented in one-hot encoding form) can be quantified using the cross-entropy loss function. The loss function is calculated as the sum of the preset category probability and the negative log-likelihood of the true label. Then, using a preset optimizer such as the Adam optimizer, the classifier's fully connected layer weights and bias parameters are updated through backpropagation based on the loss value. This process is repeated until the loss value converges or the preset number of iterations is reached, resulting in the pre-trained dynamic classifier.

[0117] Subsequently, the fused feature vector can be concatenated with the structured descriptors in the structured descriptor set, integrating the low-dimensional feature information of the fused feature vector with the prior defect feature information of the structured descriptors to obtain the concatenated feature vector. For example, concatenating the fused feature vector with the corresponding structured descriptors in dimensional order yields a concatenated feature vector whose dimension is the sum of the dimensions of the fused feature vector and the structured descriptors. During classification calculation, the concatenated feature vector is first input to the input layer of the pre-trained dynamic classifier. After linear transformation and non-linear activation function processing in the hidden layer, it is passed to the output layer. The output layer uses the Softmax activation function to map the processed features to probability values ​​corresponding to each predefined defect type, forming a probability distribution. The classification result is the defect category identifier corresponding to the highest probability value in this probability distribution.

[0118] Furthermore, based on the classification results and validation set, precision and recall can be calculated for each defect type within the predefined defect categories. For a specific defect type, precision is the ratio of true positive samples to predicted positive samples. True positive samples are the number of correctly classified samples belonging to this defect type in the validation set, while predicted positive samples are the number of samples in the validation set that the classifier classifies as belonging to this defect type. Recall is the ratio of true positive samples to actual positive samples, where actual positive samples are the total number of samples belonging to this defect type in the validation set. Additionally, for the classification results corresponding to this defect type in the validation set, the probability values ​​(i.e., confidence levels) output by the classifier can be extracted. The distribution of these confidence levels within a preset range can be statistically analyzed to obtain the sample proportion corresponding to each confidence level interval, thereby reflecting the confidence fluctuation characteristics of the defect classification results.

[0119] Subsequently, based on the principle of maximizing the harmonic mean of precision and recall, a threshold can be calculated for the confidence distribution of each type of defect to obtain the optimal decision threshold for each type of defect. The harmonic mean of precision and recall is the F1 score, calculated as follows:

[0120]

[0121] in, This represents the precision rate of this type of defect, which is the ratio of the number of true positive samples to the number of predicted positive samples. The recall rate represents the percentage of true positive samples for that type of defect, which is the ratio of the number of actual positive samples to the number of true positive samples. This score combines the performance of precision and recall; a higher value indicates better classification performance. During threshold calculation, candidate thresholds within a preset range can be iterated. For each candidate threshold, samples in the validation set with a confidence level greater than or equal to that threshold are classified as defects of that type, and the corresponding precision and recall are recalculated to obtain the F1 score for that candidate threshold. The candidate threshold with the highest F1 score can then be selected as the optimal decision threshold for that type of defect. Finally, by organizing the optimal decision thresholds for all predefined defect types in categorical order, a defect category threshold set can be obtained, providing a basis for subsequent defect detection decisions adapted to various defect types.

[0122] In one embodiment, a real-time captured image to be tested and a defect-free reference image matched with the image to be tested are acquired. The image to be tested and the defect-free reference image are input into a two-stream feature extraction network for feature extraction to obtain a test feature vector. Based on the test feature vector, a set of structured descriptors, and a dynamic classifier, a defect probability distribution is calculated, including:

[0123] Acquire the real-time captured image to be tested, and select a defect-free image with the same model as the image to be tested as a defect-free reference image;

[0124] Calculate the first gray-level mean of the image under test and the second gray-level mean of the defect-free reference image; calculate the gray-level mean ratio based on the first and second gray-level mean; perform illumination correction processing on the image under test based on the gray-level mean ratio to obtain the corrected image under test.

[0125] The corrected image to be tested is input into the defect feature stream of the dual-stream feature extraction network, and feature extraction is performed through the defect feature stream to obtain the test feature map.

[0126] A defect-free reference image is input into the reference feature stream of a dual-stream feature extraction network, and feature extraction is performed through the reference feature stream to obtain a test reference feature vector.

[0127] Adaptive instance normalization is applied to the test feature map to obtain a test space aligned feature map; the test space aligned feature map is weighted based on the channel attention mechanism to obtain a fused test feature map; global average pooling is applied to the fused test feature map to obtain a test feature vector.

[0128] The test feature vector is concatenated with the structured descriptors in the structured descriptor set to obtain the test concatenated vector; the test concatenated vector is then input into the dynamic classifier, which performs probability calculations to obtain the defect probability distribution.

[0129] Specifically, the image to be tested can be acquired in real time using an industrial camera. During acquisition, the camera frame rate and resolution must match the production line inspection rhythm and target defect size requirements. The acquired data is temporarily stored in a cache module and preprocessed. Preprocessing operations include image denoising and size standardization. Denoising can be achieved using Gaussian filtering, while size standardization can be achieved by adjusting the image to a preset fixed size using an interpolation algorithm. In selecting the defect-free reference image, model consistency can be the core constraint. The product model identifier is extracted from the image to be tested using an image recognition algorithm, allowing for the selection of a set of defect-free images of the same model from a pre-set defect-free image database. Furthermore, to further ensure the consistency of feature extraction, the image with the smallest difference in grayscale mean between the defect-free image and the image to be tested can be selected as the final defect-free reference image, ensuring that the environmental parameters such as lighting and angle of the acquisition scene are as consistent as possible.

[0130] Specifically, the grayscale distribution of the image under test can be calibrated using a grayscale benchmark of a defect-free reference image, thereby eliminating the interference of lighting changes in the real-time acquisition scene on defect feature extraction. For example, the first grayscale mean is the average of the grayscale values ​​of all pixels in the image under test, calculated as follows:

[0131]

[0132] in, This represents the first grayscale mean. N represents the height of the image to be measured, and N represents the width of the image to be measured. Indicates the first element in the image to be tested. Line 1 The grayscale value of each column pixel. The second grayscale mean is the average of the grayscale values ​​of all pixels in the defect-free reference image, calculated as follows:

[0133]

[0134] in, This represents the second grayscale mean. Indicating the first defect-free reference image Line 1 Grayscale values ​​of the column pixels. Grayscale mean ratio. The calculation formula is This ratio is used to establish the illumination relationship between the image under test and the reference image. During illumination correction, the grayscale value of each pixel in the image under test can be multiplied by the grayscale mean ratio. The grayscale values ​​of the corrected pixels are obtained, and all the corrected pixels form the corrected test image, so that the overall grayscale level of the corrected test image is consistent with the defect-free reference image.

[0135] The corrected image to be tested can then be input into the defect feature stream of the dual-stream feature extraction network. Feature extraction is performed through the defect feature stream to obtain the test feature map. Similarly, the defect feature stream adopts a depthwise separable convolutional network architecture, consistent with the structure in the previous embodiment, consisting of multiple cascaded depthwise separable convolutional blocks. After the corrected image to be tested is input into the defect feature stream, initial feature extraction and channel expansion are achieved through the first convolutional block. Subsequent convolutional blocks reduce the spatial size of the feature map while increasing the channel dimension through downsampling, resulting in a final output dimension of [dimensionality missing]. The test feature map is defined as follows: H is the feature map height, W is the feature map width, and C is the number of channels. This feature map preserves the local spatial features and texture information of the defect area in the corrected test image.

[0136] Similarly, the reference feature stream employs a lightweight Transformer network architecture, consistent with the structure in the aforementioned embodiments. It comprises multiple stacked Transformer encoding layers, each consisting of a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The defect-free reference image, after size normalization, is converted into a feature sequence. This sequence is then input into the reference feature stream. The multi-head self-attention sub-layer calculates the global association weights at each position in the feature sequence using multiple parallel attention heads, enabling the modeling of global information of the defect-free image. The feedforward neural network sub-layer performs non-linear transformations on the attention output features, enhancing the expressive power of the features. After processing by all encoding layers, a one-dimensional test reference feature vector of dimension D is output. This vector integrates the global background and structural features of the defect-free reference image.

[0137] Specifically, based on the adaptive instance normalization, weighting, and global average pooling processing of the defect feature map in the aforementioned embodiments, the test feature map can also be processed using adaptive instance normalization and channel attention mechanism weighting according to the same technical principles to obtain a fused test feature map. Then, global average pooling is performed on the fused test feature map to obtain the test feature vector. In this embodiment, the adaptive instance normalization processing uses the test reference feature vector as a statistical benchmark. First, the test reference feature vector is input into the fully connected layer built into the dual-stream feature extraction network for dimensionality adaptation, outputting a mean parameter consistent with the number of channels C of the test feature map. and variance parameter Then calculate the channel-level mean of the test feature map. and standard deviation The distribution of the test feature map is adjusted to match the distribution of the test reference feature vector using a preset formula, resulting in a test space aligned feature map, thus aligning the distribution of local defect features with the global reference features. Channel attention weighting is implemented through the attention module built into the fusion layer. First, global average pooling is performed on the test space aligned feature map to obtain a channel feature vector of dimension C. This vector is then input into a transform network consisting of two fully connected layers. After passing through the sigmoid activation function, a channel weight vector of dimension C is output. Each element of the weight vector corresponds to a test space aligned feature. Figure 1 The importance coefficients of each channel are used. Multiplying the channel weight vector by the aligned feature map of the test space channel by channel yields the fused test feature map. During global average pooling, the mean of each channel in the fused test feature map is calculated, resulting in a test feature vector of dimension C. This vector integrates key information from the aligned local features and the global reference features.

[0138] Specifically, when concatenating the test feature vector with the structured descriptors in the structured descriptor set, the test feature vector and the structured descriptors corresponding to each predefined defect type in the structured descriptor set can be concatenated in dimensional order to obtain the test concatenated vector. The dimension of this concatenated vector is the sum of the dimensions of the test feature vector and the structured descriptors. This vector simultaneously contains real-time feature information and prior defect feature information. Furthermore, the dynamic classifier is a classification network containing fully connected layers. After the test concatenated vector is input, it undergoes linear transformation and nonlinear activation processing in the hidden layers before being passed to the output layer. The output layer uses the Softmax activation function to map the processed features to probability values ​​corresponding to each predefined defect type. All probability values ​​are arranged in the order of the predefined defect types, forming a defect probability distribution. This distribution intuitively reflects the probability that the image under test belongs to each type of defect.

[0139] In one embodiment, based on the defect probability distribution and the defect category threshold set, the defect category decision result is output, including:

[0140] Analyze the defect probability distribution and extract the highest probability value and the corresponding defect category index from the defect probability distribution;

[0141] The defect probability distribution is analyzed, and the highest probability value and its corresponding first defect category index are extracted from the defect probability distribution. The second high probability value and its corresponding second defect category index are also extracted, and the difference between the highest probability value and the highest probability value is less than a preset probability difference threshold.

[0142] The first target decision threshold is obtained from the defect category threshold set based on the first defect category index; if a second defect category index exists, the second target decision threshold is obtained from the defect category threshold set based on the second defect category index.

[0143] Compare the highest probability value with the first target decision threshold. If a second target decision threshold exists, compare the second highest probability value with the corresponding second target decision threshold to obtain the comparison result.

[0144] If the comparison result is that the highest probability value is greater than or equal to the first target decision threshold, and the second highest probability value is less than the corresponding second target decision threshold, the first defect category identifier corresponding to the first defect category index is output, and the defect category decision result is obtained by combining the first defect category identifier and the highest probability value.

[0145] If the comparison result is that the highest probability value is greater than or equal to the first target decision threshold, and there is at least one second high probability value greater than or equal to the corresponding second target decision threshold, calculate the feature similarity between the structured descriptor and the test feature vector corresponding to the first defect category index and the second defect category index, respectively. Take the defect category identifier corresponding to the defect category index with the highest feature similarity as the final defect category identifier. Combine the final defect category identifier and the corresponding probability value to obtain the defect category decision result.

[0146] If the comparison result shows that the highest probability value is less than the first target decision threshold, output a defect-free label and use the defect-free label as the defect category decision result.

[0147] Specifically, the defect probability distribution is a probability sequence corresponding to each predefined defect type output by the dynamic classifier, with its dimension matching the number of predefined defect types. During analysis, this probability sequence is first traversed, and the highest probability value is selected as the highest probability value. The sequence index corresponding to this probability value is the first defect category index, and there is a one-to-one correspondence between the index and the number of the predefined defect type. Then, the highest probability value and its corresponding index are removed, and the second highest probability value is selected from the remaining probability values. The difference between the second highest probability value and the highest probability value is calculated. A preset probability difference threshold is a pre-defined critical value for determining whether the probabilities are "close," used to define whether the second highest probability value is qualified to compete with the highest probability value. If the difference is less than the preset probability difference threshold, the second highest probability value and its corresponding second defect category index are extracted. If the difference is greater than or equal to the preset probability difference threshold, it is determined that there is no second highest probability value or second defect category index that meets the criteria.

[0148] Specifically, the defect category threshold set is a sequence of optimal decision thresholds for each defect type, pre-optimized using a validation set. Its index order is identical to the predefined defect type numbers. When obtaining the target decision threshold, the first defect category index can be used as the set index to directly locate and extract the corresponding threshold as the first target decision threshold. This threshold is the optimal judgment criterion for adapting to the feature distribution of the first defect category. If a second defect category index exists, the threshold corresponding to that index is extracted from the set in the same way as the second target decision threshold, providing an adaptation criterion for determining the second highest probability value. Then, the highest probability value is compared with the first target decision threshold. If a second target decision threshold exists, the second highest probability value is also compared with its corresponding second target decision threshold to obtain the comparison result. The comparison process uses a direct numerical comparison, with the core objective being to determine whether each high probability value reaches the optimal decision threshold for its corresponding category. The comparison result includes three typical cases: first, the highest probability value reaches the first target decision threshold, but the second highest probability value does not; second, the highest probability value reaches the first target decision threshold, and the second highest probability value also reaches the second target decision threshold; third, the highest probability value does not reach the first target decision threshold. The comparison results directly determine the subsequent decision branches, providing a basis for accurate output.

[0149] Specifically, if the comparison result shows that the highest probability value is greater than or equal to the first target decision threshold, and the second highest probability value is less than the corresponding second target decision threshold, then only the first defect category meets the determination requirement of the optimal decision threshold, and there are no other competitive defect categories. During output, the predefined defect category identifier mapping table can be queried based on the first defect category index to obtain the first defect category identifier containing the defect type name and code. This identifier is then associated with the highest probability value to form a combined result of "defect category identifier - confidence level," which serves as the final defect category decision result. The highest probability value is used to characterize the confidence level of this decision result.

[0150] Specifically, if the comparison result shows that the highest probability value is greater than or equal to the first target decision threshold, and there exists at least one second high probability value greater than or equal to the corresponding second target decision threshold, then this scenario is a boundary scenario with similar probabilities, and the competing categories can be distinguished through secondary verification using feature similarity. For example, cosine similarity calculation can be used to calculate the feature similarity between the structured descriptors corresponding to the first defect category index and the second defect category index and the test feature vector, respectively, using the following formula:

[0151]

[0152] in, The value represents the feature similarity, ranging from [-1, 1]. The closer the value is to 1, the higher the similarity between the two. A represents the structured descriptor corresponding to the defect category index, which is a pre-constructed defect prior feature vector. B represents the test feature vector, which is the real-time extracted feature vector of the image to be tested.

[0153] During calculation, the structured descriptors corresponding to the first and second defect category indices and the same test feature vector can be substituted into the formula to obtain two similarity values. Then, the defect category index corresponding to the larger similarity value is selected, and its corresponding category identifier is the final defect category identifier. By associating the probability value (the highest probability value or the second highest probability value) corresponding to this identifier, the defect category decision result can be formed.

[0154] Indicatively, if the comparison result shows that the highest probability value is less than the first target decision threshold, in this scenario, the probability values ​​of all predefined defect types have not reached the corresponding optimal decision threshold, indicating that no defects conforming to the predefined standards have been detected in the image under test. In this case, a defect-free label can be output, and the defect-free label can be used as the defect category decision result. The defect-free label is a preset standardized label containing the word "qualified" and a corresponding code, indicating that the image under test is a qualified product.

[0155] Based on the same inventive concept, this application also provides a computer vision processing system for real-time industrial defect detection, used to implement the aforementioned computer vision processing method for real-time industrial defect detection. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the computer vision processing system for real-time industrial defect detection provided below can be found in the limitations of the computer vision processing method for real-time industrial defect detection described above, and will not be repeated here.

[0156] In one exemplary embodiment, such as Figure 3 As shown, a computer vision processing system 300 for real-time industrial defect detection is provided, comprising:

[0157] The feature extraction and synthesis module 301 is used to extract geometric and texture features of predefined defect types, generate a set of structured descriptors based on the geometric and texture features, and generate a set of synthesized defect images based on the set of defect-free images and the set of structured descriptors through a condition generator.

[0158] The feature fusion and threshold generation module 302 is used to call the dual-stream feature extraction network, input the synthesized defect image set into the dual-stream feature extraction network to obtain the fused feature vector; based on the fused feature vector and the set of structured descriptors, a dynamic classifier is used to perform classification calculations to generate a set of defect category thresholds;

[0159] The real-time image feature extraction module 303 is used to acquire the real-time captured image to be tested and the defect-free reference image matched with the image to be tested, and input the image to be tested and the defect-free reference image into the dual-stream feature extraction network for feature extraction to obtain the test feature vector.

[0160] The defect probability calculation and decision module 304 is used to calculate the defect probability distribution based on the test feature vector, the set of structured descriptors and the dynamic classifier; and output the defect category decision result based on the defect probability distribution and the defect category threshold set.

[0161] In one exemplary embodiment, the present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a computer vision processing method for real-time detection of industrial defects. A multi-core processor is preferred to improve the system's parallel processing capability. The memory provides sufficient temporary storage space to support program execution and data processing. The memory capacity should be large enough to accommodate large amounts of data and computational tasks.

[0162] In one exemplary embodiment, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a computer vision processing method for real-time detection of industrial defects according to the present application. The computer-readable storage medium may include: a read-only memory, a random access memory (RAM), a solid-state drive (SSD), or an optical disc, etc.

[0163] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A computer vision processing method for real-time detection of industrial defects, characterized in that, The method includes: Extract geometric and texture features of predefined defect types, and generate a set of structured descriptors based on the geometric and texture features; generate a set of synthetic defect images based on a set of defect-free images and the set of structured descriptors using a condition generator. The synthetic defect image set is input into the dual-stream feature extraction network to obtain a fused feature vector; based on the fused feature vector and the structured descriptor set, the dynamic classifier is used to perform classification calculations to generate a defect category threshold set. The test image and the defect-free reference image matched with the test image are acquired in real time. The test image and the defect-free reference image are input into the dual-stream feature extraction network for feature extraction to obtain the test feature vector. Based on the test feature vector, the set of structured descriptors, and the dynamic classifier, the defect probability distribution is calculated; based on the defect probability distribution and the set of defect category thresholds, the defect category decision result is output.

2. The method according to claim 1, characterized in that, The process involves extracting geometric and texture features of predefined defect types, generating a set of structured descriptors based on these features, and then generating a synthetic defect image set using a condition generator based on a set of defect-free images and the set of structured descriptors. Extract the geometric features of each type of defect from the predefined defect types, and construct a geometric feature vector based on the geometric features; the geometric features include aspect ratio and curvature; Extract the texture features of each type of defect from the predefined defect types, and construct a texture feature vector based on the texture features; the texture features include texture entropy values; The geometric feature vector and the texture feature vector are concatenated to generate a structured descriptor, and the structured descriptor set is composed of multiple structured descriptors. Defect-free images are selected from the set of defect-free images as input images for the condition generator; wherein, the condition generator is constructed based on the U-Net architecture; A multilayer perceptron is constructed, and the structured descriptor is input into the multilayer perceptron for encoding processing to obtain the encoded descriptor; Based on the geometric feature range of the predefined defect type, a random defect region mask is generated by random sampling; the size and shape range of the random defect region mask are matched with the aspect ratio and curvature of the geometric feature, respectively. The input image, the encoded descriptor, and the random defect region mask are input into the condition generator, and the input image is subjected to defect synthesis processing to obtain a synthesized defect image. The synthetic defect image, the corresponding structured descriptor, and the corresponding random defect region mask are associated to construct multiple triplet data sets, and the synthetic defect image set is formed by combining the triplet data sets.

3. The method according to claim 1, characterized in that, The dual-stream feature extraction network includes a defect feature stream, a reference feature stream, a fully connected layer, and a fusion layer; the defect feature stream adopts a depthwise separable convolutional network architecture, and the reference feature stream adopts a preset lightweight Transformer network architecture; The step of calling the dual-stream feature extraction network, inputting the synthetic defect image set into the dual-stream feature extraction network, and obtaining the fused feature vector includes: A synthetic defect image is selected from the set of synthetic defect images, and the synthetic defect image is input into the defect feature stream. The feature stream is used to perform feature extraction processing on the synthetic defect image to obtain a defect feature map. The defect feature map is a two-dimensional spatial feature with a preset channel dimension. A standard defect-free reference image paired with the synthetic defect image is obtained. The standard defect-free reference image is input into the reference feature stream. Feature extraction processing is performed on the standard defect-free reference image through the reference feature stream to obtain a global reference feature vector. The global reference feature vector is a one-dimensional global feature. The global reference feature vector is input into the fully connected layer of the dual-stream feature extraction network for dimension adaptation processing to obtain feature map statistical parameters; the feature map statistical parameters include mean parameters and variance parameters, and the dimensions of the mean parameters and variance parameters are matched with the preset channel dimensions of the defect feature map; Based on the statistical parameters of the feature map, the defect feature map is subjected to adaptive instance normalization processing through a preset formula to obtain a spatially aligned feature map; the spatially aligned feature map and the global reference feature vector are associated through the statistical parameters of the feature map to satisfy the spatial alignment relationship. Through the fusion layer, the spatially aligned feature map is weighted based on the channel attention mechanism to obtain a fused feature map. The fused feature map is then subjected to global average pooling to obtain the fused feature vector.

4. The method according to claim 1, characterized in that, The step of generating a defect category threshold set by performing classification calculations using the dynamic classifier based on the fused feature vector and the structured descriptor set includes: The synthetic defect image set is divided into a training set and a validation set; the dynamic classifier is pre-trained using the training set to obtain a pre-trained dynamic classifier; wherein the dynamic classifier is a classification network containing fully connected layers. The fused feature vector is concatenated with the structured descriptors in the set of structured descriptors to obtain the concatenated feature vector; The concatenated feature vector is input into the pre-trained dynamic classifier for classification calculation to obtain the classification result. Based on the classification results and the validation set, calculate the precision and recall of each type of defect in the predefined defect types, and statistically analyze the confidence distribution of each type of defect; Based on the principle of maximizing the harmonic mean of precision and recall, a threshold is calculated for the confidence distribution of each type of defect to obtain the optimal decision threshold for each type of defect. The defect category threshold set consists of multiple optimal decision thresholds.

5. The method according to claim 1, characterized in that, The process involves acquiring a real-time captured image to be tested and a defect-free reference image matching the image to be tested; inputting the image to be tested and the defect-free reference image into the dual-stream feature extraction network for feature extraction to obtain a test feature vector; and calculating the defect probability distribution based on the test feature vector, the structured descriptor set, and the dynamic classifier, including: The image to be tested is captured in real time, and a defect-free image with the same model as the image to be tested is selected as the defect-free reference image; Calculate the first grayscale mean of the image under test and the second grayscale mean of the defect-free reference image; calculate the grayscale mean ratio based on the first grayscale mean and the second grayscale mean; perform illumination correction processing on the image under test based on the grayscale mean ratio to obtain the corrected image under test; The corrected image to be tested is input into the defect feature stream of the dual-stream feature extraction network, and feature extraction is performed through the defect feature stream to obtain the test feature map. The defect-free reference image is input into the reference feature stream of the dual-stream feature extraction network, and feature extraction is performed through the reference feature stream to obtain the test reference feature vector. The test feature map is subjected to adaptive instance normalization to obtain a test space aligned feature map; the test space aligned feature map is weighted based on the channel attention mechanism to obtain a fused test feature map; the fused test feature map is subjected to global average pooling to obtain the test feature vector. The test feature vector is concatenated with the structured descriptors in the set of structured descriptors to obtain a test concatenated vector; the test concatenated vector is input into the dynamic classifier, and the probability is calculated by the dynamic classifier to obtain the defect probability distribution.

6. The method according to claim 1, characterized in that, The defect category decision result, based on the defect probability distribution and the defect category threshold set, is output, including: The defect probability distribution is analyzed, and the highest probability value and the corresponding defect category index are extracted from the defect probability distribution. The defect probability distribution is analyzed, and the highest probability value and its corresponding first defect category index are extracted from the defect probability distribution. The second high probability value and its corresponding second defect category index are also extracted, and the difference between the highest probability value and the highest probability value is less than a preset probability difference threshold. A first target decision threshold is obtained from the defect category threshold set based on the first defect category index; if a second defect category index exists, a second target decision threshold is obtained from the defect category threshold set based on the second defect category index. Compare the highest probability value with the first target decision threshold. If a second target decision threshold exists, simultaneously compare the second high probability value with the corresponding second target decision threshold to obtain a comparison result. If the comparison result is that the highest probability value is greater than or equal to the first target decision threshold, and the second high probability value is less than the corresponding second target decision threshold, the first defect category identifier corresponding to the first defect category index is output, and the defect category decision result is obtained by combining the first defect category identifier and the highest probability value. If the comparison result is that the highest probability value is greater than or equal to the first target decision threshold, and there is at least one second high probability value greater than or equal to the corresponding second target decision threshold, calculate the feature similarity between the structured descriptor and the test feature vector corresponding to the first defect category index and the second defect category index, respectively, take the defect category identifier corresponding to the defect category index with the highest feature similarity as the final defect category identifier, and combine the final defect category identifier and the corresponding probability value to obtain the defect category decision result; If the comparison result is that the highest probability value is less than the first target decision threshold, a defect-free identifier is output, and the defect-free identifier is used as the defect category decision result.

7. The method according to claim 3, characterized in that, The preset formula is: in, For adaptive instance normalization operators, This refers to the spatial alignment feature map. The defect feature map has the following dimensions: , The height of the defect feature map. The width of the defect feature map. The number of feature channels, The channel-level mean of the defect feature map in the spatial dimension. The standard deviation of the defect feature map in the spatial dimension is the channel-level standard deviation. The mean parameter of the global reference feature vector after dimensionality adaptation by the fully connected layer, and... Matching the number of channels, The variance parameter is the result of dimensionality adaptation of the global reference feature vector through the fully connected layer. This is a preset minimum value used to avoid numerical instability caused by a denominator of 0. This represents the pixel values ​​of each channel in the i-th row and j-th column of the feature map.

8. A computer vision processing system for real-time detection of industrial defects, characterized in that, The system includes: The feature extraction and synthesis module is used to extract geometric and texture features of predefined defect types, generate a set of structured descriptors based on the geometric and texture features, and generate a set of synthesized defect images based on a set of defect-free images and the set of structured descriptors through a condition generator. The feature fusion and threshold generation module is used to call the dual-stream feature extraction network, input the synthetic defect image set into the dual-stream feature extraction network to obtain the fused feature vector; based on the fused feature vector and the structured descriptor set, the dynamic classifier is used to perform classification calculation to generate a defect category threshold set; The real-time image feature extraction module is used to acquire the real-time captured image to be tested and the defect-free reference image matched with the image to be tested, and input the image to be tested and the defect-free reference image into the dual-stream feature extraction network for feature extraction to obtain the test feature vector. The defect probability calculation and decision module is used to calculate the defect probability distribution based on the test feature vector, the structured descriptor set, and the dynamic classifier; and to output the defect category decision result based on the defect probability distribution and the defect category threshold set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images