Defect detection method, device, equipment, storage medium and program product

Abstract

This application provides a defect detection method, apparatus, device, storage medium, and program product. The method includes: acquiring multiple image features from a captured image of a workpiece to be inspected; inputting the multiple image features into a first model to obtain feature classifications of the multiple image features; the feature classifications include normal features, abnormal features, and normal noise features; wherein, the first model is used to classify the multiple image features by weighted combination; and based on the image features classified as abnormal features, determining the abnormal pixel matrix in the captured image. Using this method, normal features, noise features, and abnormal features can be effectively distinguished, reducing dependence on labeled data, minimizing the impact of hardware errors, imaging position differences, and ambient light interference, improving the accuracy and robustness of surface defect detection in metal die castings, and meeting the quality control requirements of industrial production.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and industrial vision inspection technology, specifically to a defect detection method, device, equipment, storage medium, and program product. Background Technology

[0002] Detection of surface defects (such as scratches, dents, depressions, and protrusions) in die-cast metal parts is a crucial step in ensuring product quality. Currently, the mainstream detection methods are mainly divided into traditional manual inspection and automated inspection based on machine vision. Manual inspection relies on worker experience, is inefficient, highly subjective, and prone to fatigue, making it difficult to meet the needs of modern industrial production.

[0003] Existing technologies struggle to effectively overcome the challenges posed by hardware errors, differences in reflection at different locations, and ambient light interference during the imaging process of metal die-cast parts. Consequently, when faced with areas that differ from the training data, the model cannot robustly determine whether it is noise interference or a real defect, resulting in insufficient detection accuracy and stability.

[0004] Therefore, in view of the above situation, there is an urgent need to provide a method for detecting apparent defects in metal die castings based on feature decoupling, so as to overcome the shortcomings in current practical applications. Summary of the Invention

[0005] The purpose of this application is to provide a defect detection method, device, equipment, storage medium, and program product to solve the problems of high false alarm rate, poor robustness, and strong dependence on labeled data in the existing technology due to imaging interference.

[0006] In a first aspect, embodiments of this application provide a defect detection method, comprising:

[0007] Acquire multiple image features from the acquired image of the workpiece to be inspected;

[0008] The multiple image features are input into a first model to obtain feature classifications of the multiple image features; the feature classifications include normal features, abnormal features, and normal noise features; wherein, the first model is used to perform feature classification on the multiple image features by weighted combination of the multiple image features;

[0009] Based on the image features classified as abnormal features, the matrix of abnormal pixels in the acquired image is determined.

[0010] Optionally, the defect detection method further includes, before inputting the plurality of image features into the first model:

[0011] Acquire features from multiple sampled images after acquiring images of multiple sampled workpieces;

[0012] Multiple sampled image features are input into the first model, and the first model is trained according to the constraints corresponding to different feature classifications after the first model classifies the sampled image features, so as to obtain the first model for detecting abnormal pixels in the acquired image of the workpiece to be detected.

[0013] Optionally, in the defect detection method, the constraints include one or more of the following:

[0014] The first constraint is used to represent the similarity of the sampled image features classified as normal features to standard normal features;

[0015] The second constraint condition is used to represent the error between the reconstructed image and the acquired image corresponding to the sampled workpiece; wherein, the reconstructed image is obtained by superimposing the sampled image features classified as normal features and the sampled image features classified as normal noise features;

[0016] The third constraint is used to represent the classification error between the sampled image features classified as normal noise features and the sampled image features classified as abnormal features.

[0017] Optionally, in the defect detection method, determining the abnormal pixel matrix in the acquired image based on the image features classified as abnormal features includes:

[0018] Based on the image features classified as abnormal features, generate prompt points;

[0019] Based on the acquired image and the cue points in the acquired image, a second model is used to generate a foreground mask matrix of the acquired image; wherein, the second model is used to segment the acquired image and obtain the foreground mask of the acquired image based on the segmentation result;

[0020] The closed shape mask matrix corresponding to the effective area of ​​the acquired image is obtained by summing the randomly generated closed shape region based on the acquired image with the foreground mask matrix.

[0021] Generate a corresponding line mask matrix based on the edges of the closed shape region and the corresponding closed shape mask matrix;

[0022] The abnormal pixel matrix in the acquired image is determined based on the image pixel intensities corresponding to the closed shape mask matrix and the straight line mask matrix, respectively.

[0023] Optionally, in the defect detection method, training the first model based on the constraints corresponding to different feature classifications after the first model performs feature classification on the sampled image features includes:

[0024] The sum of constraint values ​​for different feature classifications is obtained after the image features are classified using the first model.

[0025] When the constraint value is less than or equal to a preset value, the first model is determined to be obtained;

[0026] When the sum of the constraint values ​​is greater than the preset value, the weight parameters corresponding to different classification features when the first model is used to classify the image features are adjusted, and the step of inputting multiple sampled image features into the first model is re-executed until the obtained constraint value is less than or equal to the preset value.

[0027] Optionally, the defect detection method further includes obtaining the first constraint condition in the following manner:

[0028] Calculate the cosine distance between the sampled image features that belong to normal features and the standard normal features to obtain the first constraint condition;

[0029] The standard normal features are obtained by clustering normal image samples in the normal sample data feature library and extracting the image features corresponding to the cluster centers.

[0030] Optionally, the defect detection method further includes obtaining the second constraint condition in the following manner:

[0031] After superimposing the sampled image features that belong to normal features with the sampled image features that belong to normal noise features to obtain the reconstructed image, the dimensions of the reconstructed image are adjusted so that the dimensional features of the reconstructed image after dimension adjustment are consistent with the dimensional features of the sampled image features.

[0032] The mean square error between the reconstructed image after dimensional adjustment and the acquired image corresponding to the sampled workpiece is calculated to obtain the second constraint condition.

[0033] Optionally, the defect detection method further includes obtaining the third constraint condition in the following manner:

[0034] The third constraint condition is obtained by weighting the label vectors corresponding to the normal noise features and the label vectors corresponding to the abnormal features using the weight coefficients corresponding to the abnormal samples.

[0035] Secondly, embodiments of this application provide a defect detection device, comprising:

[0036] The feature acquisition module is used to acquire multiple image features from the acquired image of the workpiece to be inspected;

[0037] The model processing module is used to input the multiple image features into a first model to obtain feature classifications of the multiple image features; the feature classifications include normal features, abnormal features, and normal noise features; wherein, the first model is used to perform feature classification on the multiple image features by weighted combination of the multiple image features;

[0038] The analysis module is used to determine the abnormal pixel matrix in the acquired image based on the image features that belong to abnormal features according to the feature classification.

[0039] Thirdly, embodiments of this application provide a detection device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the defect detection method as described in any of the preceding claims.

[0040] Fourthly, embodiments of this application provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the steps in the defect detection method as described in any of the preceding claims.

[0041] Fifthly, embodiments of this application provide a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the defect detection method as described in any of the preceding claims.

[0042] At least one of the above technical solutions in the specific embodiments of this application has the following beneficial effects:

[0043] By employing a feature decoupling method based on a multi-head attention mechanism, image features are decomposed and combined into three independent features: normal features, normal noise features, and abnormal features, thereby effectively distinguishing between normal image features, normal noise features, and abnormal features. A standard normal feature database is obtained by clustering normal image features to constrain normal features. The image reconstruction error by fusing normal features and noise features further constrains normal and normal noise features. A classification method is constructed to distinguish between normal noise features and abnormal features, which is then used to constrain normal noise features and abnormal features, improving the accuracy and robustness of feature recognition.

[0044] Defect data is constructed based on the surface defect characteristics and imaging patterns of metal die castings, which enhances the controllability of defect data, reduces the dependence on annotation data, and effectively reduces the impact of hardware errors, imaging position differences and ambient light interference on imaging results.

[0045] In summary, this application can solve the problems of inaccurate defect detection and poor robustness caused by hardware errors, imaging position differences and ambient light interference in the prior art, improve the accuracy and robustness of defect detection, and meet the needs of quality control of metal die castings in actual production. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the method described in the embodiments of this application.

[0048] Figure 2 This is a schematic diagram of the device described in the embodiments of this application.

[0049] Figure 3 The flowchart shows the construction process for normal features, normal noise features, and abnormal features.

[0050] Figure 4 The flowchart for constructing the reconstructed loss function.

[0051] Figure 5 Defect data generation process Figure 1 .

[0052] Figure 6 Defect data generation process Figure 2 .

[0053] Figure 7 This is an example diagram of a line mask.

[0054] Figure 8 This is an example diagram of an elliptical mask.

[0055] Figure 9 This is a diagram of a straight-line shaped defect.

[0056] Figure 10 This is a diagram of an elliptical defect.

[0057] Figure 11 This is a schematic diagram of the model training process.

[0058] Figure 12 This is a schematic diagram of the model reasoning process. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0061] This application separates the truly influential anomalous features from the overall image features, enhancing its robustness against interference and reducing false positives. Through a multi-head attention mechanism, image features are decomposed into multiple sub-features. Using different weighted combinations, these sub-features are fused into three main categories: normal features, normal noise features, and anomalous features. Anomalous features are the key to defect detection. This feature decoupling allows the model to clearly distinguish features of different natures, avoiding interference between features that could lead to false positives. Furthermore, to reduce reliance on labeled anomalous regions, a defect construction method is proposed based on the surface defect characteristics and imaging patterns of metal die-castings to generate anomalous data to assist model training. This improves the feasibility and generalization ability of model training while reducing data costs.

[0062] The present application will be further explained below with reference to specific implementation methods.

[0063] This application provides a defect detection method; please refer to [link / reference]. Figure 1 The method includes:

[0064] S1. Acquire multiple image features from the acquired image of the workpiece to be inspected;

[0065] S2. Input the multiple image features into the first model to obtain the feature classification of the multiple image features; the feature classification includes normal features, abnormal features, and normal noise features; wherein, the first model is used to perform feature classification on the multiple image features by weighted combination of the multiple image features;

[0066] S3. Based on the image features classified as abnormal features, determine the abnormal pixel matrix in the acquired image.

[0067] Optionally, the workpiece to be inspected is a metal die casting, and the defect detection method described in this application embodiment can be applied to the appearance inspection of metal die castings.

[0068] In this embodiment of the application, optionally, before inputting the plurality of image features into the first model, such as... Figure 12 As shown, the method further includes:

[0069] Acquire features from multiple sampled images after acquiring images of multiple sampled workpieces;

[0070] Multiple sampled image features are input into the first model, and the first model is trained according to the constraints corresponding to different feature classifications after the first model classifies the sampled image features, so as to obtain the first model for detecting abnormal pixels in the acquired image of the workpiece to be detected.

[0071] Specifically, in this implementation method, before performing defect detection on the workpiece to be inspected using the first model, the initial model of the first model can be trained using the collected images (original images) of multiple sampled workpieces. During the training process, the original images are fused with the abnormal pixel matrix (probabilistic abnormality generation mask) obtained in the previous training process of the first model. The features of multiple sampled images after image fusion are extracted, feature classification is performed, and the constraint conditions corresponding to different feature classifications are determined until the obtained constraint conditions reach the preset conditions.

[0072] Optionally, in this embodiment of the application, the constraint conditions include one or more of the following:

[0073] The first constraint, loss1, is used to represent the similarity between the sampled image features classified as normal features and standard normal features, i.e., normal feature constraint.

[0074] The second constraint, loss2, is used to represent the error between the reconstructed image and the acquired image corresponding to the sampled workpiece; wherein, the reconstructed image is obtained by superimposing the sampled image features classified as normal features and the sampled image features classified as normal noise features, i.e., normal noise feature constraint;

[0075] The third constraint, loss3, is used to represent the classification error between the sampled image features classified as normal noise features and the sampled image features classified as abnormal features.

[0076] In this embodiment, a high-precision industrial camera is used to acquire images of the metal die-casting (workpiece to be inspected). This application uses a 5-megapixel industrial camera for imaging. To reduce ambient light interference, a light source is added to assist imaging during the imaging process to ensure uniform illumination. This application uses a white bowl light source. To improve imaging consistency, a position calibration process is added before imaging to ensure that the imaging position of the die-casting remains basically consistent each time. This application uses the top-view imaging data of the die-casting to extract key corner points. The key corner point features are matched with the key corner point features of a pre-stored template image. The deviation angle is obtained using the matched point pairs. After rotating the die-casting counterclockwise by the deviation angle, imaging is performed at the preset imaging position to obtain the acquired image of the workpiece to be inspected.

[0077] To enhance the representation capability of basic image features, a DINOv2 vision pre-trained model is used to extract features from the obtained workpiece to be detected, obtaining multiple image features. In the technical solution of this application, after using DINOv2 for overall feature extraction, local region features are further obtained for subsequent subdivision feature extraction. Throughout the entire basic image feature extraction process, the weights of the DINOv2 model are fixed.

[0078] DINOv2 is a self-supervised vision foundation model released by Meta AI. It is trained on the Vision Transformer architecture and large-scale data and can be used to obtain general visual representations of images.

[0079] The basic image features obtained from DINOv2 are input into a multi-head attention module. Each head attention module can independently acquire information from different locations and feature dimensions of the image. For input image I, multiple subdivided features are obtained through multi-head attention. , ,..., Where n is the number of subdivision features, which is determined by the number of heads in the multi-head attention mechanism. In this application, n=8 is used for calculation.

[0080] The resulting subdivided features (multiple image features) contain data information of different dimensions. These image features are input into a first model, which performs different weighted combinations on the multiple image features to fuse them into different features. The fused features have different emphases due to the weights of each subdivided feature, thus achieving feature decoupling. This application obtains three fused features based on the subdivided features: normal features, normal noise features, and abnormal features. The weights are adjusted accordingly. , , Training with the subdivided features yields the three features mentioned above:

[0081] Normal characteristics: ;

[0082] Normal noise characteristics: ;

[0083] Abnormal characteristics: ;

[0084] By continuously adjusting the weight parameters during training, the three features mentioned above can accurately reflect the corresponding characteristics. The overall construction process is as follows: Figure 3 and Figure 12 As shown. Among them, , and The weights of normal features, normal noise features, and abnormal features are calculated for the i-th image feature.

[0085] The image input size used in this application is (N, 3, 518, 518), and the resulting basic image feature dimension is (N, 1369, 384), where N is the number of inputs; The input dimension is 384, and the output dimension is 512. The input dimension is 384, and the output dimension is 512. The input dimension is 384, and the output dimension is 256; the input dimension of the Linear corresponding to Q is 512, and the output dimension is 8*512; the input dimension of the Linear corresponding to K is 512, and the output dimension is 8*512; the input dimension of the Linear corresponding to V is 256, and the output dimension is 8*256; the Linear input dimension of the fused output features is 8*256, and the output dimension is 384. Finally, the dimensions of normal features, normal noise features, and abnormal features are all (N, 1369, 384).

[0086] To obtain the calculation parameters for normal features, a feature library was constructed by collecting a small amount of normal sample data in the early stage. This feature library was set as a standard normal feature library, in which the features in the standard normal feature library are standard normal features.

[0087] In one embodiment of this implementation, optionally, the method further includes obtaining the first constraint condition in the following manner:

[0088] Calculate the cosine distance between the sampled image features that belong to normal features and the standard normal features to obtain the first constraint condition;

[0089] The standard normal features are obtained by clustering normal image samples in the normal sample data feature library and extracting the image features corresponding to the cluster centers.

[0090] Specifically, for a small number of normal image samples collected, image features are calculated using DINOv2. Since a single image will yield a multi-dimensional block feature vector, for example, in the implementation process of this application, using 518*518 input image data will result in a 1369-dimensional block feature vector. Therefore, when calculating multiple normal image samples, a 1369*M (M is the number of images) block feature vector will be obtained, and the dimension of the block feature vector is 384, resulting in excessive memory consumption and increased computational complexity. To solve this problem, this application obtains cluster centers by clustering based on the block feature vector set obtained from all normal image data, and uses the obtained cluster centers as a standard normal feature library. This application uses the K-means clustering method, with the number of cluster centers set to 1024, ultimately obtaining a 1024*384-dimensional standard normal feature library.

[0091] For the acquired images of the die-cast parts, after basic image feature extraction and subdivision feature extraction and combination, normal features are obtained. These normal features should have a high degree of similarity to the aforementioned standard normal features. Minimizing the cosine distance is used to achieve this high similarity constraint. For the normal features... Based on the standard normal characteristics, the first constraint condition is calculated, and the constraint loss function is obtained as follows:

[0092] ;

[0093] Where S represents normal characteristics The dimensions of all block feature vectors, where K represents the dimensions of standard normal feature library block vectors. A single block feature vector representing normal features. For a single block feature vector in the standard normal feature library, This indicates that the cosine similarity between two vectors is calculated.

[0094] For normal noise features, there are two characteristics: First, for normal die-cast part image data, only normal features and normal noise features are included. Therefore, based on the fused features of normal noise features and normal features, the original normal die-cast part image data can be decoded. Second, although normal noise features differ from the standard normal feature library (due to feature differences caused by data changes), these features are still normal features and correspond to normal labels. In summary, for the constraints on normal noise features, the second constraint is applied by minimizing the image reconstruction error and classification error. For abnormal features, corresponding to abnormal labels, the third constraint is applied by minimizing the classification error.

[0095] Optionally, in this embodiment of the application, the method further includes obtaining the second constraint condition in the following manner:

[0096] After superimposing the sampled image features that belong to normal features with the sampled image features that belong to normal noise features to obtain the reconstructed image, the dimensions of the reconstructed image are adjusted so that the dimensional features of the reconstructed image after dimension adjustment are consistent with the dimensional features of the sampled image features.

[0097] The mean square error between the reconstructed image after dimensional adjustment and the acquired image corresponding to the sampled workpiece is calculated to obtain the second constraint condition.

[0098] Combination Figure 4 As shown, in one embodiment of this application, the calculated normal feature Fnormal and normal noise feature Fnorise are first added and fused to obtain the reconstructed image reshape. The dimensions of the fused reconstructed image reshape are then adjusted so that the dimensional features of the reconstructed image reshape are consistent with the dimensional features of the sampled image features. Specifically, firstly, the three-dimensional features of the reshaped image are adjusted to four-dimensional features and aligned with the corresponding positions of the input image. For example, in the implementation process of this application, after inputting N image data, the resulting fused three-dimensional feature dimensions are (N, 1369, 384). After dimension adjustment, the four-dimensional feature dimensions are (N, 37, 37, 384). Further, the four-dimensional features are indexed to ensure that the channel dimension is in the second order. For example, in the embodiment of this application, the dimensions obtained after dimension adjustment are (N, 384, 37, 37). The adjusted features are subjected to n interpolation and convolution operations (cov) to ensure that the W and H dimensions of the output features are consistent with the size of the input image (W is the corresponding feature width, and H is the height). For example, in the embodiment of this application, two operations are superimposed, interpolating by 7 times and 2 times respectively, and the number of convolution kernels in the convolution operation are 256 and 3 respectively, finally resulting in a reconstructed image dimension of (N, 3, 518, 518).

[0099] Furthermore, by applying an activation tanh to the output to constrain the output value range, the stability of the training process is improved; using this implementation method, the reconstructed image... and input image The reconstruction loss function is obtained by minimizing the mean squared error, which also yields the second constraint condition:

[0100] .

[0101] Optionally, in this embodiment of the application, the method further includes obtaining the third constraint condition in the following manner:

[0102] The third constraint condition is obtained by weighting the label vectors corresponding to the normal noise features and the label vectors corresponding to the abnormal features using the weight coefficients corresponding to the abnormal samples.

[0103] In this embodiment of the application, the third constraint can also be called classification error; the feature samples corresponding to the third constraint come from two parts, one part is normal noise features and the other part is abnormal features. The label corresponding to the normal noise features is 0, and the label corresponding to the abnormal features is the label obtained based on the abnormal mask processing, which includes 0 and 1. The specific implementation process is as follows: For normal noise features, initialize an all-zero label vector. The vector contains two dimensions: the first dimension corresponds to the number of normal noise features, and the second dimension is 1. For example, for normal noise features with dimensions (N, 1369, 384), the corresponding label vector dimension is (N*1369, 1). For abnormal features, scale the known abnormal mask matrix to the required input size, then apply a sliding window, and further calculate the maximum value within each sliding window. That is, if there is a label 1, the label of the current sliding window is defined as 1. The window size is consistent with the DINOv2 window size. The proposed window size is (14, 14). For an input with an input size of (518, 518), the final label vector dimension is (N*1369, 1), where N is the number of input images. Integrate all classification data. Since the number of normal samples is much larger than the number of abnormal samples, to balance the model performance, the loss corresponding to abnormal samples is weighted based on the binary cross-entropy loss. Optionally, the weight coefficient set in this embodiment is 20.

[0104] ;

[0105] Where w is the weight coefficient of the corresponding abnormal sample. These are the actual labels obtained after processing. For the corresponding predicted label, where, The loss corresponding to normal noise features; The loss corresponding to the abnormal features.

[0106] In one embodiment of this application, optionally, the first model is trained according to the constraints corresponding to different feature classifications after the first model performs feature classification on the sampled image features, including:

[0107] The sum of constraint values ​​for different feature classifications is obtained after the image features are classified using the first model.

[0108] When the constraint value is less than or equal to a preset value, the first model is determined to be obtained;

[0109] When the sum of the constraint values ​​is greater than the preset value, the weight parameters corresponding to different classification features when the first model is used to classify the image features are adjusted, and the step of inputting multiple sampled image features into the first model is re-executed until the obtained constraint value is less than or equal to the preset value.

[0110] Using the method of this embodiment, the first model is trained in the manner described above. During the training process, the original image is fused with the abnormal pixel matrix (probabilistic abnormal generation mask) obtained in the previous training process of the first model. Multiple sampled image features after image fusion are extracted, feature classification is performed, and the constraints corresponding to different feature classifications are determined until the constraint values ​​of each obtained constraint are less than or equal to preset values.

[0111] Optionally, in this embodiment of the application, step S3, determining the abnormal pixel matrix in the acquired image based on the image features classified as abnormal features, includes:

[0112] Based on the image features classified as abnormal features, generate prompt points;

[0113] Based on the acquired image and the cue points in the acquired image, a foreground mask matrix of the acquired image is generated using a second model; wherein, the second model is used to segment the acquired image and obtain the foreground mask of the acquired image based on the segmentation result, and the second model is a SAM model;

[0114] The closed shape mask matrix corresponding to the effective area of ​​the acquired image is obtained by summing the randomly generated closed shape region based on the acquired image with the foreground mask matrix; wherein, the closed shape can be a circle, rectangle, ellipse, etc.

[0115] Generate a corresponding line mask matrix based on the edges of the closed shape region and the corresponding closed shape mask matrix;

[0116] The abnormal pixel matrix in the acquired image is determined based on the image pixel intensities corresponding to the closed shape mask matrix and the straight line mask matrix, respectively.

[0117] Anomaly feature constraints rely on anomaly mask matrices, requiring manual contour annotation and filling of anomaly regions. This annotation process is costly, and defect data is often lacking in industrial settings. Recently, based on the success of diffusion models, some studies have proposed using diffusion models to assist in defect data generation. However, this process suffers from several problems: First, images of die-cast metal parts with different structures and imaging perspectives require separate training of a generation model, and the training process for diffusion models is very time-consuming, requiring approximately three days on an A6000 graphics card. Second, the generated anomaly data cannot be manually controlled; some generated anomalies significantly deviate from the data scenario, such as emphasizing structural morphology anomalies, which are often absent in the detection of surface defects in die-cast metal parts. Third, the generated image surface exhibits significant noise compared to the original image, affecting model training. Based on these issues, this application proposes a simulation method for defects such as scratches, dents, protrusions, and depressions, based on the imaging patterns of die-cast metal parts and the surface characteristics of defect data. The specific process is as follows: Figure 5 and Figure 6 As shown.

[0118] A. First, the foreground mask of the metal die casting image is obtained based on the SAM model (second model). The SAM model is a large image segmentation model trained on natural scene data. This application does not fine-tune the model in the metal die casting image scene. Therefore, the output result of directly using SAM has too large an error. Before the model segmentation, the position of the cue point and the corresponding label are added to obtain a foreground mask with relatively better performance.

[0119] B. To simulate the randomness of the location, size, and shape of defects, a circular mask region with random position and size is first generated in the image area. The generated circular mask matrix is ​​then ANDed with the foreground mask matrix of the die-casting part obtained in step A to obtain the circular mask matrix within the effective area. A corresponding straight line mask is generated by randomly selecting two points on the edge of each circular mask region and connecting them with a straight line. Similarly, to generate elliptical mask regions with random shapes, an elliptical mask matrix with random position, size, and angle is first generated in the image area. The resulting elliptical mask matrix is ​​then ANDed with the foreground mask matrix of the die-casting part obtained in step A to obtain the corresponding elliptical mask. The maximum length of the straight line mask cannot exceed 60 pixels, and the minimum mask length cannot be less than 5 pixels. The minimum length of the elliptical mask cannot be less than 20 pixels, and the maximum length cannot exceed 60 pixels. The minimum angle coverage cannot be less than 30 degrees, and the maximum is 360 degrees (i.e., a complete ellipse). The straight line mask and elliptical mask implemented in this application are as follows: Figure 7 and Figure 8 As shown.

[0120] C. Based on the masked area generated in process B, calculate the average pixel intensity within the masked area. If the pixel intensity is lower than a set threshold, according to imaging principles, the defect at the current location should be designed as a bright area; if it is higher than the set threshold, the defect at the current location should be designed as a dark area. To enhance data randomness, generate corresponding pixel enhancement or attenuation random numbers within the corresponding masked area and multiply them with the pixels within the corresponding masked area to obtain new image data. The pixel intensity threshold used in this application is 127, the enhancement coefficient range is [2,3], and the attenuation coefficient range is [0.2,0.5]. The resulting straight-line shape defect image (e.g.) Figure 9 (used to simulate defects such as scratches) and elliptical defect maps (such as...) Figure 10 Used to simulate defects such as dents, bumps, and scratches.

[0121] Combination Figure 1 , Figure 3 and Figure 11 The specific process of model training and inference using the method of this application embodiment is as follows:

[0122] Model training process:

[0123] Normal image data, abnormal image data generated based on the normal image data, and mask data are all input into the training model. The input abnormal image data undergoes basic image feature extraction and feature decoupling operations to generate corresponding normal features, normal noise features, and abnormal features. Normal features and a pre-extracted standard normal feature library are constrained by minimizing the cosine distance. The decoded and reconstructed image after fusing normal noise features and normal features is constrained by minimizing the mean square error of the input normal image data. Normal noise features and abnormal features are constrained by minimizing the weighted binary cross-entropy. During training, stochastic gradient descent is used to adjust the model weights to achieve overall optimization (minimum loss sum) by satisfying the sum of the above three constraints. The specific process is as follows... Figure 11 As shown in the figure, loss1 corresponds to the constraint of normal features, loss2 corresponds to the constraint of the reconstruction error of normal noise features and normal feature fusion, and loss3 corresponds to the classification error constraint of normal noise features and abnormal features.

[0124] Using the method of this application embodiment, the model inference process is as follows:

[0125] The model inference process relies solely on anomalous features. Therefore, it only requires importing and calculating the linear combination weights of the trained anomalous features to obtain the corresponding anomalous features, classifying them, and obtaining the mask and the corresponding image defect detection results. The model inference process is as follows: Figure 12 As shown.

[0126] This application also provides a defect detection device, please refer to... Figure 2 The device includes:

[0127] The feature acquisition module 201 is used to acquire multiple image features from the acquired image of the workpiece to be inspected;

[0128] Model processing module 202 is used to input the plurality of image features into a first model to obtain feature classifications of the plurality of image features; the feature classifications include normal features, abnormal features, and normal noise features; wherein, the first model is used to perform feature classification on the plurality of image features by weighted combination of the plurality of image features;

[0129] Analysis module 203 is used to determine the abnormal pixel matrix in the acquired image based on the image features that belong to abnormal features according to the feature classification.

[0130] Optionally, in the defect detection device, before inputting the plurality of image features into the first model, the model processing module 202 is further configured to:

[0131] Acquire features from multiple sampled images after acquiring images of multiple sampled workpieces;

[0132] Multiple sampled image features are input into the first model, and the first model is trained according to the constraints corresponding to different feature classifications after the first model classifies the sampled image features, so as to obtain the first model for abnormal pixel detection of the acquired image of the workpiece to be detected.

[0133] The constraints include one or more of the following:

[0134] The first constraint is used to represent the similarity of the sampled image features classified as normal features to standard normal features;

[0135] The second constraint condition is used to represent the error between the reconstructed image and the acquired image corresponding to the sampled workpiece; wherein, the reconstructed image is obtained by superimposing the sampled image features classified as normal features and the sampled image features classified as normal noise features;

[0136] The third constraint is used to represent the classification error between the sampled image features classified as normal noise features and the sampled image features classified as abnormal features.

[0137] Optionally, in the defect detection device, the analysis module 203 determines the abnormal pixel matrix in the acquired image based on the image features classified as abnormal features, including:

[0138] Based on the image features classified as abnormal features, generate prompt points;

[0139] Based on the acquired image and the cue points in the acquired image, a second model is used to generate a foreground mask matrix of the acquired image; wherein, the second model is used to segment the acquired image and obtain the foreground mask of the acquired image based on the segmentation result;

[0140] The closed shape mask matrix corresponding to the effective area of ​​the acquired image is obtained by summing the randomly generated closed shape region based on the acquired image with the foreground mask matrix.

[0141] Generate a corresponding line mask matrix based on the edges of the closed shape region and the corresponding closed shape mask matrix;

[0142] The abnormal pixel matrix in the acquired image is determined based on the image pixel intensities corresponding to the closed shape mask matrix and the straight line mask matrix, respectively.

[0143] Optionally, in the defect detection device, the model processing module 202 trains the first model based on the constraints corresponding to different feature classifications after the first model performs feature classification on the sampled image features, including:

[0144] The sum of constraint values ​​for different feature classifications is obtained after the image features are classified using the first model.

[0145] When the constraint value is less than or equal to a preset value, the first model is determined to be obtained;

[0146] When the sum of the constraint values ​​is greater than the preset value, the weight parameters corresponding to different classification features when the first model is used to classify the image features are adjusted, and the step of inputting multiple sampled image features into the first model is re-executed until the obtained constraint value is less than or equal to the preset value.

[0147] Optionally, in the defect detection device, the model processing module 202 obtains the first constraint condition in the following manner:

[0148] Calculate the cosine distance between the sampled image features that belong to normal features and the standard normal features to obtain the first constraint condition;

[0149] The standard normal features are obtained by clustering normal image samples in the normal sample data feature library and extracting the image features corresponding to the cluster centers.

[0150] Optionally, in the defect detection device, the model processing module 202 obtains the second constraint condition in the following manner:

[0151] After superimposing the sampled image features that belong to normal features with the sampled image features that belong to normal noise features to obtain the reconstructed image, the dimensions of the reconstructed image are adjusted so that the dimensional features of the reconstructed image after dimension adjustment are consistent with the dimensional features of the sampled image features.

[0152] The mean square error between the reconstructed image after dimensional adjustment and the acquired image corresponding to the sampled workpiece is calculated to obtain the second constraint condition.

[0153] Optionally, in the defect detection device, the model processing module 202 obtains the third constraint condition in the following manner:

[0154] The third constraint condition is obtained by weighting the label vectors corresponding to the normal noise features and the label vectors corresponding to the abnormal features using the weight coefficients corresponding to the abnormal samples.

[0155] This application also provides a detection device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the defect detection method described above.

[0156] This application also provides a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps in the defect detection method described above.

[0157] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the defect detection method described above.

[0158] This application uses the pre-trained model dinov2 for basic feature extraction, which can obtain a more robust basic feature representation. However, other pre-trained basic models can also be used to obtain the block feature vector set.

[0159] Based on the problem presented by real-world scenario data, this application decouples the basic features into three parts, or it can also decouple them into two parts, namely, normal noise features and abnormal features are combined into one part. Essentially, it belongs to the same series of methods as this application, but this will result in a complex distribution of these features and fewer constraints applied to them during training. The approach of this application constrains these two parts separately, which is more conducive to model training.

[0160] This method is crucial for detecting surface defects in the manufacturing process of metal die castings. Accurate identification of defects in die castings can promptly identify problems in the production process, prevent substandard products from entering the market, ensure product quality, improve production efficiency, and reduce production costs.

[0161] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A defect detection method, characterized in that, include: Acquire multiple image features from the acquired image of the workpiece to be inspected; The multiple image features are input into a first model to obtain feature classifications of the multiple image features; the feature classifications include normal features, abnormal features, and normal noise features; wherein, the first model is used to perform feature classification on the multiple image features by weighted combination of the multiple image features; Based on the image features classified as abnormal features, the matrix of abnormal pixels in the acquired image is determined.

2. The defect detection method according to claim 1, characterized in that, Before inputting the plurality of image features into the first model, the method further includes: Acquire features from multiple sampled images after acquiring images of multiple sampled workpieces; Multiple sampled image features are input into the first model, and the first model is trained according to the constraints corresponding to different feature classifications after the first model classifies the sampled image features, so as to obtain the first model for detecting abnormal pixels in the acquired image of the workpiece to be detected.

3. The defect detection method according to claim 2, characterized in that, The constraints include one or more of the following: The first constraint is used to represent the similarity of the sampled image features classified as normal features to standard normal features; The second constraint condition is used to represent the error between the reconstructed image and the acquired image corresponding to the sampled workpiece; wherein, the reconstructed image is obtained by superimposing the sampled image features classified as normal features and the sampled image features classified as normal noise features; The third constraint is used to represent the classification error between the sampled image features classified as normal noise features and the sampled image features classified as abnormal features.

4. The defect detection method according to claim 1, characterized in that, Based on the image features classified as anomalous features, the abnormal pixel matrix in the acquired image is determined, including: Based on the image features classified as abnormal features, generate prompt points; Based on the acquired image and the cue points in the acquired image, a second model is used to generate a foreground mask matrix of the acquired image; wherein, the second model is used to segment the acquired image and obtain the foreground mask of the acquired image based on the segmentation result; The closed shape mask matrix corresponding to the effective area of ​​the acquired image is obtained by summing the randomly generated closed shape region based on the acquired image with the foreground mask matrix. Generate a corresponding line mask matrix based on the edges of the closed shape region and the corresponding closed shape mask matrix; The abnormal pixel matrix in the acquired image is determined based on the image pixel intensities corresponding to the closed shape mask matrix and the straight line mask matrix, respectively.

5. The defect detection method according to claim 2 or 3, characterized in that, Based on the constraints corresponding to different feature classifications after the first model performs feature classification on the sampled image features, the first model is trained, including: The sum of constraint values ​​for different feature classifications is obtained after the image features are classified using the first model. When the constraint value is less than or equal to a preset value, the first model is determined to be obtained; When the sum of the constraint values ​​is greater than the preset value, the weight parameters corresponding to different classification features when the first model is used to classify the image features are adjusted, and the step of inputting multiple sampled image features into the first model is re-executed until the obtained constraint value is less than or equal to the preset value.

6. The defect detection method according to claim 3, characterized in that, The method further includes obtaining the first constraint condition in the following manner: Calculate the cosine distance between the sampled image features that belong to normal features and the standard normal features to obtain the first constraint condition; The standard normal features are obtained by clustering normal image samples in the normal sample data feature library and extracting the image features corresponding to the cluster centers.

7. The defect detection method according to claim 3, characterized in that, The method further includes obtaining the second constraint condition in the following manner: After superimposing the sampled image features that belong to normal features with the sampled image features that belong to normal noise features to obtain the reconstructed image, the dimensions of the reconstructed image are adjusted so that the dimensional features of the reconstructed image after dimension adjustment are consistent with the dimensional features of the sampled image features. The mean square error between the reconstructed image after dimensional adjustment and the acquired image corresponding to the sampled workpiece is calculated to obtain the second constraint condition.

8. The defect detection method according to claim 3, characterized in that, The method further includes obtaining the third constraint condition in the following manner: The third constraint condition is obtained by weighting the label vectors corresponding to the normal noise features and the label vectors corresponding to the abnormal features using the weight coefficients corresponding to the abnormal samples.

9. A defect detection device, characterized in that, include: The feature acquisition module is used to acquire multiple image features from the acquired image of the workpiece to be inspected; The model processing module is used to input the multiple image features into a first model to obtain feature classifications of the multiple image features; the feature classifications include normal features, abnormal features, and normal noise features; wherein, the first model is used to perform feature classification on the multiple image features by weighted combination of the multiple image features; The analysis module is used to determine the abnormal pixel matrix in the acquired image based on the image features that belong to abnormal features according to the feature classification.

10. A testing device, comprising: A processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the defect detection method as described in any one of claims 1-8.

11. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the defect detection method as described in any one of claims 1-8.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the defect detection method as described in any one of claims 1-8.

Images