A knowledge-guided, finely adjustable method and apparatus for locating workpiece defects.

Abstract

This invention discloses a knowledge-guided, finely adjustable workpiece defect localization method and apparatus, relating to the field of industrial inspection technology. The method includes: training a target residual flow model using a set of normal workpiece images; constructing an initial knowledge-guided edge refinement network based on the target residual flow model, and training the target knowledge-guided edge refinement network using a set of normal workpiece images and a set of pseudo-defect workpiece images; constructing a difficult-to-detect defect knowledge base based on the difficult-to-detect image features of each workpiece's difficult-to-detect defect images; determining image features of suspected defect areas and prior defect knowledge based on the image of the workpiece to be tested and the difficult-to-detect defect knowledge base; using the target knowledge-guided edge refinement network, fine-tuning the difficult samples of the workpiece image to be tested using the suspected defect area image features combined with prior defect knowledge; and enhancing the texture structure anomaly features of the workpiece image to be tested based on the target residual flow model, outputting a workpiece defect localization image. Overall, this improves the accuracy of workpiece defect localization.

Description

Technical Field

[0001] This invention relates to the field of industrial inspection technology, and in particular to a knowledge-guided, finely adjustable method for locating workpiece defects. Background Technology

[0002] In industrial production, workpieces may develop product defects during the production processes such as raw material processing, rough treatment, cutting, and grinding. Therefore, the detection of surface defects is an important part of product quality control. Since the causes of defects are diverse, the forms of defects are unpredictable and even novel defects may appear.

[0003] Traditional methods for detecting surface defects in workpieces require a large amount of defect anomaly data and labeling costs. However, defect anomaly data is becoming increasingly rare in increasingly mature industrial production lines, and complex processes make it easier for novel defects to appear. To address the issues of insufficient defect anomaly data and difficulty in detecting novel anomalies, existing technologies detect defect anomaly data in a set of homogeneous images based on pre-training. However, these technologies suffer from domain adaptation problems and lack the ability to adapt to the target domain dataset, resulting in low accuracy in workpiece defect detection. Summary of the Invention

[0004] This invention provides a knowledge-guided, finely adjustable method and apparatus for locating workpiece defects, which solves the technical problem that existing pre-trained workpiece surface defect detection methods lack the ability to adapt to target domain datasets, resulting in low accuracy in workpiece defect detection.

[0005] The first aspect of this invention provides a knowledge-guided, finely adjustable workpiece defect localization method, comprising:

[0006] Obtain a set of normal images of the workpiece, and perform image preprocessing on the set of normal images to generate a set of false defect images of the workpiece;

[0007] The initial residual flow model is trained using the set of normal images of the workpiece to determine the target residual flow model;

[0008] An initial knowledge-guided edge refinement network is constructed based on the target residual flow model, and the initial knowledge-guided edge refinement network is trained using the normal image set and the pseudo defect image set of the workpiece to determine the target knowledge-guided edge refinement network.

[0009] Acquire multiple images of difficult-to-detect defects in workpieces, and construct a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each workpiece image;

[0010] When an image of a workpiece to be tested is received, the image features of the suspected defect area and the prior knowledge of defects in the image of the workpiece to be tested are determined based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects.

[0011] The edge refinement network is guided by the target knowledge, and the image features of the suspected defect area are combined with the defect prior knowledge to fine-tune the hard samples of the image of the workpiece to be tested. Based on the target residual flow model, the image channels of the image of the workpiece to be tested are augmented to enhance the texture structure anomaly features, and the workpiece defect localization image is output.

[0012] Optionally, the step of preprocessing the normal image set of the workpiece to generate a false defect image set of the workpiece includes:

[0013] Generating noisy images based on Berlin noise;

[0014] The noisy image is fused with the normal image set of the workpiece using the Poisson fusion algorithm to obtain the false defect image set of the workpiece.

[0015] Optionally, the step of training the initial residual flow model using the normal image set of the workpiece to determine the target residual flow model includes:

[0016] The normal sample image of the workpiece is input into a preset ImageNet pre-trained model for feature extraction, and the normal image features are output.

[0017] The initial residual flow model is trained using the normal image features, and the target residual flow model is output.

[0018] Optionally, the step of acquiring multiple images of difficult-to-detect defects in workpieces and constructing a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each of the workpiece images includes:

[0019] Multiple images of difficult-to-detect defects in workpieces are acquired, and features are extracted from each of the images using a preset feature extractor to obtain multiple difficult-to-detect image features.

[0020] The true label binary mask of each workpiece's difficult-to-detect defect image is multiplied element-wise with the corresponding difficult-to-detect image features, and then global pooling is performed to output multiple difficult-to-detect defect region image features.

[0021] The image features of each of the aforementioned difficult-to-detect defect regions are stitched together to construct a knowledge base for difficult-to-detect defects.

[0022] Optionally, when receiving an image of a workpiece to be tested, determining the image features of suspected defect areas and prior knowledge of defects in the image of the workpiece to be tested based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects includes:

[0023] When a workpiece image to be tested is received, the image features to be tested are extracted using a preset feature extractor.

[0024] A first similarity matrix is ​​constructed based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and global max pooling is performed on the first similarity matrix to generate a first similar image.

[0025] The first similarity image is multiplied element-wise and global pooled with the features of the image to be tested to output the suspected defect area image features of the workpiece image to be tested.

[0026] A second similarity matrix is ​​constructed using the cosine similarity between the suspected defect region image features and each of the difficult-to-detect image features, and global max pooling is performed on the second similarity matrix to obtain a second similarity map;

[0027] If the intersection of the real label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, then the first similarity image is used as the defect prior knowledge.

[0028] Optionally, the target knowledge-guided edge refinement network includes a semantic feature extractor, a parallel dilated convolution module, a target residual flow model, and a decoder; the step of using the target knowledge-guided edge refinement network to fine-tune the hard samples of the workpiece image under test by combining the image features of the suspected defect region with the prior knowledge of the defect, and augmenting the image channels of the workpiece image under test based on the target residual flow model to enhance the texture structure anomaly features, and outputting a workpiece defect localization image, includes:

[0029] The image of the workpiece to be tested is input into a semantic feature extractor for feature extraction to obtain semantic features;

[0030] The semantic features are sampled and fused in parallel using a parallel dilated convolution module to output multi-scale features.

[0031] The semantic features and the multi-scale features are respectively input into the target residual flow model for feature processing and then normalized to determine the first anomaly score normalization map and the second anomaly score normalization map.

[0032] The semantic features are multiplied element-wise with the first anomaly score normalization map, and the multi-scale features are multiplied element-wise with the second anomaly score normalization map, respectively, to output the first weighted feature and the second weighted feature;

[0033] The semantic features, the first anomaly score normalization map, and the first weighted features are concatenated by channels to obtain the first enhanced features;

[0034] The second enhanced feature is generated by channel splicing using the multi-scale feature, the second anomaly score normalization map, and the second weighted feature;

[0035] After concatenating the semantic features, the prior knowledge of defects, and the image features of the suspected defect region through channels, the first attention feature is determined by adding it element-by-element to the first enhancement feature.

[0036] The prior knowledge of the defect is added element-wise to the second enhancement feature to determine the second attention feature;

[0037] After the first attention feature and the second attention feature are extracted and fused by the decoder, they are added element by element to the prior knowledge of the defect to output the workpiece defect location image.

[0038] Optionally, the decoder includes a first convolutional layer and a second convolutional layer; the step of extracting and fusing the first and second attention features through the decoder, and then adding them element-wise with the prior knowledge of the defect to output a workpiece defect localization image includes:

[0039] After performing convolution operations on the second attention features based on the first convolutional layer, upsampling is performed to output the detection features;

[0040] After the first attention feature is convolved by the first convolutional layer, it is concatenated with the detection feature to determine the fused feature.

[0041] After performing convolution calculation on the fused features using a second convolutional layer and then upsampling, the features are added element-wise with the prior knowledge of the defects to output a workpiece defect location image.

[0042] A second aspect of the present invention provides a knowledge-guided, finely adjustable workpiece defect location device, comprising:

[0043] The image processing module is used to acquire a set of normal images of the workpiece and perform image preprocessing on the set of normal images to generate a set of false defect images of the workpiece.

[0044] The residual flow model training module is used to train the initial residual flow model using the normal image set of the workpiece, and to determine the target residual flow model.

[0045] The edge refinement network training module is used to construct an initial knowledge-guided edge refinement network based on the target residual flow model, and to train the initial knowledge-guided edge refinement network using the normal image set and the pseudo-defect image set of the workpiece, so as to determine the target knowledge-guided edge refinement network.

[0046] The knowledge base construction module is used to acquire multiple images of difficult-to-detect defects in workpieces and construct a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each of the workpiece images.

[0047] The prior knowledge determination module is used to determine the image features of the suspected defect area and the prior knowledge of defects in the image of the workpiece to be tested based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects when the image of the workpiece to be tested is received.

[0048] The workpiece defect localization module is used to guide the edge refinement network through the target knowledge, use the image features of the suspected defect area combined with the defect prior knowledge to fine-tune the difficult samples of the workpiece image to be tested, and augment the image channels of the workpiece image to be tested based on the target residual flow model to enhance the texture structure anomaly features, and output the workpiece defect localization image.

[0049] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any of the preceding claims.

[0050] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the knowledge-guided fine-tunable workpiece defect location method as described in any of the preceding claims.

[0051] As can be seen from the above technical solutions, the present invention has the following advantages:

[0052] The above-mentioned technical solution of the present invention provides a knowledge-guided fine-tunable workpiece defect localization method, comprising: acquiring a set of normal workpiece images; performing image preprocessing on the normal workpiece image set to generate a set of pseudo-defect workpiece images; training an initial residual flow model using the normal workpiece image set to determine a target residual flow model; constructing an initial knowledge-guided edge refinement network based on the target residual flow model, and training the initial knowledge-guided edge refinement network using the normal workpiece image set and the set of pseudo-defect workpiece images to determine a target knowledge-guided edge refinement network; acquiring multiple images of difficult-to-detect defects in workpieces, and constructing a difficult-to-detect defect knowledge base based on the difficult-to-detect image features of each difficult-to-detect defect image; when receiving an image of a workpiece to be tested, determining the suspected defect region image features and defect prior knowledge of the image of the workpiece to be tested based on the image of the workpiece to be tested and the difficult-to-detect defect knowledge base; fine-tuning the difficult samples of the image of the workpiece to be tested using the suspected defect region image features combined with defect prior knowledge through the target knowledge-guided edge refinement network, and augmenting the image channels of the image of the workpiece to be tested based on the target residual flow model to enhance the texture structure anomaly features, and outputting a workpiece defect localization image. Based on the above scheme, the residual flow model is trained using an easily accessible set of normal workpiece images. A set of pseudo-defect images is generated based on image preprocessing to simulate the rich variations of real defects in industrial scenarios. The normal workpiece image set is used as the training set to train the knowledge-guided edge refinement network, which avoids the need for manual sample annotation and thus effectively reduces labor costs. This helps to locate and segment real defects. The prior knowledge of defects is determined by extracting difficult-to-detect image features from images of difficult-to-detect defects and comparing them with the image of the workpiece to be tested. The knowledge is then fine-tuned to guide the edge refinement network to focus on the features of difficult-to-detect defects, thereby reducing missed and false detections and improving the overall accuracy of workpiece defect location. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating the steps of a knowledge-guided, finely adjustable workpiece defect localization method provided in this embodiment of the invention;

[0055] Figure 2 A framework diagram of a knowledge-guided edge refinement network provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the knowledge-guided feature enhancement structure of the knowledge-guided edge refinement network provided in an embodiment of the present invention;

[0057] Figure 4 This is a structural block diagram of a knowledge-guided, finely adjustable workpiece defect location device provided in an embodiment of the present invention. Detailed Implementation

[0058] This invention provides a knowledge-guided, finely adjustable workpiece defect localization method and apparatus to address the technical problem that existing pre-trained workpiece surface defect detection methods lack the ability to adapt to target domain datasets, resulting in low accuracy in workpiece defect detection.

[0059] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0060] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a knowledge-guided, finely adjustable workpiece defect location method provided in this embodiment of the invention.

[0061] This invention provides a knowledge-guided, finely adjustable workpiece defect localization method, comprising:

[0062] Step 101: Obtain the normal image set of the workpiece, and perform image preprocessing on the normal image set of the workpiece to generate the false defect image set of the workpiece.

[0063] It should be noted that the normal workpiece image set contains multiple normal workpiece images, that is, workpiece images without defects. After acquiring the normal workpiece images, preprocessing such as center cropping can be performed before assembling the normal workpiece image set. On the other hand, the pseudo-defect workpiece image set contains multiple pseudo-defect workpiece images, which refer to images with simulated defects added to the normal workpiece images by computer generation or other methods when there are no actual defects in the normal workpiece images.

[0064] Step 101 includes the following sub-steps:

[0065] Generating noisy images based on Berlin noise;

[0066] The Poisson fusion algorithm is used to fuse noisy images with a set of normal workpiece images to obtain a set of false defect images of the workpiece.

[0067] It should be noted that the Perlin noise algorithm generates noisy images with random shapes and diverse texture patterns. Based on the Poisson fusion algorithm, the noisy images generated by the Poisson fusion algorithm are fused with each normal image of the workpiece in the normal image set to generate multiple workpiece pseudo-defect images with seamless edges and form a workpiece pseudo-defect image set.

[0068] Step 102: Train the initial residual flow model using a set of normal workpiece images to determine the target residual flow model.

[0069] Step 102 includes the following sub-steps:

[0070] Input normal sample images of workpieces into a pre-trained ImageNet model for feature extraction, and output normal image features;

[0071] The initial residual flow model is trained using normal image features, and the target residual flow model is output.

[0072] It should be noted that using readily available normal images of workpieces to train the residual flow model can effectively reduce costs. The residual flow model can distinguish between normal and abnormal images, and the abnormal images can be understood as images with defects.

[0073] Step 103: Construct an initial knowledge-guided edge refinement network based on the target residual flow model, and train the initial knowledge-guided edge refinement network using a set of normal workpiece images and a set of pseudo-defect workpiece images to determine the target knowledge-guided edge refinement network.

[0074] It should be noted that a knowledge-guided edge thinning network based on semantic segmentation and dilated convolution is used for defect detection in workpiece images, such as... Figure 2 As shown, the Knowledge Guided Edge Refinement Network (KGERN) adopts the classic encoder-decoder architecture in the semantic segmentation field, including an encoder, a decoder, and a target residual flow model. The encoder includes a semantic feature extractor and a parallel dilated convolution module (ASPP). The parallel dilated convolution module performs feature processing and fusion output through parallel 1×1 convolutional layers, 3×3 convolutional layers (dilation rate 6), 3×3 convolutional layers (dilation rate 12), 3×3 convolutional layers (dilation rate 18), and average pooling layers. The decoder includes a first convolutional layer (1×1 convolution) and a second convolutional layer (3×3 convolution).

[0075] During the network training phase, a set of normal workpiece images and a set of pseudo-defect workpiece images are used as the initial knowledge-guided edge refinement network for training. The diversity of pseudo-defect images indirectly improves the adaptability to unknown defect types in real detection scenarios. A cross-entropy loss is designed. ) and Dice Loss ( As a synergistic effect of the network loss function, it guides the network to finely segment the edges of abnormal regions. When the network loss function value converges, the target knowledge-guided edge refinement network is obtained. The network loss function... include:

[0076] ;

[0077] In the formula, These are hyperparameters, and are generally taken as follows: .

[0078] Understandably, the specific calculation process for cross-entropy loss and Dice loss can be found in existing technologies, and will not be elaborated here.

[0079] Step 104: Obtain multiple images of difficult-to-detect defects in workpieces, and construct a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each workpiece image.

[0080] It should be noted that the image of a workpiece with a difficult-to-detect defect refers to an image of a workpiece with a defect that is relatively difficult to detect. The knowledge base of difficult-to-detect defects contains the difficult-to-detect image features of multiple workpiece images with difficult-to-detect defects.

[0081] Step 104 includes the following sub-steps:

[0082] S11. Acquire multiple images of difficult-to-detect defects in workpieces, and extract features from each image using a preset feature extractor to obtain multiple difficult-to-detect image features.

[0083] It should be noted that N images of difficult-to-detect defects in workpieces are used as reference images, and the feature extractor extracts the difficult-to-detect image features of the reference images. The feature extractor can be a semantic feature extractor in the target knowledge-guided edge refinement network, or it can be any other model that can extract image features, without any restrictions.

[0084] S12. The binary mask of the real label of each workpiece's difficult-to-detect defect image is multiplied element-wise with the corresponding difficult-to-detect image features, and then global pooling is performed to output multiple difficult-to-detect defect region image features.

[0085] It should be noted that the true label binary mask marks which pixels in the image belong to the defect area and the normal area. This allows for the mapping of the true label binary mask to the image of the difficult-to-detect defect in the workpiece. Corresponding difficult-to-detect image features Multiply and perform global pooling to obtain image features of hard-to-detect defect regions with multiple dimensions [1, C]. .

[0086] S13. Perform channel stitching on the image features of each difficult-to-detect defect area to construct a knowledge base for difficult-to-detect defects.

[0087] It should be noted that by stitching together the image features of the difficult-to-detect defect regions of each workpiece's difficult-to-detect defect image, a knowledge base of difficult-to-detect defects with dimensions [N, C] is obtained.

[0088] Step 105: When the image of the workpiece to be tested is received, the image features of the suspected defect area and the prior knowledge of defects in the image of the workpiece to be tested are determined based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects.

[0089] It should be noted that the image of the workpiece to be tested refers to the image of the workpiece for which defect localization and detection are required. The image features of the suspected defect area and the prior knowledge of defects corresponding to the image of the workpiece to be tested are determined by the knowledge base of difficult-to-detect defects. This facilitates the subsequent feature enhancement to better guide the target knowledge to guide the edge refinement network to focus on the features of difficult-to-detect defects.

[0090] Step 105 includes the following sub-steps:

[0091] S21. When the image of the workpiece to be tested is received, the image features of the workpiece to be tested are extracted by a preset feature extractor.

[0092] It should be noted that for the image of the workpiece to be tested... The corresponding features of the image to be tested are extracted using a feature extractor. This feature extractor can employ a semantic feature extractor in a target knowledge-guided edge refinement network, or it can employ other models that can extract image features.

[0093] S22. Construct a first similarity matrix based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and perform global max pooling on the first similarity matrix to generate a first similarity image.

[0094] It should be noted that the features of the image to be tested The image features at each coordinate are calculated once and compared with the image features of the hard-to-detect defect region in the hard-to-detect defect knowledge base. The cosine similarity is calculated, and then multiple cosine similarities are used to construct a first similarity matrix with dimensions [N, H, W]. Max pooling is then performed on the first similarity matrix along the channel dimension to obtain a first similarity image with dimensions [1, H, W]. The first similarity image represents the region in the image features to be tested that is similar to the image features of the difficult-to-detect defect region.

[0095] S23. Perform element-wise multiplication and global pooling on the first similarity image and the features of the image to be tested to output the suspected defect area image features of the workpiece image to be tested.

[0096] It should be noted that the first similarity image As a mask and features of the image to be tested Multiply and perform global pooling to obtain the image features of the suspected defect area of ​​the workpiece image under test. .

[0097] S24. Construct a second similarity matrix using the cosine similarity between the image features of the suspected defective region and the features of each difficult-to-detect image, and perform global max pooling on the second similarity matrix to obtain the second similarity map.

[0098] It should be noted that the image features of the suspected defect area are... Conversely, the difficult-to-detect image features of each workpiece's difficult-to-detect defect images Calculate the cosine similarity and construct a second similarity matrix. Then, perform global max pooling on the second similarity matrix to obtain a second similarity image with dimensions [1, H, W]. .

[0099] S25. If the intersection of the real label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, then the first similarity image is taken as the defect prior knowledge.

[0100] It should be noted that if the second similarity image Binary masks with corresponding ground truth labels for multiple workpiece images with difficult-to-detect defects associated with a difficult-to-detect defect knowledge base. If the intersection between the two is not empty, it indicates that the image features of the suspected defect area found in the image of the workpiece under test do indeed match the image features of the difficult-to-detect defect area of ​​the workpiece image. By designing a two-way verification method for defect features, we can ensure that the obtained first similarity image accurately describes the location of a specific type of defect. Therefore, the first similarity image can be used to verify the location of the defect. As prior knowledge of defects, it is used for subsequent feature enhancement; otherwise, Set to 0.

[0101] Step 106: Guide the edge refinement network with target knowledge, use image features of suspected defect areas combined with defect prior knowledge to fine-tune the difficult samples of the workpiece image under test, and augment the image channels of the workpiece image under test based on the target residual flow model to enhance the texture structure abnormal features, and output the workpiece defect location image.

[0102] It should be noted that the target knowledge-guided edge refinement network includes a semantic feature extractor, a parallel dilated convolution module, a target residual flow model, and a decoder. The workpiece defect localization image maps the defect situation of the workpiece.

[0103] Step 106 includes the following sub-steps:

[0104] S31. Input the image of the workpiece to be tested into the semantic feature extractor for feature extraction to obtain semantic features.

[0105] S32. The semantic features are sampled and fused in parallel through the parallel dilated convolution module to output multi-scale features.

[0106] S33. Input the semantic features and multi-scale features into the target residual flow model for feature processing and then normalize them to determine the first anomaly score normalization map and the second anomaly score normalization map.

[0107] It should be noted that after inputting semantic features and multi-scale features into the target residual flow model, the first anomaly score map and the second anomaly score map are output respectively. The two anomaly score maps are normalized and their value range is restricted to [0, 1], thus obtaining the first anomaly score normalization map and the second anomaly score normalization map.

[0108] S34. Perform element-wise multiplication of semantic features with the first anomaly score normalization map and multi-scale features with the second anomaly score normalization map, respectively, and output the first weighted feature and the second weighted feature.

[0109] It should be noted that by performing element-wise multiplication of semantic features with the first anomaly score normalization map and multi-scale features with the second anomaly score normalization map, the feature representation of the anomaly region is preserved after multiplication because the score value corresponding to the anomaly region is larger, while the score value corresponding to the normal region is smaller, thereby suppressing redundant information in the image background.

[0110] S35. Concatenate the semantic features, the first anomaly score normalization map, and the first weighted features to obtain the first enhanced features.

[0111] S36. Channel splicing is performed using multi-scale features, the second anomaly score normalization map, and the second weighted features to generate the second enhanced feature.

[0112] It should be noted that, as Figure 3 As shown, the process of obtaining the first or second enhanced feature can be referred to as follows:

[0113] ;

[0114] ;

[0115] In the formula, For semantic features or multi-scale features, For dot product calculation, This is an anomaly rating chart. For normalization, for, This is a channel-level merging operation. To enhance features.

[0116] S37. After concatenating the semantic features, prior knowledge of defects, and image features of suspected defect areas through channels, add them element-by-element to the first enhancement feature to determine the first attention feature.

[0117] S38. Add the prior knowledge of the defect to the second enhancement feature element by element to determine the second attention feature.

[0118] S39. After extracting and fusing the first and second attention features through the decoder, the features are added element by element with the prior knowledge of defects to output the workpiece defect location image.

[0119] The decoder includes a first convolutional layer and a second convolutional layer; sub-step S39 includes:

[0120] The second attention features are upsampled after convolution operation on the first convolutional layer to output the detection features;

[0121] After the first attention feature is convolved by the first convolutional layer, it is concatenated with the detection feature to determine the fused feature.

[0122] After performing convolution calculation on the fused features using the second convolutional layer and then upsampling, the features are added element-by-element with the prior knowledge of defects to output the workpiece defect location image.

[0123] It should be noted that, in order to realize the defect prior knowledge, i.e., the first similarity image A deeper and more comprehensive fusion of image features with the image of the workpiece under test is achieved. Feature fusion pathways are set at different positions of the target knowledge-guided edge refinement network. At the intermediate feature layer, the target knowledge-guided edge refinement network is guided to focus on hard-to-detect defect features. At the output end, the defect prior knowledge or the output result of the target knowledge-guided edge refinement network is added element by element with a certain weight as a result post-processing method. At this time, the abnormal localization result of the target knowledge-guided edge refinement network and the hard-to-detect defect localization result complement each other. By using defect prior knowledge in multiple ways to enhance image feature guidance fine-tuning without retraining, the defect localization accuracy is improved.

[0124] In this embodiment of the invention, a residual flow model is trained using an easily accessible set of normal workpiece images. A set of pseudo-defect images is generated based on image preprocessing to simulate the rich variations of real defects in industrial scenarios. The normal workpiece image set is used as a training set to train the knowledge-guided edge refinement network, avoiding the need for manual sample annotation and thus effectively reducing labor costs. This helps to locate and segment real defects. The prior knowledge of defects is determined by extracting difficult-to-detect image features from images of difficult-to-detect defects and comparing them with the image of the workpiece to be tested. The fine-tuning of the knowledge guides the edge refinement network to focus on the features of difficult-to-detect defects, thereby reducing missed and false detections. Overall, it can be well applied to any type of abnormal workpiece defects, has strong generalization performance, and improves the accuracy of workpiece defect location.

[0125] Please see Figure 4 , Figure 4 This is a structural block diagram of a knowledge-guided, finely adjustable workpiece defect location device provided in an embodiment of the present invention.

[0126] This invention provides a knowledge-guided, finely adjustable workpiece defect location device, comprising:

[0127] Image processing module 401 is used to acquire a normal image set of the workpiece and perform image preprocessing on the normal image set of the workpiece to generate a false defect image set of the workpiece.

[0128] The residual flow model training module 402 is used to train the initial residual flow model using a set of normal workpiece images to determine the target residual flow model.

[0129] The edge refinement network training module 403 is used to construct an initial knowledge-guided edge refinement network based on the target residual flow model, and to train the initial knowledge-guided edge refinement network using a set of normal workpiece images and a set of pseudo-defect workpiece images to determine the target knowledge-guided edge refinement network.

[0130] The knowledge base construction module 404 is used to acquire multiple images of difficult-to-detect defects in workpieces and construct a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each workpiece image.

[0131] The prior knowledge determination module 405 is used to determine the image features of the suspected defect area and the prior knowledge of defects in the image of the workpiece under test based on the image of the workpiece under test and the knowledge base of difficult-to-detect defects when the image of the workpiece under test is received.

[0132] The workpiece defect localization module 406 is used to guide the edge refinement network through target knowledge, use image features of suspected defect areas combined with defect prior knowledge to fine-tune the difficult samples of the workpiece image to be tested, and augment the image channels of the workpiece image to be tested based on the target residual flow model to enhance the texture structure abnormal features, and output the workpiece defect localization image.

[0133] Furthermore, the image processing module 401 is specifically used for:

[0134] Generating noisy images based on Berlin noise;

[0135] The Poisson fusion algorithm is used to fuse noisy images with a set of normal workpiece images to obtain a set of false defect images of the workpiece.

[0136] Furthermore, the residual flow model training module 402 is specifically used for:

[0137] Input normal sample images of workpieces into a pre-trained ImageNet model for feature extraction, and output normal image features;

[0138] The initial residual flow model is trained using normal image features, and the target residual flow model is output.

[0139] Furthermore, the knowledge base construction module 404 is specifically used for:

[0140] Multiple images of difficult-to-detect defects in workpieces are acquired, and features of each image are extracted using a preset feature extractor to obtain multiple difficult-to-detect image features.

[0141] The binary mask of the real label of each workpiece's difficult-to-detect defect image is multiplied element-wise with the corresponding difficult-to-detect image features and then global pooled to output multiple difficult-to-detect defect region image features.

[0142] Image features of each difficult-to-detect defect region are stitched together to construct a knowledge base for difficult-to-detect defects.

[0143] Furthermore, the prior knowledge determination module 405 is specifically used for:

[0144] When the image of the workpiece to be tested is received, the image features to be tested are extracted by a preset feature extractor.

[0145] The first similarity matrix is ​​constructed based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and global max pooling is performed on the first similarity matrix to generate the first similarity image.

[0146] The first similarity image and the features of the image to be tested are multiplied element-wise and global pooled to output the suspected defect area image features of the workpiece image to be tested.

[0147] A second similarity matrix is ​​constructed using the cosine similarity between the features of the suspected defective region image and the features of each difficult-to-detect image. Global max pooling is then performed on the second similarity matrix to obtain the second similarity map.

[0148] If the intersection of the binary mask of the real label corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, then the first similarity image is used as the defect prior knowledge.

[0149] Furthermore, the target knowledge-guided edge refinement network includes a semantic feature extractor, a parallel dilated convolution module, a target residual flow model, and a decoder; the workpiece defect localization module 406 is specifically used for:

[0150] The image of the workpiece to be tested is input into the semantic feature extractor for feature extraction to obtain semantic features;

[0151] The semantic features are sampled and fused in parallel using a parallel dilated convolution module to output multi-scale features.

[0152] Semantic features and multi-scale features are input into the target residual flow model for feature processing and then normalized to determine the first anomaly score normalization map and the second anomaly score normalization map.

[0153] Element-wise multiplication is performed on the semantic features and the first anomaly score normalization map, and on the multi-scale features and the second anomaly score normalization map, respectively, to output the first weighted feature and the second weighted feature;

[0154] The semantic features, the first anomaly score normalization map, and the first weighted features are concatenated by channels to obtain the first enhanced features;

[0155] The second enhanced feature is generated by channel stitching using multi-scale features, a second anomaly score normalization map, and a second weighted feature.

[0156] After concatenating the semantic features, prior knowledge of defects, and image features of suspected defect areas through channels, the first attention feature is determined by adding it element-by-element to the first enhancement feature.

[0157] The defect prior knowledge is added element-by-element to the second enhancement feature to determine the second attention feature;

[0158] After feature extraction and fusion of the first and second attention features by the decoder, the feature is added element by element with the prior knowledge of defects to output the workpiece defect location image.

[0159] Furthermore, the decoder includes a first convolutional layer and a second convolutional layer; after feature extraction and fusion of the first and second attention features by the decoder, the features are added element-wise with the prior knowledge of defects to output a workpiece defect localization image, including:

[0160] The second attention features are upsampled after convolution operation on the first convolutional layer to output the detection features;

[0161] After the first attention feature is convolved by the first convolutional layer, it is concatenated with the detection feature to determine the fused feature.

[0162] After performing convolution calculation on the fused features using the second convolutional layer and then upsampling, the features are added element-by-element with the prior knowledge of defects to output the workpiece defect location image.

[0163] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any of the above embodiments.

[0164] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implement the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any of the above embodiments.

[0165] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any of the above embodiments.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge-guided, finely adjustable method for locating workpiece defects, characterized in that, include: Obtain a set of normal images of the workpiece, and perform image preprocessing on the set of normal images to generate a set of false defect images of the workpiece; The initial residual flow model is trained using the set of normal images of the workpiece to determine the target residual flow model; An initial knowledge-guided edge refinement network is constructed based on the target residual flow model, and the initial knowledge-guided edge refinement network is trained using the normal image set and the pseudo defect image set of the workpiece to determine the target knowledge-guided edge refinement network. Acquire multiple images of difficult-to-detect defects in workpieces, and construct a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each workpiece image; When an image of a workpiece to be tested is received, the image features of the suspected defect area and the prior knowledge of defects in the image of the workpiece to be tested are determined based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects. The edge refinement network is guided by the target knowledge, and the image features of the suspected defect area are combined with the defect prior knowledge to fine-tune the difficult samples of the image of the workpiece to be tested. The image channels of the image of the workpiece to be tested are augmented based on the target residual flow model to enhance the texture structure anomaly features and output the workpiece defect localization image. The knowledge-guided edge refinement network includes an encoder, a decoder, and a target residual flow model. The encoder includes a semantic feature extractor and a parallel dilated convolution module. The parallel dilated convolution module includes a 1×1 convolutional layer, 3×3 convolutional layers with different dilation rates, and an average pooling layer. The decoder includes a first convolutional layer and a second convolutional layer.

2. The knowledge-guided, finely adjustable workpiece defect location method according to claim 1, characterized in that, The step of preprocessing the normal image set of the workpiece to generate a false defect image set includes: Generating noisy images based on Berlin noise; The noisy image is fused with the normal image set of the workpiece using the Poisson fusion algorithm to obtain the false defect image set of the workpiece.

3. The knowledge-guided, finely adjustable workpiece defect location method according to claim 1, characterized in that, The step of training the initial residual flow model using the normal image set of the workpiece to determine the target residual flow model includes: The normal sample image of the workpiece is input into a preset ImageNet pre-trained model for feature extraction, and the normal image features are output. The initial residual flow model is trained using the normal image features, and the target residual flow model is output.

4. The knowledge-guided, finely adjustable workpiece defect location method according to claim 1, characterized in that, The process of acquiring multiple images of difficult-to-detect defects in workpieces and constructing a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each image includes: Multiple images of difficult-to-detect defects in workpieces are acquired, and features are extracted from each of the images using a preset feature extractor to obtain multiple difficult-to-detect image features. The true label binary mask of each workpiece's difficult-to-detect defect image is multiplied element-wise with the corresponding difficult-to-detect image features, and then global pooling is performed to output multiple difficult-to-detect defect region image features. The image features of each of the aforementioned difficult-to-detect defect regions are stitched together to construct a knowledge base for difficult-to-detect defects.

5. The knowledge-guided, finely adjustable workpiece defect location method according to claim 1, characterized in that, When a workpiece image to be tested is received, the process involves determining the image features of the suspected defect area and prior defect knowledge of the workpiece image based on the workpiece image and the difficult-to-detect defect knowledge base, including: When a workpiece image to be tested is received, the image features to be tested are extracted using a preset feature extractor. A first similarity matrix is ​​constructed based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and global max pooling is performed on the first similarity matrix to generate a first similar image. The first similarity image is multiplied element-wise and global pooled with the features of the image to be tested to output the suspected defect area image features of the workpiece image to be tested. A second similarity matrix is ​​constructed using the cosine similarity between the suspected defect region image features and each of the difficult-to-detect image features, and global max pooling is performed on the second similarity matrix to obtain a second similarity map; If the intersection of the real label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, then the first similarity image is used as the defect prior knowledge.

6. The knowledge-guided, fine-tunable workpiece defect location method according to claim 1, characterized in that, The process involves using the target knowledge-guided edge refinement network, combining the image features of the suspected defect region with the prior defect knowledge to fine-tune the difficult samples of the workpiece image under test, and augmenting the image channels of the workpiece image under test based on the target residual flow model to enhance texture structure anomaly features, outputting a workpiece defect localization image, including: The image of the workpiece to be tested is input into a semantic feature extractor for feature extraction to obtain semantic features; The semantic features are sampled and fused in parallel using a parallel dilated convolution module to output multi-scale features. The semantic features and the multi-scale features are respectively input into the target residual flow model for feature processing and then normalized to determine the first anomaly score normalization map and the second anomaly score normalization map. The semantic features are multiplied element-wise with the first anomaly score normalization map, and the multi-scale features are multiplied element-wise with the second anomaly score normalization map, respectively, to output the first weighted feature and the second weighted feature; The semantic features, the first anomaly score normalization map, and the first weighted features are concatenated by channels to obtain the first enhanced features; The second enhanced feature is generated by channel splicing using the multi-scale feature, the second anomaly score normalization map, and the second weighted feature; After concatenating the semantic features, the prior knowledge of defects, and the image features of the suspected defect region through channels, the first attention feature is determined by adding it element-by-element to the first enhancement feature. The prior knowledge of the defect is added element-wise to the second enhancement feature to determine the second attention feature; After the first attention feature and the second attention feature are extracted and fused by the decoder, they are added element by element to the prior knowledge of the defect to output the workpiece defect location image.

7. The knowledge-guided, fine-tunable workpiece defect location method according to claim 6, characterized in that, The step of extracting and fusing the first and second attention features through a decoder, and then adding them element-by-element to the prior knowledge of the defect to output a workpiece defect location image includes: After performing convolution operations on the second attention features based on the first convolutional layer, upsampling is performed to output the detection features; After the first attention feature is convolved by the first convolutional layer, it is concatenated with the detection feature to determine the fused feature. After performing convolution calculation on the fused features using a second convolutional layer and then upsampling, the features are added element-wise with the prior knowledge of the defects to output a workpiece defect location image.

8. A knowledge-guided, finely adjustable workpiece defect location device, characterized in that, include: The image processing module is used to acquire a set of normal images of the workpiece and perform image preprocessing on the set of normal images to generate a set of false defect images of the workpiece. The residual flow model training module is used to train the initial residual flow model using the normal image set of the workpiece, and to determine the target residual flow model. The edge refinement network training module is used to construct an initial knowledge-guided edge refinement network based on the target residual flow model, and to train the initial knowledge-guided edge refinement network using the normal image set and the pseudo-defect image set of the workpiece, so as to determine the target knowledge-guided edge refinement network. The knowledge base construction module is used to acquire multiple images of difficult-to-detect defects in workpieces and construct a knowledge base of difficult-to-detect defects based on the difficult-to-detect image features of each of the workpiece images. The prior knowledge determination module is used to determine the image features of the suspected defect area and the prior knowledge of defects in the image of the workpiece to be tested based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects when the image of the workpiece to be tested is received. The workpiece defect localization module is used to guide the edge refinement network through the target knowledge, use the image features of the suspected defect area combined with the defect prior knowledge to fine-tune the difficult samples of the workpiece image to be tested, and augment the image channel of the workpiece image to be tested based on the target residual flow model to enhance the texture structure anomaly features, and output the workpiece defect localization image. The knowledge-guided edge refinement network includes an encoder, a decoder, and a target residual flow model. The encoder includes a semantic feature extractor and a parallel dilated convolution module. The parallel dilated convolution module includes a 1×1 convolutional layer, 3×3 convolutional layers with different dilation rates, and an average pooling layer. The decoder includes a first convolutional layer and a second convolutional layer.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any one of claims 1-7.

Images