Workpiece defect positioning method and device capable of being finely adjusted based on knowledge guidance

Abstract

The invention discloses a fine-adjustable workpiece defect positioning method and device based on knowledge guidance, and relates to the technical field of industrial detection, and the method comprises the steps: employing a workpiece normal image set to train and determine a target residual flow model; constructing an initial knowledge-guided edge refining network based on the target residual flow model, and training and determining a target knowledge-guided edge refining network by adopting the workpiece normal image set and the workpiece pseudo-defect image set; based on the difficult-to-detect image features of the difficult-to-detect image of each workpiece, constructing a difficult-to-detect defect knowledge base; determining suspected defect area image features and defect priori knowledge according to the to-be-detected workpiece image and the difficult-to-detect defect knowledge base; and guiding an edge refining network through the target knowledge, performing difficult sample fine adjustment on the to-be-detected workpiece image by combining the suspected defect area image features with the defect priori knowledge, performing texture structure abnormal feature enhancement on the to-be-detected workpiece image based on the target residual flow model, and outputting a workpiece defect positioning image. And the workpiece defect positioning accuracy is integrally improved.

Description

Technical Field

[0001] The invention relates to the technical field of industrial detection, and in particular to a knowledge-guided fine-tunable workpiece defect positioning method. Background Art

[0002] In industrial production, product defects may occur in the production processes such as raw materials, rough processing, cutting, and grinding. Therefore, workpiece surface defect detection is an important part of product quality control. Due to the various causes of defects, the form of defects is unpredictable and even new defects may appear.

[0003] Traditional workpiece surface defect detection methods require a large amount of defect anomaly data and annotation costs. However, in the gradually mature industrial production lines, defect anomaly data are becoming increasingly rare, and complex processes make it more likely for new defects to appear. In order to solve the above-mentioned problems such as insufficient defect anomaly data and difficulty in detecting new anomalies, the existing technology detects defect anomaly data in a set of homogeneous images based on pre-training. However, there is a domain adaptation problem and a lack of adaptability of the target domain data set, resulting in low accuracy in workpiece defect detection. Summary of the invention

[0004] The present invention provides a knowledge-guided fine-tunable workpiece defect positioning method and device, which solves the technical problem that the existing pre-training-based workpiece surface defect detection method lacks the target domain data set adaptability, resulting in low workpiece defect detection accuracy.

[0005] A first aspect of the present invention provides a knowledge-guided fine-tunable workpiece defect location method, comprising:

[0006] Acquire a workpiece normal image set, and perform image preprocessing on the workpiece normal image set to generate a workpiece pseudo-defect image set;

[0007] Using the workpiece normal image set to train an initial residual flow model to determine a target residual flow model;

[0008] An initial knowledge-guided edge refinement network is constructed based on the target residual flow model, and the normal workpiece image set and the pseudo-defect workpiece image set are used to train the initial knowledge-guided edge refinement network to determine a target knowledge-guided edge refinement network;

[0009] Acquire multiple images of difficult-to-detect defects of workpieces, and build a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects of workpieces;

[0010] When receiving the image of the workpiece to be tested, determining the image features of the suspected defect area of ​​the image of the workpiece to be tested and the defect priori knowledge according to the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects;

[0011] The target knowledge is used to guide the edge refinement network, and the image features of the suspected defect area are combined with the defect prior knowledge to perform fine-tuning on the difficult samples of the workpiece image to be tested. The image channel of the workpiece image to be tested is augmented based on the target residual flow model to enhance the texture structure abnormality features, and the workpiece defect positioning image is output.

[0012] Optionally, performing image preprocessing on the workpiece normal image set to generate the workpiece pseudo-defect image set includes:

[0013] Generate a noisy image based on Perlin noise;

[0014] The noise image is fused with the workpiece normal image set by using a Poisson fusion algorithm to obtain a workpiece pseudo-defect image set.

[0015] Optionally, the adopting the workpiece normal image set to train an initial residual flow model to determine a target residual flow model includes:

[0016] Inputting the normal sample image of the workpiece into a preset ImageNet pre-trained model for feature extraction, and outputting normal image features;

[0017] The normal image features are used to train an initial residual flow model, and a target residual flow model is output.

[0018] Optionally, the acquiring of a plurality of images of difficult-to-detect defects of workpieces and constructing a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects of workpieces includes:

[0019] Acquire multiple images of workpieces with difficult-to-detect defects, and extract features from each of the images of workpieces with difficult-to-detect defects using a preset feature extractor to obtain multiple difficult-to-detect image features;

[0020] The real label binary mask of each of the hard-to-detect defect images of the workpiece is multiplied element by element with the corresponding hard-to-detect image features, and then global pooling is performed to output a plurality of hard-to-detect defect area image features accordingly;

[0021] The image features of each of the difficult-to-detect defect areas are channel-joined to construct a knowledge base of difficult-to-detect defects.

[0022] Optionally, when receiving the workpiece image to be tested, determining the image features of the suspected defect area and defect priori knowledge of the workpiece image to be tested according to the workpiece image to be tested and the difficult-to-detect defect knowledge base, includes:

[0023] When receiving the image of the workpiece to be measured, extracting the image features of the workpiece to be measured by a preset feature extractor;

[0024] Constructing a first similarity matrix based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and performing global maximum pooling on the first similarity matrix to generate a first similarity image;

[0025] Performing element-by-element multiplication and global pooling on the first similarity image and the features of the image to be tested, and outputting image features of the suspected defect area of ​​the image of the workpiece to be tested;

[0026] A second similarity matrix is ​​constructed by using the cosine similarity between the image features of the suspected defect area and each of the difficult-to-detect image features, and the second similarity matrix is ​​subjected to global maximum pooling to obtain a second similarity graph;

[0027] If the intersection of the real label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, the first similarity map is used as defect prior knowledge.

[0028] Optionally, the target knowledge guided edge refinement network includes a semantic feature extractor, a parallel hole convolution module, a target residual flow model and a decoder; the target knowledge guided edge refinement network uses the image features of the suspected defect area combined with the defect prior knowledge to perform difficult sample fine-tuning on the image of the workpiece to be tested, and based on the target residual flow model, the image channel of the workpiece image to be tested is augmented to enhance the texture structure abnormality feature, and the workpiece defect positioning image is output, including:

[0029] Inputting the image of the workpiece to be tested into a semantic feature extractor for feature extraction to obtain semantic features;

[0030] The semantic features are sampled and fused in parallel through a parallel atrous convolution module to output multi-scale features;

[0031] Inputting the semantic features and the multi-scale features into the target residual flow model respectively for feature processing and then normalizing them to determine a first anomaly score normalized map and a second anomaly score normalized map;

[0032] Respectively performing element-by-element multiplication on the semantic feature and the first anomaly score normalized map, and on the multi-scale feature and the second anomaly score normalized map, and outputting a first weighted feature and a second weighted feature;

[0033] Channel-joining the semantic feature, the first anomaly score normalized map, and the first weighted feature to obtain a first enhanced feature;

[0034] Using the multi-scale feature, the second anomaly score normalized map, and the second weighted feature to perform channel splicing to generate a second enhanced feature;

[0035] After channel splicing the semantic feature, the defect prior knowledge and the image feature of the suspected defect area, the semantic feature, the defect prior knowledge and the image feature of the suspected defect area are added element by element with the first enhanced feature to determine a first attention feature;

[0036] Adding the defect prior knowledge and the second enhanced feature element by element to determine a second attention feature;

[0037] After the decoder extracts and fuses the first attention feature and the second attention feature, they are added element by element with the defect prior knowledge to output a workpiece defect positioning image.

[0038] Optionally, the decoder includes a first convolution layer and a second convolution layer; after extracting and fusing the first attention feature and the second attention feature through the decoder, adding them element by element with the defect prior knowledge, and outputting a workpiece defect location image, including:

[0039] Performing a convolution operation on the second attention feature based on the first convolution layer and then upsampling it, outputting a detection feature;

[0040] After convolution processing is performed on the first attention feature through a first convolution layer, channel splicing is performed with the detection feature to determine a fusion feature;

[0041] The second convolutional layer is used to perform convolution calculation on the fusion feature and then upsample it, and then it is added element by element with the defect prior knowledge to output a workpiece defect positioning image.

[0042] A second aspect of the present invention provides a knowledge-guided fine-tunable workpiece defect locating device, comprising:

[0043] An image processing module is used to obtain a workpiece normal image set, and perform image preprocessing on the workpiece normal image set to generate a workpiece pseudo-defect image set;

[0044] A residual flow model training module is used to train an initial residual flow model using the normal image set of the workpiece to determine a target residual flow model;

[0045] An edge refinement network training module is used to construct an initial knowledge-guided edge refinement network based on the target residual flow model, and train the initial knowledge-guided edge refinement network using the workpiece normal image set and the workpiece pseudo-defect image set to determine a target knowledge-guided edge refinement network;

[0046] A knowledge base construction module is used to obtain multiple images of difficult-to-detect defects in workpieces, and to construct a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects in workpieces;

[0047] A priori knowledge determination module is used to determine image features of suspected defect areas and prior knowledge of defects in the image of the workpiece to be tested based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects when receiving the image of the workpiece to be tested;

[0048] The workpiece defect localization module is used to guide the edge refinement network through the target knowledge, use the image features of the suspected defect area in combination with the defect prior knowledge to perform difficult sample fine-tuning on the workpiece image to be tested, and based on the target residual flow model, augment the image channel of the workpiece image to be tested to enhance the texture structure abnormality features, and output a workpiece defect localization image.

[0049] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any one of the above items.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the knowledge-guided fine-tunable workpiece defect location method as described in any one of the above items.

[0051] It can be seen from the above technical solutions that the present invention has the following advantages:

[0052] The above technical scheme of the present invention provides a workpiece defect localization method that can be fine-tuned based on knowledge guidance, including: obtaining a set of normal images of the workpiece, performing image preprocessing on the normal image set of the workpiece to generate a set of pseudo-defect images of the workpiece; using the normal image set of the workpiece to train the initial residual flow model, and determining the target residual flow model; constructing an initial knowledge-guided edge refinement network based on the target residual flow model, and using the normal image set of the workpiece and the pseudo-defect image set of the workpiece to train the initial knowledge-guided edge refinement network, and determining the target knowledge-guided edge refinement network; obtaining multiple workpiece difficult-to-detect defect images, and constructing a difficult-to-detect defect knowledge base based on the difficult-to-detect image features of each workpiece difficult-to-detect defect image; when receiving the workpiece image to be tested, determining the image features of the suspected defect area and defect prior knowledge of the workpiece image to be tested according to the workpiece image to be tested and the difficult-to-detect defect knowledge base; through the target knowledge-guided edge refinement network, using the image features of the suspected defect area combined with the defect prior knowledge to fine-tune the difficult samples of the workpiece image to be tested, and based on the target residual flow model, augmenting the image channel of the workpiece image to be tested to enhance the texture structure abnormality features, and outputting the workpiece defect localization image. Based on the above scheme, a residual flow model is trained using a set of easily accessible normal images of workpieces, and a set of pseudo-defect images is generated based on image preprocessing to simulate the rich variations of real defects in industrial scenes. The normal image set of workpieces is used as a training set to train a knowledge-guided edge refinement network, which avoids the need for manual annotation of samples and effectively reduces labor costs. It is helpful to achieve the positioning and segmentation of real defects. The difficult-to-detect image features extracted from the difficult-to-detect defect images of the workpieces and the defect prior knowledge determined by the images of the workpieces to be tested are used to fine-tune the guided knowledge to guide the edge refinement network to focus on the difficult-to-detect defect features to reduce missed detections and false detections, thereby improving the accuracy of workpiece defect positioning as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A flowchart of a method for locating workpiece defects that can be fine-tuned based on knowledge guidance is provided in an embodiment of the present invention;

[0055] Figure 2 A framework diagram of a knowledge-guided edge refinement network provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of a knowledge-guided feature enhancement structure of a knowledge-guided edge thinning network provided by an embodiment of the present invention;

[0057] Figure 4 A structural block diagram of a knowledge-guided fine-tunable workpiece defect locating device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The embodiments of the present invention provide a knowledge-guided fine-tunable workpiece defect localization method and device, which are used to solve the technical problem that the existing pre-training-based workpiece surface defect detection method lacks the target domain data set adaptability, resulting in low workpiece defect detection accuracy.

[0059] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] See also Figure 1 , Figure 1 A flowchart of the steps of a knowledge-guided and fine-tunable workpiece defect location method provided in an embodiment of the present invention.

[0061] The present invention provides a knowledge-guided fine-tunable workpiece defect location method, comprising:

[0062] Step 101: Acquire a workpiece normal image set, and perform image preprocessing on the workpiece normal image set to generate a workpiece pseudo-defect image set.

[0063] It should be noted that the workpiece normal image set contains multiple workpiece normal images, that is, workpiece images without defects. After the workpiece normal images are collected, they can be pre-processed such as center cropping before forming the workpiece normal image set, while the workpiece pseudo-defect image set contains multiple workpiece pseudo-defect images, which refers to images with simulated defects added to the workpiece normal images through computer generation and other methods when there are no actual defects in the workpiece normal images.

[0064] Step 101 includes the following sub-steps:

[0065] Generate a noisy image based on Perlin noise;

[0066] The noise image is fused with the normal image set of the workpiece by Poisson fusion algorithm to obtain the workpiece pseudo-defect image set.

[0067] It should be noted that a noise image with random shapes and diverse texture patterns is generated by the Perlin noise algorithm, and the Poisson fusion algorithm noise image is fused with each normal image of the workpiece in the normal image set based on the Poisson fusion algorithm to generate multiple workpiece pseudo-defect images with seamless edges and constitute a workpiece pseudo-defect image set.

[0068] Step 102: Use a set of normal workpiece images to train an initial residual flow model to determine a target residual flow model.

[0069] Step 102 includes the following sub-steps:

[0070] Input the normal sample image of the workpiece into the preset ImageNet pre-trained model for feature extraction, and output the normal image features;

[0071] The normal image features are used to train the initial residual flow model and output the target residual flow model.

[0072] It should be noted that using easily accessible normal images of workpieces to train the residual flow model can effectively reduce costs, and the residual flow model can distinguish between normal and abnormal images, and the abnormal images can be understood as images with defects.

[0073] Step 103: construct an initial knowledge-guided edge refinement network based on the target residual flow model, and train the initial knowledge-guided edge refinement network using a workpiece normal image set and a workpiece pseudo-defect image set to determine a target knowledge-guided edge refinement network.

[0074] It should be noted that the knowledge-guided edge refinement network based on semantic segmentation and dilated convolution is used to detect defects in workpiece images, such as Figure 2 As shown in the figure, the knowledge-guided edge refinement network (KGERN for short) adopts the classic encoder-decoder architecture in the field of semantic segmentation, including an encoder, a decoder and a target residual flow model. The encoder includes a semantic feature extractor and a parallel atrous convolution module (ASPP). The parallel atrous convolution module performs feature processing through parallel 1×1 convolution layers, 3×3 convolution layers (atrous rate 6), 3×3 convolution layers (atrous rate 12), 3×3 convolution layers (atrous rate 18) and average pooling layers, and then fuses the output. The decoder includes the first convolution layer (1×1 convolution) and the second convolution layer (3×3 convolution).

[0075] In the network training stage, the normal workpiece image set and the workpiece pseudo-defect image set are used as the training set input to guide the edge refinement network to train the initial knowledge. The diverse pseudo-defects of the workpiece pseudo-defect image set indirectly improve the adaptability to unknown defect types in real detection scenarios, and the cross entropy loss is designed ( ) and Dice Loss ( ) acts as a network loss function to guide the network to finely segment the edges of abnormal areas. When the network loss function value converges, the target knowledge is obtained to guide the edge refinement network. Among them, the network loss function include:

[0076] ;

[0077] In the formula, is a hyperparameter, usually .

[0078] It can be understood that the specific calculation process of cross entropy loss and Dice Loss can be referred to the prior art and will not be repeated here.

[0079] Step 104 : acquiring a plurality of images of difficult-to-detect defects of workpieces, and constructing a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects of workpieces.

[0080] It should be noted that the workpiece difficult-to-detect defect images refer to workpiece images whose defects are relatively difficult to detect, and the difficult-to-detect defect knowledge base contains difficult-to-detect image features of multiple workpiece difficult-to-detect defect images.

[0081] Step 104 includes the following sub-steps:

[0082] S11, acquiring a plurality of workpiece difficult-to-detect defect images, and performing feature extraction on each workpiece difficult-to-detect defect image using a preset feature extractor to obtain a plurality of difficult-to-detect image features.

[0083] It should be noted that N images of workpieces with difficult-to-detect defects are used as reference images and input into the feature extractor to extract the difficult-to-detect image features of the reference images. ,The feature extractor can use the target knowledge to guide the semantic feature extractor in the edge ,refinement network, or can use other models that can extract image features without ,restriction.

[0084] S12, using the real label binary mask of each workpiece's difficult-to-detect defect image and the corresponding difficult-to-detect image feature to perform element-by-element multiplication and then perform global pooling, and correspondingly output multiple difficult-to-detect defect area image features.

[0085] It should be noted that the true label binary mask marks which pixels in the image belong to the defect area and the normal area. And the corresponding difficult-to-detect image features Multiply and perform global pooling operations to obtain multiple image features of difficult-to-detect defect areas with dimensions [1, C] .

[0086] S13, channel-joining the image features of each difficult-to-detect defect area to build a knowledge base of difficult-to-detect defects.

[0087] It should be noted that the image features of the difficult-to-detect defect areas of the difficult-to-detect defect images of each workpiece are spliced ​​to obtain a difficult-to-detect defect knowledge base with a dimension of [N, C].

[0088] Step 105 : when receiving the image of the workpiece to be tested, determining the image features of the suspected defect area of ​​the image of the workpiece to be tested and prior knowledge of defects according to the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects.

[0089] It should be noted that the workpiece image to be tested refers to the workpiece image that currently needs to be detected for defect location, and the image features of the suspected defect area corresponding to the workpiece image to be tested and the defect prior knowledge are determined through the difficult-to-detect defect knowledge base, which facilitates subsequent feature enhancement to better guide the target knowledge to guide the edge refinement network to focus on the difficult-to-detect defect features.

[0090] Step 105 includes the following sub-steps:

[0091] S21. When receiving the image of the workpiece to be measured, extracting the image features of the workpiece to be measured by a preset feature extractor.

[0092] It should be noted that for the workpiece image to be tested , extract the corresponding image features to be tested through the feature extractor , the feature extractor can use target knowledge to guide the semantic feature extractor in the edge refinement network, or it can use other models that can extract image features.

[0093] S22: construct a first similarity matrix based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and perform global maximum pooling on the first similarity matrix to generate a first similarity image.

[0094] It should be noted that the image features to be tested The image features at each coordinate are calculated once and compared with the image features of the difficult-to-detect defect area in the difficult-to-detect defect knowledge base. The cosine similarity of the first similarity matrix with the dimension of [N, H, W] is constructed by using multiple cosine similarities, and the first similarity matrix is ​​subjected to maximum pooling in the channel dimension to obtain the first similarity image with the dimension of [1, H, W]. The first similarity image represents the region in the image to be tested that is similar to the image features of the difficult-to-detect defect region.

[0095] S23, performing element-by-element multiplication and global pooling on the first similarity image and the features of the image to be tested, and outputting image features of the suspected defect area of ​​the image of the workpiece to be tested.

[0096] It should be noted that the first similarity image As a mask and the image features to be tested Multiply and perform global pooling to obtain the image features of the suspected defect area of ​​the workpiece image to be tested .

[0097] S24, constructing a second similarity matrix using the cosine similarity between the image features of the suspected defect area and the features of each difficult-to-detect image, and performing global maximum pooling on the second similarity matrix to obtain a second similarity graph.

[0098] It should be noted that the image features of the suspected defect area In turn, the difficult-to-detect image features of the difficult-to-detect defect images of each workpiece Calculate the cosine similarity and construct the second similarity matrix. Perform global maximum pooling on the second similarity matrix to obtain the second similarity image with dimensions [1, H, W]. .

[0099] S25. If the intersection of the real label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, the first similarity map is used as defect prior knowledge.

[0100] It should be noted that if the second similarity image The corresponding true label binary masks of multiple workpiece difficult-to-detect defect images associated with the difficult-to-detect defect knowledge base If the intersection between them is not empty, it means that the image features of the suspected defect area found in the workpiece image to be tested do meet the image features of the difficult-to-detect defect area of ​​the workpiece difficult-to-detect defect image. By designing a two-way verification method for defect features to ensure that the first similarity image obtained accurately describes the location of a specific category of defects, the first similarity image can be As defect prior knowledge for subsequent feature enhancement, otherwise Set to 0.

[0101] Step 106, guiding the edge refinement network through target knowledge, using the image features of the suspected defect area combined with defect prior knowledge to fine-tune the difficult samples of the workpiece image to be tested, and augmenting the image channel of the workpiece image to be tested based on the target residual flow model to enhance the texture structure abnormality features, and outputting the workpiece defect positioning image.

[0102] It should be noted that the target knowledge-guided edge refinement network includes a semantic feature extractor, a parallel hole convolution module, a target residual flow model and a decoder, and the workpiece defect location image maps the defect situation of the workpiece.

[0103] Step 106 includes the following sub-steps:

[0104] S31, inputting the image of the workpiece to be tested into a semantic feature extractor for feature extraction to obtain semantic features.

[0105] S32. Perform parallel sampling and fusion of semantic features through parallel atrous convolution modules to output multi-scale features.

[0106] S33, inputting the semantic features and the multi-scale features into the target residual flow model respectively for feature processing and then normalizing them to determine a first anomaly score normalized map and a second anomaly score normalized map.

[0107] It should be noted that after the semantic features and multi-scale features are respectively input into the target residual flow model, the first anomaly score map and the second anomaly score map are output accordingly, and the two anomaly score maps are normalized and their value range is limited to [0, 1], thereby obtaining the first anomaly score normalized map and the second anomaly score normalized map.

[0108] S34, performing element-by-element multiplication of the semantic feature and the first anomaly score normalization map, and the multi-scale feature and the second anomaly score normalization map, respectively, to output a first weighted feature and a second weighted feature.

[0109] It should be noted that the semantic features are multiplied element-by-element with the first anomaly score normalized map, and the multi-scale features are multiplied element-by-element with the second anomaly score normalized map. Since the score value corresponding to the abnormal area is larger, the feature representation of the abnormal area is retained after multiplication, while the score value corresponding to the normal area is smaller, thereby suppressing the redundant information of the image background.

[0110] S35. Channel-concatenate the semantic feature, the first abnormal score normalization map, and the first weighted feature to obtain a first enhanced feature.

[0111] S36. Use the multi-scale features, the second anomaly score normalization map, and the second weighted features to perform channel splicing to generate a second enhanced feature.

[0112] It should be noted that if Figure 3 As shown, the process of obtaining the first enhancement feature or the second enhancement feature can be referred to as follows:

[0113] ;

[0114] ;

[0115] In the formula, is a semantic feature or a multi-scale feature, is the dot product calculation, is the abnormality score graph, For normalization, for, is the channel dimension merging operation, To enhance the features.

[0116] S37, after channel-joining the semantic features, defect prior knowledge and the image features of the suspected defect area, they are added element-by-element with the first enhanced feature to determine the first attention feature.

[0117] S38. Add the defect prior knowledge and the second enhanced feature element by element to determine the second attention feature.

[0118] S39, after extracting and fusing the first attention feature and the second attention feature through the decoder, they are added element by element with the defect prior knowledge to output a workpiece defect location image.

[0119] The decoder includes a first convolutional layer and a second convolutional layer; sub-step S39 includes:

[0120] Based on the first convolutional layer, the second attention feature is convolved and then upsampled to output the detection feature;

[0121] After the first attention feature is convolved by the first convolutional layer, it is channel-joined with the detection feature to determine the fusion feature;

[0122] The second convolutional layer is used to perform convolution calculation on the fused features and then upsample them, and then they are added element by element with the defect prior knowledge to output the workpiece defect location image.

[0123] It should be noted that in order to realize the defect prior knowledge, that is, the first similarity image A deeper and more comprehensive fusion with the relevant image features of the workpiece image to be tested, setting feature fusion paths at different positions of the target knowledge guided edge refinement network, guiding the target knowledge guided edge refinement network to focus on the difficult-to-detect defect features at the intermediate feature layer, and adding the defect prior knowledge or the output result of the target knowledge guided edge refinement network with a certain weight element by element at the output end as a result post-processing means. At this time, the abnormal positioning result of the target knowledge guided edge refinement network and the difficult-to-detect defect positioning result complement each other, and the image feature guided fine-tuning is enhanced by using defect prior knowledge in a variety of ways without the need for retraining, thereby improving the defect positioning accuracy.

[0124] In an embodiment of the present invention, a residual flow model is trained using a set of normal images of workpieces that are easily available, a set of pseudo-defect images is generated based on image preprocessing to simulate the rich variations of real defects in industrial scenarios, and a knowledge-guided edge refinement network is trained in combination with the set of normal images of workpieces as a training set, thereby avoiding the need for manual annotation of samples and effectively reducing labor costs, and facilitating the positioning and segmentation of real defects. Prior knowledge of defects is determined by using difficult-to-detect image features extracted from images of difficult-to-detect defects of workpieces and the images of workpieces to be tested, and the edge refinement network is guided by fine-tuning the guided knowledge to focus on difficult-to-detect defect features to reduce missed detections and false detections. Overall, this method is well applicable to any type of abnormal defects in workpieces, has strong generalization performance, and improves the accuracy of workpiece defect positioning.

[0125] See also Figure 4 , Figure 4 A structural block diagram of a knowledge-guided fine-tunable workpiece defect locating device provided in an embodiment of the present invention.

[0126] The present invention provides a workpiece defect positioning device that can be fine-tuned based on knowledge guidance, comprising:

[0127] The image processing module 401 is used to obtain a workpiece normal image set, and perform image preprocessing on the workpiece normal image set to generate a workpiece pseudo-defect image set;

[0128] The residual flow model training module 402 is used to train the initial residual flow model using a normal workpiece image set to determine a target residual flow model;

[0129] The edge refinement network training module 403 is used to construct an initial knowledge-guided edge refinement network based on the target residual flow model, and train the initial knowledge-guided edge refinement network using a workpiece normal image set and a workpiece pseudo-defect image set to determine a target knowledge-guided edge refinement network;

[0130] A knowledge base construction module 404 is used to obtain multiple workpiece difficult-to-detect defect images and construct a difficult-to-detect defect knowledge base based on difficult-to-detect image features of each workpiece difficult-to-detect defect image;

[0131] A priori knowledge determination module 405 is used to determine image features of suspected defect areas and priori knowledge of defects in the image of the workpiece to be tested based on the image of the workpiece to be tested and a knowledge base of difficult-to-detect defects when receiving the image of the workpiece to be tested;

[0132] The workpiece defect localization module 406 is used to guide the edge refinement network through target knowledge, use the image features of the suspected defect area combined with the defect prior knowledge to fine-tune the difficult samples of the workpiece image to be tested, and augment the image channel of the workpiece image to be tested based on the target residual flow model to enhance the texture structure abnormality features, and output the workpiece defect localization image.

[0133] Furthermore, the image processing module 401 is specifically used for:

[0134] Generate a noisy image based on Perlin noise;

[0135] The noise image is fused with the normal image set of the workpiece by Poisson fusion algorithm to obtain the workpiece pseudo-defect image set.

[0136] Furthermore, the residual flow model training module 402 is specifically used for:

[0137] Input the normal sample image of the workpiece into the preset ImageNet pre-trained model for feature extraction, and output the normal image features;

[0138] The normal image features are used to train the initial residual flow model and output the target residual flow model.

[0139] Furthermore, the knowledge base construction module 404 is specifically used for:

[0140] Acquire multiple images of difficult-to-detect defects of workpieces, and extract features of each of the difficult-to-detect defect images of workpieces using a preset feature extractor to obtain multiple difficult-to-detect image features;

[0141] The binary mask of the real label of each workpiece's difficult-to-detect defect image is multiplied element by element with the corresponding difficult-to-detect image feature, and then global pooling is performed to output multiple difficult-to-detect defect area image features;

[0142] The image features of each difficult-to-detect defect area are channel-joined to build a knowledge base of difficult-to-detect defects.

[0143] Furthermore, the prior knowledge determination module 405 is specifically used for:

[0144] When receiving the image of the workpiece to be tested, extracting the image features of the workpiece to be tested through a preset feature extractor;

[0145] Constructing a first similarity matrix based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and performing global maximum pooling on the first similarity matrix to generate a first similarity image;

[0146] Performing element-by-element multiplication and global pooling on the first similarity image and the features of the image to be tested, and outputting image features of the suspected defect area of ​​the image of the workpiece to be tested;

[0147] A second similarity matrix is ​​constructed by using the cosine similarity between the image features of the suspected defect area and the features of each difficult-to-detect image, and the second similarity matrix is ​​subjected to global maximum pooling to obtain a second similarity graph;

[0148] If the intersection of the true label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, the first similarity map is used as defect prior knowledge.

[0149] Furthermore, the target knowledge guided edge refinement network includes a semantic feature extractor, a parallel hole convolution module, a target residual flow model and a decoder; the workpiece defect location module 406 is specifically used for:

[0150] The image of the workpiece to be tested is input into the semantic feature extractor for feature extraction to obtain semantic features;

[0151] The semantic features are sampled and fused in parallel through parallel atrous convolution modules to output multi-scale features.

[0152] The semantic features and the multi-scale features are respectively input into the target residual flow model for feature processing and then normalized to determine a first anomaly score normalized map and a second anomaly score normalized map;

[0153] Respectively perform element-by-element multiplication of the semantic feature and the first anomaly score normalization map, and the multi-scale feature and the second anomaly score normalization map, and output a first weighted feature and a second weighted feature;

[0154] Channel-concatenate the semantic feature, the first anomaly score normalized map, and the first weighted feature to obtain a first enhanced feature;

[0155] The multi-scale features, the second anomaly score normalization map and the second weighted features are used for channel splicing to generate the second enhanced features;

[0156] After channel concatenation of the semantic features, defect prior knowledge and image features of the suspected defect area, they are added element by element with the first enhanced feature to determine the first attention feature;

[0157] Determine a second attention feature by adding the defect prior knowledge to the second enhanced feature element by element;

[0158] After the decoder extracts and fuses the first attention feature and the second attention feature, they are added element by element with the defect prior knowledge to output the workpiece defect location image.

[0159] Furthermore, the decoder includes a first convolutional layer and a second convolutional layer; after the decoder extracts and fuses the first attention feature and the second attention feature, the first attention feature and the second attention feature are added element by element with the defect prior knowledge, and the workpiece defect location image is output, including:

[0160] Based on the first convolutional layer, the second attention feature is convolved and then upsampled to output the detection feature;

[0161] After the first attention feature is convolved by the first convolutional layer, it is channel-joined with the detection feature to determine the fusion feature;

[0162] The second convolutional layer is used to perform convolution calculation on the fused features and then upsample them, and then they are added element by element with the defect prior knowledge to output the workpiece defect location image.

[0163] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the knowledge-guided fine-tunable workpiece defect positioning method as described in any of the above embodiments.

[0164] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any of the above embodiments are implemented.

[0165] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the knowledge-guided fine-tunable workpiece defect location method as in any of the above embodiments.

[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0171] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge-guided fine-tunable workpiece defect location method, characterized in that: include: Acquire a workpiece normal image set, and perform image preprocessing on the workpiece normal image set to generate a workpiece pseudo-defect image set; Using the workpiece normal image set to train an initial residual flow model to determine a target residual flow model; An initial knowledge-guided edge refinement network is constructed based on the target residual flow model, and the normal workpiece image set and the pseudo-defect workpiece image set are used to train the initial knowledge-guided edge refinement network to determine a target knowledge-guided edge refinement network; Acquire multiple images of difficult-to-detect defects of workpieces, and build a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects of workpieces; When receiving the image of the workpiece to be tested, determining the image features of the suspected defect area of ​​the image of the workpiece to be tested and the defect priori knowledge according to the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects; The target knowledge is used to guide the edge refinement network, and the image features of the suspected defect area are combined with the defect prior knowledge to perform fine-tuning on the difficult samples of the workpiece image to be tested. The image channel of the workpiece image to be tested is augmented based on the target residual flow model to enhance the texture structure abnormality features, and the workpiece defect positioning image is output.

2. The knowledge-guided fine-tunable workpiece defect location method according to claim 1, characterized in that: The step of performing image preprocessing on the workpiece normal image set to generate the workpiece pseudo-defect image set includes: Generate a noisy image based on Perlin noise; The noise image is fused with the workpiece normal image set by using a Poisson fusion algorithm to obtain a workpiece pseudo-defect image set.

3. The knowledge-guided fine-tunable workpiece defect location method according to claim 1, characterized in that: The step of training the initial residual flow model using the normal image set of the workpiece to determine the target residual flow model includes: Inputting the normal sample image of the workpiece into a preset ImageNet pre-trained model for feature extraction, and outputting normal image features; The normal image features are used to train an initial residual flow model, and a target residual flow model is output.

4. The knowledge-guided fine-tunable workpiece defect location method according to claim 1, characterized in that: The step of acquiring a plurality of images of difficult-to-detect defects of workpieces and constructing a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects of workpieces comprises: Acquire multiple images of workpieces with difficult-to-detect defects, and extract features from each of the images of workpieces with difficult-to-detect defects using a preset feature extractor to obtain multiple difficult-to-detect image features; The real label binary mask of each of the hard-to-detect defect images of the workpiece is multiplied element by element with the corresponding hard-to-detect image features, and then global pooling is performed to output a plurality of hard-to-detect defect area image features accordingly; The image features of each of the difficult-to-detect defect areas are channel-joined to construct a knowledge base of difficult-to-detect defects.

5. The knowledge-guided fine-tunable workpiece defect location method according to claim 1, characterized in that: When receiving the workpiece image to be tested, determining the image features of the suspected defect area and defect prior knowledge of the workpiece image to be tested according to the workpiece image to be tested and the difficult-to-detect defect knowledge base, including: When receiving the image of the workpiece to be measured, extracting the image features of the workpiece to be measured by a preset feature extractor; Constructing a first similarity matrix based on the cosine similarity between the features of the image to be tested and the knowledge base of difficult-to-detect defects, and performing global maximum pooling on the first similarity matrix to generate a first similarity image; Performing element-by-element multiplication and global pooling on the first similarity image and the features of the image to be tested, and outputting image features of the suspected defect area of ​​the image of the workpiece to be tested; A second similarity matrix is ​​constructed by using the cosine similarity between the image features of the suspected defect area and each of the difficult-to-detect image features, and the second similarity matrix is ​​subjected to global maximum pooling to obtain a second similarity graph; If the intersection of the real label binary mask corresponding to the difficult-to-detect defect knowledge base and the second similarity map is not empty, the first similarity map is used as defect prior knowledge.

6. The knowledge-guided fine-tunable workpiece defect location method according to claim 1, characterized in that: The target knowledge guided edge refinement network includes a semantic feature extractor, a parallel hole convolution module, a target residual flow model and a decoder; the target knowledge guided edge refinement network uses the image features of the suspected defect area combined with the defect prior knowledge to perform difficult sample fine-tuning on the image of the workpiece to be tested, and based on the target residual flow model, the image channel of the workpiece to be tested is augmented to enhance the texture structure abnormality feature, and the workpiece defect positioning image is output, including: Inputting the image of the workpiece to be tested into a semantic feature extractor for feature extraction to obtain semantic features; The semantic features are sampled and fused in parallel through a parallel atrous convolution module to output multi-scale features; Inputting the semantic features and the multi-scale features into the target residual flow model respectively for feature processing and then normalizing them to determine a first anomaly score normalized map and a second anomaly score normalized map; Respectively performing element-by-element multiplication on the semantic feature and the first anomaly score normalized map, and on the multi-scale feature and the second anomaly score normalized map, and outputting a first weighted feature and a second weighted feature; Channel-joining the semantic feature, the first anomaly score normalized map, and the first weighted feature to obtain a first enhanced feature; Using the multi-scale feature, the second anomaly score normalized map, and the second weighted feature to perform channel splicing to generate a second enhanced feature; After channel splicing the semantic feature, the defect prior knowledge and the image feature of the suspected defect area, the semantic feature, the defect prior knowledge and the image feature of the suspected defect area are added element by element with the first enhanced feature to determine a first attention feature; Adding the defect prior knowledge and the second enhanced feature element by element to determine a second attention feature; After the decoder extracts and fuses the first attention feature and the second attention feature, they are added element by element with the defect prior knowledge to output a workpiece defect positioning image.

7. The knowledge-guided fine-tunable workpiece defect location method according to claim 6, characterized in that: The decoder comprises a first convolutional layer and a second convolutional layer; after extracting and fusing the first attention feature and the second attention feature through the decoder, the first attention feature and the second attention feature are added element by element with the defect prior knowledge to output a workpiece defect location image, including: Performing a convolution operation on the second attention feature based on the first convolution layer and then upsampling it, outputting a detection feature; After convolution processing is performed on the first attention feature through a first convolution layer, channel splicing is performed with the detection feature to determine a fusion feature; The second convolutional layer is used to perform convolution calculation on the fusion feature and then upsample it, and then it is added element by element with the defect prior knowledge to output a workpiece defect positioning image.

8. A knowledge-guided fine-tunable workpiece defect location device, characterized in that: include: An image processing module is used to obtain a workpiece normal image set, and perform image preprocessing on the workpiece normal image set to generate a workpiece pseudo-defect image set; A residual flow model training module is used to train an initial residual flow model using the normal image set of the workpiece to determine a target residual flow model; An edge refinement network training module is used to construct an initial knowledge-guided edge refinement network based on the target residual flow model, and train the initial knowledge-guided edge refinement network using the workpiece normal image set and the workpiece pseudo-defect image set to determine a target knowledge-guided edge refinement network; A knowledge base construction module is used to obtain multiple images of difficult-to-detect defects in workpieces, and to construct a knowledge base of difficult-to-detect defects based on difficult-to-detect image features of each of the images of difficult-to-detect defects in workpieces; A priori knowledge determination module is used to determine image features of suspected defect areas and prior knowledge of defects in the image of the workpiece to be tested based on the image of the workpiece to be tested and the knowledge base of difficult-to-detect defects when receiving the image of the workpiece to be tested; The workpiece defect localization module is used to guide the edge refinement network through the target knowledge, use the image features of the suspected defect area in combination with the defect prior knowledge to perform difficult sample fine-tuning on the workpiece image to be tested, and based on the target residual flow model, augment the image channel of the workpiece image to be tested to enhance the texture structure abnormality features, and output a workpiece defect localization image.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the knowledge-guided fine-tunable workpiece defect positioning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the knowledge-guided fine-tunable workpiece defect location method as described in any one of claims 1 to 7 are implemented.

Images