A product defect detection method and apparatus

Abstract

The embodiment of the application provides a product defect detection method, which can realize general training-free product defect detection based on image prompts. The method comprises the following steps: first, obtaining a sample image; then, extracting features from the sample image by using a mixed feature extractor to obtain sample features; then, performing image reconstruction based on the extracted sample features and normal image prompts in an image prompt library to obtain reconstructed features, that is, performing feature reconstruction by combining normal images and sample features; comparing the sample features with the reconstructed features to obtain a detection result; then, obtaining an inspection result for at least one detection result, wherein the inspection result is used to indicate whether the detection result is correct; and if the inspection result indicates that the detection result is incorrect, updating the image prompt library according to the sample image.

Description

Technical Field

[0001] This application relates to the field of product quality inspection, and in particular to a method and apparatus for detecting product defects. Background Technology

[0002] Machine vision-based defect detection refers to using machine vision algorithms to determine whether a given product image contains defects (flaws) and to identify their locations. This shift in quality inspection methods from manual labor and traditional machine vision to AI-powered machine vision significantly improves detection accuracy while reducing reliance on human intervention.

[0003] Commonly used defect detection algorithms typically extract features using feature extractors and determine and locate defects based on differences in feature representation between normal and abnormal locations. However, training these models with large amounts of data is required for different products, resulting in very low practicality for actual deployment. Therefore, improving the generalization ability of product defect detection applications has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a product defect detection method that can achieve general, training-free product defect detection based on image prompts.

[0005] In view of the above, in a first aspect, this application provides a product defect detection method, comprising: first, acquiring a sample image, which includes a sample to be detected, the specific product type of which can be determined according to the actual application scenario; then, extracting features from the sample image using a hybrid feature extractor to obtain sample features; then, performing image reconstruction based on the extracted sample features and normal image prompts in an image prompt library to obtain reconstructed features, that is, combining normal image and sample features for feature reconstruction; comparing the sample features and the reconstructed features to obtain a detection result. Typically, if a defect exists in the sample, there will be a difference between the features reconstructed based on the normal image and sample features and the sample features. Therefore, the presence of a defect in the sample can be identified by comparing the reconstructed features and the sample features, such as the presence of flaws, foreign objects, or deformation; then, obtaining a sampling inspection result for at least one detection result, which is used to indicate whether the detection result is correct; if the sampling inspection result indicates that the detection result is incorrect, updating the image prompt library according to the sample image.

[0006] Therefore, in this embodiment, the image prompt library can be updated by sampling the detection results. Only by updating the image prompt library can the product defect detection capability corresponding to the newly added images be increased without retraining. There is no need to retrain the model parameters, which has a very strong adaptability and generalization to application scenarios.

[0007] In one possible implementation, the aforementioned image reconstruction based on sample features and normal image prompts in the image prompt library to obtain reconstructed features includes: extracting features from normal image prompts using a hybrid feature extractor to obtain prompt features; furthermore, if there are multiple normal image prompts, features can be extracted from multiple normal image prompts using a hybrid feature extractor to obtain multiple features, and these multiple features are encoded into representative features as prompt features; subsequently, the sample features and prompt features are input into a feature reconstructor to output reconstructed features. Therefore, in this embodiment, the same feature extractor can be used to extract features from both normal image prompts and sample images, thereby extracting features from the same dimension, making the extracted features more reflective of the differences between defective parts and normal image features.

[0008] In one possible implementation, the aforementioned feature reconstructor includes an encoder based on a cross-attention mechanism. The feature reconstructor encodes the input sample features and cue features based on the cross-attention mechanism to obtain reconstructed features. Therefore, in this embodiment, the cross-attention mechanism can be used to model the interaction relationship between mixed features. Normal image features can usually be reconstructed, while features of defective parts cannot be reconstructed. Therefore, the output defect response map, i.e., the reconstructed features, contains the defect response part, so that the defective part can be identified subsequently.

[0009] In one possible implementation, the aforementioned step of updating the image cue library based on sample images if the sampling inspection result indicates an incorrect detection result specifically includes: if the sampling inspection result indicates a false alarm or missed detection, then using the sample image as a normal image cue in the image cue library. Therefore, in this embodiment, for sample images that are false alarms or missed detections, the sample images can be directly added to the image cue library, enabling the model to increase its defect detection capability for newly added products without training model parameters.

[0010] In one possible implementation, the aforementioned hybrid feature extractor includes multiple feature extraction networks, which are used to extract features for different defect types respectively. Therefore, in this embodiment, corresponding feature extraction networks can be set for different types of products to adapt to products with different defect types, exhibiting very strong generalization ability.

[0011] In one possible implementation, the aforementioned hybrid feature extractor includes multiple expert feature extraction networks for feature extraction based on various product defects of different forms.

[0012] In one possible implementation, the aforementioned hybrid feature extractor includes multiple deep feature extraction networks, which are pre-trained using training data. Therefore, in this embodiment, the available feature extraction network structure can be selected according to the actual application scenario, exhibiting very strong generalization ability.

[0013] In one possible implementation, the aforementioned plurality of feature extraction networks include a selected feature extraction network and an unselected feature extraction network. The selected feature extraction network is the feature extraction network corresponding to the normal prompt images in the image prompt library, and the unselected feature extraction network is the feature extraction network that does not correspond to the normal prompt images in the image prompt library.

[0014] After updating the image cue library based on the sample images, the aforementioned method further includes: selecting a feature extraction network from the unselected feature extraction networks for the unselected sample images to obtain the selected feature extraction network corresponding to the sample images.

[0015] In this embodiment of the application, if a new image is added to the image prompt library during the online update process, the new image can be used as an abnormal image prompt to select an appropriate feature extraction network. This can enhance the detection capability of the hybrid feature extractor through the abnormal prompt image, without the need for training to adjust the network parameters of the feature extractor, and can adapt to the scenario of new products very efficiently.

[0016] Secondly, this application provides a product defect detection device, comprising:

[0017] The input module is used to acquire sample images;

[0018] The feature extraction module is used to extract features from the sample image through a hybrid feature extractor to obtain sample features;

[0019] The feature reconstruction module is used to reconstruct images based on sample features and normal image cues in the image cue library to obtain reconstructed features.

[0020] The detection module is used to compare the sample features with the reconstructed features to obtain the detection results;

[0021] The sampling inspection module is used to obtain the sampling inspection results for at least one test result. The sampling inspection results are used to indicate whether the test result is correct.

[0022] The update module is used to update the image prompt library based on the sample images if the sampling results indicate that the test results are incorrect.

[0023] The effects achieved by the second aspect or any optional implementation of the second aspect of this application can be referred to the description of the first aspect or any optional implementation of the first aspect above, and will not be repeated here.

[0024] In one possible implementation, the aforementioned feature reconstruction module is specifically used to: extract features from normal image prompts using a hybrid feature extractor to obtain prompt features; input the sample features and prompt features into a feature reconstructor to output reconstructed features.

[0025] In one possible implementation, the aforementioned feature reconstructor includes an encoder based on a cross-attention mechanism. The feature reconstructor is used to encode the input sample features and cue features based on the cross-attention mechanism to obtain reconstructed features.

[0026] In one possible implementation, the aforementioned update module, which updates if the sampling result indicates that the detection result is incorrect, is specifically used to: if the sampling result indicates that the detection result is a false alarm or a missed detection, then use the sample image as a normal image prompt in the image prompt library.

[0027] In one possible implementation, the aforementioned hybrid feature extractor includes multiple feature extraction networks for extracting features of different defect types.

[0028] In one possible implementation, the aforementioned hybrid feature extractor includes multiple expert feature extraction networks for feature extraction based on various product defects of different forms.

[0029] In one possible implementation, the aforementioned hybrid feature extractor includes multiple deep feature extraction networks, which are networks pre-trained using training data.

[0030] In one possible implementation, the aforementioned plurality of feature extraction networks include a selected feature extraction network and an unselected feature extraction network. The selected feature extraction network is the feature extraction network corresponding to the normal prompt images in the image prompt library, and the unselected feature extraction network is the feature extraction network that does not correspond to the normal prompt images in the image prompt library.

[0031] After updating the image cue library based on the sample images, the update module is also used to: select a feature extraction network from the unselected feature extraction networks for the unselected sample images, and obtain the selected feature extraction network corresponding to the sample images.

[0032] Thirdly, this application provides a product defect detection device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via circuits, and the processor calls program code in the memory to execute processing-related functions in the product defect detection method shown in any of the first aspects above. Optionally, the product defect detection device may be a chip.

[0033] Fourthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtains program instructions through the communication interface, the program instructions are executed by the processing unit, and the processing unit is used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0034] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0035] In a sixth aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0037] Figure 2 A schematic flowchart illustrating a product defect detection method provided in an embodiment of this application;

[0038] Figure 3 A schematic diagram of an initialization architecture provided for an embodiment of this application;

[0039] Figure 4 A schematic diagram of another initialization architecture provided for an embodiment of this application;

[0040] Figure 5 A schematic flowchart of another product defect detection method provided in an embodiment of this application;

[0041] Figure 6 A schematic diagram of a sample image provided in an embodiment of this application;

[0042] Figure 7 A schematic diagram of a defect in a sample image provided in an embodiment of this application;

[0043] Figure 8 This is a schematic diagram of the reconstruction process of a feature reconstructor provided in an embodiment of this application;

[0044] Figure 9 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0045] Figure 10 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0046] Figure 11A schematic diagram illustrating an online update process provided in this application embodiment;

[0047] Figure 12 This is a schematic diagram of a product defect detection result provided in an embodiment of this application;

[0048] Figure 13 This is another schematic diagram of product defect detection results provided in an embodiment of this application;

[0049] Figure 14 This is another schematic diagram of product defect detection results provided in an embodiment of this application;

[0050] Figure 15 This is a schematic diagram of the structure of a product defect detection device provided in an embodiment of this application;

[0051] Figure 16 This is a schematic diagram of another product defect detection device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0053] First, for ease of understanding, some terms or concepts involved in the methods provided in this application will be explained.

[0054] (1) Deep Neural Networks

[0055] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple intermediate layers. Based on the position of these layers, the internal neural network of a DNN can be divided into three categories: input layer, intermediate layers, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are considered intermediate layers, or hidden layers.

[0056] Although DNNs appear complex, each layer can be represented as a linear relational expression: in, It is the input vector. It is the output vector. is the offset vector, also known as the bias parameter; w is the weight matrix (also called coefficients); and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient w as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0057] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0058] It's important to note that the input layer does not have a W parameter. In deep neural networks, more intermediate layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0059] (2) Convolutional Neural Network

[0060] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0061] (3) transformer

[0062] A transformer structure is a feature extraction network that includes both an encoder and a decoder (classified as a convolutional neural network). Of course, in some cases, a transformer structure may not include an encoder but may include a decoder.

[0063] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0064] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0065] For example, a Transformer layer structure may include an attention network and a feedforward network module. Taking natural language processing as an example, the attention network obtains corresponding weight values ​​by calculating the relevance between words based on the attention mechanism, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, residual layers (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0066] (4) Attention mechanism

[0067] Attention mechanisms can quickly extract important features from sparse data. They provide an effective modeling approach for capturing global contextual information through QKV (Queries, Keys, Values). Assuming the input is Q(query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0068] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0069]

[0070] Where d is the dimension of matrix Q,K.

[0071] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0072] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, has the advantage of global focus; it can obtain global spatial information about the feature map through simple lookups and assignments.

[0073] (5) Multi-head attention mechanism (MHA)

[0074] Given the same set of queries, keys, and values, the desired model is to learn different behaviors based on the same attention mechanism, and then combine these different behaviors as knowledge, such as capturing dependencies of various ranges within a sequence (e.g., short-range and long-range dependencies). Therefore, the attention mechanism combines different representation subspaces of queries, keys, and values. Compared to self-attention models, multi-head attention models increase the number of heads. For example, Q, K, and V first pass through a linear layer, then are input into scaled dot-product attention, repeated h times (i.e., multiple heads, each time calculating one head, with no parameter sharing between heads; the parameters W for the linear transformation of Q, K, and V are different each time). The results of the h-times scaled dot-product attention are then concatenated, and a final linear transformation (linear layer) is performed to obtain the result of the multi-head attention.

[0075] Machine vision-based defect detection refers to using machine vision algorithms to determine whether a given product image contains defects (flaws) and to identify their locations. This shift in quality inspection methods from manual labor and traditional machine vision to AI-powered machine vision significantly improves detection accuracy while reducing reliance on human intervention.

[0076] Existing defect detection algorithms typically rely on features extracted by feature extractors to determine and locate defects based on differences in feature representation between normal and abnormal locations. The challenge lies in the rapid product iteration cycles and high requirements for model changeover efficiency. When the image imaging, product layout, or image appearance of a new product in the same scenario changes, existing solutions need to be retrained to detect the new product. Furthermore, it is difficult to collect a sufficient amount of comprehensive defect data offline. When a product encounters new degrees or types of defects, existing solutions need to be retrained to detect these new defects.

[0077] For example, an existing feature-matching-based approach obtains a defect response map by directly comparing features of normal and abnormal images. It utilizes the feature extraction capabilities of a CNN pre-trained on the public ImageNet dataset, first compressing normal features into a memory, and then detecting and locating anomalies by comparing the features of the input image with the normal features stored in memory. However, this approach lacks learnable parameters; both the feature extractor and feature matching module are fixed, making it impossible to adjust for different detection scenarios, resulting in insufficient detection capability.

[0078] For example, an existing reconstruction-based approach uses normal samples to transfer knowledge from the teacher model to the student model. Since the student model has not received knowledge from anomalous samples, the reconstructed features of the anomalous samples differ from those of the teacher model, enabling the discovery and localization of defects. However, its problems lie in poor generality. Specific product information is stored in the network, making it difficult to adapt to products not covered during training; information leakage is also prone to occur, with features at the defect location being reconstructed, leading to the inability to detect defects.

[0079] For example, an existing method based on a multimodal large model aims to achieve general defect detection. However, the generalization ability of this method relies on the versatility of CLIP (a multimodal pre-trained model). When the feature extractor cannot reflect the differences between normal and abnormal regions, or when the text prompt does not contain the defect, defect detection becomes difficult. Image feature matching, like PatchCore (an existing product anomaly detection solution), is based on nearest neighbor search and lacks learnable parameters, making it difficult to handle complex situations. Furthermore, the models are relatively large and complex, making deployment in actual production difficult.

[0080] Therefore, embodiments of this application provide a product defect detection method that can perform general, training-free product defect detection based on image prompts.

[0081] The method provided in this application can be applied to various product testing scenarios, specifically to the detection of defects or other anomalies in various industrial products, such as the detection of textile products with various textures, packaged circuits, packaged chips, or other industrial products. It can be used to detect the quality of products during the production process, thereby screening out products that meet the requirements.

[0082] The method provided in this application embodiment can be deployed on the product production line or in the cloud.

[0083] In one possible scenario, the method provided in this application embodiment can be deployed in a product production line. For example, an electronic device can be installed at the end of the production line, and the method provided in this application embodiment can be deployed in the electronic device to scan product images. Using the method provided in this application embodiment, an image cue library can be used to identify whether the product has defects or abnormalities. Users can perform online sampling of the detection results to identify false positives or false negatives, and use the images of the false positives or false negatives to update the image cue library. This allows the model to store images of products that have been falsely detected or falsely detected, improving the detection accuracy for products that have already been falsely detected or falsely detected. Even if the product is updated, only the image cue library needs to be updated; there is no need to retrain the model.

[0084] In another possible scenario, the method provided in this application embodiment can also be deployed in the cloud. Users can send product images to the cloud, which uses an image cue library to identify whether the product has defects or anomalies and provides feedback on the detection results to the user. Users can perform online sampling of the detection results to identify false positives or false negatives, and use these images to update the image cue library. This allows the model to store images of products that have been falsely detected or missed, improving the detection accuracy for products that have already been falsely detected or missed. Even if the product is updated, only the image cue library needs to be updated; the model does not need to be retrained.

[0085] For example, in the scenario of textile fabric defect detection, the pattern of fabric is iterated quickly, the defect morphology varies greatly, and different customers have different detection requirements. The method provided in this application embodiment can be trained offline for a specific scenario and generalized online to the detection of new patterns and new defects without training. Different detection standards can be achieved through a user-defined image prompt library.

[0086] For example, in advanced packaging defect detection scenarios, chip design patterns can be used as initial image templates, and new image prompts can be added to adapt to and generalize different detection requirements in different production environments and different parts. The general training-free product defect detection method provided in this application can be used without the need for customized detection models.

[0087] Exemplary examples show that the application scenarios of the methods provided in this application can be as follows: Figure 1 As shown. The method provided in this application is deployed in an electronic device, using the acquired sample image as input to the electronic device, or directly acquiring the sample image using the electronic device, and outputting the detection result, such as displaying the detection result on a screen, which can be used to indicate whether there are defects in the sample.

[0088] Users can perform random checks on the detection results. When false alarms or missed detections are found, the image prompt library in the electronic device can be updated online using the sample images of the false alarms or missed detections. This allows the electronic device to output more accurate detection results when it detects the sample images again.

[0089] The method flow provided in the embodiments of this application is described below.

[0090] See Figure 2 The following is a flowchart illustrating a product defect detection method provided in this application embodiment.

[0091] 201. Obtain sample images.

[0092] The sample image can be an image containing a sample, specifically an image acquired by an image sensor or a received image.

[0093] Typically, the sample images collected in different application scenarios may also be different. For example, in the scenario of defect detection of textile products, the sample image may include the textile products that need to be defect detected; in the scenario of component product detection, such as screws or nuts, the sample image may include the component products that need to be defect detected; in the scenario of packaged chip detection, the sample image may include images of packaged chips, etc.

[0094] 202. Extract features from the sample image using a hybrid feature extractor to obtain sample features.

[0095] This hybrid feature extractor can include multiple feature extraction networks to extract features from the input sample image and output sample features. These multiple feature extraction networks can be used for feature extraction based on various defect morphologies. Defect morphologies can be categorized based on dimensions such as length, width, or texture to extract features from different defect morphologies. Correspondingly, the output sample features can also include multiple features extracted based on different defect morphologies, thereby filtering out potential defect morphology dimensions from multiple defect morphologies.

[0096] The feature extraction network can be a user-defined network or a network trained using training data, allowing for a more flexible implementation of the feature extraction network.

[0097] In one possible implementation, the hybrid feature extractor includes multiple expert feature extraction networks. These networks are used to extract features based on various types of product defects, and are typically determined based on expert experience or historical data. Therefore, in this embodiment, the feature extraction network can be manually configured and can be started with little or no training data.

[0098] In one possible implementation, the hybrid feature extractor includes multiple deep feature extraction networks, which are pre-trained using training data. Therefore, in this embodiment, the network structure in the hybrid feature extractor can be selected according to the actual application scenario, adapting to various different situations.

[0099] In addition, in this embodiment of the application, an image prompt library is also deployed, which may include normal image prompts. Normal image prompts represent images of at least one type of product, and the feature distributions corresponding to different types of products may be different.

[0100] Optionally, when extracting features from a sample image using a hybrid feature extractor, the sample image and the normal image cue can be used as inputs to the hybrid feature extractor, outputting sample features and cue features. In this embodiment, when the hybrid feature extractor extracts features from a sample image, the same feature extractor can be used to extract features from both the normal image cue and the sample image. Thus, by using the same feature extraction method, when the two types of input data have the same or similar distributions, two features with the same or similar distributions can be extracted. Conversely, when the two types of input data have different distributions, two features with different distributions can be extracted, facilitating the reconstruction of the differing regions during subsequent feature reconstruction.

[0101] This can be understood as follows: the method provided in this application embodiment can be deployed in an image prompt library, which may include image prompts for one or more types of products. The products in the image prompts are defect-free and can therefore be called normal image prompts. In the feature extractor, a corresponding feature extraction network can be selected for different products. For example, a first feature extraction network can be selected for the first normal image prompt to extract features based on the feature distribution of the product in the first normal image prompt; a second feature extraction network can be selected for the second normal image prompt to extract features based on the feature distribution of the product in the second normal image prompt, and so on. That is, it can be assumed that the feature extraction network has the ability to identify normal and abnormal regions in an image based on the features extracted from the normal image prompt. When there is a difference between the features extracted by the same feature extraction network for the normal image prompt and the input sample image, it can be considered that there are abnormal region features in the sample image. Therefore, for the feature extractor, no training is required; the difference between the features extracted by the feature extractor for the normal image prompt and the sample image can be used to subsequently identify whether there are defects in the sample image through feature reconstruction.

[0102] For feature extraction networks not selected in the feature extractor, they can be used during online updates. Newly added images can be used as anomaly alerts, allowing for the selection of a feature extraction network corresponding to the anomaly alert from the unselected networks. Therefore, when new samples are added or missed detections occur, no additional model training is required; a suitable feature extraction network can be directly selected for product defect detection. In other words, the detection capability of the hybrid feature extractor can be enhanced through anomaly alert images without the need for training to adjust the network parameters.

[0103] 203. Based on the sample features and normal image cues in the image cue library, perform image reconstruction to obtain reconstructed features.

[0104] The sample features and normal image cues from the image cue library can be input into the feature reconstructor to output reconstructed features.

[0105] Among them, normal image prompts are images containing product features. The feature reconstructor can reconstruct sample features based on the guidance of normal image prompts and output reconstructed features.

[0106] In one possible implementation, features can be extracted from normal image cues. The sample features and the features of the normal image cues are then input into a feature reconstructor, which outputs reconstructed features. Specifically, there can be one or more normal image cues. When multiple normal image cues exist, features can be extracted from them to obtain multiple features. These multiple features are then encoded into representative features. These representative features are then used as extraction features and input into the feature reconstructor for feature reconstruction, resulting in reconstructed features that better reflect the differences.

[0107] Optionally, the feature reconstructor can employ an encoder based on a cross-attention mechanism. That is, the feature reconstructor can reconstruct the input sample features and normal image cue features based on a cross-attention mechanism, outputting reconstructed features. Specifically, the features of the normal image cue can be encoded to obtain a cue code. The feature reconstructor can use a cross-attention mechanism to obtain the similarity between the cue code and the sample features corresponding to the same position, and update the feature value at that position based on this similarity as a query feature. Therefore, in this embodiment, feature reconstruction can be performed based on a cross-attention mechanism, achieving more accurate feature reconstruction based on normal image cue.

[0108] Alternatively, the feature reconstructor can also be implemented using other neural networks, such as CNNs. CNNs can be used to extract features from normal image cues and sample features respectively, and then further fuse the extracted features to output reconstructed features, which can achieve feature reconstruction more flexibly.

[0109] 204. Output the detection results.

[0110] After obtaining the sample features, the sample features can be compared with the reconstructed features. Based on the difference between the sample features and the reconstructed features, or whether there is a difference between the sample features and the reconstructed features, the detection result is obtained. This detection result can indicate whether there are defects or flaws in the input sample image.

[0111] For example, optionally, the sample features and reconstructed features can be compared element by element. When the number of elements with differences is greater than a first threshold, a detection result indicating that there is a defect in the sample image can be output. Alternatively, the sample features and reconstructed features can be compared according to a specific size of region granularity. When there are differences between one or more regions, a detection result indicating that there is a defect in the sample image can be output. Alternatively, the similarity between the sample features and reconstructed features can be calculated. When the similarity is less than a second threshold, a detection result indicating that there is a defect in the sample image can be output. The appropriate comparison method can be selected according to the actual application scenario.

[0112] 205. Obtain the sampling results based on the test results.

[0113] The sampling inspection result can be used to indicate whether the detection result is correct. For example, it may indicate that a normal sample image is detected as defective, or that there are false positives or false negatives. Specifically, the sampling inspection result can be input by the user, or it can be generated based on a model trained with a large amount of data related to the sample. This application will exemplify the example of the sampling inspection result being determined based on user input.

[0114] For example, an interactive interface can be provided for users. After obtaining the detection results, the results can be displayed to the user on the interactive interface. Sample images can also be displayed to the user, and the user can select whether the detection results are correct on the interactive interface.

[0115] After obtaining the sampling results, it can be determined whether the sampling results indicate that the test results are correct. If the test results are correct, the test results can continue to be output. If the test results are incorrect, step 206 can be executed, that is, the image prompt library is updated according to the sample images.

[0116] 206. Update the image cue library based on the sample images.

[0117] In some scenarios, if the sampling results indicate that the test results are uncertain, the sample images can be used to update the image cue library.

[0118] For example, if the detection results indicate that a sample image has been falsely reported or missed, such as identifying a sample image without defects as a sample image with defects, the sample image is added to the image prompt library as a normal image prompt, or all sample images of a certain type are identified as images with defects (that is, the image prompt library does not include product images of that type), then the falsely reported or missed images can be added to the image prompt library as normal image prompts.

[0119] Therefore, in this embodiment, random sampling can be used to check for missed detections or false alarms, and the sampling results can be used to update the image cue library. This enables the model to detect defects in samples that have been falsely reported or missed, thus enriching the model's product defect detection capabilities by updating the image cue library online, achieving training-free product defect detection. For example, when the product layout changes or the detection environment (such as lighting) changes, the method provided in this embodiment does not require changing the model parameters; only the image cue library needs to be updated.

[0120] Furthermore, the aforementioned feature extraction networks include selected feature extraction networks and unselected feature extraction networks. The selected feature extraction networks are those corresponding to normal prompt images in the image prompt library, while the unselected feature extraction networks are those not corresponding to normal prompt images in the image prompt library. After updating the image prompt library, feature extraction networks can be selected from unselected feature extraction networks for sample images to obtain selected feature extraction networks corresponding to sample images. These selected feature extraction networks can then be used to extract features of possible defect morphologies in the sample image, thereby enabling the feature extractor to detect defects in the product corresponding to the sample image.

[0121] Therefore, in this embodiment, based on the hybrid feature extraction using abnormal image cues, offline training utilizes normal image cues to reconstruct features using a full-scale hybrid feature extractor. Online inference incrementally updates the hybrid feature extractor using abnormal image cues, without further training of the feature reconstructor. For anomaly detection based on normal image cues, normal image features are compressed into representative features, and these representative features are used to reconstruct input image features. Differences are calculated by comparing the input features to obtain a defect response map that can indicate the presence or absence of a defect.

[0122] The foregoing has described the method flow provided in the embodiments of this application. The following will further describe the method provided in the embodiments of this application in more detail in conjunction with specific application scenarios.

[0123] The method steps provided in this application embodiment can be divided into an initialization stage, an online inference stage, and an online update stage. The process of each stage is described below.

[0124] I. Initialization

[0125] For example, the architecture that needs to be initialized during the initialization phase is such as... Figure 3 As shown, this can include initializing the image cue library, as well as model building and pre-training.

[0126] 1. Image cue library

[0127] First, before performing online inference, an image cue library can be initialized. This image cue library can include at least one normal image cue, which can be used to guide the feature reconstructor to perform feature extraction and feature reconstruction based on multiple types of products. Thus, under the guidance of normal images, even if there are defects in the input product to be detected, the reconstructed features that differ from the features of the product to be detected can be reconstructed.

[0128] During initialization, normal image hints can be added to the image hint library, allowing product defect detection capabilities to be initiated with less data. Normal image hints are images of products without defects.

[0129] 2. Model Building and Pre-training

[0130] The model in this application embodiment may include a hybrid feature extractor and a feature reconstructor.

[0131] Specifically, the structure of the hybrid feature extractor and the feature reconstructor can be deployed according to the actual application scenario.

[0132] In one possible implementation, the hybrid feature extractor can employ a set of expert feature extractors, including multiple CNN filters or filters of other structures selected for different defect morphologies. For defect morphologies, the regions can be divided according to the length, width, or texture of areas where defects may exist or image regions of interest, thereby allowing for the targeted selection of feature extractors for feature extraction based on the defect morphology.

[0133] Accordingly, for the feature reconstructor, it can be deployed based on the maximum and minimum values ​​of the features encoded by the feature task, providing statistical information on the normal image response in different expert filters, and reconstructing features based on the input features encoded by the task.

[0134] Accordingly, when pre-training the model using a hybrid expert feature extractor and a feature reconstructor, the parameters of the hybrid expert feature extractor are specified so that the feature reconstructor can be adjusted offline using the training data, thereby completing the model initialization.

[0135] In another possible implementation, a hybrid deep feature extraction network and a feature reconstructor can be used. The hybrid feature extractor can employ a set of deep feature extraction networks, which can be pre-trained using in-scene or out-of-scene data. The feature reconstructor is a learnable feature reconstructor, with normal image cue features p + The task is encoded as maximizing the minimum distance feature set. As p + Representative characteristics. Based on task coding. For input image features p in Perform feature reconstruction MHCA(Query, Key, Value) is a multi-head cross-attention mechanism, and Refiner(·) is a local detail optimization improver to adapt to normal variations that may cause differences in image performance. The feature reconstructor is trained using normal images from both within and outside the domain, eliminating the need for defective data labeled with elastic deformation or illumination changes.

[0136] Furthermore, during the overall pre-training process of the model, such as Figure 4 As shown, the parameters of the hybrid feature extractor can be fixed, meaning the model parameters do not change during the pre-training process. For products with different defect morphologies, an appropriate feature extraction network can be selected. During the initialization phase, the user can choose an appropriate feature extraction network for each defect morphology. A portion of the output features of the hybrid feature extractor is randomly masked, and the masked features are input into the feature reconstructor. The defect response map is obtained based on the reconstructed features output by the feature reconstructor, and the feature reconstructor is updated based on the defect response. That is, in this embodiment, the preset hybrid feature extractor can distinguish between normal and abnormal regions, and its extracted features are stitched together pixel by pixel, eliminating the need for parameter adjustment of the hybrid feature extractor. During training, a portion of the hybrid feature extractor is randomly masked, and during inference, the selected feature extractor can be directly used for feature extraction. The unselected feature extraction networks can be used when a new image is added as an abnormality warning image, for example, using the new image as an abnormality warning image and selecting the corresponding feature extraction network from the unselected feature extraction networks.

[0137] For details on the online inference and online update phases, please refer to [link / reference]. Figure 5 The present application provides a schematic flowchart of another product defect detection method.

[0138] II. Online Reasoning

[0139] 501. Obtain the sample to be tested.

[0140] In different application scenarios, the type of sample to be detected may be different. In this embodiment of the application, taking the defect detection scenario of textile products as an example, the sample to be detected may be an image of the textile product that has been collected, that is, the aforementioned sample image.

[0141] For example, the image of the sample to be detected can be as follows: Figure 6 As shown, different textile products may exhibit different textures, such as checks, diamonds, thick yarns, or calendering. Defects that may occur in each product can be identified as follows: Figure 7 As shown, it can be expressed as white dots, stripes, wood grain patterns, or dyed vertical stripes, etc.

[0142] 502. Hybrid feature extraction based on abnormal image cues.

[0143] The process involves inputting both normal image prompts from the image prompt library and the samples to be detected into a hybrid feature extractor, which outputs the prompt features corresponding to normal avatar prompts and the sample features corresponding to the samples to be detected.

[0144] It should be understood that the abnormal image cues referred to here are those that utilize the characteristics of a specific distribution to select the corresponding feature extractor, thereby enabling the direct detection of defects in newly added product types without the need for model training.

[0145] Typically, a hybrid feature extractor includes multiple feature extraction networks, each capable of extracting features of a specific defect type. If a new defect type is introduced, a feature extraction network adapted to the new defect type can be selected from the previously unselected networks for feature extraction, eliminating the need for separate network training. This improves the solution's adaptability to new products, especially in scenarios with rapid product updates, enabling faster learning of new product defect detection capabilities.

[0146] 503. Anomaly detection based on normal image cues.

[0147] Both sample features and normal image cues are input into the feature reconstructor, which outputs reconstructed features.

[0148] 504. Output the detection results.

[0149] By comparing the reconstructed features with the features of the sample to be detected, the detection results can be output based on the comparison results.

[0150] The hybrid feature extractor and feature reconstructor can adopt different model structures, and the corresponding online inference process may also be different. The following will introduce them in conjunction with steps 502 to 504.

[0151] Specifically, the feature reconstruction process of the feature reconstructor can be as follows: Figure 8 As shown, the feature reconstructor includes multiple network layers, such as... Figure 8 The diagram shows Layers 1 to M. Its input consists of an image of the sample to be detected and representative features extracted from normal image cues, and its output is a reconstructed image. For example, N normal image cues and the input sample image are processed by a hybrid feature extractor to obtain corresponding features p. + and p in Normal image suggests feature p + Representative features will be obtained through task encoding and compression. Feature reconstructors can be used to reconstruct based on representative features. Features p of the sample image to be detected in Outputting reconstructed features can speed up inference and reduce memory usage.

[0152] Feature reconstructors can model the interaction relationships between mixed features and utilize representative features. For input image features p in Reconstruction is performed to obtain reconstructed features p rec ,because It only contains normal image features, the original feature p in The abnormal regions could not be reconstructed by comparing p rec and p in The defect response diagram is obtained.

[0153] Implementation Method 1: Expert Feature Extractor + Feature Recombinator

[0154] Taking the defect detection scenario of woven chemical fiber fabrics as an example, such as Figure 9 As shown.

[0155] For different types of defects, corresponding CNN filters are set, with normal image prompts and sample images to be detected as inputs. The normal image prompts can include a set of product images containing various types of product feature distributions, which can be used to guide the CNN filters to filter the image to be detected based on the input normal sample images, thereby extracting features that fit each product feature distribution.

[0156] like Figure 9 As shown, for ease of comparison, normal sample images and defective sample images can be input into the feature extractor respectively. The feature extractor then outputs features with various distributions through multiple CNN filters, such as... Figure 9 Features 1, 2, and 3 correspond to the normal sample image and the sample image to be detected, respectively. Features 1, 2, and 3 corresponding to the normal sample image are also the aforementioned cue features, which can be used as representative features of the product. Features 1, 2, and 3 corresponding to the image to be detected are also the aforementioned sample features, which can be used to represent the features in the image to be detected. Among these, in the distribution category of feature 3, due to defects in the sample to be detected, there are certain differences between the features of the normal sample image and the sample image to be detected in the same CNN output features.

[0157] Subsequently, features 1, 2, and 3 corresponding to the normal sample image and the image to be detected, respectively, are input into the feature reconstructor, and the reconstructed features are output. Specifically, for different CNN filters, when performing feature reconstruction, the maximum or minimum value of the input encoded as a feature can be used as statistical information of its response in different CNN filters to output the reconstructed features.

[0158] The reconstructed features are then compared with the features of the image to be detected to obtain the detection results.

[0159] In this embodiment, an expert feature extractor can be used as a hybrid feature extractor. The feature extractor can be constructed based on historical experience, which can reduce the need for training data or eliminate the need for model training, thereby reducing the model training cost.

[0160] Implementation Method 2: Deep Feature Extraction Network + Feature Recombinator

[0161] For example, taking the defect detection scenario of lace fabric as an example, such as Figure 10 As shown, the hybrid feature extractor employs multiple deep feature extraction networks, which are pre-trained using in-scene or out-of-scene data. The feature reconstructor is trained using normal images from both within and outside the scene, eliminating the need for labeled defect data.

[0162] Among them, the normal image prompt feature p + Encoded as a feature set that maximizes minimum distance As p + Representative characteristics. Based on task coding. The image features p of the input sample to be detected in Feature reconstruction can be performed, and the reconstruction process can be represented as follows: MHCA(Query,Key,Value) is a multi-head cross-attention mechanism, and Refiner(·) is a local detail optimization improver to adapt to normal changes that cause differences in image performance, such as elastic deformation and illumination changes. The improver is an optional module.

[0163] This can be understood as a feature reconstructor modeling the interaction relationships between mixed features, utilizing representative features. For input image features p in Reconstruction is performed to obtain reconstructed features p rec ,because It only contains normal image features, the original feature p in The abnormal regions could not be reconstructed by comparing p rec and p in The defect response diagram is obtained.

[0164] In this embodiment, the hybrid feature extractor can employ a deep feature extraction network. Given training data, multiple networks suitable for feature extraction can be obtained through training, allowing users to flexibly choose a hybrid feature extractor deployment method that best suits their specific environment. Therefore, the method provided in this embodiment has very strong generalization capabilities.

[0165] III. Online Updates

[0166] 505. Online random inspection.

[0167] After completing the online inference, the detection results output by the online inference can be further sampled online to verify their accuracy.

[0168] Optionally, users can conduct random sampling during product defect detection, or conduct sampling periodically, or conduct sampling when the product yield is high or low, or conduct sampling when new products are produced, etc. The specific online sampling can be determined according to the actual application scenario, and this application does not limit it in this regard.

[0169] For example, the interactive interface can display the image of the input sample to be detected and the corresponding detection results to the user. The user can choose whether the detection results are accurate, and the sampling results can be obtained based on the user's feedback.

[0170] 506. Add false alarm and missed detection images to the image prompt library.

[0171] In cases of false alarms or missed detections, the image cue library can be updated using the product images associated with those false alarms or missed detections. For example, the product images corresponding to the false alarms or missed detections can be added to the image cue library as normal cue images for use in the next online inference process.

[0172] Specifically, the online update process can be as follows: Figure 11 As shown, for example, when adding a new missed image, the unselected feature extraction networks from multiple feature extraction networks are considered as candidates and added to the existing feature extraction network. The reconstructed defect response map of the missed image is calculated using the features of the normal image prompt, and simultaneously, the reconstructed defect response map of the normal image under the normal image prompt is calculated. The feature extraction network corresponding to the largest or relatively large relative defect response is selected and added to the feature extraction network used in the online inference process to enhance the network's ability to detect various types of product defects. That is, for a newly added missed image, both the missed image and the normal prompt image are input into the hybrid feature extractor. Feature reconstruction is performed separately based on the sample features output by each unselected feature extraction network in the hybrid feature extractor, and the distance between the defect response maps of each feature extraction network is calculated. The feature extraction network with the largest or relatively large defect region distance is selected as the feature extraction network corresponding to the missed image. In other words, as normal prompt images are added to the image prompt library, the number of selected feature extraction networks in the hybrid feature extractor can be updated synchronously to detect new types of product defects using a feature extraction network that does not require retraining.

[0173] Therefore, in this embodiment, the image cue library can be updated online. During online inference, the updated image cue library can be used for product defect detection without model training. This allows for the addition of new products, false positives, or missed detections at a lower cost, increasing the range of products for which the model can detect defects. For example, in the scenario of adding new products, images of the new products can be added to the image cue library to enable defect detection of the new products.

[0174] For example, the following section will introduce a specific defect detection scenario using a concrete application example.

[0175] See Figure 12 This application provides a schematic diagram of a product defect detection result.

[0176] The system can add image cues to define detection types. When a normal image is used as the cue, both abnormal vertical lines and dot patterns can be detected. When a vertical line image is added as a cue image, dot patterns can be detected normally, but horizontal lines are no longer detected. Further addition of dot pattern images will result in both types no longer being detected. This embodiment of the application can utilize custom image cues to achieve different detection types without training, thereby reducing overdetection.

[0177] See Figure 13 This application provides another schematic diagram of product defect detection results.

[0178] Different levels of defect warning images can be added to define the detection level. When a normal image is used as the warning image, both mild and moderate dirt can be detected. After adding a mildly dirty image to the warning image, moderate dirt can still be detected normally, but mild dirt is no longer detected. Adding more moderately dirty images will cause both to be no longer detected. This embodiment of the application can utilize custom image warnings to achieve different detection levels without training, thereby reducing overdetection.

[0179] See Figure 14 This application provides another schematic diagram of product defect detection results.

[0180] This method enhances the feature extraction capabilities of defect-indicating images, thereby detecting previously undetectable defects. In the "Improved Defect Response" column, defect regions in the input image now respond in the score map. The method provided in this application can leverage custom anomaly image cues to enhance the detection capabilities of a hybrid feature extractor without requiring training, thus reducing missed detections.

[0181] In this embodiment, normal image prompts can be used to define the current defect, detection task, and detection criteria. Image prompt updates are supported, and detection criteria can be updated without training, enabling scene-level large-scale replication. This reduces training costs, increases deployment efficiency, and enhances detection controllability. Furthermore, incremental detection capabilities are supported, achieving general and efficient defect detection without relying on large, comprehensive models, thus reducing algorithm deployment and maintenance costs.

[0182] The foregoing has described the method flow provided in the embodiments of this application. The following describes the structure of the apparatus for executing the foregoing method flow.

[0183] See Figure 15 This application provides a schematic diagram of the structure of a product defect detection device, which includes:

[0184] Input module 1501 is used to acquire sample images;

[0185] Feature extraction module 1502 is used to extract features from sample images through a hybrid feature extractor to obtain sample features;

[0186] The feature reconstruction module 1503 is used to reconstruct images based on sample features and normal image cues in the image cue library to obtain reconstructed features;

[0187] The detection module 1504 is used to compare the sample features with the reconstructed features to obtain the detection results;

[0188] The sampling inspection module 1505 is used to obtain the sampling inspection result for at least one test result, and the sampling inspection result is used to indicate whether the test result is correct.

[0189] The update module 1506 is used to update the image prompt library based on the sample image if the sampling results indicate that the detection results are incorrect.

[0190] In one possible implementation, the aforementioned feature reconstruction module 1503 is specifically used to: extract features from normal image prompts through a hybrid feature extractor to obtain prompt features; input the sample features and prompt features into a feature reconstructor and output reconstructed features.

[0191] In one possible implementation, the aforementioned feature reconstructor includes an encoder based on a cross-attention mechanism. The feature reconstructor is used to encode the input sample features and cue features based on the cross-attention mechanism to obtain reconstructed features.

[0192] In one possible implementation, the aforementioned update module 1506, which updates if the sampling result indicates that the detection result is incorrect, is specifically used to: if the sampling result indicates that the detection result is a false alarm or a missed detection, then use the sample image as a normal image prompt in the image prompt library.

[0193] In one possible implementation, the aforementioned hybrid feature extractor includes multiple feature extraction networks for extracting features of different defect types.

[0194] In one possible implementation, the aforementioned hybrid feature extractor includes multiple expert feature extraction networks for feature extraction based on various product defects of different forms.

[0195] In one possible implementation, the aforementioned hybrid feature extractor includes multiple deep feature extraction networks, which are networks pre-trained using training data.

[0196] In one possible implementation, the aforementioned plurality of feature extraction networks include a selected feature extraction network and an unselected feature extraction network. The selected feature extraction network is the feature extraction network corresponding to the normal prompt images in the image prompt library, and the unselected feature extraction network is the feature extraction network that does not correspond to the normal prompt images in the image prompt library.

[0197] After updating the image cue library based on the sample images, the update module 1506 is also used to: select a feature extraction network from the unselected feature extraction networks for the unselected sample images, and obtain the selected feature extraction network corresponding to the sample images.

[0198] like Figure 16 The diagram shown is a hardware structure schematic of a product defect detection device 160 provided in an embodiment of this application. This product defect detection device 160 can be used to implement the aforementioned... Figures 2 to 14 The steps of the method in the text.

[0199] Figure 16 The product defect detection device 160 shown may include: a processor 1601, a memory 1602, a communication interface 1603, and a bus 1604. The processor 1601, the memory 1602, and the communication interface 1603 can be connected via the bus 1604.

[0200] The processor 1601 is the control center of the product defect detection device 160. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0201] As an example, processor 1601 may include one or more CPUs, and may also include other processors, such as... Figure 16 The CPU, NPU, or GPU shown are examples of such devices.

[0202] The memory 1602 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0203] In one possible implementation, the memory 1602 can exist independently of the processor 1601. The memory 1602 can be connected to the processor 1601 via a bus 1604, and is used to store data, instructions, or program code. When the processor 1601 calls and executes the instructions or program code stored in the memory 1602, it can implement the methods provided in the embodiments of this application, for example, Figures 2 to 14 The method shown.

[0204] In another possible implementation, the memory 1602 can also be integrated with the processor 1601.

[0205] The communication interface 1603 is used for the product defect detection device 160 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 1603 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0206] The 1604 bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, and control bus, etc. For ease of representation, Figure 16 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0207] It should be pointed out that, Figure 16 The structure shown does not constitute a limitation on the product defect detection device 160, except... Figure 16 In addition to the components shown, the product defect detection device 160 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a data quantization device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0209] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0210] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform the aforementioned... Figures 2 to 14 All or part of the steps in the method described in the embodiments shown.

[0211] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0212] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0213] The target sensing device or target sensing device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to perform the above-mentioned operations. Figures 2-14The method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register, cache, etc. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.

[0214] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0215] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0217] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0218] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0219] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A product defect detection method, characterized in that, include Acquire sample images; Features are extracted from the sample images using a hybrid feature extractor to obtain sample features; Image reconstruction is performed based on the sample features and normal image prompts in the image prompt library to obtain reconstructed features; The sample features are compared with the reconstructed features to obtain the detection results; Obtain a sampling result for at least one of the test results, the sampling result being used to indicate whether the test result is correct; If the sampling results indicate that the detection results are incorrect, the image cue library is updated based on the sample images.

2. The method according to claim 1, characterized in that, The process of reconstructing images based on the sample features and normal image prompts in the image prompt library to obtain reconstructed features includes: The hybrid feature extractor extracts features from the normal image prompt to obtain prompt features; The sample features and the cue features are input into the feature reconstructor, and the reconstructed features are output.

3. The method according to claim 2, characterized in that, The feature reconstructor includes an encoder based on a cross-attention mechanism. The feature reconstructor is used to encode the input sample features and the cue features based on the cross-attention mechanism to obtain the reconstructed features.

4. The method according to any one of claims 1-3, characterized in that, If the sampling result indicates that the detection result is incorrect, then updating the image cue library based on the sample image includes: If the sampling results indicate that the detection result is a false alarm or a missed detection, then the sample image will be used as a normal image prompt in the image prompt library.

5. The method according to any one of claims 1-4, wherein the hybrid feature extractor includes multiple feature extraction networks, the multiple feature extraction networks being used to extract features of different defect types respectively.

6. The method according to claim 5, characterized in that, The hybrid feature extractor includes multiple expert feature extraction networks, which are used to extract features based on various types of product defects.

7. The method according to claim 5, characterized in that, The hybrid feature extractor includes multiple deep feature extraction networks, which are networks pre-trained using training data.

8. The method according to any one of claims 5-7, characterized in that, The plurality of feature extraction networks include a selected feature extraction network and an unselected feature extraction network. The selected feature extraction network is the feature extraction network that corresponds to the normal prompt image in the image prompt library, and the unselected feature extraction network is the feature extraction network that does not correspond to the normal prompt image in the image prompt library. After updating the image cue library based on the sample images, the method further includes: From the unselected feature extraction networks, a feature extraction network is selected for the sample image to obtain the selected feature extraction network corresponding to the sample image.

9. A product defect detection device, characterized in that, include: The input module is used to acquire sample images; The feature extraction module is used to extract features from the sample image using a hybrid feature extractor to obtain sample features; The feature reconstruction module is used to reconstruct images based on the sample features and normal image prompts in the image prompt library to obtain reconstructed features; The detection module is used to compare the sample features with the reconstructed features to obtain the detection result; A sampling inspection module is used to obtain sampling inspection results for at least one of the test results, the sampling inspection results being used to indicate whether the test results are correct; An update module is used to update the image prompt library based on the sample image if the sampling result indicates that the detection result is incorrect.

10. The apparatus according to claim 9, characterized in that, The feature reconstruction module is specifically used for: The hybrid feature extractor extracts features from the normal image prompt to obtain prompt features; The sample features and the cue features are input into the feature reconstructor, and the reconstructed features are output.

11. The apparatus according to claim 10, characterized in that, The feature reconstructor includes an encoder based on a cross-attention mechanism. The feature reconstructor is used to encode the input sample features and the cue features based on the cross-attention mechanism to obtain the reconstructed features.

12. The apparatus according to any one of claims 9-11, characterized in that, If the sampling result indicates that the test result is incorrect, the update module is specifically used for: If the sampling results indicate that the detection result is a false alarm or a missed detection, then the sample image will be used as a normal image prompt in the image prompt library.

13. The apparatus according to any one of claims 9-12, wherein the hybrid feature extractor comprises a plurality of feature extraction networks, the plurality of feature extraction networks being used to extract features of different defect types respectively.

14. The apparatus according to claim 13, characterized in that, The hybrid feature extractor includes multiple expert feature extraction networks, which are used to extract features based on various types of product defects.

15. The apparatus according to claim 13, characterized in that, The hybrid feature extractor includes multiple deep feature extraction networks, which are networks pre-trained using training data.

16. The apparatus according to any one of claims 12-15, characterized in that, The plurality of feature extraction networks include a selected feature extraction network and an unselected feature extraction network. The selected feature extraction network is the feature extraction network that corresponds to the normal prompt image in the image prompt library, and the unselected feature extraction network is the feature extraction network that does not correspond to the normal prompt image in the image prompt library. After updating the image cue library based on the sample images, the update module is further configured to: From the unselected feature extraction networks, a feature extraction network is selected for the sample image to obtain the selected feature extraction network corresponding to the sample image.

17. A product defect detection device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1-8.

18. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images