Industrial product defect detection method and apparatus, device, and medium

Abstract

Disclosed in the present application are an industrial product defect detection method and apparatus, a device, and a medium, which belong to the field of image processing, and can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and aided driving. The method comprises: acquiring a first product image and a second product image, the first product image being an image of a defect-free industrial product, and the second product image being an image of an industrial product to undergo detection; separately performing feature extraction on the first product image and the second product image to obtain a first image feature and a second image feature; merging the first image feature and the second image feature to obtain a first intermediate feature; inputting the first intermediate feature into a defect detection model to obtain an inference feature, the defect detection model being obtained by means of training by using preset images of articles having different appearances and modified images obtained after modifying the preset images; performing up-sampling on the inference feature to obtain a second intermediate feature; and, on the basis of the second intermediate feature, obtaining information of the position of a defect. The present application achieves cross-category defect detection by utilizing the inference capability of large models.

Description

Methods, devices, equipment and media for detecting defects in industrial products

[0001] This application claims priority to Chinese Patent Application No. 2023111553036, filed on September 8, 2023, entitled "Method, Apparatus, Equipment and Medium for Detecting Defects in Industrial Products", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and in particular to a method, apparatus, equipment and medium for detecting defects in industrial products.

[0003] Background of the Invention

[0004] In industrial production settings, industrial products often exhibit various defects for different reasons. These include uneven coloring of dyed fabrics, the presence of abnormal white / black spots, holes, and inconsistent patterns. Therefore, defect detection is necessary for manufactured industrial products.

[0005] In related technologies, defect detection is performed by establishing a feature library. This involves acquiring images of defect-free products, storing the features of these images in the feature library, and then acquiring an image of the product to be inspected. If the features of the image of the product to be inspected are not in the feature library, the product is considered to have a defect.

[0006] However, the feature library approach is only applicable to a single product category. When using the feature library to detect defects in another product category, the relevant technology requires retraining the model.

[0007] Summary of the Invention

[0008] This application provides a method, apparatus, equipment, and medium for detecting defects in industrial products, and provides a defect detection architecture based on a large model. This architecture utilizes the reasoning capabilities of the large model, enabling the overall architecture to detect defects across product categories. The technical solution includes the following:

[0009] According to one aspect of this application, a method for detecting defects in industrial products is provided, the method comprising the following steps.

[0010] The electronic device acquires a first product image and a second product image, wherein the first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected.

[0011] The electronic device performs feature extraction on the first product image to obtain first image features; and performs feature extraction on the second product image to obtain second image features.

[0012] The electronic device merges the first image feature and the second image feature to obtain a first intermediate feature; the first intermediate feature is input into a defect detection model to obtain inference features. The defect detection model is trained using preset images of various items with different appearances and modified images obtained by modifying the preset images.

[0013] The electronic device upsamples the inference features to obtain a second intermediate feature.

[0014] The electronic device obtains information about the location of the defect in the second product image based on the second intermediate feature.

[0015] According to another aspect of this application, an industrial product defect detection device is provided, the device comprising the following modules.

[0016] An acquisition module is used to acquire a first product image and a second product image. In some embodiments, the first product image and the second product image are images of products with the same or similar appearance. The first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected.

[0017] The feature extraction module is used to extract features from the first product image to obtain first image features; and to extract features from the second product image to obtain second image features.

[0018] The processing module is used to add the first image feature and the second image feature to obtain a first intermediate feature; and input the first intermediate feature into the defect detection model to obtain inference features. The defect detection model is trained using images of various items with different appearances and modified images obtained by modifying the preset image.

[0019] The processing module is further configured to upsample the inference features to obtain a second intermediate feature.

[0020] The prediction module is used to obtain information about the location of the defect in the second product image based on the second intermediate feature.

[0021] According to one aspect of this application, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described method for detecting defects in industrial products.

[0022] According to another aspect of this application, a computer-readable storage medium is provided, which stores a computer program that is loaded and executed by a processor to implement the above-described method for detecting defects in industrial products.

[0023] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned method for detecting defects in industrial products.

[0024] The beneficial effects of the technical solutions provided in this application include at least the following:

[0025] A first intermediate feature is obtained by adding the first image feature corresponding to the first product image and the second image feature corresponding to the second product image; the first intermediate feature is input into the defect detection model to obtain the inference feature; the inference feature is upsampled to obtain the second intermediate feature; and the location of the defect is predicted based on the second intermediate feature. The defect detection model satisfies at least one of the following conditions: the number of parameters reaches a parameter threshold and the number of network layers reaches a layer threshold, i.e., the defect detection model is a large model.

[0026] In other words, this application provides a defect detection architecture based on a large model. The input to the defect detection architecture is an image of a defect-free product and an image of the product to be inspected. The defect detection architecture utilizes the reasoning power of the large model, which enables the overall architecture to perform defect detection across product categories. Compared to related technologies that can only perform defect detection for single product categories, the defect detection architecture provided in this application has universality.

[0027] Furthermore, in related technologies, the model needs to be retrained for each new product category. In actual use, products are updated and replaced quickly (such as fabric dyeing), and retraining the model for each new product category seriously delays production schedules. The defect detection architecture provided in this application utilizes the inference capabilities of a large model. The inference capabilities of the large model enable the overall architecture to detect defects across product categories. The overall defect detection architecture does not require retraining and deployment. Regardless of changes in the product category, only images of defect-free products and images of products to be detected are needed, thereby improving the overall production efficiency of products.

[0028] Furthermore, the defect detection model is trained on images from multiple industrial product categories, which helps improve the generalization ability of the defect detection model and thus facilitates the defect detection model to perform cross-category defect detection.

[0029] Brief description of the attached figures

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 is a schematic diagram of an algorithm for image detection and image segmentation using a large model in related technologies.

[0032] Figure 2 is a schematic diagram of the detection principle of industrial product defects provided in one embodiment of this application.

[0033] Figure 3 is a flowchart of an industrial product defect detection method provided in one embodiment of this application.

[0034] Figure 4 is a schematic diagram of a defect detection architecture provided in an exemplary embodiment of this application.

[0035] Figure 5 is a schematic diagram of a defect detection architecture provided in another exemplary embodiment of this application.

[0036] Figure 6 is a schematic diagram of the defect detection results provided in an exemplary embodiment of this application.

[0037] Figure 7 is a flowchart of a training method for a defect detection model provided in an exemplary embodiment of this application.

[0038] Figure 8 is a schematic diagram of a defect detection architecture provided in another exemplary embodiment of this application.

[0039] Figure 9 is a structural block diagram of an industrial product defect detection device provided in one embodiment of this application.

[0040] Figure 10 is a structural block diagram of a computer device provided in one embodiment of this application.

[0041] Figure 11 is a structural block diagram of a computer device provided in another embodiment of this application.

[0042] Methods of implementing the present invention

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0044] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0045] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] Unsupervised anomaly detection: Defect detection is a crucial part of the industrial manufacturing process. The most common detection method involves providing only images of a defect-free product and the product to be inspected, allowing a neural network model to determine whether the product to be inspected is abnormal. Unsupervised anomaly detection means that the neural network model is not trained using real defect images. The training samples for the neural network model do not require manual annotation; the model only needs to use easily obtainable normal images during training.

[0048] Large models typically refer to models with a large number of parameters and deep network layers. They are machine learning models with a vast amount of parameters and computational resources. These models require massive amounts of data and computing power during training and can have millions to billions of parameters. The design purpose of large models is to improve their representational power and performance, enabling them to better capture patterns and regularities in data when handling complex tasks.

[0049] Related technologies provide unsupervised anomaly detection methods for industrial products. Methods such as PatchCore, DREAM, and SimpleNet all possess the ability to infer whether an input image is abnormal based on a normal image. PatchCore uses a feature library for anomaly detection. The PatchCore method stores features of normal images in the feature library; if the features of the input image to be detected are not in the feature library, the image is considered abnormal. PatchCore is only applicable to a single product category. For example, if the feature library for the first type of fabric does not contain a wavy pattern, the wavy pattern will be considered a defect during detection. When producing the second type of fabric, which includes a wavy pattern, the feature library of the first type cannot be used for defect detection. DREAM uses normal images for training and reconstruction. If the model in DREAM has not seen the abnormal region of the input image to be detected, the model cannot reconstruct the image to be anomaly repaired. SimpleNet also uses a similar reconstruction method. The difference between SimpleNet and DREAM is that SimpleNet considers the feature level of the image. If the model has not seen the abnormal features of the input image to be detected, the model cannot reconstruct the abnormal features into normal features.

[0050] Understandably, the unsupervised anomaly detection methods provided by the aforementioned technologies cannot detect anomalies in unseen images and lack generalization ability. These technologies can only be applied to images of a single category.

[0051] Related technologies provide algorithms for image detection and segmentation using large models. For example, Painter and SegGPT use a given example (containing input and output images) to predict new input images through imitation, and the model outputs the corresponding detection and segmentation results. Figure 1 illustrates the model prediction method provided by Painter in related technologies. The leftmost part of Figure 1 shows the given task example, which includes an input image and an output image. The middle part of Figure 1 shows the new input image, and the right side of Figure 1 shows the output result of the model's prediction of the new input image based on the given task example.

[0052] Understandably, image detection and segmentation using large models in related technologies rely on the model's mimicry capabilities. However, unsupervised anomaly detection requires the model to determine whether the image to be detected is abnormal based on a given normal image, necessitating reasoning ability, which current research has not yet extended to.

[0053] Figure 2 is a schematic diagram illustrating the detection principle of industrial product defects according to an exemplary embodiment of this application. The computer system shown in Figure 2 includes a device 201 for using the defect detection architecture and a training device 202 for the defect detection architecture. The training device 202 provides the trained defect detection architecture to the device 201. In some embodiments, the device 201 and the training device 202 are the same computer device. In some embodiments, the device 201 and the training device 202 communicate via wireless or wired means.

[0054] Figure 2 illustrates the usage process 210 and training process 220 of the defect detection architecture. In some embodiments, an end-to-end approach is used to predict the location of defects in a product image.

[0055] Figure 2 illustrates the usage process 210 of the defect detection architecture. A first product image 211 is acquired, and feature output (also called feature extraction) is performed on the first product image 211 to obtain first image features 212. A second product image 213 is acquired, and feature extraction is performed on the second product image 213 to obtain second image features 214. The first product image 211 and the second product image 213 are images of products within the same industrial product category. Products within the same industrial product category refer to products with the same or similar appearance, such as products from the same batch, products of the same model, or products from the same series, etc. For example, fabrics of the same model (with the same or similar patterns), or printed materials from the same batch (with the same or similar patterns). The first product image is an image of a defect-free product (also called a normal image or standard image), and the second product image is an image of the product to be inspected.

[0056] The first image feature 212 and the second image feature 214 are merged to obtain the first intermediate feature 215. The first intermediate feature 215 is input into the defect detection model 216, and the output is the inference feature 217. In some embodiments, the defect detection model 216 satisfies at least one of the following conditions: the number of parameters is not less than a parameter threshold and the number of network layers is not less than a layer threshold; that is, the defect detection model 216 is a large model. In some embodiments, the defect detection model 216 is a large model that has been tested and supports the execution of defect detection methods for general product categories. It can be understood that the defect detection model 216 is used to compare the image of the product to be detected with the image of the product without defects, and the inference feature 217 represents the comparison result. In some embodiments, the first image feature 212 and the second image feature 214 can be merged by adding them together. In some embodiments, the first image feature 212 and the second image feature 214 can be merged by averaging them together. In some embodiments, when merging the first image feature 212 and the second image feature 214, a preset weight can be used to weight the two features, such as weighted summation, weighted averaging, etc. In other embodiments, the first image feature 212 and the second image feature 214 can be merged by adding them together in any other feasible manner.

[0057] The inference feature 217 is upsampled to obtain a second intermediate feature 218. The upsampling operation is used to enlarge the size of the inference feature 217 compressed by the defect detection model 216. Based on the second intermediate feature 218, the location 219 of the defect in the second product image is predicted.

[0058] In each embodiment, the defect detection model 216 is trained using preset images of items with different appearances from multiple industrial product categories and modified images obtained by modifying the preset images.

[0059] In some embodiments, the image content of another image (e.g., a preset designated image) can be overlaid on a preset area of ​​a preset image to obtain a modified image. The image content of the designated image differs from the image content in the preset area of ​​the preset image.

[0060] In some embodiments, the values ​​of pixels in a preset region of a preset image can be modified to preset values ​​to obtain a modified image. For example, the preset region can be cropped, or the values ​​of all pixels in it can be set to preset values ​​(e.g., pixel values ​​corresponding to preset colors).

[0061] In some embodiments, a preset pattern can be overlaid on a preset area of ​​a preset image to obtain a modified image. For example, the image content in the preset area can be replaced with the preset pattern. Alternatively, the preset pattern can be overlaid on the pattern content in the preset area.

[0062] In each embodiment, the size of the preset area is smaller than the size of the preset image.

[0063] Figure 2 also illustrates the training process 220 of the defect detection architecture. A fourth product image 221 is acquired, and feature extraction is performed on the fourth product image 221 to obtain fourth image features 222. A fifth product image 223 is acquired, and data augmentation (i.e., modifying the image content of the preset region) is performed on a portion of the fifth product image 223 (e.g., a preset region, or a region selected from multiple preset regions) to obtain an augmented sixth product image 224. Feature extraction is performed on the sixth product image 224 to obtain sixth image features 225. The fourth product image 221 and the fifth product image 223 are images of defect-free products within the same industrial product category. The size of the preset region is smaller than the size of the fifth product image. In some embodiments, the training data used by the defect detection architecture comes from multiple datasets. Multiple datasets are beneficial for improving the generality of the defect detection architecture across product categories, thereby enabling defect detection for multiple product categories.

[0064] The fourth image feature 222 and the sixth image feature 225 are merged (e.g., added) to obtain a fourth intermediate feature 226. The fourth intermediate feature 226 is input into the defect detection model 216, and the output is a training feature 227. The training feature 227 is upsampled to obtain a fifth intermediate feature 228. The location 229 of the defect in the sixth product image is predicted based on the fifth intermediate feature 228. The location of the defect in the sixth product image is provided to the defect detection model so that the defect detection model adjusts the parameters in the defect detection model 216 based on the error between the predicted location 229 of the defect in the sixth product image and the location of the data-enhanced portion of the image.

[0065] It is understandable that the training process 220 will perform data augmentation on a portion of the fifth product image (e.g., by covering a portion of the fifth product image with the image content of a specified image), thereby achieving an unsupervised approach. The entire defect detection architecture achieves unsupervised anomaly detection.

[0066] It is also understood that the defect detection model 216 is a large model. This application provides a defect detection architecture based on a large model for detecting defects in industrial products. By utilizing the reasoning capabilities of the large model, the defect detection architecture supports the detection of defects in multiple product categories. The defect detection architecture provided in this application is universal for product categories.

[0067] In the above text, the training device 201 and the device 202 for using the defect detection architecture can be electronic devices with machine learning capabilities, such as computer devices, which can be terminals or servers.

[0068] In some embodiments, the user device 201 and the training device 202 may be the same electronic device, or they may be different electronic devices. Furthermore, when the user device 201 and the training device 202 are different devices, they may be of the same type, such as both being servers; or they may be of different types. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a mobile phone, computer, intelligent voice interaction device, smart home appliance, vehicle terminal, etc., but is not limited to these. The terminal and server may be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0069] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the product images involved in this application were obtained with full authorization.

[0070] Furthermore, when dealing with relevant information, the relevant information processor will adhere to the principles of legality, legitimacy, and necessity, clearly define the purpose, method, and scope of the relevant information processing, obtain the consent of the relevant information subject, and take necessary technical and organizational measures to ensure the security of the relevant information.

[0071] Figure 3 shows a flowchart of an industrial product defect detection method provided in an exemplary embodiment of this application, illustrated by an example of the method being performed by the device 201 shown in Figure 2. The method includes the following steps.

[0072] Step 310: Obtain the first product image and the second product image.

[0073] The first product image and the second product image are images of products that have the same or similar appearance (e.g., products of the same industrial product category).

[0074] Industrial product categories are determined based on the degree of similarity between industrial products in appearance. In some embodiments, industrial products of the same production model are grouped into the same industrial product category.

[0075] Understandably, when the production goal of industrial products of the same model is to produce completely identical, defect-free industrial products, then industrial products of the same model produced are considered to belong to the same industrial product category. For example, if model one is a standard fabric containing a lotus pattern and model two is a standard fabric containing a wave pattern, then the fabric produced of model one belongs to the same industrial product category, while the fabric produced of model two belongs to a different industrial product category.

[0076] In industrial production settings, various defects exist, necessitating defect detection. For example, fabric dyeing defect detection aims to identify discrepancies between the dyed fabric and the customer's sample, such as uneven coloring, white or black spots, or holes. Because fabric patterns are highly variable, factories need to produce fabrics with different patterns every few days.

[0077] For example, similar to fabric dyeing defect detection, cardboard printing defect detection requires checking whether the printed cardboard differs from the sample cardboard provided by the customer, such as uneven color, white or black spots, holes, or other defects. Furthermore, due to the variety of printing patterns, manufacturing plants frequently need to produce cardboard with different designs.

[0078] The first product image is an image of a defect-free industrial product. It should be noted that "defect-free" here means that the defects of the product are so few as to be negligible. In the defect detection method to be performed in this application, the first product image will be used as a standard image for comparison with the product image to be detected.

[0079] The second product image is an image of the industrial product to be inspected. The second product image can be an image of a defect-free industrial product or an image of a defective industrial product. The objective of this application is to detect the location of the defect when the second product image is an image of a defective industrial product.

[0080] It should be noted that the first product image and the second product image can be acquired simultaneously or at different times, and this application does not impose any restrictions on this.

[0081] Step 320: Extract features from the first product image to obtain the first image features.

[0082] In some embodiments, the first product image can be input into several convolutional layers to obtain the first image features, which are feature representations of the first product image.

[0083] Step 330: Extract features from the second product image to obtain the second image features.

[0084] In some embodiments, the second product image can be input into several convolutional layers to obtain second image features, which are feature representations of the second product image.

[0085] Step 340: Merge the first image features and the second image features to obtain the first intermediate features.

[0086] For example, if the first image feature and the second image feature have the same size, the first image feature and the second image feature are added together to obtain the first intermediate feature, which is used as the input feature of the defect detection model.

[0087] Step 350: Input the first intermediate feature into the defect detection model to obtain the inference feature.

[0088] A defect detection model is used to compare a first product image and a second product image. Inference features are used to characterize the comparison results between the first and second product images. The defect detection model satisfies at least one of the following conditions: the number of parameters is not less than a parameter threshold, and the number of network layers is not less than a layer threshold. That is, the defect detection model is a large model, or in other words, the defect detection model is the backbone network (the network that plays the main role) of a large model.

[0089] In some embodiments, the defect detection model is a large, tested model that supports image detection methods across product categories. In some embodiments, the defect detection model may be selected from Vit large, Vit Huge, etc.

[0090] The defect detection model is trained using preset images of various items with different appearances and modified images obtained by modifying the preset images. For example, the defect detection model can be trained using images of multiple industrial product categories separately. That is, when training the defect detection model, using product images of multiple categories (i.e., images of various items with different appearances) helps improve the generalization ability of the defect detection model, thereby helping the defect detection model to perform cross-category defect detection.

[0091] In some embodiments, images of multiple industrial product categories used to train the defect detection model are derived from multiple datasets, such as images from both the MVTec dataset and the ViSA dataset. This also helps improve the defect detection model's ability to perform cross-category defect detection.

[0092] Step 360: Upsample the inference features to obtain the second intermediate features.

[0093] Upsampling is used to amplify the size of the inference features obtained by compressing the defect detection model.

[0094] Step 370: Based on the second intermediate feature, obtain information about the location of the defect in the second product image.

[0095] In one embodiment, the number of channels of the second intermediate feature is compressed to three, namely red, green, and blue channels, to obtain the third intermediate feature. The length and width of the third intermediate feature are the same as the size of the pixel matrix of the second product image. For example, if the third intermediate feature is 3×h×w, the second product image is also represented as 3×h×w.

[0096] The third intermediate feature is exponentially normalized to obtain a segmentation map. The pixel value of each pixel in the segmentation map represents the probability that the pixel is a defective pixel. For example, a softmax calculation is performed on the third intermediate feature to obtain the segmentation map. Depending on the requirements, pixels with a value greater than 0.3 (or 0.5) are identified as defective pixels. The locations of all defective pixels constitute the locations of the defects.

[0097] The formula is expressed as F = softmax(Convs(x)), where x is the second intermediate feature, Convs is the convolution operation used to compress the number of channels to three, F is the segmentation map, and softmax is the exponential normalization function.

[0098] In summary, a first intermediate feature is obtained by adding the first image feature corresponding to the first product image and the second image feature corresponding to the second product image; the first intermediate feature is then input into the defect detection model to obtain inference features; the inference features are upsampled to obtain second intermediate features; and the location information of the defect is obtained based on the second intermediate features. The defect detection model satisfies at least one of the following conditions: the number of parameters reaches a parameter threshold and the number of network layers reaches a layer threshold, i.e., the defect detection model is a large model.

[0099] In other words, this application provides a defect detection architecture based on a large model. The input to the defect detection architecture is an image of a defect-free product and an image of the product to be inspected. The defect detection architecture utilizes the reasoning power of the large model, which enables the overall architecture to perform defect detection across product categories. Compared to related technologies that can only perform defect detection for single product categories, the defect detection architecture provided in this application has universality.

[0100] Furthermore, in related technologies, the model needs to be retrained for each new product category. In actual use, products are updated and replaced quickly (such as fabric dyeing), and retraining the model for each new product category seriously delays the production schedule. The defect detection architecture provided in this application does not require retraining and deployment. Regardless of how the product category changes, it only requires images of defect-free products and images of products to be detected, thereby improving the overall production efficiency of products.

[0101] Furthermore, the defect detection model is trained on images from multiple industrial product categories, which helps improve its generalization ability and facilitates cross-category defect detection. Additionally, as described above, information about the defect location is obtained through segmentation maps. The generation of segmentation maps is relatively simple, and they can intuitively and accurately present defect pixels, thus providing a complete picture of product defects.

[0102] Based on the embodiment shown in Figure 3, Figure 4 illustrates a defect detection architecture of one embodiment.

[0103] (1) Obtain a first product image 401 and a second product image 402. Perform feature extraction on the first product image 401 (in some embodiments, this is performed by some convolutional layers) to obtain a first image feature 403. Perform feature extraction on the second product image 402 (in some embodiments, this is performed by some convolutional layers) to obtain a second image feature 404. The first image feature 403 and the second image feature 404 have the same shape.

[0104] Schematic representation: the first image feature 403 is represented as c×h×w, and the second image feature 404 is represented as c×h×w. c is the number of channels of the feature, h is the width of the feature, and w is the length of the feature.

[0105] Schematic illustration: The size of the first product image 401 is 3×h×w, where 3 represents the red, green, and blue channels of the image, h represents the width of the image, and w represents the height of the image. c is an integer greater than 3. The first image feature 403 is used to increase the number of channels of the image and to characterize the image. Schematic illustration: The size of the second product image 402 is 3×h×w, where 3 represents the red, green, and blue channels of the image, h represents the width of the image, and w represents the height of the image. c is an integer greater than 3. The second image feature 404 is used to increase the number of channels of the image and to characterize the image.

[0106] (2) The first image feature 403 and the second image feature 404 are added together to obtain the first intermediate feature 405. The first intermediate feature 405 has the same shape as the first image feature 403 and the second image feature 404.

[0107] Schematic, the first image feature 403, represented as c×h×w, and the second image feature 404, represented as c×h×w, are added together to obtain the first intermediate feature 405, represented as c×h×w.

[0108] (3) The first intermediate feature 405 is input into the defect detection model 406, and the inference feature 407 is output. In some embodiments, the defect detection model 406 is used to compress the first intermediate feature 405 to obtain the inference feature 407. The size of the inference feature 407 is smaller than the size of the second image feature 404 (or the first image feature 403). In some embodiments, the defect detection model 406 is used to compress the length and width of the first intermediate feature 405 to the same degree to obtain the inference feature 407. The length of the inference feature 407 is smaller than the length of the second image feature 404 (or the first image feature 403), and the width of the inference feature 407 is smaller than the width of the second image feature 404 (or the first image feature 403).

[0109] Schematic illustration: The defect detection model is used to compress the first intermediate feature 405, represented as c×h×w, to obtain the inference feature 407, represented as c×(h / k)×(w / k). k is a positive integer. Schematic illustration: The first intermediate feature 405 is represented as c×(h / 32)×(w / 32) or c×(h / 16)×(w / 16).

[0110] (4) Input the reasoning feature 407 into the decoding network 408 and output the second intermediate feature 409.

[0111] The decoding network 408 is used to upsample the inference feature 407 to restore the features, resulting in a second intermediate feature 409. The feature size of the second intermediate feature 409 is the same as that of the first intermediate feature 405. The size of the inference feature 407 is changed to the size of the second image feature 404 (or the first image feature 403) through upsampling to obtain the second intermediate feature 409.

[0112] In some embodiments, the decoding network 408 is used to upsample the length and width of the inference feature 407 to restore the features to the same degree, resulting in a second intermediate feature 409. The length of the inference feature 407 is changed to the length of the second image feature 404 (or the first image feature 403) through upsampling, and the width of the inference feature 407 is changed to the width of the second image feature 404 (or the first image feature 403).

[0113] Schematic, the decoding network 408 is used to upsample the inference feature 407, represented as c×(h / k)×(w / k), to obtain a second intermediate feature 409, represented as c×h×w. In some embodiments, the decoding network 408 is a decoder in MAE (a paper titled Masked Autoencoders Are Scalable Vision Learners).

[0114] (5) Based on the second intermediate feature 409, perform the step of obtaining information 410 about the location of the defect.

[0115] In summary, the above embodiments provide feature map dimensions for each stage of the defect detection architecture, and further provide an overall structural design for the defect detection architecture, enabling defect detection to be achieved by simply inputting an image of a defect-free product and an image of the product to be inspected.

[0116] Based on the defect detection architecture shown in Figure 4, Figure 5 illustrates a further defect detection architecture.

[0117] Figure 5 illustrates that during defect detection, an additional product image 411 is also acquired. The additional product image 411 is an image of another defect-free industrial product within the same industrial product category as the first product image 401. Feature extraction is performed on the additional product image 411 (in some embodiments, this is performed by convolutional layers) to obtain additional image features 412.

[0118] Based on the additional image feature 412 and the first image feature 403, a template image feature 413 is obtained. The template image feature 413 and the second image feature 404 are merged (e.g., added) to obtain the first intermediate feature 405.

[0119] In some embodiments, the first image feature 403 and the additional image feature 412 have the same shape. The average value of the first image feature 403 and the additional image feature 412 is calculated to obtain the template image feature 413, which has the same shape as the first image feature 403 and the additional image feature 412.

[0120] Schematic, both the first image feature 403 and the additional image feature 412 are represented as c×h×w, where c is the number of channels of the feature, h is the width of the feature, and w is the length of the feature, and c, h, and w are positive integers. The average value of the first image feature 403 represented as c×h×w and the additional image feature 412 represented as c×h×w is calculated to obtain the template image feature 413 represented as c×h×w.

[0121] In some embodiments, the first product image 401 and the additional image 411 share a convolutional layer for feature extraction.

[0122] Figure 5 also illustrates the steps of reconstructing the image 411 based on the second intermediate feature 409. Image reconstruction based on the second intermediate feature yields a third product image, which represents the image after defect repair in the second product image. The third product image can be provided to the defect detection model, which adjusts its parameters using the differences between the third product image and the first product image. In one embodiment, the number of channels in the second intermediate feature is compressed to three, i.e., red, green, and blue channels, before image reconstruction is performed.

[0123] The formula is F = Convs(x), where x is the second intermediate feature, Convs is the convolution operation used to compress the number of channels to three, and F is the third product image (reconstruction result).

[0124] In summary, the above embodiments obtain the features of a template image based on the image features of multiple defect-free products. Different defect-free product images can have different features, and the template image will then fuse features from multiple conditions to obtain standard features. For example, one image of a defect-free product was taken under strong light conditions (such as a sunny day), while another image of a defect-free product was taken under weak light conditions (such as a rainy day). The fused template image will have lighting characteristics that more closely resemble the standard for defect-free products, thereby improving the contrast between the defect-free product image and the image of the product to be inspected, and making the defect detection results more accurate.

[0125] Furthermore, image reconstruction is performed based on the second intermediate features. The reconstructed image can also be used to repair defects in the product image to be inspected.

[0126] Figure 6 shows a schematic diagram of the defect detection results provided by an exemplary embodiment of this application.

[0127] Figure 6(A) shows an image of the defect-free product (i.e., the first product image), Figure 6(B) shows an image of the product to be inspected (here, an image of the defective product is shown), and Figure 6(C) shows the location of the defects. Figure 6(C) is the segmentation diagram described above. Figure 6(D) shows the reconstructed image, specifically the image of the second product image after defect repair. It can be seen that the defect detection architecture has predicted all defects, and the reconstructed image is defect-free.

[0128] After testing, this application achieves an AUROC (Area Under the Receiver Operating Characteristic Curve) of 90 on the MVtec dataset using the Vit Large (defect detection model), which can be simply understood as an accuracy of 90%. This meets normal production line requirements. Fine-tuning the output defect threshold can yield even better results. To ensure versatility, the defect threshold is set to 0.5.

[0129] Figure 7 illustrates a flowchart of a training method for a defect detection model provided in an exemplary embodiment of this application. Figure 7 shows the training of the defect detection model in an unsupervised manner. In some embodiments, defect locations are predicted end-to-end, and all neural networks in the defect detection framework are trained during training. Figure 7 illustrates a training method for the defect detection model, exemplified by the method being performed by the training device 202 in Figure 2. The method includes the following steps.

[0130] Step 710: Obtain the fourth product image and the fifth product image.

[0131] The fourth and fifth product images are images of defect-free products within the same industrial product category. For example, the fourth and fifth product images could be images of a camera lens or images of fabric. The fourth and fifth product images serve as training samples. In some embodiments, the fourth and fifth product images are images from the MVTec dataset. Alternatively, the fourth and fifth product images are images from the ViSA dataset.

[0132] The MVTec dataset contains 5354 high-resolution color images of different objects and textures. It includes normal (i.e., defect-free) images for training and anomalous images for testing. The MVTec dataset contains 70 different types of defects, such as scratches, dents, stains, and various structural variations.

[0133] The ViSA dataset contains 12 subsets, each representing a different object. It contains 10,821 images, including 9,621 normal samples and 1,200 anomalous samples.

[0134] Step 720: Extract features from the fourth product image to obtain the features of the fourth image.

[0135] In some embodiments, the fourth product image is input into several convolutional layers to obtain fourth image features. The fourth image features are a feature representation of the fourth product image.

[0136] Step 730: Perform data enhancement (i.e., modification) on a preset portion of the fifth product image; for specific modification methods, please refer to the description above to obtain the sixth product image.

[0137] In some embodiments, the image content of a designated image is overlaid on a preset portion of the fifth product image to obtain a sixth product image, wherein the image content of the designated image is different from the image content of the preset portion of the fifth product image.

[0138] To illustrate, a preset area on the fifth product image is cropped, and the image content of a specified image is copied and pasted onto the preset area on the fifth product image to obtain the sixth product image.

[0139] Step 740: Extract features from the sixth product image to obtain the features of the sixth image.

[0140] In some embodiments, the sixth product image is input into several convolutional layers to obtain the sixth image features, which are feature representations of the sixth product image.

[0141] Step 750: Merge the fourth image feature and the sixth image feature (e.g., add them together) to obtain the fourth intermediate feature.

[0142] For example, if the fourth image feature and the sixth image feature have the same size, the fourth image feature and the sixth image feature are added together to obtain the fourth intermediate feature, which is used as the input feature of the defect detection model.

[0143] Step 760: Input the fourth intermediate feature into the defect detection model to obtain the training features.

[0144] The fourth intermediate feature is input into the defect detection model to obtain the training features.

[0145] Step 770: Upsample the training features to obtain the fifth intermediate feature.

[0146] Upsampling is used to amplify the size of the inference features obtained by compressing the defect detection model.

[0147] Step 780: Based on the fifth intermediate feature, obtain information on the location of the defect in the sixth product image.

[0148] Based on the fifth intermediate feature, information about the location of the defect in the sixth product image is obtained.

[0149] Step 790: The location of the defect in the sixth product image is provided to the defect detection model so that the defect detection model can adjust the parameters in the defect detection model based on the error between the obtained location of the defect and the location of the preset area.

[0150] In some embodiments, the defect detection model is adjusted based on the error between the pixel coordinates of the defect's location and the pixel coordinates of a portion of the region. This error allows the defect detection model to optimize its ability to predict defect locations.

[0151] In one embodiment, the training device 201 further performs image reconstruction based on the fifth intermediate feature to obtain a seventh product image; based on the error between the seventh product image and the fifth product image, a defect detection model is trained. The error between the reconstructed image and the original image is used to help the defect detection model optimize its ability to recognize images of defect-free products. Furthermore, the defect detection model also optimizes its recognition of structural information in images of defect-free products.

[0152] It should be noted that the training process and usage process of the defect detection architecture are similar. For other details regarding the training process of the defect detection architecture, please refer to the above introduction to the usage process.

[0153] In summary, the above embodiments provide a training method for an unsupervised defect detection model. By performing data augmentation on a portion of the model and training the defect detection model based on the error between this portion and the predicted defect location, the characteristics of unsupervised anomaly detection are satisfied.

[0154] Furthermore, the above embodiments utilize the reconstruction error between the reconstructed image and the original image to train the defect detection model. The reconstruction error not only helps the defect detection model recognize that the first product image is a defect-free product image (normal image), but also helps the defect detection model learn the structural information of the defect-free product image, thereby helping to predict the location of defects.

[0155] Figure 8 shows a schematic diagram of an industrial product defect detection framework provided by an exemplary embodiment of this application.

[0156] (1) Template image branch: Given N template images 801 (i.e., images of normal products), input them into the template shared convolutional block (i.e., some convolutional layers, which are not fixed and can vary). The size of the output feature of each image is c×h×w (where c is the number of channels, h is the width of the image feature after convolution, and w is the length of the image feature after convolution). Perform template image feature merging (multiple images are directly averaged) to obtain a c×h×w feature.

[0157] (2) Input graph branch: The input graph 803 passes through the input convolution block (that is, some convolutional layers, which are not fixed and can change), and the feature size of the input graph is also c×h×w.

[0158] (3) Large Model Backbone Network 805: The features of the input graph and the template graph are directly added together, and the feature shape is still c×h×w. Then, the features are extracted by the large model backbone network 805 (the large model backbone network is a network with a large number of parameters, such as Vit Large and Vit Huge).

[0159] (4) Decoding Network 806: Because the output of the large model backbone network 805 will compress the features to a relatively small size, generally c×h×w relative to the input will become c×(h / 32)×(w / 32) or c×(h / 16)×(w / 16), etc., it is necessary to perform upsampling operations through some convolutional layers. In some embodiments, the decoding network 806 is the decoder in MAE.

[0160] The main function of the decoding network is to restore the features to the size of the image. The last layer of the decoding network increases the number of channels and then limits the output features to the size of the input image 803.

[0161] (5) Reconstruction Branch 807: The reconstruction branch aims to restore the input image 803 to a defect-free image. Because unsupervised anomaly detection does not have a supervision signal, but the large model must understand the features of the input template image in order to infer anomalies, the reconstruction branch 807 uses the features of the backbone network 805 of the large model to reconstruct the original image, thereby helping the large model to have the cognitive ability of the input template image.

[0162] During training, since it is unsupervised training, all images are normal and without defects. By adding some data augmentation, such as copying and pasting a region of a specified image into the input image 803, directly cropping some black areas, and then the reconstruction branch 807 can directly use the augmented image to reconstruct the input image 803 before augmentation.

[0163] Because of the emphasis on general applicability, model training is best done across datasets. In some implementations, MvTec and ViSA are used as training datasets.

[0164] (6) Predicting Defect Location Branch 808: This branch directly outputs a segmentation map of the same size as the original image. Each pixel has an abnormal branch, which reflects end-to-end defect detection.

[0165] Because the model can directly predict defect locations end-to-end (defect location prediction branch), after inputting template image 801, input the image to be detected (input image 803), and the segmentation image predicted by defect location prediction branch 808 can be directly obtained. The length and width of the segmentation image are consistent with the original image, and each pixel value represents the probability that a single pixel is a defect, with the probability value belonging to [0, 1]. Whether a pixel is a defect pixel is determined by applying a threshold. For example, a threshold of 0.3 or 0.5 can be used depending on the actual needs.

[0166] During the training process, since it is unsupervised training, all images are normal and without defects. By adding some data augmentation, such as copying and pasting a region of a specified image into the input image 803, some black areas are directly cropped out. Then, the defect location prediction branch 808 predicts the cropped area.

[0167] Figure 9 shows a structural block diagram of an industrial product defect detection device provided in an exemplary embodiment of this application. The device includes the following modules.

[0168] The acquisition module 901 is used to acquire a first product image and a second product image. The first product image and the second product image are images of products with the same or similar appearance (e.g., products under the same industrial product category). The first product image is an image of a defect-free industrial product, and the second product image is an image of the industrial product to be inspected.

[0169] The feature extraction module 902 is used to extract features from the first product image to obtain first image features; and to extract features from the second product image to obtain second image features.

[0170] The processing module 903 is used to add the first image features and the second image features to obtain a first intermediate feature; and input the first intermediate feature into the defect detection model to obtain inference features. The defect detection model is trained based on preset images of various items with different appearances (e.g., products from multiple industrial product categories) and modified images obtained by modifying the preset images.

[0171] The processing module 903 is also used to upsample the inference features to obtain the second intermediate features.

[0172] The prediction module 904 is used to obtain information about the location of defects in the second product image based on the second intermediate features.

[0173] In one embodiment, the size of the inference feature is smaller than the size of the second image feature. The processing module 903 is further configured to upsample the size of the inference feature to the size of the second image feature to obtain a second intermediate feature.

[0174] In one embodiment, the length of the inference feature is less than the length of the second image feature, and the width of the inference feature is less than the width of the second image feature. The processing module 903 is configured to upsample the length of the inference feature to the length of the second image feature and the width of the inference feature to the width of the second image feature to obtain a second intermediate feature.

[0175] In one embodiment, the acquisition module 901 is further configured to acquire an additional product image, which is an image of another defect-free industrial product belonging to the same industrial product category as the first product image. The feature extraction module 902 is further configured to extract features from the additional product image to obtain additional image features, and to combine the additional image features and the first image features to obtain template image features. The processing module 903 is further configured to add the template image features and the second image features to obtain a first intermediate feature.

[0176] In one embodiment, the first image feature and the additional image feature have the same size. The feature extraction module 902 is further configured to calculate the average value of the first image feature and the additional image feature to obtain a template image feature, wherein the template image feature has the same size as the first image feature and the additional image feature.

[0177] In one embodiment, the prediction module 904 is further configured to compress the number of channels of the second intermediate feature to three to obtain a third intermediate feature, wherein the length and width of the third intermediate feature are the same as the size of the pixel array of the second product image.

[0178] The third intermediate feature is subjected to exponential normalization to obtain the segmentation map. The pixel value of each pixel in the segmentation map represents the probability that the pixel is a defective pixel.

[0179] In one embodiment, the apparatus further includes a reconstruction module 905. The reconstruction module 905 is used to perform image reconstruction based on a second intermediate feature to obtain a third product image, the third product image representing an image after defect repair in the second product image.

[0180] In one embodiment, the acquisition module 901 is further configured to acquire a fourth product image and a fifth product image, wherein the fourth product image and the fifth product image are images of defect-free products under the same industrial product category; and to perform data augmentation on a portion of the fifth product image to obtain a sixth product image.

[0181] The feature extraction module 902 is also used to extract features from the fourth product image to obtain the fourth image features; and to extract features from the sixth product image to obtain the sixth image features.

[0182] The processing module 903 is also used to merge the fourth image feature and the sixth image feature to obtain the fourth intermediate feature; input the fourth intermediate feature into the defect detection model to obtain the training feature; and upsample the training feature to obtain the fifth intermediate feature.

[0183] The prediction module 904 is also used to obtain information about the location of defects in the sixth product image based on the fifth intermediate feature.

[0184] The device also includes a training module 906. The training module 906 is used to train a defect detection model based on the error between the predicted location of the defect and the location of a partial area.

[0185] In one embodiment, the acquisition module 901 is further configured to overlay the image content of a specified image onto a portion of the fifth product image to obtain a sixth product image, wherein the image content of the specified image is different from the image content of a portion of the fifth product image.

[0186] In one embodiment, the reconstruction module 905 is further configured to provide the location of the defect in the sixth product image to the defect detection model, so that the defect detection model can reconstruct the image based on the fifth intermediate feature to obtain the seventh product image. The training module 906 is further configured to train the defect detection model based on the error between the seventh product image and the fifth product image.

[0187] In one embodiment, the images of multiple industrial product categories used to train the defect detection model are derived from multiple datasets. For example, the acquisition module 901 can acquire the fourth product image and the fifth product image from multiple datasets, which include images of various items with different appearances.

[0188] In summary, the first intermediate feature is obtained by adding the first image feature corresponding to the first product image and the second image feature corresponding to the second product image; the first intermediate feature is input into the defect detection model to obtain the inference feature; the inference feature is upsampled to obtain the second intermediate feature; and the location of the defect is predicted based on the second intermediate feature. The defect detection model satisfies at least one of the following conditions: the number of parameters reaches a parameter threshold and the number of network layers reaches a layer threshold, i.e., the defect detection model is a large model.

[0189] In other words, this application provides a defect detection architecture based on a large model. The input to the defect detection architecture is an image of a defect-free product and an image of the product to be inspected. The defect detection architecture utilizes the reasoning power of the large model, which enables the overall architecture to perform defect detection across product categories. Compared to related technologies that can only perform defect detection for single product categories, the defect detection architecture provided in this application has universality.

[0190] Furthermore, in related technologies, the model needs to be retrained for each new product category. In actual use, products are updated and replaced quickly (such as fabric dyeing), and retraining the model for each new product category seriously delays the production schedule. The defect detection architecture provided in this application does not require retraining and deployment. Regardless of how the product category changes, it only requires images of defect-free products and images of products to be detected, thereby improving the overall production efficiency of products.

[0191] Furthermore, the defect detection model is trained on images from multiple industrial product categories, which helps improve the generalization ability of the defect detection model and thus facilitates the defect detection model to perform cross-category defect detection.

[0192] Figure 10 is a schematic diagram of a computer device according to an exemplary embodiment. The computer device 1000 includes a Central Processing Unit (CPU) 1001, a system memory 1004 including Random Access Memory (RAM) 1002 and Read-Only Memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a Basic Input / Output (I / O) system 1006 to facilitate information transfer between various devices within the computer device, and a mass storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0193] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0194] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer device-readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0195] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0196] According to various embodiments of this disclosure, the computer device 1000 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1011 via a network interface unit 1012 connected to the system bus 1005, or the network interface unit 1012 can be used to connect to other types of networks or remote computer device systems (not shown).

[0197] The memory also includes one or more programs stored in the memory, and the central processing unit 1001 executes the one or more programs to implement all or part of the steps of the above-mentioned industrial product defect detection method.

[0198] Figure 11 shows a structural block diagram of a computer device 1100 provided in an exemplary embodiment of this application. The computer device 1100 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 1100 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0199] Typically, computer device 1100 includes a processor 1101 and a memory 1102.

[0200] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0201] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 is used to store at least one instruction, which is executed by the processor 1101 to implement the industrial product defect detection method provided in the method embodiments of this application.

[0202] In some embodiments, the computer device 1100 may optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. For example, the peripheral device may include at least one of the following: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, a power supply 1108, and one or more sensors 1109. The one or more sensors 1109 include, but are not limited to: an accelerometer 1110, a gyroscope 1111, a pressure sensor 1112, an optical sensor 1113, and a proximity sensor 1114.

[0203] Those skilled in the art will understand that the structure shown in FIG11 does not constitute a limitation on the computer device 1100, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0204] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the industrial product defect detection method provided in the above method embodiments.

[0205] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the industrial product defect detection method provided in the above-described method embodiments.

[0206] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0207] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0208] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0209] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for detecting defects in industrial products, comprising: The electronic device acquires a first product image and a second product image, wherein the first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected; The electronic device performs feature extraction on the first product image to obtain a first image feature; and performs feature extraction on the second product image to obtain a second image feature; The electronic device merges the first image feature and the second image feature to obtain a first intermediate feature; the first intermediate feature is input into a defect detection model to obtain inference features. The defect detection model is trained using preset images of various items with different appearances and modified images obtained by modifying the preset images. The electronic device upsamples the inference features to obtain a second intermediate feature; The electronic device obtains information about the location of the defect in the second product image based on the second intermediate feature.

2. The method according to claim 1, wherein the size of the inference feature is smaller than the size of the second image feature; The step of upsampling the inference features to obtain the second intermediate features includes: The size of the inference feature is changed to the size of the second image feature by upsampling, thus obtaining the second intermediate feature.

3. The method according to claim 2, wherein the length of the inference feature is less than the length of the second image feature, and the width of the inference feature is less than the width of the second image feature; The step of changing the size of the inference feature to the size of the second image feature through upsampling to obtain the second intermediate feature includes: The second intermediate feature is obtained by upsampling, which changes the length of the inference feature to the length of the second image feature and changes the width of the inference feature to the width of the second image feature.

4. The method according to any one of claims 1 to 3, further comprising: Acquire additional product images, wherein the additional product images are images of defect-free industrial products that have the same or similar appearance as the first product image; Feature extraction is performed on the additional product image to obtain additional image features; The additional image features and the first image features are combined to obtain the template image features; The first image feature and the second image feature are merged to obtain the first intermediate feature, which includes: The template image features and the second image features are merged to obtain the first intermediate feature.

5. The method according to claim 4, wherein the first image feature and the additional image feature have the same size; the step of obtaining the template image feature based on the additional image feature and the first image feature includes: The average value of the first image feature and the additional image feature is calculated to obtain the template image feature, wherein the template image feature has the same size as the first image feature and the additional image feature.

6. The method according to any one of claims 1 to 3, wherein obtaining information about the location of the defect in the second product image based on the second intermediate feature includes: The number of channels of the second intermediate feature is compressed to three to obtain the third intermediate feature, and the size of the third intermediate feature is the same as that of the pixel matrix of the second product image. The third intermediate feature is subjected to exponential normalization to obtain a segmentation map, and the pixel value of the pixel in the segmentation map represents the probability that the pixel is a defective pixel.

7. The method according to any one of claims 1 to 3, further comprising: Image reconstruction is performed based on the second intermediate feature to obtain a third product image, which represents the image after defect repair in the second product image. The third product image is provided to the defect detection model, which then uses the differences between the third product image and the first product image to adjust the parameters in the defect detection model.

8. The method according to any one of claims 1 to 3, further comprising: Acquire a fourth product image and a fifth product image, wherein the fourth product image and the fifth product image are images of a defect-free product; The image content of a preset area of ​​the fifth product image is modified to obtain a sixth product image, wherein the size of the preset area is smaller than the size of the fifth product image; The feature extraction model is used to extract features from the fourth product image to obtain fourth image features; and the feature extraction model is used to extract features from the sixth product image to obtain sixth image features; The fourth image feature and the sixth image feature are merged to obtain the fourth intermediate feature; the fourth intermediate feature is then input into the defect detection model to obtain the training feature. The training features are upsampled to obtain the fifth intermediate feature; Based on the fifth intermediate feature, information about the location of the defect in the sixth product image is obtained; The location of the defect in the sixth product image is provided to the defect detection model so that the defect detection model adjusts the parameters in the defect detection model based on the error between the location of the defect and the location of the preset area.

9. The method according to any one of claims 1 to 3, wherein, The modified image includes at least one of the following: The modified image is obtained by overlaying the image content of a specified image onto a preset area of ​​the preset image, wherein the image content of the specified image is different from the image content of the preset area on the preset image; The modified image is obtained by modifying the pixel values ​​in a preset region of the preset image to preset values; The modified image is obtained by overlaying a preset pattern onto a preset area of ​​the preset image; The size of the preset region is smaller than the size of the preset image.

10. The method of claim 8, further comprising: Image reconstruction is performed based on the fifth intermediate feature to obtain the seventh product image; The seventh product image is provided to the defect detection model so that the defect detection model adjusts the parameters in the defect detection model based on the error between the seventh product image and the fifth product image.

11. The method according to any one of claims 1 to 3, wherein, The preset images are selected from multiple datasets, which include images of various items with different appearances.

12. A device for detecting defects in industrial products, comprising: The acquisition module is used to acquire a first product image and a second product image, wherein the first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected; The feature extraction module is used to extract features from the first product image to obtain first image features; and to extract features from the second product image to obtain second image features; The processing module is used to merge the first image features and the second image features to obtain a first intermediate feature; and input the first intermediate feature into a defect detection model to obtain inference features. The defect detection model is trained using preset images of various items with different appearances and modified images obtained by modifying the preset images. The processing module is further configured to upsample the inference features to obtain a second intermediate feature; The prediction module is used to obtain information about the location of the defect in the second product image based on the second intermediate feature.

13. The apparatus according to claim 12, wherein, The feature extraction module is used for: Acquire additional product images, wherein the additional product images are images of defect-free industrial products that have the same or similar appearance as the first product image; Feature extraction is performed on the additional product image to obtain additional image features; The additional image features and the first image features are combined to obtain the template image features; The processing module is used for: The template image features and the second image features are merged to obtain the first intermediate feature.

14. The apparatus of claim 12, wherein the prediction module is configured to: The number of channels of the second intermediate feature is compressed to three to obtain the third intermediate feature, and the size of the third intermediate feature is the same as that of the pixel matrix of the second product image. The third intermediate feature is subjected to exponential normalization to obtain a segmentation map, and the pixel value of the pixel in the segmentation map represents the probability that the pixel is a defective pixel.

15. The apparatus of claim 12, further comprising a reconstruction module for: Image reconstruction is performed based on the second intermediate feature to obtain a third product image, which represents the image after defect repair in the second product image. The third product image is provided to the defect detection model, which then uses the differences between the third product image and the first product image to adjust the parameters in the defect detection model.

16. The apparatus according to any one of claims 12 to 15, wherein, The acquisition module is further configured to acquire a fourth product image and a fifth product image, wherein the fourth product image and the fifth product image are images of defect-free products; modify the image content of a preset area of ​​the fifth product image to obtain a sixth product image, wherein the size of the preset area is smaller than the size of the fifth product image; The feature extraction module is further configured to: extract features from the fourth product image to obtain fourth image features; and extract features from the sixth product image to obtain sixth image features; The processing module is further configured to: merge the fourth image feature and the sixth image feature to obtain a fourth intermediate feature; input the fourth intermediate feature into the defect detection model to obtain training features; and upsample the training features to obtain a fifth intermediate feature. The prediction module is further used to obtain information about the location of the defect in the sixth product image based on the fifth intermediate feature; The device further includes a training module for providing the location of the defect in the sixth product image to the defect detection model, so that the defect detection model adjusts the parameters in the defect detection model based on the error between the predicted location of the defect and the location of the preset area.

17. The apparatus according to claim 16, wherein, The acquisition module is used for: The modified image is obtained by overlaying the image content of a specified image onto a preset area of ​​the fifth product image, wherein the image content of the specified image is different from the image content of the preset area on the fifth product image; The modified image is obtained by modifying the pixel values ​​in a preset area of ​​the fifth product image to preset values; The modified image is obtained by overlaying a preset pattern onto a preset area of ​​the fifth product image.

18. The apparatus according to claim 16, wherein, The acquisition module is used for: The fourth and fifth product images are obtained from multiple datasets, which include images of various items with different appearances.

19. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the method for detecting defects in industrial products as described in any one of claims 1 to 11.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method for detecting defects in industrial products as described in any one of claims 1 to 11.

Images