Industrial product defect detection method and apparatus, device, and medium

Abstract

Disclosed in the present application are an industrial product defect detection method and apparatus, a device, and a medium, which belong to the field of image processing, and can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and aided driving. The method comprises: acquiring a first product image and a second product image, the first product image being an image of a defect-free industrial product, and the second product image being an image of an industrial product to undergo detection; separately performing feature extraction on the first product image and the second product image to obtain a first image feature and a second image feature; merging the first image feature and the second image feature to obtain a first intermediate feature; inputting the first intermediate feature into a defect detection model to obtain an inference feature, the defect detection model being obtained by means of training by using preset images of articles having different appearances and modified images obtained after modifying the preset images; performing up-sampling on the inference feature to obtain a second intermediate feature; and, on the basis of the second intermediate feature, obtaining information of the position of a defect. The present application achieves cross-category defect detection by utilizing the inference capability of large models.

Description

Industrial product defect detection method, device, equipment and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 8, 2023, with application number 2023111553036 and invention name “Methods, devices, equipment and media for detecting industrial product defects”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing, and in particular to a method, device, equipment and medium for detecting defects in industrial products.

[0003] Background of the Invention

[0004] In industrial production scenarios, due to various reasons, industrial products often have various defects. For example, dyed cloth may have uneven color, abnormal white or black spots, holes in the cloth, inconsistent patterns, etc. Therefore, defect detection of industrial products is necessary.

[0005] Related technologies use a feature library to perform defect detection. This involves capturing images of non-defective products and storing their features in a feature library. Next, an image of the product to be inspected is captured. If the features of the image are not in the library, the product is considered defective.

[0006] However, the method of using a feature library can only be applied to a single category of products. When the feature library is used to detect defects in another category of products, the relevant technology requires retraining the model.

[0007] Summary of the Invention

[0008] This application provides a method, apparatus, device, and medium for detecting industrial product defects, and a large-scale model-based defect detection architecture. This architecture leverages the reasoning power of large-scale models, enabling the overall architecture to detect defects across product categories. The technical solution includes the following.

[0009] According to one aspect of the present application, a method for detecting defects in industrial products is provided, the method comprising the following steps.

[0010] The electronic device acquires a first product image and a second product image, wherein the first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected.

[0011] The electronic device performs feature extraction on the first product image to obtain first image features; and performs feature extraction on the second product image to obtain second image features.

[0012] The electronic device combines the first image feature and the second image feature to obtain a first intermediate feature; inputs the first intermediate feature into a defect detection model to obtain an inference feature, and the defect detection model is trained using preset images of multiple objects with different appearances and modified images obtained by modifying the preset images.

[0013] The electronic device upsamples the inference feature to obtain a second intermediate feature.

[0014] The electronic device obtains information about a location of the defect in the second product image based on the second intermediate feature.

[0015] According to another aspect of the present application, a device for detecting defects in industrial products is provided, and the device includes the following modules.

[0016] An acquisition module is configured to acquire a first product image and a second product image. In some embodiments, the first product image and the second product image are images of products having the same or similar appearance. The first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected.

[0017] The feature extraction module is configured to perform feature extraction on the first product image to obtain first image features; and perform feature extraction on the second product image to obtain second image features.

[0018] The processing module is configured to add the first image feature and the second image feature to obtain a first intermediate feature, and input the first intermediate feature into a defect detection model to obtain an inferred feature. The defect detection model is trained using images of multiple objects with different appearances and modified images obtained by modifying the preset image.

[0019] The processing module is further used to upsample the inference feature to obtain a second intermediate feature.

[0020] A prediction module is configured to obtain information about a location of a defect in the second product image based on the second intermediate feature.

[0021] According to one aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the above industrial product defect detection method.

[0022] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the above industrial product defect detection method.

[0023] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described industrial product defect detection method.

[0024] The beneficial effects brought about by the technical solutions provided in the embodiments of the present application include at least the following.

[0025] A first intermediate feature is obtained by adding a first image feature corresponding to the first product image and a second image feature corresponding to the second product image. The first intermediate feature is input into the defect detection model to obtain an inferred feature. The inferred feature is upsampled to obtain a second intermediate feature. The defect location is predicted based on the second intermediate feature. The defect detection model is considered large if it meets at least one of the following conditions: the number of parameters reaches a parameter threshold and the number of network layers reaches a layer threshold.

[0026] Specifically, this application provides a large-scale model-based defect detection architecture. The inputs to the defect detection architecture are images of defect-free products and images of products to be inspected. This architecture leverages the reasoning power of the large-scale model, enabling the overall architecture to detect defects across product categories. Compared to related technologies that can only detect defects in a single product category, the defect detection architecture provided by this application is universal.

[0027] Moreover, in the related art, the model needs to be retrained for each new product category. In actual use, products are updated very quickly (such as cloth dyeing, etc.). Every time a new product category is produced, the model needs to be retrained, which seriously delays the production progress. The defect detection architecture provided by this application utilizes the reasoning ability of the large model. The reasoning ability of the large model enables the overall architecture to have the defect detection capability of cross-category products. The overall defect detection architecture does not need to be retrained and deployed. Regardless of how the product category changes, it only needs to provide images of defect-free products and images of products to be detected, thereby improving the overall production efficiency of the product.

[0028] In addition, the defect detection model is trained based on images of multiple industrial product categories, which helps to improve the generalization of the defect detection model and thus facilitates the defect detection model to perform cross-category defect detection.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] FIG1 is a schematic diagram of an algorithm for image detection and image segmentation using a large model in related art.

[0032] FIG2 is a schematic diagram of a detection principle of industrial product defects provided by an embodiment of the present application.

[0033] FIG3 is a flow chart of a method for detecting industrial product defects provided by one embodiment of the present application.

[0034] FIG4 is a schematic diagram of a defect detection architecture provided by an exemplary embodiment of the present application.

[0035] FIG5 is a schematic diagram of a defect detection architecture provided by another exemplary embodiment of the present application.

[0036] FIG6 is a schematic diagram of a defect detection result provided by an exemplary embodiment of the present application.

[0037] FIG7 is a flowchart of a method for training a defect detection model provided by an exemplary embodiment of the present application.

[0038] FIG8 is a schematic diagram of a defect detection architecture provided by another exemplary embodiment of the present application.

[0039] FIG9 is a structural block diagram of an industrial product defect detection device provided by an embodiment of the present application.

[0040] FIG10 is a structural block diagram of a computer device provided in one embodiment of the present application.

[0041] FIG11 is a structural block diagram of a computer device provided in another embodiment of the present application.

[0042] Modes for Carrying Out the Invention

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0045] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0046] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0047] Unsupervised anomaly detection: Defect detection is a critical component of the industrial manufacturing process. The primary detection method involves presenting only images of defect-free products and the product being tested, allowing a neural network model to determine whether the product being tested has an anomaly. Unsupervised anomaly detection means that the neural network model is not trained on real defect images. Training samples for the neural network model do not require manual labeling, and the model is trained using only readily available normal images.

[0048] Large models generally refer to models with large parameters and deep network layers. Large models are machine learning models with a large number of parameters and computing resources. These models require vast amounts of data and computing power to train and may have millions to billions of parameters. Large models are designed to improve their representational capabilities and performance, enabling them to better capture patterns and regularities in data when handling complex tasks.

[0049] Related technologies provide unsupervised anomaly detection methods for industrial products. PatchCore, DREAM, SimpleNet, and others all have the ability to infer whether an input image is abnormal based on a normal image. PatchCore uses a feature library for anomaly detection. The PatchCore method stores the features of normal images in the feature library. If the features of the input image to be detected are not in the feature library, the image to be detected is considered abnormal. PatchCore's approach can only be applied to a single category of products. For example, if a wave pattern does not exist in the feature library of a first model of fabric, the wave pattern will be considered a defect during detection. When a second model of fabric is produced, a wave pattern design is added to the second model of fabric. In this case, the feature library of the first model of fabric cannot be used to detect defects in the second model of fabric. DREAM uses a normal image training and reconstruction method. If the DREAM model has never seen the abnormal area of ​​the input image to be detected, the model cannot reconstruct the image to be detected into an image with the abnormality repaired. SimpleNet also uses a similar reconstruction method. The difference from DREAM is that SimpleNet considers the feature level of the image. If the model has never seen the abnormal features of the input image to be detected, the model cannot reconstruct the abnormal features into normal features.

[0050] It is understandable that the unsupervised anomaly detection methods provided by the above-mentioned related technologies are unable to detect anomalies in unseen images and are not generalizable. The above-mentioned related technologies can only be applied to images of a single category.

[0051] In the related art, algorithms for image detection and image segmentation using large models are provided. For example, Painter and SegGPT will use a given example (including an input image and an output image) to predict a new input image by imitation, and the model outputs corresponding detection results and segmentation results. Schematically, Figure 1 shows the model prediction method provided by Painter in the related art. The leftmost side of Figure 1 is a given task example. A task example includes an input image and an output image. The middle of Figure 1 is a new input image. The right side of Figure 1 is the output result of the model predicting the new input image based on the given task example.

[0052] Understandably, the related art of image detection and segmentation using large models relies on their imitation capabilities. However, unsupervised anomaly detection requires the model to determine whether the image being tested is abnormal based on a given normal image, which requires reasoning capabilities, and current research has not yet expanded to this area.

[0053] FIG2 is a schematic diagram illustrating the principle of industrial product defect detection provided by an exemplary embodiment of the present application. The computer system shown in FIG2 includes a device 201 for using a defect detection architecture and a device 202 for training the defect detection architecture. The training device 202 provides the trained defect detection architecture to the device 201. In some embodiments, the device 201 and the training device 202 are the same computer device. In some embodiments, data transmission between the device 201 and the training device 202 is performed wirelessly or wired.

[0054] 2 illustrates a process 210 of using a defect detection framework and a process 220 of training the defect detection framework. In some embodiments, an end-to-end approach is used to predict the location of defects in product images.

[0055] Figure 2 shows the use process 210 of the defect detection architecture. A first product image 211 is obtained, and feature output (also called feature extraction) is performed on the first product image 211 to obtain a first image feature 212. A second product image 213 is obtained, and feature extraction is performed on the second product image 213 to obtain a second image feature 214. The first product image 211 and the second product image 213 are images of products in the same industrial product category. Products in the same industrial product category refer to products with the same or similar appearance, such as products from the same batch, products of the same model, or products from the same series, etc. For example, cloth of the same model (with the same or similar patterns), printed materials of the same batch (with the same or similar patterns). The first product image is an image of a defect-free product (also called a normal image, standard image), and the second product image is an image of the product to be inspected.

[0056] The first image feature 212 and the second image feature 214 are merged to obtain a first intermediate feature 215. The first intermediate feature 215 is input into the defect detection model 216, which outputs an inferred feature 217. In some embodiments, the defect detection model 216 satisfies at least one of the following conditions: the number of parameters is not less than a parameter threshold and the number of network layers is not less than a layer threshold; that is, the defect detection model 216 is a large model. In some embodiments, the defect detection model 216 is a large model that has been tested and supports defect detection methods for a common product category. It is understood that the defect detection model 216 is used to compare images of products to be inspected with images of non-defective products, and the inferred feature 217 represents the comparison result. In some embodiments, the first image feature 212 and the second image feature 214 can be merged by adding them together. In some embodiments, the first image feature 212 and the second image feature 214 can be merged by averaging them. In some embodiments, when merging the first image feature 212 and the second image feature 214, the two features can be weighted using preset weights, such as weighted summation or weighted averaging. In other embodiments, any other feasible manner may be adopted to combine the first image feature 212 and the second image feature 214 by adding them together.

[0057] The inferred feature 217 is upsampled to obtain a second intermediate feature 218. The upsampling operation is used to enlarge the size of the inferred feature 217 compressed by the defect detection model 216. Based on the second intermediate feature 218, the location 219 of the defect in the second product image is predicted.

[0058] In various embodiments, the defect detection model 216 is trained using preset images of items of different appearances in a plurality of industrial product categories and modified images obtained by modifying the preset images.

[0059] In some embodiments, the image content of another image (e.g., a preset designated image) may be overlaid on the preset area of ​​the preset image to obtain a modified image, wherein the image content of the designated image is different from the image content in the preset area of ​​the preset image.

[0060] In some embodiments, the values ​​of pixels in a preset area of ​​a preset image may be modified to preset values ​​to obtain a modified image. For example, the preset area may be cropped or the values ​​of all pixels therein may be set to preset values ​​(e.g., pixel values ​​corresponding to a preset color).

[0061] In some embodiments, a preset pattern may be overlaid on a preset area of ​​a preset image to obtain a modified image. For example, the image content in the preset area may be replaced with a preset pattern. In another example, the preset pattern may be overlaid on the pattern content in the preset area.

[0062] In various embodiments, the size of the preset area is smaller than the size of the preset image.

[0063] Figure 2 also shows the training process 220 of the defect detection architecture. A fourth product image 221 is obtained, and feature extraction is performed on the fourth product image 221 to obtain a fourth image feature 222. Also, a fifth product image 223 is obtained, and data enhancement is performed on a partial area (for example, a preset area, or an area selected from multiple preset areas) in the fifth product image 223 (i.e., the image content of the preset area is modified) to obtain an enhanced sixth product image 224. Feature extraction is performed on the sixth product image 224 to obtain a sixth image feature 225. The fourth product image 221 and the fifth product image 223 are images of defect-free products in the same industrial product category. The size of the preset area is smaller than the size of the fifth product image. In some embodiments, the training data used by the defect detection architecture comes from multiple data sets, and multiple data sets are conducive to improving the versatility of the defect detection architecture in product categories to achieve defect detection of multiple categories of products.

[0064] Fourth image feature 222 and sixth image feature 225 are combined (e.g., added) to obtain fourth intermediate feature 226. Fourth intermediate feature 226 is input into defect detection model 216, which outputs training feature 227. Training feature 227 is upsampled to obtain fifth intermediate feature 228. A defect location 229 in the sixth product image is predicted based on fifth intermediate feature 228. The defect location in the sixth product image is provided to the defect detection model, so that the defect detection model adjusts parameters in defect detection model 216 based on the error between predicted defect location 229 in the sixth product image and the location of the partial region subjected to data augmentation.

[0065] It can be understood that the training process 220 will perform data enhancement on part of the area in the fifth product image (such as covering part of the area in the fifth product image by specifying the image content of the image), thereby realizing an unsupervised approach, and the entire defect detection architecture realizes unsupervised anomaly detection.

[0066] It can also be understood that the defect detection model 216 is a large model. The present application provides a defect detection architecture based on the large model for detecting defects in industrial products. By utilizing the reasoning ability of the large model, the defect detection architecture supports the detection of defects in multiple categories of products. The defect detection architecture provided by the present application is universal for product categories.

[0067] In the above, the training device 201 of the defect detection architecture and the using device 202 of the defect detection architecture can be electronic devices with machine learning capabilities, such as computer devices, and the electronic device can be a terminal or a server.

[0068] In some embodiments, the user device 201 and the training device 202 may be the same electronic device, or the user device 201 and the training device 202 may be different electronic devices. Furthermore, when the user device 201 and the training device 202 are different devices, the user device 201 and the training device 202 may be the same type of device, such as both the user device 201 and the training device 202 may be servers; or, the user device 201 and the training device 202 may be different types of devices. The server may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0069] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the product images involved in this application were obtained with full authorization.

[0070] Moreover, when it comes to relevant information, the relevant information processors will follow the principles of legality, legitimacy and necessity, clarify the purpose, method and scope of relevant information processing, obtain the consent of the relevant information subjects, and take necessary technical and organizational measures to ensure the security of relevant information.

[0071] FIG3 shows a flow chart of a method for detecting industrial product defects provided by an exemplary embodiment of the present application. The method is illustrated by an example in which the method is executed by the device 201 shown in FIG2 . The method includes the following steps.

[0072] Step 310: Acquire a first product image and a second product image.

[0073] The first product image and the second product image are images of products having the same or similar appearance (eg, products of the same industrial product category).

[0074] Industrial product categories are obtained by classifying industrial products based on the degree of similarity between their appearances. In some embodiments, industrial products of the same production model are classified into the same industrial product category.

[0075] It is understood that the production goal of the same model of industrial products is to produce identical, defect-free products. In this case, the same model of industrial products are considered to be of the same industrial product category. For example, if the first model is a standard cloth with a lotus pattern, and the second model is a standard cloth with a wavy pattern, the first model of cloth is considered to be of the same industrial product category, while the second model of cloth is considered to be of a different industrial product category.

[0076] In industrial production scenarios, a variety of defects are common, necessitating defect detection. For example, fabric dyeing defect detection aims to detect any discrepancies between the dyed fabric and the customer's sample, such as uneven color, white or black spots, and holes. Due to the wide variety of fabric patterns, production plants must produce fabric with different patterns every few days.

[0077] For example, similar to fabric dyeing defect detection, cardboard printing defect detection involves inspecting the printed cardboard for discrepancies with the customer's sample, such as uneven color, white or black spots, or holes or defects. Furthermore, due to the wide variety of printing patterns, manufacturers often need to produce cardboard with different patterns.

[0078] The first product image is an image of a defect-free industrial product. It should be noted that "defect-free" here should be considered as the product having defects so few that they can be ignored. In the defect detection method to be performed in this application, the first product image will be used as a standard image for comparison with the product image to be detected.

[0079] The second product image is an image of the industrial product to be inspected. The second product image can be an image of a defect-free industrial product or an image of an industrial product with defects. The objective of this application is to detect the location of defects when the second product image is an image of an industrial product with defects.

[0080] It should be noted that the first product image and the second product image can be acquired simultaneously or at different times, and this application does not impose any limitation on this.

[0081] Step 320: extract features from the first product image to obtain first image features.

[0082] In some embodiments, the first product image may be input into several convolutional layers to obtain first image features, where the first image features are feature representations of the first product image.

[0083] Step 330: extract features from the second product image to obtain second image features.

[0084] In some embodiments, the second product image may be input into several convolutional layers to obtain second image features, where the second image features are feature representations of the second product image.

[0085] Step 340: Merge the first image feature and the second image feature to obtain a first intermediate feature.

[0086] For example, the first image feature and the second image feature have the same size, and the first image feature and the second image feature are added to obtain a first intermediate feature, which is used as an input feature of the defect detection model.

[0087] Step 350: Input the first intermediate feature into the defect detection model to obtain an inference feature.

[0088] The defect detection model is used to compare the first product image and the second product image. The inference features are used to characterize the comparison results between the first product image and the second product image. The defect detection model meets at least one of the following conditions: the number of parameters is not less than a parameter threshold, and the number of network layers is not less than a layer threshold. In other words, the defect detection model is a large model, or in other words, the defect detection model is the backbone network (the main functional network) of the large model.

[0089] In some embodiments, the defect detection model is a large model obtained through testing that supports image detection methods across product categories. In some embodiments, the defect detection model can be selected from Vit large, Vit Huge, etc.

[0090] The defect detection model is trained using pre-set images of various objects with different appearances, as well as modified images obtained by modifying the pre-set images. For example, the defect detection model can be trained using images of multiple industrial product categories. Training the defect detection model using images of multiple product categories (i.e., images of multiple objects with different appearances) helps improve the generalization of the defect detection model and, in turn, its ability to detect defects across multiple categories.

[0091] In some embodiments, images of multiple industrial product categories used to train the defect detection model come from multiple datasets, for example, images from both the MVTec dataset and the ViSA dataset, which also helps improve the defect detection model's ability to perform cross-category defect detection.

[0092] Step 360: upsample the inference feature to obtain a second intermediate feature.

[0093] The upsampling operation is used to enlarge the size of the inference features obtained by compressing the defect detection model.

[0094] Step 370: Obtain information about the location of the defect in the second product image based on the second intermediate feature.

[0095] In one embodiment, the number of channels in the second intermediate feature is compressed to three (red, green, and blue) to obtain a third intermediate feature. The length and width of the third intermediate feature are the same as the pixel matrix size of the second product image. For example, the third intermediate feature is 3 × h × w, and the second product image is also represented as 3 × h × w.

[0096] The third intermediate feature is exponentially normalized to generate a segmentation map. The pixel values ​​on the segmentation map represent the probability that the pixel is a defective pixel. For example, a softmax calculation is performed on the third intermediate feature to generate the segmentation map. If required, pixels with values ​​greater than 0.3 (or 0.5) are identified as defective pixels. The locations of all defective pixels constitute the defect location.

[0097] The formula is expressed as F=softmax(Convs(x)), where x is the second intermediate feature, Convs is a convolution operation used to compress the number of channels to three, F is a segmentation map, and softmax is an exponential normalization function.

[0098] In summary, the first intermediate feature is obtained by adding the first image feature corresponding to the first product image and the second image feature corresponding to the second product image; the first intermediate feature is input into the defect detection model to obtain the inferred feature; the inferred feature is upsampled to obtain the second intermediate feature; and information about the defect location is obtained based on the second intermediate feature. The defect detection model is considered large if it meets at least one of the following conditions: the number of parameters reaches a parameter threshold or the number of network layers reaches a layer threshold.

[0099] Specifically, this application provides a large-scale model-based defect detection architecture. The inputs to the defect detection architecture are images of defect-free products and images of products to be inspected. This architecture leverages the reasoning power of the large-scale model, enabling the overall architecture to detect defects across product categories. Compared to related technologies that can only detect defects in a single product category, the defect detection architecture provided by this application is universal.

[0100] Furthermore, in related technologies, the model needs to be retrained for each new product category. In practice, products are updated rapidly (e.g., cloth dyeing), and retraining the model for each new product category seriously delays production progress. The defect detection architecture provided in this application does not require retraining and deployment. Regardless of the changes in the product category, it only needs to provide images of defect-free products and images of products to be inspected, thereby improving overall product production efficiency.

[0101] Furthermore, the defect detection model is trained on images from multiple industrial product categories, which helps improve its generalization and facilitates cross-category defect detection. Furthermore, as previously described, segmentation maps are used to obtain defect location information. Generating segmentation maps is relatively simple, and they intuitively and accurately represent defect pixels, thereby fully revealing product defects.

[0102] Based on the embodiment shown in FIG3 , FIG4 shows a defect detection architecture according to an embodiment.

[0103] (1) A first product image 401 and a second product image 402 are obtained, and feature extraction is performed on the first product image 401 (in some embodiments, performed by some convolutional layers) to obtain a first image feature 403. Feature extraction is performed on the second product image 402 (in some embodiments, performed by some convolutional layers) to obtain a second image feature 404. The first image feature 403 and the second image feature 404 have the same shape.

[0104] Schematically, the first image feature 403 is represented as c×h×w, and the second image feature 404 is represented as c×h×w, where c is the number of channels of the feature, h is the width of the feature, and w is the length of the feature.

[0105] Illustratively, the dimensions of first product image 401 are 3 × h × w, where 3 represents the image's red, green, and blue channels, h represents the image's width, and w represents the image's height. c is an integer greater than 3, and first image feature 403 is used to expand the number of image channels and characterize the image. Illustratively, the dimensions of second product image 402 are 3 × h × w, where 3 represents the image's red, green, and blue channels, h represents the image's width, and w represents the image's height. c is an integer greater than 3, and second image feature 404 is used to expand the number of image channels and characterize the image.

[0106] (2) The first image feature 403 and the second image feature 404 are added to obtain a first intermediate feature 405 . The first intermediate feature 405 has the same shape as the first image feature 403 and the second image feature 404 .

[0107] Illustratively, the first image feature 403 represented as c×h×w and the second image feature 404 represented as c×h×w are added to obtain a first intermediate feature 405 represented as c×h×w.

[0108] (3) Input the first intermediate feature 405 into the defect detection model 406 and output the inferred feature 407. In some embodiments, the defect detection model 406 is used to perform feature compression on the first intermediate feature 405 to obtain the inferred feature 407. The size of the inferred feature 407 is smaller than the size of the second image feature 404 (or the first image feature 403). In some embodiments, the defect detection model 406 is used to perform feature compression on the length and width of the first intermediate feature 405 to the same degree to obtain the inferred feature 407. The length of the inferred feature 407 is smaller than the length of the second image feature 404 (or the first image feature 403), and the width of the inferred feature 407 is smaller than the width of the second image feature 404 (or the first image feature 403).

[0109] Illustratively, the defect detection model is used to perform feature compression on first intermediate features 405 represented as c×h×w to obtain inferred features 407 represented as c×(h / k)×(w / k). k is a positive integer. Illustratively, first intermediate features 405 are represented as c×(h / 32)×(w / 32) and c×(h / 16)×(w / 16).

[0110] (4) Input the inference feature 407 into the decoding network 408 and output the second intermediate feature 409.

[0111] Decoding network 408 is used to restore the inferred feature 407 by upsampling to obtain a second intermediate feature 409. The feature size of second intermediate feature 409 is the same as the feature size of first intermediate feature 405. The size of inferred feature 407 is changed to the size of second image feature 404 (or first image feature 403) by upsampling to obtain second intermediate feature 409.

[0112] In some embodiments, the decoding network 408 is configured to restore the length and width of the inferred feature 407 to the same degree by upsampling, thereby obtaining a second intermediate feature 409. The upsampling is used to change the length of the inferred feature 407 to the length of the second image feature 404 (or the first image feature 403), and to change the width of the inferred feature 407 to the width of the second image feature 404 (or the first image feature 403).

[0113] Illustratively, the decoding network 408 is used to restore the inference feature 407 represented as c×(h / k)×(w / k) by upsampling to obtain a second intermediate feature 409 represented as c×h×w. In some embodiments, the decoding network 408 is a decoder in MAE (a paper titled Masked Autoencoders Are Scalable Vision Learners).

[0114] (5) Based on the second intermediate feature 409 , a step of obtaining information 410 on the location of the defect is performed.

[0115] In summary, the above embodiments provide feature map sizes for each stage in the defect detection architecture, and further provide an overall structural design of the defect detection architecture, so that defect detection can be achieved by simply inputting images of defect-free products and images of products to be inspected.

[0116] Based on the defect detection architecture shown in FIG4 , FIG5 shows a further defect detection architecture.

[0117] FIG5 shows that during defect detection, additional product images 411 are acquired. Additional product images 411 are images of other non-defective industrial products in the same industrial product category as first product image 401. Feature extraction is performed on additional product images 411 (in some embodiments, performed by convolutional layers) to obtain additional image features 412.

[0118] Based on the additional image feature 412 and the first image feature 403 , a template image feature 413 is obtained. The template image feature 413 and the second image feature 404 are combined (eg, added) to obtain a first intermediate feature 405 .

[0119] In some embodiments, the first image feature 403 and the additional image feature 412 have the same shape. The average of the first image feature 403 and the additional image feature 412 is calculated to obtain a template image feature 413. The template image feature 413 has the same shape as the first image feature 403 and the additional image feature 412.

[0120] Illustratively, first image feature 403 and additional image feature 412 are both represented as c×h×w, where c is the number of channels of the feature, h is the width of the feature, and w is the length of the feature, and c, h, and w are positive integers. The average of first image feature 403 represented as c×h×w and additional image feature 412 represented as c×h×w is calculated to obtain template image feature 413 represented as c×h×w.

[0121] In some embodiments, the first product image 401 and the additional image 411 share a convolutional layer for feature extraction.

[0122] Figure 5 also illustrates the step of reconstructing an image 411 based on the second intermediate feature 409. Image reconstruction is performed based on the second intermediate feature to obtain a third product image, which represents the image after defects in the second product image have been repaired. This third product image can be provided to the defect detection model, which uses the differences between the third product image and the first product image to adjust parameters within the defect detection model. In one embodiment, the number of channels in the second intermediate feature is compressed to three (red, green, and blue) before image reconstruction is performed.

[0123] The formula is expressed as F=Convs(x), where x is the second intermediate feature, Convs is a convolution operation used to compress the number of channels to three, and F is the third product image (reconstruction result).

[0124] In summary, the above embodiment derives the features of a template image from the features of multiple images of defect-free products. Different images of defect-free products may have different features, and the template image then fuses the features from multiple conditions to obtain standard features. For example, if one image of a defect-free product is taken under strong light conditions (such as a sunny day), and another image of a defect-free product is taken under low light conditions (such as a rainy day), the fused template image will have lighting features that are closer to the standard defect-free standard, thereby improving the contrast between the defect-free product image and the image of the product to be inspected, resulting in more accurate defect detection results.

[0125] Furthermore, image reconstruction is performed based on the second intermediate feature. The reconstructed image can also be used to repair defects in the image of the product to be inspected.

[0126] FIG6 is a schematic diagram showing a defect detection result provided by an exemplary embodiment of the present application.

[0127] Part (A) of Figure 6 shows an image of a defect-free product (i.e., the first product image). Part (B) of Figure 6 shows an image of the product to be inspected (here, an image of a product with defects). Part (C) of Figure 6 shows the locations of the defects obtained. Part (C) of Figure 6 is the segmentation diagram described above. Part (D) of Figure 6 shows the reconstructed image, i.e., Part (D) of Figure 6 shows the image of the second product image after defect repair. It can be seen that the defect detection architecture has predicted all defects, and the reconstructed image is defect-free.

[0128] After testing, this application achieved an AUROC (Area Under the Receiver Operating Characteristic Curve) of 90 using the Vit Large defect detection model on the MVtec dataset. This simply translates to an accuracy rate of 90%. This approach meets the needs of actual production lines. Fine-tuning the output defect threshold can yield even better results. To ensure universality, the defect threshold is set to 0.5.

[0129] Figure 7 illustrates a flow chart of a defect detection model training method provided by an exemplary embodiment of the present application. Figure 7 illustrates training a defect detection model in an unsupervised manner. In some embodiments, defect locations are predicted end-to-end, and all neural networks in the defect detection framework are trained during training. Figure 7 illustrates a defect detection model training method, illustrated by the training device 202 in Figure 2 . The method includes the following steps.

[0130] Step 710: Acquire a fourth product image and a fifth product image.

[0131] The fourth product image and the fifth product image are images of non-defective products from the same industrial product category. For example, the fourth product image and the fifth product image are images of camera lenses or images of cloth. The fourth product image and the fifth product image are training samples. In some embodiments, the fourth product image and the fifth product image are images from the MVTec dataset. Alternatively, the fourth product image and the fifth product image are images from the ViSA dataset.

[0132] The MVTec dataset contains 5,354 high-resolution color images of various objects and texture types. It includes normal (defect-free) images for training and abnormal images for testing. The abnormalities in the MVTec dataset include 70 different types of defects, such as scratches, dents, contamination, and various structural variations.

[0133] The ViSA dataset contains 12 subsets corresponding to 12 different objects, with a total of 10,821 images, including 9,621 normal samples and 1,200 abnormal samples.

[0134] Step 720: Perform feature extraction on the fourth product image to obtain fourth image features.

[0135] In some embodiments, the fourth product image is input into a plurality of convolutional layers to obtain fourth image features, which are feature representations of the fourth product image.

[0136] In step 730 , data enhancement (ie, modification, for specific modification methods, see the above description) is performed on a predetermined partial area of ​​the fifth product image to obtain a sixth product image.

[0137] In some embodiments, the image content of a designated image is overlaid on a preset partial area of ​​the fifth product image to obtain a sixth product image, and the image content of the designated image is different from the image content of the preset partial area on the fifth product image.

[0138] Illustratively, a preset partial area on the fifth product image is cropped, and the image content of the designated image is copied and pasted to the preset partial area on the fifth product image to obtain a sixth product image.

[0139] Step 740: extract features from the sixth product image to obtain sixth image features.

[0140] In some embodiments, the sixth product image is input into several convolutional layers to obtain a sixth image feature, which is a feature representation of the sixth product image.

[0141] Step 750: Combine (for example, add) the fourth image feature and the sixth image feature to obtain a fourth intermediate feature.

[0142] For example, the fourth image feature and the sixth image feature have the same size, and the fourth image feature and the sixth image feature are added to obtain a fourth intermediate feature, which is used as an input feature of the defect detection model.

[0143] Step 760: Input the fourth intermediate feature into the defect detection model to obtain a training feature.

[0144] The fourth intermediate feature is input into the defect detection model to obtain a training feature.

[0145] Step 770: upsample the training features to obtain the fifth intermediate features.

[0146] The upsampling operation is used to enlarge the size of the inference features obtained by compressing the defect detection model.

[0147] Step 780: Obtain information about the location of the defect in the sixth product image based on the fifth intermediate feature.

[0148] Based on the fifth intermediate feature, information about the location of the defect in the sixth product image is obtained.

[0149] Step 790: Provide the location of the defect in the sixth product image to the defect detection model, so that the defect detection model adjusts parameters in the defect detection model based on the error between the obtained location of the defect and the location of the preset area.

[0150] In some embodiments, the defect detection model is adjusted based on the error between the pixel coordinates of the defect location and the pixel coordinates of the partial area. The error can be used to optimize the defect detection model's ability to predict the defect location.

[0151] In one embodiment, training device 201 further reconstructs an image based on the fifth intermediate feature to obtain a seventh product image. The defect detection model is then trained based on the error between the seventh product image and the fifth product image. The error between the reconstructed image and the original image helps the defect detection model optimize its ability to recognize images of non-defective products. Furthermore, the defect detection model optimizes its understanding of the structural information in images of non-defective products.

[0152] It should be noted that the training process and usage process of the defect detection architecture are similar. For other content about the training process of the defect detection architecture, please refer to the introduction of the usage process above.

[0153] In summary, the above embodiment provides an unsupervised defect detection model training method. By performing data augmentation on a partial area and training the defect detection model based on the error between the partial area and the predicted defect location, the characteristics of unsupervised anomaly detection are met.

[0154] Furthermore, the above embodiment utilizes the reconstruction error between the reconstructed image and the original image to train the defect detection model. This reconstruction error not only helps the defect detection model recognize that the first product image is a defect-free product image (a normal image), but also helps the defect detection model learn the structural information of the defect-free product image, thereby facilitating the prediction of defect locations.

[0155] FIG8 shows a schematic diagram of an industrial product defect detection framework provided by an exemplary embodiment of the present application.

[0156] (1) Template image branch: Given N template images 801 (i.e., images of normal products), they are input into the template shared convolution block (i.e., some convolution layers, which are not fixed and can be changed). The size of the feature output of each image is c×h×w (where c is the number of channels, h is the width of the image feature after convolution, and w is the length of the image feature after convolution). Template image feature merging is performed (direct averaging of multiple images) to obtain a c×h×w feature.

[0157] (2) Input graph branch: The input graph 803 passes through the input convolution block (that is, some convolution layers, which are not fixed and can change), and the feature size of the obtained input graph is also c×h×w.

[0158] (3) Large model backbone network 805: The features of the input image and the template image are directly added, and the feature shape is still c×h×w. Then, the features are extracted through the large model backbone network 805 (the large model backbone network is a network with a relatively large number of parameters, such as Vit Large, Vit Huge).

[0159] (4) Decoding Network 806: Because the output of the large model backbone network 805 compresses the features to a relatively small size, generally, the input c×h×w will become c×(h / 32)×(w / 32) or c×(h / 16)×(w / 16), etc., so it is necessary to perform upsampling operations through some convolutional layers. In some embodiments, the decoding network 806 is the decoder in the MAE.

[0160] The main function of the decoding network is to restore the features to the size of the image. The number of channels of the last network layer of the decoding network is increased, and the output features are limited to the size of the input image 803.

[0161] (5) Reconstruction branch 807: The reconstruction branch aims to repair the input image 803 into a defect-free image. Because unsupervised anomaly detection does not have a supervisory signal, but the large model must understand the features of the input template image to infer anomalies, the reconstruction branch 807 uses the features of the large model backbone network 805 to reconstruct the original image, helping the large model to acquire the cognitive ability of the input template image.

[0162] During the training process, because of unsupervised training, all images are normal and have no defects. By adding some data enhancement, such as copying and pasting the specified image area to the input image 803, directly cropping some black areas, and then reconstructing the enhanced image in the reconstruction branch 807 to reconstruct the input image 803 before enhancement.

[0163] Due to the emphasis on generality, model training is best performed across datasets. In some embodiments, MvTec and ViSA are used as training datasets.

[0164] (6) Defect location prediction branch 808: This branch directly outputs a segmentation map of the same size as the original image. Each pixel has an abnormal branch, which reflects end-to-end defect detection.

[0165] Because the model can directly predict defect locations end-to-end (defect location prediction branch), after directly inputting the template image 801, the image to be inspected (input image 803) is input, and the segmentation map predicted by the defect location prediction branch 808 is used. The length and width of the segmentation map are consistent with the original image, and each pixel value is the probability that the pixel is a defect, with the probability value in the range [0, 1]. Whether a pixel is a defective pixel is determined by using a threshold. For example, a threshold of 0.3 or 0.5 can be used according to actual needs.

[0166] During the training process, because of unsupervised training, all images are normal and have no defects. By adding some data enhancement, such as copying and pasting the area of ​​the specified image to the input image 803, some black areas are directly cropped out, and then the defect location prediction branch 808 predicts the cropped area.

[0167] FIG9 shows a structural block diagram of an industrial product defect detection device provided by an exemplary embodiment of the present application. The device includes the following modules.

[0168] Acquisition module 901 is used to acquire a first product image and a second product image. The first product image and the second product image are images of products with the same or similar appearance (for example, products under the same industrial product category). The first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected.

[0169] The feature extraction module 902 is configured to perform feature extraction on the first product image to obtain first image features; and perform feature extraction on the second product image to obtain second image features.

[0170] Processing module 903 is configured to add the first image feature and the second image feature to obtain a first intermediate feature, and input the first intermediate feature into a defect detection model to obtain an inferred feature. The defect detection model is trained using preset images of various items with different appearances (e.g., products from multiple industrial product categories) and modified images obtained by modifying the preset images.

[0171] The processing module 903 is further configured to upsample the inference feature to obtain a second intermediate feature.

[0172] The prediction module 904 is configured to obtain information about the location of the defect in the second product image based on the second intermediate feature.

[0173] In one embodiment, the size of the inference feature is smaller than the size of the second image feature. The processing module 903 is further configured to change the size of the inference feature to the size of the second image feature by upsampling to obtain a second intermediate feature.

[0174] In one embodiment, the length of the inferred feature is smaller than the length of the second image feature, and the width of the inferred feature is smaller than the width of the second image feature. Processing module 903 is configured to upsample the length of the inferred feature to the length of the second image feature and the width of the inferred feature to the width of the second image feature, thereby obtaining a second intermediate feature.

[0175] In one embodiment, the acquisition module 901 is further configured to acquire additional product images, which are images of other non-defective industrial products belonging to the same industrial product category as the first product image. The feature extraction module 902 is further configured to extract features from the additional product images to obtain additional image features, and then combine the additional image features with the first image features to obtain template image features. The processing module 903 is further configured to add the template image features and the second image features to obtain a first intermediate feature.

[0176] In one embodiment, the first image feature and the additional image feature have the same size. The feature extraction module 902 is further configured to calculate an average value of the first image feature and the additional image feature to obtain a template image feature, wherein the template image feature has the same size as the first image feature and the additional image feature.

[0177] In one embodiment, the prediction module 904 is further configured to compress the number of channels of the second intermediate feature to three to obtain a third intermediate feature, wherein the length and width of the third intermediate feature are the same as the size of the pixel matrix of the second product image;

[0178] The third intermediate feature is subjected to an exponential normalization operation to obtain a segmentation map, and the pixel value of a pixel point on the segmentation map represents the probability that the pixel point is a defective pixel point.

[0179] In one embodiment, the apparatus further includes a reconstruction module 905. The reconstruction module 905 is configured to perform image reconstruction based on the second intermediate feature to obtain a third product image, where the third product image represents an image after defects in the second product image are repaired.

[0180] In one embodiment, the acquisition module 901 is also used to acquire a fourth product image and a fifth product image, where the fourth product image and the fifth product image are images of defect-free products under the same industrial product category; and data enhancement is performed on a partial area of ​​the fifth product image to obtain a sixth product image.

[0181] The feature extraction module 902 is further configured to perform feature extraction on the fourth product image to obtain a fourth image feature; and perform feature extraction on the sixth product image to obtain a sixth image feature.

[0182] The processing module 903 is further used to merge the fourth image feature and the sixth image feature to obtain a fourth intermediate feature; input the fourth intermediate feature into the defect detection model to obtain a training feature; and upsample the training feature to obtain a fifth intermediate feature.

[0183] The prediction module 904 is further configured to obtain information about a location of a defect in the sixth product image based on the fifth intermediate feature.

[0184] The apparatus further includes a training module 906. The training module 906 is configured to train a defect detection model based on the predicted locations of the defects and the errors between the locations of the partial regions.

[0185] In one embodiment, the acquisition module 901 is further configured to overlay the image content of a designated image on a partial area of ​​the fifth product image to obtain a sixth product image, where the image content of the designated image is different from the image content of the partial area of ​​the fifth product image.

[0186] In one embodiment, the reconstruction module 905 is further configured to provide the defect location in the sixth product image to the defect detection model, so that the defect detection model performs image reconstruction based on the fifth intermediate feature to obtain a seventh product image. The training module 906 is further configured to train the defect detection model based on the error between the seventh product image and the fifth product image.

[0187] In one embodiment, the images of multiple industrial product categories used to train the defect detection model are derived from multiple datasets. For example, the acquisition module 901 may acquire the fourth product image and the fifth product image from multiple datasets, wherein the multiple datasets include images of multiple items with different appearances.

[0188] In summary, the first intermediate feature is obtained by adding the first image feature corresponding to the first product image and the second image feature corresponding to the second product image; the first intermediate feature is input into the defect detection model to obtain the inferred feature; the inferred feature is upsampled to obtain the second intermediate feature; and the defect location is predicted based on the second intermediate feature. The defect detection model is considered large if it meets at least one of the following conditions: the number of parameters reaches a parameter threshold or the number of network layers reaches a layer threshold.

[0189] Specifically, this application provides a large-scale model-based defect detection architecture. The inputs to the defect detection architecture are images of defect-free products and images of products to be inspected. This architecture leverages the reasoning power of the large-scale model, enabling the overall architecture to detect defects across product categories. Compared to related technologies that can only detect defects in a single product category, the defect detection architecture provided by this application is universal.

[0190] Furthermore, in related technologies, the model needs to be retrained for each new product category. In practice, products are updated rapidly (e.g., cloth dyeing), and retraining the model for each new product category seriously delays production progress. The defect detection architecture provided in this application does not require retraining and deployment. Regardless of the changes in the product category, it only needs to provide images of defect-free products and images of products to be inspected, thereby improving overall product production efficiency.

[0191] In addition, the defect detection model is trained based on images of multiple industrial product categories, which helps to improve the generalization of the defect detection model and thus facilitates the defect detection model to perform cross-category defect detection.

[0192] Figure 10 is a schematic diagram of the structure of a computer device according to an exemplary embodiment. The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory (RAM) 1002 and a read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 for facilitating information transmission between various components within the computer device, and a mass storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0193] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009, such as a mouse or keyboard, for user input. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0194] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable medium provide non-volatile storage for the computer device 1000. In other words, the mass storage device 1007 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0195] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer device storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0196] According to various embodiments of the present disclosure, the computer device 1000 may also be connected to a remote computer device on a network such as the Internet for operation. That is, the computer device 1000 may be connected to the network 1011 via the network interface unit 1012 connected to the system bus 1005, or the network interface unit 1012 may be used to connect to other types of networks or remote computer device systems (not shown).

[0197] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1001 implements all or part of the steps of the above-mentioned industrial product defect detection method by executing the one or more programs.

[0198] FIG11 illustrates a block diagram of a computer device 1100 according to an exemplary embodiment of the present application. Computer device 1100 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Computer device 1100 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other similar names.

[0199] Typically, the computer device 1100 includes a processor 1101 and a memory 1102 .

[0200] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0201] Memory 1102 may include one or more computer-readable storage media, which may be non-transitory. Memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1102 is used to store at least one instruction, which is executed by processor 1101 to implement the industrial product defect detection method provided in the method embodiment of the present application.

[0202] In some embodiments, the computer device 1100 may optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1103 via a bus, signal lines, or circuit boards. For example, the peripheral device may include at least one of a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, a power supply 1108, and one or more sensors 1109. The one or more sensors 1109 include, but are not limited to, an accelerometer 1110, a gyroscope sensor 1111, a pressure sensor 1112, an optical sensor 1113, and a proximity sensor 1114.

[0203] Those skilled in the art will appreciate that the structure shown in FIG11 does not limit the computer device 1100 and may include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.

[0204] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the industrial product defect detection method provided in the above method embodiment.

[0205] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the industrial product defect detection method provided in the above method embodiment.

[0206] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0207] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0208] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0209] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for detecting defects in industrial products, comprising: The electronic device acquires a first product image and a second product image, wherein the first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected; The electronic device extracts features from the first product image to obtain first image features; and extracts features from the second product image to obtain second image features; The electronic device combines the first image feature and the second image feature to obtain a first intermediate feature; inputs the first intermediate feature into a defect detection model to obtain an inference feature, wherein the defect detection model is trained using preset images of multiple objects with different appearances and modified images obtained by modifying the preset images; The electronic device upsamples the inference feature to obtain a second intermediate feature; The electronic device obtains information about a location of a defect in the second product image based on the second intermediate feature.

2. The method according to claim 1, wherein the size of the inference feature is smaller than the size of the second image feature; The upsampling of the inference feature to obtain a second intermediate feature includes: The size of the inference feature is changed to the size of the second image feature through the upsampling to obtain the second intermediate feature.

3. The method according to claim 2, wherein the length of the inference feature is smaller than the length of the second image feature, and the width of the inference feature is smaller than the width of the second image feature; The step of changing the size of the inference feature to the size of the second image feature by upsampling to obtain the second intermediate feature includes: The upsampling is used to change the length of the inference feature to the length of the second image feature, and to change the width of the inference feature to the width of the second image feature, so as to obtain the second intermediate feature.

4. The method according to any one of claims 1 to 3, further comprising: Acquire an additional product image, where the additional product image and the first product image are images of defect-free industrial products having the same or similar appearance; Extracting features from the additional product image to obtain additional image features; Combining the additional image feature with the first image feature to obtain a template image feature; The first image feature and the second image feature are combined to obtain a first intermediate feature, including: The template image feature and the second image feature are combined to obtain the first intermediate feature.

5. The method according to claim 4, wherein the first image feature and the additional image feature have the same size; and the step of combining the additional image feature and the first image feature to obtain the template image feature comprises: An average value of the first image feature and the additional image feature is calculated to obtain the template image feature, and the template image feature, the first image feature, and the additional image feature are all the same in size.

6. The method according to any one of claims 1 to 3, wherein obtaining information about the location of the defect in the second product image based on the second intermediate feature comprises: Compressing the number of channels of the second intermediate feature to three to obtain a third intermediate feature, wherein the third intermediate feature has the same size as the pixel matrix of the second product image; The third intermediate feature is subjected to an exponential normalization operation to obtain a segmentation map, wherein the pixel value of a pixel point on the segmentation map represents the probability that the pixel point is a defective pixel point.

7. The method according to any one of claims 1 to 3, further comprising: Reconstructing an image based on the second intermediate feature to obtain a third product image, where the third product image represents an image after defects in the second product image are repaired; The third product image is provided to the defect detection model, and the defect detection model uses the difference between the third product image and the first product image to adjust parameters in the defect detection model.

8. The method according to any one of claims 1 to 3, further comprising: acquiring a fourth product image and a fifth product image, wherein the fourth product image and the fifth product image are images of a non-defective product; Modifying the image content of the preset area of ​​the fifth product image to obtain a sixth product image, wherein the size of the preset area is smaller than the size of the fifth product image; Using the feature extraction model to perform feature extraction on the fourth product image to obtain a fourth image feature; and performing feature extraction on the sixth product image to obtain a sixth image feature; Combining the fourth image feature and the sixth image feature to obtain a fourth intermediate feature; inputting the fourth intermediate feature into the defect detection model to obtain a training feature; Upsampling the training feature to obtain a fifth intermediate feature; Based on the fifth intermediate feature, obtaining information about a location of a defect in the sixth product image; The location of the defect in the sixth product image is provided to the defect detection model, so that the defect detection model adjusts parameters in the defect detection model based on an error between the location of the defect and the location of the preset area.

9. The method according to any one of claims 1 to 3, wherein: The modified image includes at least one of the following: The modified image is obtained by overlaying the image content of a designated image on the preset area of ​​the preset image, wherein the image content of the designated image is different from the image content of the preset area on the preset image; The modified image is obtained by modifying the values ​​of pixels in a preset area of ​​the preset image to preset values; Obtaining the modified image by overlaying a preset pattern on a preset area of ​​the preset image; Wherein, the size of the preset area is smaller than the size of the preset image.

10. The method according to claim 8, further comprising: Performing image reconstruction based on the fifth intermediate feature to obtain a seventh product image; The seventh product image is provided to the defect detection model so that the defect detection model adjusts parameters in the defect detection model based on an error between the seventh product image and the fifth product image.

11. The method according to any one of claims 1 to 3, wherein: The preset images are selected from a plurality of data sets, and the plurality of data sets include images of a plurality of objects with different appearances.

12. An industrial product defect detection device, comprising: An acquisition module, used to acquire a first product image and a second product image, wherein the first product image is an image of a defect-free industrial product, and the second product image is an image of an industrial product to be inspected; a feature extraction module, configured to extract features from the first product image to obtain first image features; and to extract features from the second product image to obtain second image features; a processing module, configured to combine the first image feature and the second image feature to obtain a first intermediate feature; input the first intermediate feature into a defect detection model to obtain an inference feature, wherein the defect detection model is trained using preset images of multiple objects with different appearances and modified images obtained by modifying the preset images; The processing module is further used to upsample the inference feature to obtain a second intermediate feature; A prediction module is used to obtain information about the location of the defect in the second product image based on the second intermediate feature.

13. The device according to claim 12, wherein: The feature extraction module is used for: Acquire an additional product image, where the additional product image and the first product image are images of defect-free industrial products having the same or similar appearance; Extracting features of the additional product image to obtain additional image features; Combining the additional image feature with the first image feature to obtain a template image feature; Wherein, the processing module is used for: The template image feature and the second image feature are combined to obtain the first intermediate feature.

14. The apparatus according to claim 12, wherein the prediction module is configured to: Compressing the number of channels of the second intermediate feature to three to obtain a third intermediate feature, wherein the third intermediate feature has the same size as the pixel matrix of the second product image; The third intermediate feature is subjected to an exponential normalization operation to obtain a segmentation map, wherein the pixel value of a pixel point on the segmentation map represents the probability that the pixel point is a defective pixel point.

15. The apparatus according to claim 12, further comprising a reconstruction module, configured to: Reconstructing an image based on the second intermediate feature to obtain a third product image, where the third product image represents an image after defects in the second product image are repaired; The third product image is provided to the defect detection model, and the defect detection model uses the difference between the third product image and the first product image to adjust parameters in the defect detection model.

16. The device according to any one of claims 12 to 15, wherein: The acquisition module is further used to acquire a fourth product image and a fifth product image, wherein the fourth product image and the fifth product image are images of defect-free products; modify the image content of a preset area of ​​the fifth product image to obtain a sixth product image, wherein the size of the preset area is smaller than the size of the fifth product image; The feature extraction module is further used to perform feature extraction on the fourth product image to obtain a fourth image feature; and perform feature extraction on the sixth product image to obtain a sixth image feature; The processing module is further used to merge the fourth image feature and the sixth image feature to obtain a fourth intermediate feature; input the fourth intermediate feature into the defect detection model to obtain a training feature; and upsample the training feature to obtain a fifth intermediate feature; The prediction module is further used to obtain information about the location of the defect in the sixth product image based on the fifth intermediate feature; The device further includes a training module for providing the location of the defect in the sixth product image to the defect detection model so that the defect detection model adjusts the parameters in the defect detection model based on the error between the predicted location of the defect and the location of the preset area.

17. The device according to claim 16, wherein: The acquisition module is used for: The modified image is obtained by overlaying the image content of a designated image on the preset area of ​​the fifth product image, wherein the image content of the designated image is different from the image content of the preset area on the fifth product image; The modified image is obtained by modifying the values ​​of pixels in a preset area of ​​the fifth product image to preset values; The modified image is obtained by overlaying a preset pattern on a preset area of ​​the fifth product image.

18. The device according to claim 16, wherein: The acquisition module is used for: The fourth product image and the fifth product image are obtained from a plurality of data sets including images of a plurality of items having different appearances.

19. A computer device, characterized in that: The computer device comprises: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the method for detecting industrial product defects according to any one of claims 1 to 11.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the method for detecting industrial product defects according to any one of claims 1 to 11.

Images