Product defect detection method and related apparatus

Abstract

This application discloses a product defect detection method and related apparatus. The product defect detection method includes: extracting features from an actual product image and a standard product image respectively, obtaining actual product features and standard product features, where the actual product image is taken of the target product in its actual state, and the standard product image is taken of the target product in its standard state; stitching the actual product features and standard product features along the channel direction to obtain stitched product features; using a distance metric model based on the stitched product features to obtain a distance map between the actual product image and the standard product image; and determining whether the target product in its actual state has a defect based on the distance map. This solution can improve the accuracy of product defect detection results and save storage space and computational overhead.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a product defect detection method and related apparatus. Background Technology

[0002] Image-based product defect detection aims to detect product defects during the quality inspection stage by examining product images. Product defect types can be categorized into background defects and character defects. Character defects include missing characters and misprinted characters. Background defects include background cracks, dirt, ink defects, and broken backgrounds.

[0003] The product defect detection methods in related technologies have insufficient accuracy in obtaining product defect detection results, require a large amount of storage space, and require a large amount of computational overhead. Summary of the Invention

[0004] This application provides a product defect detection method and related apparatus, which can solve the problems of insufficient accuracy of product defect detection results, large storage space requirements, and large computational overhead in related technologies.

[0005] This application provides a product defect detection method, comprising: extracting features from an actual product image and a standard product image respectively to obtain actual product features and standard product features, wherein the actual product image is a photograph of the target product in its actual state, and the standard product image is a photograph of the target product in its standard state; stitching the actual product features and the standard product features along the channel direction to obtain stitched product features; using a distance metric model based on the stitched product features to obtain a distance map between the actual product image and the standard product image; and determining whether the target product in its actual state has defects based on the distance map.

[0006] This application provides a product defect detection device, including: a feature extraction module, a stitching module, a distance measurement module, and a determination module. The feature extraction module extracts features from an actual product image and a standard product image, respectively, to obtain actual product features and standard product features. The actual product image is taken of the target product in its actual state, and the standard product image is taken of the target product in its standard state. The stitching module stitches the actual product features and standard product features along their respective channel directions to obtain stitched product features. The distance measurement module uses a distance measurement model based on the stitched product features to obtain a distance map between the actual product image and the standard product image. The determination module determines whether the target product in its actual state has defects based on the distance map.

[0007] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described method.

[0008] This application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described method.

[0009] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0010] The above scheme stitches together actual product features and standard product features along the channel direction to obtain stitched product features. A distance metric model is then used to generate a distance map based on these stitched product features. Compared to the fixed and simple distance metric method of feature level difference, this approach offers several advantages. First, the distance metric model, through training, can learn a more complex and non-linear adaptive distance metric. This adaptive metric can suppress reasonable minor differences caused by non-defects, thus exhibiting high robustness to reasonable minor differences and resulting in a more accurate distance map. This, in turn, leads to more accurate product defect detection results and reduces the false positive rate for defects with reasonable minor differences. Second, the distance metric model has encountered diverse defects during training, resulting in good generalization ability. Only one or a few standard state product features are needed to obtain a more accurate distance map, eliminating the need to store all standard state product image features, saving storage space, and eliminating the need for comparison with all standard state product image features, thus saving computational overhead.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0013] Figure 1 This is a schematic flowchart of an embodiment of the product defect detection method provided in this application; Figure 2 This is a structural schematic diagram of a specific example of the product defect detection model of this application; Figure 3 This is a schematic diagram of the standard product image and the actual product image in this application; Figure 4 This is a structural diagram of a specific example of the spatial alignment model of this application; Figure 5 This is a structural diagram of a specific instance of the distance metric model in this application; Figure 6 This is a schematic diagram of the binarized distance map of this application; Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 8 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0014] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0015] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0016] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. The term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. Finally, the term "several" in this document means any integer greater than 0, such as 1, 2, 3, 4, 5, ...

[0017] Image-based product defect detection aims to detect product defects during the quality inspection stage by examining product images. Product defect types can be categorized into background defects and character defects. Character defects include missing characters and misprinted characters. Background defects include background cracks, dirt, ink defects, and broken backgrounds.

[0018] In related technologies, image-based product defect detection methods can be divided into two categories: The first type involves taking pictures of the target product in its actual state to obtain an image of the actual product, and taking pictures of the target product in its absolute standard state to obtain a standard product image (template). The actual product image and the standard product image are then compared at the pixel level to obtain a pixel-level distance map. Based on the pixel-level distance map, it is determined whether the product in its actual state has defects.

[0019] The second approach involves pre-storing multiple standard product images and their corresponding standard product features in an image feature library. These standard product images are taken from target products in multiple standard states. Feature-level differences are then performed between the actual product features in the actual product images and the multiple standard product features to obtain multiple feature-level distance maps. Based on the feature-level distance map between the actual product image and the standard product feature most similar to the actual product feature, it is determined whether the actual product has defects.

[0020] The inventors of this application have discovered, through research, that the aforementioned Class I and Class II methods have at least the following technical problems: Minor differences in lighting, font, shooting angle, and color are reasonable minor differences. As long as the actual changes are reasonable minor differences, there is no defect.

[0021] Category 1: This category requires extremely high registration / spatial alignment accuracy between actual product images and standard product images. Furthermore, it requires that the actual defect remain completely unchanged relative to the absolute standard state to be considered defect-free. It cannot tolerate reasonable minor differences not caused by defects, and is overly sensitive and reacts too significantly to such minor differences, meaning that reasonable minor differences may be identified as defects, leading to a high false positive rate. Therefore, it is too rigid, inflexible, and lacks robustness.

[0022] The second category: Actual / standard product features possess a certain degree of spatial invariance relative to actual / standard product images, thus requiring lower spatial alignment between actual and standard product images. Furthermore, by comparing with standard product features from multiple standard states (absolute standard states and all relative standard states, where absolute standard states are variants of standard states with reasonably minor differences), reasonably minor differences can be tolerated. However, this approach consumes a large amount of storage space and the feature comparison process is computationally time-consuming.

[0023] Therefore, this application proposes a new method for detecting product defects.

[0024] Figure 1 This is a schematic flowchart of an embodiment of the product defect detection method provided in this application. Figure 1 As shown, in this embodiment, the product defect detection method may include the following steps: S110: Extract features from the actual product image and the standard product image respectively to obtain the actual product features and the standard product features.

[0025] The execution subject of this embodiment is a product defect detection device, which can be any electronic device with product defect detection capability.

[0026] Actual product images are taken from the target product in its actual state, while standard product images are taken from the target product in its standard state.

[0027] In some embodiments, the standard product image is a single image, which is a photograph of a target product in an absolute standard state or a relative standard state. The relative standard state is a variation of the absolute standard state within a reasonably small range of differences.

[0028] In some embodiments, there are multiple standard product images, which are obtained by capturing images of the target product in multiple standard states. These multiple standard states may include absolute standard states, multiple absolute standard states, or only multiple absolute standard states.

[0029] In some embodiments, the actual product feature is a single-depth actual product feature. The standard product feature is a single-depth actual product feature.

[0030] In some embodiments, actual product features include multiple depths of actual product features (actual product feature pyramid), and standard product features include multiple depths of standard product features (standard product feature pyramid).

[0031] Understandably, actual / standard product features with lower depth contain more detailed information, while actual / standard product features with higher depth contain more global semantic information. Actual / standard product features with multiple depths can more comprehensively express the actual / standard state of the product.

[0032] S120: The actual product features and standard product features are spliced ​​together in the channel direction to obtain the spliced ​​product features.

[0033] In some embodiments, the spliced ​​product feature is a single-depth spliced ​​product feature, which is obtained by splicing single-depth actual product features with single-depth actual product features in the channel direction.

[0034] In some embodiments, the spliced ​​product features include spliced ​​product features of multiple depths, and a spliced ​​product feature of one depth is obtained by splicing together actual product features of the corresponding depth and standard product features of the corresponding depth.

[0035] Understandably, multi-depth splicing of product features helps capture subtle differences and identify small-sized defects. Furthermore, multi-depth splicing of product features allows for a more comprehensive representation of the same defect from multiple depths.

[0036] S130: Using a distance metric model based on stitched product features, a distance map is obtained between the actual product image and the standard product image.

[0037] In this distance map, the distance value of each pixel represents the degree of difference between corresponding pixels in the actual product image and the standard product image. The distance value is positively correlated with the degree of difference.

[0038] S140: Determine whether the target product in its actual state has defects based on the distance map.

[0039] In some embodiments, the defect type includes at least one of product background defects and product character defects. Product background defects refer to defects in the background area of ​​the product in an actual product image, while product character defects refer to defects in the character area of ​​the product in an actual product image. The product character area is the area containing the product characters, which can be numbers, letters, symbols, text, etc. The product background area is the area excluding the product characters. Product character defects include missing character printing defects and misprinted character defects. Product background defects include background crack defects, background dirt defects, background ink defects, and background damage defects. The defect type includes product character defects only when there are characters on the surface of the target product.

[0040] The above scheme stitches together actual product features and standard product features along the channel direction to obtain stitched product features. A distance metric model is then used to generate a distance map based on these stitched product features. Compared to the fixed and simple distance metric method of feature level difference, this approach offers several advantages. First, the distance metric model, through training, can learn a more complex and non-linear adaptive distance metric. This adaptive metric can suppress reasonable minor differences caused by non-defects, thus exhibiting high robustness to reasonable minor differences and resulting in a more accurate distance map. This, in turn, leads to more accurate product defect detection results and reduces the false positive rate for defects with reasonable minor differences. Second, the distance metric model has encountered diverse defects during training, resulting in good generalization ability. Only one or a few standard state product features are needed to obtain a more accurate distance map, eliminating the need to store all standard state product image features, saving storage space, and eliminating the need for comparison with all standard state product image features, thus saving computational overhead.

[0041] In some embodiments, S130 includes S131-S132. S131: Perform difference measurement on the features of the spliced ​​products to obtain a multi-channel distance map. S132: Compress the multi-channel distance map to obtain a single-channel distance map.

[0042] In the multi-channel distance map, each pixel has a multi-channel distance value, which represents the response intensity of the pixel corresponding to the actual product feature and the standard product feature in multiple implicit difference modes.

[0043] Understandably, since the difference metric is learnable during the training phase, the distance metric model can learn multiple implicit difference patterns between defective real-world states and standard states during training. Compared to explicit difference patterns set manually based on experience, implicit difference patterns can more accurately distinguish between defective and undefective real-world states, thus obtaining a more accurate multi-channel distance map.

[0044] In the single-channel distance map, each pixel has a single-channel distance value, which is obtained by compressing the multi-channel distance values.

[0045] Understandably, since compression is learnable during the training phase, the distance metric model can learn the compression weights for each implicit difference pattern during training. Compared to manually set compression weights based on experience, the learned compression weights can more accurately quantify the importance of each difference pattern and the correlation between them, thus obtaining a more accurate single-channel distance map.

[0046] In some embodiments, the difference measure is implemented through a first convolution operation.

[0047] The parameters of the first convolution kernel corresponding to the first convolution operation are learned during the training phase. The first convolution kernel can be 3... 3. Convolution, etc.

[0048] In some embodiments, compression is achieved through a second convolution operation.

[0049] The parameters of the second convolution kernel corresponding to the second convolution operation are learned during the training phase. The second convolution kernel can be 1... 1. Convolution.

[0050] In some embodiments, when there are stitched product features at multiple depths, S131 includes S1311-S1312. S1311: Perform difference measurement on the stitched product features at each depth to obtain a multi-channel distance map for each depth; S1312: Fuse the multi-channel distance maps at each depth to obtain the final multi-channel distance map. The fusion can be achieved, but is not limited to, through stitching and convolution operations.

[0051] In some embodiments, S1312 includes: upsampling the multi-channel distance maps at various depths to the dimensions of the actual / standard product image and then fusing them to obtain the final multi-channel distance map.

[0052] In some embodiments, the multi-channel feature maps at each depth include a multi-channel feature map at a first depth, a multi-channel feature map at a second depth, and a multi-channel feature map at a third depth, with the first depth, second depth, and third depth increasing sequentially.

[0053] In some embodiments, S1312 includes: upsampling the multi-channel feature map of the third depth and fusing it with the multi-channel feature map of the second depth to obtain a first intermediate multi-channel feature map; upsampling the intermediate multi-channel feature map and fusing it with the multi-channel feature map of the first depth to obtain a second intermediate multi-channel feature map; and upsampling the second intermediate multi-channel feature map to obtain a final multi-channel distance map.

[0054] In some embodiments, when there are multiple standard product images, S140 includes: performing a statistical analysis on the distance map between the actual product image and the multiple standard product images to obtain a final distance map. The statistical analysis can be, but is not limited to, averaging.

[0055] In some embodiments, the final distance map may also be normalized to normalize the distance values ​​of each pixel to the range of (0, 1).

[0056] In some embodiments, S140 includes: determining whether there are pixels in the actual product image whose distance value is greater than a distance threshold; in response to the existence of pixels whose distance value is greater than the distance threshold, determining that the target product in the actual state has a defect, and taking the image area formed by each pixel whose distance value is greater than the distance threshold as the area with a defect; in response to the absence of pixels whose distance value is greater than the distance threshold, determining that the target product in the actual state has no defect.

[0057] In some embodiments, before S110, the method further includes: stitching the actual product image and the standard product image in the channel direction to obtain a stitched product image; performing a third convolution operation, a pooling operation, and a fully connected operation on the stitched product image in sequence to obtain spatial transformation parameters of the actual product image relative to the standard product image; and using the spatial transformation parameters to perform spatial transformation on the actual product image to update the actual product image.

[0058] Among them, the parameters of the third convolution operation, pooling operation, and fully connected operation are all learnable parameters.

[0059] In some embodiments, spatial transformation is performed on the actual product image using spatial transformation parameters to update the actual product image. This includes: obtaining the coordinates of each first pixel in the updated actual product image in the original actual product image based on the spatial transformation parameters; using the first pixel with integer coordinates as the second pixel, and using the pixel value of the pixel at the integer coordinate in the original actual product image as the pixel value of the second pixel in the updated actual product image; using the first pixel with floating-point coordinates as the third pixel, and performing interpolation calculations on the pixel values ​​of the pixels surrounding the floating-point coordinate in the original actual product image to obtain the pixel value of the third pixel in the updated actual product image. The interpolation calculation can be, but is not limited to, implemented using bilinear interpolation sampling.

[0060] In some embodiments, the product defect detection method is implemented based on a product defect detection network, which is trained using a set of product image pairs. This set includes several product image sample pairs of various products, including the target product and other products. A product image sample pair includes a standard product image sample and its corresponding defective product image sample. The standard product image sample represents a product in a standard state, while the defective product image sample represents a product in an actual state with defects. The defective product image samples in different product image sample pairs of the same product cover multiple types of defects for the same product. Training the product defect detection network using multiple types of defects from various products improves its generalization ability regarding products and defects.

[0061] In some embodiments, the training of the product defect detection network is divided into multiple training rounds. The training steps of the product defect detection network include: using at least one training round as a perturbation training round; before training the perturbation training round, perturbing the defective product image sample in at least one product image sample pair of the product image pair set to obtain a perturbed product image pair set; the perturbation operation includes at least one of translation, rotation, distortion transformation, contrast transformation, and brightness transformation with a preset amplitude; and training the product defect detection network using the perturbed product image pair set for the perturbation training round. The perturbation operation can improve the product defect detection network's generalization ability to defects.

[0062] In some embodiments, before training in at least one training round, the standard product image sample and the defective product image sample in at least one product image sample pair of the product image pair set can be randomly resized. The resized sample is within a preset size range, such as (480, 640). It is understood that random resizing can increase the size diversity of the product image sample pairs, enabling the product defect detection network to learn defect detection under images of diverse sizes, thereby improving the generalization ability of the product defect detection network.

[0063] In some embodiments, the step of obtaining a defective product image sample corresponding to at least one product background defect includes: applying product background defect material to the product background area of ​​a standard product image sample to obtain a defective product image sample corresponding to the product background defect.

[0064] Furthermore, image processing such as size transformation, rotation, distortion, and transparency adjustment can be performed on the product background defect material before application.

[0065] To make the product background defects in the defective product image sample more realistic, the edges of the product background defect material can be optimized after application. This ensures that the grayscale values ​​of the product background defect material's edges linearly transition along the contour normal direction to the product background pixel values ​​of the standard product image sample. Product background pixel values ​​refer to the pixel values ​​of the product background area where no defects exist.

[0066] In some embodiments, the step of obtaining a defective product image sample corresponding to at least one product character defect includes: replacing some product character pixel values ​​in a standard product image sample with product background pixel values ​​in a standard product image sample to obtain a defective product image sample corresponding to a missing character defect, wherein the product background pixel values ​​are representative pixel values ​​in the product background area of ​​the standard product image sample, and the product character pixel values ​​are pixel values ​​of pixels in the product character area of ​​the standard product image sample.

[0067] In some embodiments, the step of obtaining a defective product image sample corresponding to at least one product character defect includes: using a partial copy of a first product character sub-region of a standard product image sample to a second product character sub-region to obtain a defective product image sample corresponding to a character misprint defect.

[0068] Understandably, the aforementioned method for generating defective product image samples not only increases the diversity of defective product image samples but also compensates for situations where the number of real defective product image samples is insufficient or difficult to collect. Furthermore, since the generated defect types and locations are controllable, the ground truth labels of the generated defective product image samples can be directly determined without additional manual annotation, reducing annotation costs.

[0069] To facilitate understanding, the product defect detection method provided in this application will be illustrated below with a specific example: Figure 2 This is a structural diagram of a specific example of the product defect detection model of this application. For example... Figure 2As shown, the product defect detection network includes a spatial alignment model, a feature extraction model, and a distance metric model. The feature extraction model is a feature pyramid subnetwork, which can be, but is not limited to, Dinov3. Product defect detection methods based on the product defect detection network include: I. Image Acquisition.

[0070] The process involves capturing images of the target product in its actual state, and obtaining standard product images (1-N) from an image feature library, each captured for a target product in one of N standard states. Examples of standard product images and actual product images are provided below. Figure 3 , Figure 3 The image on the left is a standard product image, where the product character "103" indicates that there is no defect. The image on the right is an actual product image, where the "1" in the product character "103" indicates a defect.

[0071] II. Spatial alignment using spatial alignment models.

[0072] Figure 4 This is a structural diagram of a specific example of the spatial alignment model in this application. For example... Figure 4 As shown, the spatial alignment model includes a transformation parameter acquisition module, a grid generator, and an interpolation sampling module (e.g., a bilinear interpolation sampling module). The transformation parameter acquisition module includes several convolutional pooling modules (e.g., four) and fully connected layers (FClayers). The convolutional pooling module includes convolutional layers (e.g., 3×3 convolutions) and pooling layers (e.g., max pooling).

[0073] Spatial alignment methods based on spatial alignment models include: 1. Stitch the actual product image and the standard product image i (i=1, ..., N) in the channel direction to obtain the stitched product image.

[0074] 2. The spliced ​​product image is processed using the transformation parameter acquisition module to obtain the spatial transformation parameter θ of the actual product image relative to the standard product image i.

[0075] 3. Using a mesh generator based on the spatial transformation parameter θ, calculate the coordinates of each first pixel in the updated actual product image in the original actual product image.

[0076] 4. Using the interpolation sampling module, the first pixel with integer coordinates is taken as the second pixel. The pixel value of the pixel at the integer coordinate in the original product image is taken as the pixel value of the second pixel in the updated product image. The first pixel with floating-point coordinates is taken as the third pixel. Interpolation calculations are performed on the pixel values ​​of the pixels surrounding the floating-point coordinate in the original product image to obtain the pixel value of the third pixel in the updated product image, thus obtaining the updated product image.

[0077] III. Feature extraction using feature extraction models.

[0078] Feature extraction is performed on the actual product image to obtain actual product features, which include actual product features at the first depth, second depth, and third depth: Feature1 block6, Feature1 block8, and Feature1 block10.

[0079] Obtain standard product features 1-N of the target product from the standard product feature library. Standard product feature i is obtained by feature extraction from standard product image i. Standard product feature i includes actual product features at the first depth, second depth, and third depth: Feature2 block6, Feature2 block8, and Feature2 block10.

[0080] IV. Using distance metric models for distance measurement.

[0081] Figure 5 This is a structural diagram of a specific instance of the distance metric model in this application. For example... Figure 5 As shown, the distance metric model includes a splicing module, a distance metric module, and a compression module.

[0082] Distance metric methods based on distance metric models include: 1. Using the splicing module, the actual product features and standard product features of the first depth are spliced ​​together in the channel direction to obtain the first depth spliced ​​product feature Feature12 block6; the actual product features and standard product features of the second depth are spliced ​​together in the channel direction to obtain the second depth spliced ​​product feature Feature12 block8; the actual product features and standard product features of the third depth are spliced ​​together in the channel direction to obtain the third depth spliced ​​product feature Feature12 block10.

[0083] 2. Use the distance metric module to perform a 3×3 convolution on Feature12 block10 to obtain the first depth multi-channel distance map block10_featurePair.

[0084] 3. Perform a 3×3 convolution on Feature12 block8 using the distance metric module to obtain the first depth multi-channel distance map block8_featurePair. Upsample block10_featurePair and fuse it with block8_featurePair (concatenate and convolve) to obtain the first intermediate multi-channel feature map block8_10_featureFusion.

[0085] 4. Using the distance metric module, perform a 3×3 convolution on Feature12 block10 to obtain the first-depth multi-channel distance map block6_featurePair. Upsample block8_10_featureFusion and fuse it with block6_featurePair (concatenate and convolve) to obtain the second intermediate multi-channel feature map. Upsample the second intermediate multi-channel feature map to obtain the final multi-channel distance map.

[0086] 5. Using the compression module, perform a 1×1 convolution on the final multi-channel distance map to obtain a single-channel distance map i (size [1,H,W]) between the actual product image and the standard product image i. Normalize the single-channel distance map i (sigmoid) to normalize the distance values ​​of each pixel to the (0,1) interval.

[0087] 6. The single-channel distance between the actual product image and the standard product images 1-N. Figure 1 The final single-channel distance map is obtained by averaging the values ​​of -N.

[0088] V. Defect Judgment.

[0089] 1. Binarize the final single-channel distance map using a distance threshold. Distance values ​​greater than the threshold are binarized into first distance values, and distance values ​​less than or equal to the threshold are binarized into second distance values, resulting in a binarized distance map. See the example of a binarized distance map below. Figure 6 .

[0090] 2. Obtain product defect detection results based on the binarized distance map. If a first distance value exists in the binarized distance map, it is determined that the actual product has a defect, and the image region formed by each pixel of the first distance value is taken as the defective region; if the first distance value does not exist, it is determined that the actual product does not have a defect.

[0091] VI. Training of the product defect detection network.

[0092] 1. Obtain a set of product images.

[0093] (1) Obtain standard product image samples of various products from a general dataset. For example, the general dataset can be MSRA-TD500, CTW, ICDAR, etc. Standard product image samples can be flat text images, street sign text images, certificate text images, product description text images, and online advertising text images.

[0094] (2) The standard product image sample includes the standard product background area and the standard product character area. Both the standard product background area and the standard product character area are free of defects.

[0095] (3) Apply the product background defect material to the standard product background area to obtain the defective product image sample corresponding to the product background defect.

[0096] (4) Using the product background pixel values ​​in the standard product background area, replace some of the product character pixel values ​​in the standard product character area to obtain a defective product image sample corresponding to the missing character printing defect. For example, replace 10%-50% of the product character pixel values.

[0097] (5) By partially copying the first product character sub-region of the standard product character area to the second product character sub-region, a defective product image sample corresponding to the character misprint defect is obtained. For example, 10%-50% of product character A is copied to product character B.

[0098] (6) Standard product image samples and defective product image samples are combined to form product image sample pairs. Product image sample pairs corresponding to various defects of various products constitute a product image pair set.

[0099] 2. Train the product defect detection network using product image pairs.

[0100] (1) In the first stage, the product defect detection network is pre-trained, and the parameters of the feature extraction model are frozen after pre-training.

[0101] (2) In the second stage, the spatial alignment model and distance metric model are fine-tuned and trained.

[0102] In this process, at least one training round in the first and second stages is a perturbation training round. Before the perturbation training round, at least one defective product image sample in the product image pair is perturbed to obtain a perturbed product image pair. The perturbation operations include small-amplitude translation, rotation, distortion transformation, contrast transformation, and brightness transformation. The distance metric model of the perturbed product image pair is then used to train the perturbation training round.

[0103] This application also provides a product defect detection device. The product defect detection device includes a feature extraction module, a stitching module, a distance measurement module, and a determination module. Wherein: The feature extraction module is used to extract features from the actual product image and the standard product image respectively, and obtain the actual product features and the standard product features. The actual product image is taken from the target product in its actual state, and the standard product image is taken from the target product in its standard state.

[0104] The splicing module is used to splice actual product features and standard product features along the channel direction to obtain spliced ​​product features.

[0105] The distance measurement module is used to obtain a distance map between the actual product image and the standard product image based on the stitched product features using a distance measurement model.

[0106] The determination module is used to determine whether the target product in its actual state has defects based on the distance map.

[0107] For further detailed descriptions of the product defect detection device, please refer to other embodiments, which will not be repeated here.

[0108] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Figure 7 As shown, the electronic device 50 includes a memory 51 and a processor 52. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above method embodiments. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include a laptop computer, a tablet computer, or other carrier device, which is not limited here.

[0109] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above method embodiments. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 52 may be implemented using integrated circuit chips.

[0110] Please see Figure 8 , Figure 8 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 60 stores program instructions 601 thereon, which, when executed by a processor, implement the steps in any of the above method embodiments.

[0111] This application also provides a computer program product comprising a computer program that, when executed by a processor, can implement the steps of the methods described in any of the foregoing embodiments. Specifically, the computer program product can be a software or program product containing a computer program, capable of running on a computing device or stored on any available medium.

[0112] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0113] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. In another image location, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A product defect detection method, characterized in that, include: Feature extraction is performed on the actual product image and the standard product image respectively to obtain the actual product features and the standard product features. The actual product image is obtained by taking pictures of the target product in its actual state, and the standard product image is obtained by taking pictures of the target product in its standard state. The actual product features and the standard product features are spliced ​​together along the channel direction to obtain spliced ​​product features; Using a distance metric model based on the stitched product features, a distance map is obtained between the actual product image and the standard product image; Based on the distance map, it is determined whether the target product in the actual state has defects.

2. The method according to claim 1, characterized in that, The step of using a distance metric model based on the stitched product features to obtain a distance map between the actual product image and the standard product image includes: The differences in the features of the spliced ​​products are measured to obtain a multi-channel distance map; The multi-channel distance map is compressed to obtain a single-channel distance map.

3. The method according to claim 2, characterized in that, The difference metric is implemented through a first convolution operation, and the compression is implemented through a second convolution operation; and / or, The actual product features include actual product features at multiple depths, the standard product features include standard product features at multiple depths, and the stitched product features include stitched product features at multiple depths. A stitched product feature at a given depth is obtained by stitching together actual product features and standard product features corresponding to that depth. The step of measuring the differences in the stitched product features to obtain a multi-channel distance map includes: The differences in the features of the stitched products at each depth are measured to obtain the multi-channel distance map for each depth. The multi-channel distance maps at each depth are fused to obtain the final multi-channel distance map.

4. The method according to claim 3, characterized in that, The multi-channel feature maps at each depth include a first depth multi-channel feature map, a second depth multi-channel feature map, and a third depth multi-channel feature map, with the first depth, second depth, and third depth increasing sequentially; the fusion of the multi-channel distance maps at each depth to obtain the final multi-channel distance map includes: After upsampling the multi-channel feature map of the third depth, it is fused with the multi-channel feature map of the second depth to obtain the first intermediate multi-channel feature map. After upsampling the intermediate multi-channel feature map, it is fused with the multi-channel feature map of the first depth to obtain the second intermediate multi-channel feature map; The second intermediate multi-channel feature map is upsampled to obtain the final multi-channel distance map.

5. The method according to claim 1, characterized in that, The standard product images are multiple, each image being taken of a target product in a different standard state; determining whether the target product in its actual state has defects based on the distance map includes: The distance maps between the actual product image and multiple standard product images are statistically analyzed to obtain the final distance map; And / or, the distance values ​​of each pixel in the distance map represent the degree of difference between the corresponding pixels of the actual product image and the standard product image; Determine whether there are any pixels in the actual product image whose distance value is greater than the distance threshold; In response to the presence of pixels with distance values ​​greater than a distance threshold, it is determined that the target product in the actual state has a defect, and the image region formed by each pixel with a distance value greater than the distance threshold is taken as the defective region.

6. The method according to claim 1, characterized in that, Before performing feature extraction on the actual product image and the standard product image respectively to obtain the actual product features and the standard product features, the process also includes: The actual product image and the standard product image are stitched together in the channel direction to obtain a stitched product image; The stitched product image is sequentially subjected to a third convolution operation, a pooling operation, and a fully connected operation to obtain the spatial transformation parameters of the actual product image relative to the standard product image. The actual product image is spatially transformed using the spatial transformation parameters to update the actual product image.

7. The method according to claim 6, characterized in that, The step of performing a spatial transformation on the actual product image using the spatial transformation parameters to update the actual product image includes: Based on the spatial transformation parameters, obtain the coordinates of each first pixel in the updated actual product image in the original actual product image; The first pixel point whose corresponding coordinates are integer coordinates is taken as the second pixel point, and the pixel value of the pixel point at the integer coordinates in the actual product image before the update is taken as the pixel value of the second pixel point in the actual product image after the update. The first pixel with the corresponding floating-point coordinates is taken as the third pixel. The pixel values ​​of the pixels around the floating-point coordinates in the actual product image before the update are interpolated to obtain the pixel value of the third pixel in the updated actual product image.

8. The method according to claim 1, characterized in that, The product defect detection method is implemented based on a product defect detection network, which is trained using a product image pair set. The product image pair set includes several product image sample pairs of various products, including the target product and other products. Each product image sample pair includes a standard product image sample and its corresponding defective product image sample. The standard product image sample belongs to a product in a standard state, while the defective product image sample belongs to a product in an actual state with defects. The defective product image samples in different product image sample pairs of the same product cover multiple types of defects of the same product.

9. The method according to claim 8, characterized in that, The training of the product defect detection network is divided into multiple training rounds, and the training steps of the product defect detection network include: At least one training wheel is used as a perturbation training wheel. Before training the perturbation training wheel, a perturbation operation is performed on the defective product image sample in at least one of the product image sample pairs of the product image pair set to obtain a perturbation product image pair set. The perturbation operation includes at least one of translation, rotation, distortion transformation, contrast transformation, and brightness transformation with a preset amplitude. The product defect detection network is trained using the disturbed product image set through the disturbance training round.

10. The method according to claim 8, characterized in that, The defects include product background defects and product character defects. The step of obtaining at least one defective product image sample corresponding to the product background defect includes: Apply the product background defect material to the product background area of ​​the standard product image sample to obtain the defective product image sample corresponding to the product background defect; And / or, The steps for obtaining at least one defective product image sample corresponding to a product character defect include: By using the product background pixel values ​​in the standard product image sample, some product character pixel values ​​in the standard product image sample are replaced to obtain a defective product image sample corresponding to the missing character printing defect. The product background pixel values ​​are representative pixel values ​​in the product background area of ​​the standard product image sample, and the product character pixel values ​​are pixel values ​​of the pixels in the product character area of ​​the standard product image sample. By partially copying the first product character sub-region of the standard product image sample to the second product character sub-region, a defective product image sample corresponding to the character misprint defect is obtained.

11. An electronic device, characterized in that, It includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method of any one of claims 1-10.

12. A computer-readable storage medium / program product, characterized in that, It stores program instructions / computer programs that, when executed by a processor, implement the method of any one of claims 1-10.

Images