Defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a defect detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a template image and a product image of a to-be-detected product; performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on feature point pairs obtained by feature matching to obtain an alignment image of which the geometric view angle is consistent with that of the template image; and defect detection is carried out based on the aligned image and the template image to obtain a detection result, so that the problem of image deformation caused by the fact that the surface of the to-be-detected product is a curved surface and the shooting angle is not correct in a traditional scheme is effectively solved, and local and global deformation of the product image can be corrected, thereby greatly improving the alignment precision of defect detection, and improving the detection accuracy of the to-be-detected product. Therefore, the real difference of the product surface can be more focused during defect detection, non-defect geometric deformation interference is effectively shielded, and the accuracy and robustness of defect detection of complex surfaces of household electrical appliances and the like are greatly improved.

Description

Defect detection methods, devices, electronic equipment and storage media Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a defect detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the home appliance manufacturing industry, product appearance quality is a key factor in measuring craftsmanship, with the printing or affixing quality of areas such as logos and nameplates being particularly critical. As home appliance designs become increasingly diversified, the appearance of products under test is becoming more complex, encompassing various regular and irregular surface shapes, and the materials and colors of logos are also exhibiting rich diversity. Furthermore, the variability in shooting angles during production and transportation further increases the difficulty of inspection.

[0003] Currently, most logo defect detection in the industry relies on template matching, which involves directly comparing the measured image with a template image. While simple to implement, this method struggles to handle complex operating conditions and environmental changes in real-world production lines, easily leading to false positives and false negatives, resulting in extremely low reliability of the detection results. Furthermore, this method requires manual adjustments during production line switchovers and environmental changes, making it extremely cumbersome and unsuitable for the rapid changeover patterns of small-batch, multi-variety products in home appliance manufacturing, thus failing to meet the quality inspection needs of modern home appliance production. Summary of the Invention

[0004] This invention provides a defect detection method, apparatus, electronic device, and storage medium to solve the problem of inaccurate detection in current industrial inspections due to the complex surface morphology of products and fluctuating shooting conditions.

[0005] This invention provides a defect detection method, comprising: acquiring a template image and a product image of a product to be tested; performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; performing defect detection based on the aligned image and the template image to obtain a detection result.

[0006] According to a defect detection method provided by the present invention, the feature matching of the template image and the product image includes: determining a detection region in the template image and a detection region in the product image; the detection region contains the detection target of the product to be tested; extracting features from the detection regions in the template image and the detection regions in the product image to obtain template feature points and their feature point descriptions, and product feature points and their feature point descriptions; and performing feature matching on the template feature points and the product feature points based on the feature point descriptions to obtain feature point pairs.

[0007] According to a defect detection method provided by the present invention, the step of performing non-rigid alignment on the product image based on feature point pairs obtained by feature matching to obtain an aligned image whose geometric perspective is consistent with that of the template image includes: constructing a non-rigid transformation model between the detection region in the template image and the detection region in the product image based on the feature point pairs; and remapping the pixel coordinates within the detection region in the product image based on the non-rigid transformation model to obtain the aligned image.

[0008] According to a defect detection method provided by the present invention, the non-rigid transformation model is composed of multiple local homography matrices; the step of constructing a non-rigid transformation model between the detection area in the template image and the detection area in the product image based on the feature point pairs includes: dividing the detection area in the product image into multiple local grids; determining the grid center point of each local grid, and determining the distance weight of the feature point pair for each local grid based on the distance between the product feature point in the feature point pair and the grid center point; determining the local homography matrix of each local grid based on the feature point pair and its distance weight for each local grid; and constructing the non-rigid transformation model based on the local homography matrix of each local grid.

[0009] According to a defect detection method provided by the present invention, the defect detection based on the aligned image and the template image to obtain a detection result includes: performing semantic segmentation on the aligned image to obtain a product target probability map; binarizing the product target probability map to obtain a product semantic mask; performing masking processing on the aligned image based on the product semantic mask to obtain a product foreground image; and performing defect detection based on the product foreground image and the template image to obtain a detection result.

[0010] According to a defect detection method provided by the present invention, the defect detection based on the product foreground image and the template image to obtain a detection result includes: performing pixel-level difference processing on the product foreground image and the template image to obtain a difference image; performing threshold segmentation on the difference image to obtain a binarized defect candidate image; performing connected component analysis on the defect candidate image, and determining the detection result based on the result obtained from the connected component analysis.

[0011] According to a defect detection method provided by the present invention, the step of performing pixel-level differential processing on the product foreground image and the template image to obtain a differential image includes: performing semantic segmentation on the template image to obtain a template target probability map; binarizing the template target probability map to obtain a template semantic mask; performing masking processing on the template image based on the template semantic mask to obtain a template foreground image; and performing pixel-level differential processing on the product foreground image and the template foreground image to obtain a differential image.

[0012] The present invention also provides a defect detection device, comprising: an image acquisition unit for acquiring a template image and a product image of a product to be tested; an image alignment unit for performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; and a defect detection unit for performing defect detection based on the aligned image and the template image to obtain a detection result.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the defect detection method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the defect detection method as described above.

[0015] The defect detection method, apparatus, electronic device, and storage medium provided by this invention introduce a non-rigid alignment mechanism after image acquisition. This effectively solves the image distortion problem caused by curved surfaces of the product under test and incorrect shooting angles in traditional solutions. It can adaptively correct local and global deformations of the product image based on feature point pairs, generating an aligned image with a geometric viewpoint highly consistent with the template image. This greatly improves the alignment accuracy of defect detection, allowing for greater focus on the real differences on the product surface during defect detection. It effectively shields non-defect-related geometric deformation interference, thereby significantly improving the accuracy and robustness of defect detection on complex surfaces such as home appliances. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 is a flowchart of the defect detection method provided by the present invention; Figure 2 is an overall flowchart of the defect detection method provided by the present invention; Figure 3 is a structural schematic diagram of the defect detection device provided by the present invention; Figure 4 is a structural schematic diagram of the electronic device provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] In the home appliance manufacturing process, the appearance quality of a product is a crucial indicator of its manufacturing process, with the printing or affixing quality of areas such as logos and nameplates being particularly critical. As home appliance designs become increasingly diverse, the shapes of products under test are becoming more complex, encompassing flat, cylindrical, curved, and irregular cubic surfaces. Furthermore, the materials and colors of logos vary significantly, and the shooting angles often change during assembly line transport. Therefore, achieving efficient and accurate logo defect detection under complex conditions is a pressing issue that the industry needs to address.

[0020] Currently, defect detection for logos primarily employs template matching. The process involves manually creating a standard logo, then using edge features or Normalized Cross Correlation (NCC) algorithms for coarse localization at the whole-image level in the image to be inspected. Finally, the located regions are matched against a template to identify any defects. While this method is simple to implement and requires minimal training data, it suffers from significant drawbacks in the complex environments of actual production lines. Specifically, it is extremely sensitive to changes in lighting. Since NCC and edge features essentially rely on the consistency of image grayscale or gradients, fluctuations in the direction or intensity of the light source on the production line cause a shift in the statistical feature distribution of the template and the image to be inspected, leading to a decrease in matching accuracy and a high likelihood of false alarms. Consequently, engineers often need to adjust the defect threshold individually for each production line and even each time period, resulting in extremely high maintenance costs.

[0021] Secondly, background texture interference is significant, and it's difficult to handle curved surface deformation. That is, current matching methods involve pixel subtraction, which cannot distinguish between background texture, reflective noise, and the actual foreground area. Furthermore, relaxing the defect threshold during detection will result in missed detection of real defects, while tightening it will lead to a surge in false positives due to texture interference. More seriously, for home appliances with curved or rounded surfaces, simple rigid template matching cannot address image deformation caused by differences in viewing angle or surface curvature, leading to mismatches and consequently, batch misjudgments.

[0022] Finally, current inspection methods lack adaptability. That is, the entire process is based on fixed templates and manually set hard thresholds, lacking data-driven learning capabilities. Whenever the logo is changed, the material is altered, or other parameters are modified, it is necessary to readjust the defect thresholds and other parameters, making it difficult to adapt to the rapid switching patterns of small-batch, multi-variety manufacturing in modern home appliances.

[0023] To address this issue, this invention provides a defect detection method aimed at solving the problem of inaccurate detection in current industrial inspections caused by the complexity of product surface morphology and fluctuating shooting conditions. By combining feature matching and non-rigid alignment techniques, interference caused by variations in product surface curvature, shooting angle, and lighting is eliminated. This achieves sub-pixel-level precise correction and defect detection of the product image under test, thereby significantly improving the robustness and accuracy of complex surface defect detection without the need for frequent manual adjustments. Here, the product under test can be a household appliance, such as a washing machine, refrigerator, or rice cooker; a consumer electronics product, such as a mobile phone, tablet, or laptop; or any industrial product involving appearance quality inspection, such as office equipment, furniture, or automobile manufacturing. This invention does not specifically limit the specific types of products tested.

[0024] Figure 1 is a flowchart of the defect detection method provided by the present invention. As shown in Figure 1, the method includes: step 110, acquiring a template image and a product image of the product to be tested; step 120, performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; step 130, performing defect detection based on the aligned image and the template image to obtain the detection result.

[0025] Specifically, before defect detection, it is first necessary to identify the product to be inspected, i.e., the product under test, and acquire product images of the product. In practical applications, the product under test can be the exterior parts of household appliances, such as washing machines, refrigerators, rice cookers, etc., especially the logos, nameplates, or decorative areas on these parts; it can also be products from other fields, such as consumer electronics, office equipment, home furniture, automobile manufacturing, etc., and this embodiment of the invention does not specifically limit this. Considering industrial realities, these product images can be acquired in real time by industrial cameras set up on the production line. However, because the surface of the product under test may have curvature, i.e., a curved surface, or there may be posture deviations during transmission, the acquired product images may have a certain angular tilt or geometric deformation relative to the standard state.

[0026] Therefore, in order to perform effective comparison, this embodiment of the invention also requires obtaining a template image of the product to be tested. This template image is pre-stored image data of the product under ideal, defect-free conditions. The template image is a high-resolution image acquired under standard lighting and a direct viewing angle, serving as a reference for subsequently assessing whether the product image contains defects.

[0027] After obtaining the product image and template image, in order to eliminate geometric differences in the product image caused by factors such as shooting angle and surface curvature, this embodiment of the invention first performs feature matching on the template image and the product image. Specifically, this may involve extracting representative visual key points, such as corner points and texture centers, from both images and establishing a correspondence between these visual key points in the two images, thereby obtaining multiple sets of feature point pairs. Each set of feature point pairs locks the position coordinates of the same physical point on both images.

[0028] Next, the spatial positional differences reflected by these multiple sets of feature points can be used to perform non-rigid alignment of the product image. It's important to note that this non-rigid alignment is designed for the curved surface features that the product under test may possess. Unlike traditional rigid transformations that can only handle translation or rotation, non-rigid alignment allows local areas of the image to undergo adaptive distortion or stretching correction based on actual deformation. Through this refined correction process, the product image, which might have previously exhibited perspective distortion or surface warping, is flattened or reprojected, resulting in a new product image—the aligned image. In this aligned image, every pixel of the product under test is highly consistent with the template image in terms of spatial geometry, i.e., geometric viewpoint. This ensures that subsequent comparisons are performed under the same spatial reference, thus providing a crucial data foundation for improving detection accuracy.

[0029] After this, defect detection can be performed based on the aligned image and the template image to obtain the detection results. That is, when the product image is corrected to become an aligned image with the template image in terms of geometric perspective, the difference between the two theoretically only stems from physical defects on the surface of the product under test. Therefore, at this point, in-depth analysis and comparison can be performed directly based on these two images to achieve defect detection. For example, the pixel differences and structural similarity between the aligned image and the template image at corresponding positions can be calculated, or a pre-trained model can be used to identify the difference areas between the aligned image and the template image to obtain the detection results. The detection results can be qualitative conclusions expressed as pass / fail, or quantitative data containing rich information, such as specific defect types, such as scratches, missing parts, color differences, etc., the coordinate position of the defect in the product image, the area size of the defect region, or even a visual feedback of directly highlighting the defect region on the product image to facilitate subsequent processing by quality inspectors or automated equipment.

[0030] The defect detection method provided by this invention introduces a non-rigid alignment mechanism after image acquisition, which effectively solves the image deformation problem caused by curved surfaces of the product under test and incorrect shooting angles in traditional solutions. It can adaptively correct local and global deformations of the product image based on feature point pairs, generating an aligned image with a geometric viewpoint highly consistent with the template image. This greatly improves the alignment accuracy of defect detection, allowing for a greater focus on the real differences on the product surface during defect detection. It effectively shields non-defect-related geometric deformation interference, thereby significantly improving the accuracy and robustness of defect detection on complex surfaces such as home appliances.

[0031] Based on the above embodiments, in step 120, feature matching is performed on the template image and the product image, including: determining the detection area in the template image and the detection area in the product image; the detection area contains the detection target of the product to be tested; extracting features from the detection area in the template image and the detection area in the product image to obtain template feature points and their feature point descriptions, as well as product feature points and their feature point descriptions; and performing feature matching on the template feature points and product feature points based on the feature point descriptions to obtain feature point pairs.

[0032] Considering that in practical applications, directly processing the entire high-resolution product image is often inefficient and easily affected by background clutter, this embodiment of the invention first extracts the core region, i.e., the detection region, when performing feature matching.

[0033] In detail, to ensure that defect detection focuses on core content, this embodiment of the invention can first identify the region of interest (ROI) from the panoramic image. Specifically, this can involve first determining the detection area in the template image and the product image; this area, also known as the ROI, contains the detection target, i.e., the actual object to be inspected on the product, such as brand logos, model numbers, or specific decorative patterns on appliance casings. By locating the detection area, large areas of background in the product image unrelated to the detection target, such as conveyor belts and machine supports, can be excluded from the calculation range, thereby greatly reducing the subsequent computational load and lowering the risk of false detections due to background noise.

[0034] Specifically, the detection region can be divided / located using a preset coordinate cropping box, or it can be located using object detection algorithms such as the YOLO (You Only Look Once) series of algorithms or deep learning-based object detection models. Preferably, to quickly locate the detection region in the image, this embodiment of the invention uses a deep learning-based object detection model, such as YOLOv12, to perform object detection and obtain the detection region. It is worth noting that before applying this model for object detection, it can be pre-trained using a large number of sample images labeled with bounding boxes of the target objects, enabling it to recognize and locate targets in complex backgrounds.

[0035] Next, feature extraction can be performed on the detection areas. For example, the SuperPoint deep learning model can be used to extract features from the detection areas in the template image and the product image respectively, to delve into the pixel level and uncover the detailed information of the two images. This allows us to obtain the visual key points belonging to the template image, i.e., template feature points, and the visual key points belonging to the product image, i.e., product feature points. Here, both template feature points and product feature points are pixels in the image with significant visual characteristics, such as corner points, edge endpoints, and texture centers, which have high stability under changes in lighting or viewing angle.

[0036] Meanwhile, to achieve point-to-point matching, this embodiment of the invention also generates corresponding descriptive information for each feature point, i.e., a feature point description. This feature point description is not a simple textual description, but a high-dimensional numerical vector that encodes image information within the neighborhood of the feature point, such as gradient direction and brightness distribution.

[0037] Furthermore, based on this feature point description, feature matching can be performed on the template feature points and product feature points to obtain feature point pairs. Specifically, this can involve calculating the similarity between the feature point descriptions of the template and product feature points, such as calculating the Euclidean distance or cosine similarity between vectors, and determining which two feature points are actually projections of the same physical point onto different images. When the similarity between the feature point descriptions of two feature points meets a preset condition, such as being closest and less than a specific threshold, it can be confirmed that they correspond to the same physical point, and these two feature points constitute a feature point pair.

[0038] In this embodiment of the invention, the detection area containing the target is first determined, and then feature extraction and feature matching are performed within the detection area. This not only effectively eliminates background interference and reduces the computational load of feature extraction, but also enables the more accurate finding of corresponding feature point pairs under complex lighting or changing viewing angles, thereby ensuring the accuracy of subsequent image alignment.

[0039] Based on the above embodiments, based on the feature point pairs obtained by feature matching, the product image is non-rigidly aligned to obtain an aligned image whose geometric viewpoint is consistent with that of the template image. This includes: constructing a non-rigid transformation model between the detection area in the template image and the detection area in the product image based on the feature point pairs; and remapping the pixel coordinates within the detection area in the product image based on the non-rigid transformation model to obtain the aligned image.

[0040] Specifically, the process of non-rigid alignment of the product image based on feature point pairs can include: In order to accurately describe the complex deformation of the product image relative to the template image, in this embodiment of the invention, it is necessary to first construct a non-rigid transformation model. It should be noted that, unlike global rigid transformations that can only describe overall translation, rotation, or scaling, the non-rigid transformation model in this embodiment of the invention can describe locally differentiated deformations and is a mapping relationship reflecting local differences.

[0041] Specifically, this can be achieved by using the feature point pairs obtained through feature matching as constraints. The model must map the coordinates of product feature points in the product image to the corresponding template feature points in the template image as accurately as possible. By solving this constraint problem, a non-rigid transformation model can be constructed. This model defines a dense correspondence field or transformation mesh from the source image space (the detection area of ​​the product image) to the target image space (the aligned image). It allows different local regions of the image to have different transformation parameters, thereby simulating and canceling nonlinear geometric distortions caused by the surface curvature of the product under test, such as the curved surface of an appliance logo or uneven perspective.

[0042] After obtaining the non-rigid transformation model, in this embodiment of the invention, image correction can be performed accordingly, that is, remapping the pixel coordinates within the detection area of ​​the product image. This process can be understood as performing pixel-level interpolation on the product image, that is, traversing each pixel coordinate in the alignment image to be generated, and using the constructed non-rigid transformation model to find or calculate the corresponding sampling position of the pixel coordinate in the detection area of ​​the product image. Since the calculated corresponding position is often not an integer coordinate, it is also necessary to combine bilinear interpolation or bicubic interpolation to obtain the pixel value at that position and fill it into the corresponding position in the alignment image.

[0043] Through this pixel-by-pixel coordinate remapping, the pixel coordinates that were originally distorted in the product image due to curved surfaces or deformed due to the tilt of the shooting angle are reorganized and restored to the same geometric shape as the template image. In the end, an aligned image with the same geometric perspective as the template image is generated, that is, an aligned image that is visually flattened or straightened.

[0044] Compared to traditional global transformation methods, this embodiment of the invention constructs a non-rigid transformation model and performs pixel remapping, which enables independent and continuous adjustment of different local regions of the image, achieving high-precision correction of complex deformations. It is particularly suitable for curved logo detection scenarios commonly found in the home appliance industry, and can eliminate perspective projection errors caused by curved surfaces, making the generated aligned image highly consistent with the template image in terms of geometric structure.

[0045] Based on the above embodiments, the non-rigid transformation model is composed of multiple local homography matrices. Based on feature point pairs, a non-rigid transformation model is constructed between the detection region in the template image and the detection region in the product image, including: dividing the detection region in the product image into multiple local grids; determining the grid center point of each local grid, and determining the distance weight of the feature point pair for each local grid based on the distance between the product feature point and the grid center point; determining the local homography matrix of each local grid based on the feature point pair and its distance weight for each local grid; and constructing a non-rigid transformation model based on the local homography matrices of each local grid.

[0046] Considering that product surfaces such as home appliance logos often exhibit non-uniform surface deformation, a simple global transformation matrix often cannot account for the deformation differences across various parts of the image. Therefore, this invention proposes a strategy of breaking down the model into smaller parts; that is, the non-rigid transformation model is essentially composed of multiple local homography matrices targeting different minute regions.

[0047] Based on this, the process of constructing a non-rigid transformation model specifically includes the following steps: In order to capture subtle local deformations in the image, in this embodiment of the invention, the detection area in the product image can first be divided into a grid to divide the complete detection area into several tiny units. For example, the detection area can be divided into multiple neatly arranged local grids according to a preset number of rows and columns. Each local grid represents a small area of ​​the surface of the product to be tested. Through this discretization process, the originally complex overall curved surface is approximately decomposed into several tiny planar segments, thus providing a spatial basis for subsequent independent deformation calculations for each small area.

[0048] After dividing the grid, it is necessary to determine the transformation rules for each local grid. Since product feature points closer to a local grid better reflect the deformation trend in the vicinity of that local grid, we can first calculate the geometric center of each local grid, i.e., the grid center point. Then, based on the distance between each product feature point and the grid center point of the local grid, we assign a weight value to each product feature point (feature point pair) for that local grid, i.e., distance weight. The closer the distance, the larger the weight value, and the farther the distance, the smaller the weight value, for example, following a Gaussian distribution decay. This ensures that the deformation parameters of each local grid mainly consider the surrounding feature points, thus achieving localized constraints in deformation calculation.

[0049] Based on the aforementioned weight allocation, in this embodiment of the invention, the deformation parameters, i.e., the local homography matrix, can be calculated separately for each local grid. Unlike traditional schemes where the entire image shares a single transformation matrix, the local homography matrix in this embodiment is obtained by solving for all feature point pairs and their distance weights relative to the local grid using methods such as weighted least squares. This means that each local grid possesses a matrix describing how it maps from the product image space to the template image space. For example, the local homography matrix of a local grid located at the curved area in the upper left corner of the image will focus on correcting the specific distortion in the upper left corner; while the local homography matrix of a local grid located in the flat area in the center of the image reflects the transformation relationship of the central region.

[0050] Following this, a non-rigid transformation model can be constructed based on the local homography matrices of each local grid. That is, the local homography matrices corresponding to all local grids are combined to form a complete non-rigid transformation model. This model is no longer a single mathematical formula, but a matrix field composed of numerous local homography transformation matrices. When this model is applied to the entire image, it guides the coordinates of each pixel in the image to move according to the transformation rules of its local grid, thereby achieving smooth, continuous, and non-rigid alignment of the entire image that conforms to the characteristics of local surfaces.

[0051] In this embodiment of the invention, a local homography matrix and distance weights are introduced to decompose the complex global nonlinear deformation problem into multiple simple local linear transformation problems. This divide-and-conquer strategy, namely the APAP (As-Projective-As-Possible) idea, can accurately correct the local distortion caused by the complex curvature of the surface of home appliances, while maintaining the overall continuity and naturalness of the image. This greatly improves the alignment accuracy and effectively avoids the phenomenon of correcting the middle but distorting the edges, providing high-quality alignment data for subsequent high-precision defect detection.

[0052] Based on the above embodiments, step 130 includes: performing semantic segmentation on the aligned image to obtain a product target probability map; binarizing the product target probability map to obtain a product semantic mask; performing masking processing on the aligned image based on the product semantic mask to obtain a product foreground image; and performing defect detection based on the product foreground image and the template image to obtain a detection result.

[0053] Figure 2 is a flowchart of the overall defect detection method provided by the present invention. As shown in Figure 2, after completing the high-precision geometric alignment of the image, in this embodiment of the invention, a semantic understanding mechanism based on deep learning is introduced during defect detection to solve the interference of complex backgrounds, such as brushed textures and reflections, on the detection of minor defects, thereby improving the accuracy of detection. Based on this, the defect detection process based on the aligned image and the template image specifically includes: Although the aligned image has been geometrically corrected, the image may still contain background textures or light reflections that do not belong to the detection target itself. In order to eliminate these interferences, in this embodiment of the invention, semantic segmentation can be performed on the aligned image. For example, the aligned image can be input into a pre-trained semantic segmentation model, such as a deep learning segmentation model like SegFormer, for processing. The semantic segmentation model will classify and distinguish each pixel in the image, calculate the probability that the pixel belongs to the detection target, such as the foreground strokes of a logo, or the background area, thereby obtaining a probability map, i.e., a product target probability map. In this probability map, the value of each pixel is no longer a color value, but a value between 0 and 1, which represents the confidence or probability that the corresponding position is the detection target.

[0054] After determining the product target probability map, in this embodiment of the invention, the probability map also needs to be binarized, that is, a discrimination threshold is set, such as 0.5. Pixels in the probability map that are greater than the discrimination threshold are judged as foreground (marked as 1, usually displayed as white), and pixels that are less than the discrimination threshold are judged as background (marked as 0, usually displayed as black).

[0055] After this processing, the continuously changing product target probability map is transformed into a clear black-and-white product semantic mask. This mask clearly outlines the exact shape and contour of the detected target, such as a logo or character, pinpointing the detection area to the pixel level.

[0056] Furthermore, this product semantic mask can be used to mask the original aligned image, retaining only the image content corresponding to the white area in the mask, while forcing the image content corresponding to the black area in the mask to zero or filling it with a solid color background. In this way, the complex metal brushed texture, environmental reflections on the shell, and other noise of non-detected targets in the aligned image can be removed, thereby generating a clean product foreground image.

[0057] After this, defect detection can be performed based on the product foreground image and the template image to obtain the detection results. That is, the product foreground image is compared and analyzed with the standard template image. Since the background interference has been removed at this time, any difference signal, such as color change or abnormal brightness, can be attributed with high confidence to real defects on the product surface, such as scratches, paint peeling, or discoloration, thus obtaining the final detection result.

[0058] In this embodiment of the invention, semantic segmentation and masking mechanisms are introduced after geometric alignment, which can accurately identify and extract small foreground targets such as logos, while powerfully filtering out complex background noise such as metal brushed textures and curved surface reflections commonly found on the surface of home appliances. This greatly reduces the false alarm rate in traditional differential detection, and can accurately capture tiny appearance defects even under harsh conditions such as poor lighting and complex background textures.

[0059] Based on the above embodiments, defect detection is performed based on the product foreground image and the template image to obtain the detection result, including: performing pixel-level difference processing on the product foreground image and the template image to obtain a difference image; performing threshold segmentation on the difference image to obtain a binarized defect candidate image; performing connected component analysis on the defect candidate image, and determining the detection result based on the results obtained from the connected component analysis.

[0060] Specifically, the aforementioned process of defect detection based on the product foreground image and the template image may include: First, pixel-level difference processing can be performed on the product foreground image and the template image, that is, calculating the absolute value of the difference between the pixel values ​​of the two images at each coordinate point, such as grayscale values ​​or RGB channel values. If the product under test is intact, the difference in pixel values ​​at the corresponding coordinate point should be close to zero; however, if scratches, stains, or defects exist, the pixel values ​​at the corresponding coordinate point will show significant differences. Traversing the entire image yields a difference image representing the difference in pixel values ​​at all coordinate points. In this image, the background and intact areas appear black (low pixel values), while potential defective areas appear as bright white patches (high pixel values), thus visually exposing hidden defects.

[0061] Next, thresholding can be performed on the difference image to obtain a binarized defect candidate image.

[0062] Specifically, the differential image may contain weak noise points caused by sensor noise or minor illumination fluctuations, which are not actual defects. To remove these interferences and clearly define the defect outline, in this embodiment of the invention, the differential image can be thresholded. Points with a difference value higher than a preset grayscale threshold are identified as outliers, i.e., potential defects, and assigned a value of 1. Points with a difference value lower than the grayscale threshold are identified as normal points, i.e., background or noise, and assigned a value of 0. After this processing, the original differential image with continuously varying grayscale is converted into a black-and-white defect candidate image. In the image, white areas represent candidate locations that may be defects, while black areas are normal areas.

[0063] Next, connected component analysis can be performed on the defect candidate image, and the detection result can be determined based on the results of the connected component analysis. That is, although a defect candidate image is obtained, the white pixels in the image may be scattered. In order to obtain a defect description with practical significance, in this embodiment of the invention, connected component analysis can be performed on the defect candidate image, that is, adjacent white pixels in the image can be clustered to combine interconnected white pixels together and mark them as an independent connected component. Each connected component represents a specific defect, such as a scratch, a spot, etc.

[0064] Subsequently, the geometric properties of these connected components, such as area, length, width, and centroid position, can be statistically analyzed. Based on these statistical results, the final detection result can be determined. For example, the area of ​​the connected component can be compared with a preset defect standard. Components with an area less than 5 pixels are considered noise and filtered out, while those with an area greater than 5 pixels are determined to be real defects. The final detection result is then output, and a specific report can be provided, such as "3 defects were detected, with coordinates (x1, y1)... and areas s1...", thus achieving the visualization and quantification of defects.

[0065] In this embodiment of the invention, a complete defect detection chain from pixel difference to connected component analysis is constructed. First, pixel-level difference is used to capture subtle differences with high sensitivity. Then, threshold segmentation is used to effectively filter out random noise. Finally, connected component analysis is used to transform discrete pixels into physically meaningful defect entities. Through this progressive processing method, not only can the defect location be accurately located, but also quantitative indicators such as the number and area of ​​defects can be provided. This provides objective and quantifiable data support for the quality control of home appliances leaving the factory.

[0066] Based on the above embodiments, pixel-level differential processing is performed on the product foreground image and the template image to obtain a differential image, including: semantic segmentation of the template image to obtain a template target probability map; binarization of the template target probability map to obtain a template semantic mask; masking processing of the template image based on the template semantic mask to obtain a template foreground image; and pixel-level differential processing is performed on the product foreground image and the template foreground image to obtain a differential image.

[0067] Specifically, to ensure the rigor of the comparison and avoid false differences between the template image and the pure product foreground image due to background information contained in the template image, in this embodiment of the invention, semantic segmentation processing can also be performed on the template image before differential calculation. Based on this, the above-mentioned pixel-level differential processing of the product foreground image and the template image specifically includes: consistent with the logic of processing the product image, firstly, semantic segmentation is performed on the template image. For example, by inputting the template image into the same semantic segmentation model for processing, a template target probability map indicating the probability that each pixel in the template image belongs to the detection target can be obtained. Subsequently, the same discrimination threshold, such as 0.5, can be used to binarize the probability map. Pixels in the probability map with a value greater than the discrimination threshold are judged as foreground (marked as 1, usually displayed as white), and pixels with a value less than the discrimination threshold are judged as background (marked as 0, usually displayed as black). In this way, a black-and-white template semantic mask can be obtained. This mask accurately defines the standard contour range of the detection target in the template image.

[0068] Furthermore, the generated template semantic mask can be used to mask the original template image, retaining only the image content corresponding to the white area in the mask, and forcibly setting the image content corresponding to the black area, i.e. the background area in the template image that does not belong to the detection target, such as the base color and texture, to zero or filling it with a solid color background, thereby obtaining the template foreground image.

[0069] After both images have completed background masking, in this embodiment of the invention, pixel-level differential processing can be formally performed. That is, the product foreground image and the template foreground image are compared and analyzed, such as by performing point-to-point subtraction. Since both images have now had background interference removed and only the foreground is retained, the calculated difference value will focus entirely on the shape or surface quality changes of the target itself. The final differential image can reflect the true defects of the product under test relative to the standard template with an extremely high signal-to-noise ratio.

[0070] In this embodiment of the invention, by performing the same semantic segmentation and masking processing on the template image as on the product image, a bidirectional cleaning differential mechanism is constructed. This ensures that the differential operation is performed in the purity dimension of foreground to foreground, completely eliminating the error introduced by the inconsistency between the background of the template itself and the background of the product, further improving the purity of the differential image, and enabling extremely high sensitivity and accuracy when facing minor scratches or color differences.

[0071] The defect detection device provided by the present invention is described below. The defect detection device described below and the defect detection method described above can be referred to in correspondence.

[0072] Figure 3 is a schematic diagram of the defect detection device provided by the present invention. As shown in Figure 3, the device includes: an image acquisition unit 310, used to acquire a template image and a product image of the product to be tested; an image alignment unit 320, used to perform feature matching on the template image and the product image, and perform non-rigid alignment on the product image based on the feature point pairs obtained by feature matching, to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; and a defect detection unit 330, used to perform defect detection based on the aligned image and the template image, and obtain a detection result.

[0073] The defect detection device provided by this invention introduces a non-rigid alignment mechanism after image acquisition, which effectively solves the image deformation problem caused by curved surfaces of the product under test and incorrect shooting angles in traditional solutions. It can adaptively correct local and global deformations of the product image based on feature point pairs, generating an aligned image with a geometric viewpoint highly consistent with the template image. This greatly improves the alignment accuracy of defect detection, allowing for greater focus on the real differences on the product surface during defect detection. It effectively shields non-defective geometric deformation interference, thereby significantly improving the accuracy and robustness of defect detection on complex surfaces such as home appliances.

[0074] Based on the above embodiments, the image alignment unit 320 is used to: determine the detection region in the template image and the detection region in the product image; the detection region contains the detection target of the product to be tested; perform feature extraction on the detection region in the template image and the detection region in the product image to obtain template feature points and their feature point descriptions, as well as product feature points and their feature point descriptions; and perform feature matching on the template feature points and the product feature points based on the feature point descriptions to obtain feature point pairs.

[0075] Based on the above embodiments, the image alignment unit 320 is used to: construct a non-rigid transformation model between the detection region in the template image and the detection region in the product image based on the feature point pair; and remap the pixel coordinates in the detection region of the product image based on the non-rigid transformation model to obtain the aligned image.

[0076] Based on the above embodiments, the non-rigid transformation model is composed of multiple local homography matrices; the image alignment unit 320 is used to: divide the detection area in the product image into grids to obtain multiple local grids; determine the grid center point of each local grid, and determine the distance weight of the feature point pair for each local grid based on the distance between the product feature point in the feature point pair and the grid center point; determine the local homography matrix of each local grid based on the feature point pair and its distance weight for each local grid; and construct the non-rigid transformation model based on the local homography matrix of each local grid.

[0077] Based on the above embodiments, the defect detection unit 330 is used to: perform semantic segmentation on the aligned image to obtain a product target probability map; binarize the product target probability map to obtain a product semantic mask; perform masking processing on the aligned image based on the product semantic mask to obtain a product foreground image; and perform defect detection based on the product foreground image and the template image to obtain a detection result.

[0078] Based on the above embodiments, the defect detection unit 330 is used to: perform pixel-level difference processing on the product foreground image and the template image to obtain a difference image; perform threshold segmentation on the difference image to obtain a binarized defect candidate image; perform connected component analysis on the defect candidate image, and determine the detection result based on the results obtained from the connected component analysis.

[0079] Based on the above embodiments, the defect detection unit 330 is used to: perform semantic segmentation on the template image to obtain a template target probability map; binarize the template target probability map to obtain a template semantic mask; perform masking processing on the template image based on the template semantic mask to obtain a template foreground image; and perform pixel-level difference processing on the product foreground image and the template foreground image to obtain a difference image.

[0080] Figure 4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 4, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, communication interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a defect detection method. This method includes: acquiring a template image and a product image of the product to be tested; performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching to obtain an aligned image whose geometric perspective is consistent with that of the template image; and performing defect detection based on the aligned image and the template image to obtain a detection result.

[0081] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the defect detection method provided by the above methods, the method including: acquiring a template image and a product image of a product to be tested; performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on the feature point pairs obtained by feature matching to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; performing defect detection based on the aligned image and the template image to obtain a detection result.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the defect detection method provided by the above methods. The method includes: acquiring a template image and a product image of a product to be tested; performing feature matching on the template image and the product image, and performing non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; performing defect detection based on the aligned image and the template image to obtain a detection result.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A defect detection method, characterized in that, include: Obtain a template image and a product image of the product to be tested; perform feature matching on the template image and the product image, and perform non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching to obtain an aligned image whose geometric viewpoint is consistent with that of the template image; Defect detection is performed based on the aligned image and the template image to obtain the detection results.

2. The defect detection method according to claim 1, characterized in that, The feature matching of the template image and the product image includes: determining the detection region in the template image and the detection region in the product image; the detection region contains the detection target of the product to be tested; performing feature extraction on the detection region in the template image and the detection region in the product image to obtain template feature points and their feature point descriptions, as well as product feature points and their feature point descriptions; and performing feature matching on the template feature points and the product feature points based on the feature point descriptions to obtain feature point pairs.

3. The defect detection method according to claim 2, characterized in that, The step of performing non-rigid alignment on the product image based on feature point pairs obtained from feature matching to obtain an aligned image whose geometric perspective is consistent with that of the template image includes: constructing a non-rigid transformation model between the detection region in the template image and the detection region in the product image based on the feature point pairs; and remapping the pixel coordinates within the detection region in the product image based on the non-rigid transformation model to obtain the aligned image.

4. The defect detection method according to claim 3, characterized in that, The non-rigid transformation model is composed of multiple local homography matrices. The step of constructing a non-rigid transformation model between the detection region in the template image and the detection region in the product image based on the feature point pairs includes: dividing the detection region in the product image into multiple local grids; determining the grid center point of each local grid, and determining the distance weight of the feature point pair for each local grid based on the distance between the product feature point in the feature point pair and the grid center point; determining the local homography matrix of each local grid based on the feature point pair and its distance weight for each local grid; and constructing the non-rigid transformation model based on the local homography matrices of each local grid.

5. The defect detection method according to any one of claims 1 to 4, characterized in that, The defect detection based on the aligned image and the template image to obtain the detection result includes: performing semantic segmentation on the aligned image to obtain a product target probability map; binarizing the product target probability map to obtain a product semantic mask; performing masking processing on the aligned image based on the product semantic mask to obtain a product foreground image; and performing defect detection based on the product foreground image and the template image to obtain the detection result.

6. The defect detection method according to claim 5, characterized in that, The defect detection based on the product foreground image and the template image to obtain the detection result includes: performing pixel-level difference processing on the product foreground image and the template image to obtain a difference image; performing threshold segmentation on the difference image to obtain a binarized defect candidate image; performing connected component analysis on the defect candidate image, and determining the detection result based on the results obtained from the connected component analysis.

7. The defect detection method according to claim 6, characterized in that, The step of performing pixel-level differential processing on the product foreground image and the template image to obtain a differential image includes: performing semantic segmentation on the template image to obtain a template target probability map; binarizing the template target probability map to obtain a template semantic mask; performing masking processing on the template image based on the template semantic mask to obtain a template foreground image; and performing pixel-level differential processing on the product foreground image and the template foreground image to obtain a differential image.

8. A defect detection device, characterized in that, include: The image acquisition unit is used to acquire template images and product images of the product to be tested. An image alignment unit is used to perform feature matching on the template image and the product image, and to perform non-rigid alignment on the product image based on the feature point pairs obtained from the feature matching, so as to obtain an aligned image whose geometric viewpoint is consistent with that of the template image. The defect detection unit is used to perform defect detection based on the aligned image and the template image to obtain the detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the defect detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the defect detection method as described in any one of claims 1 to 7.

Images