Thin surface appearance defect detection method based on template learning and comparison

Abstract

A thin-surface appearance defect detection method based on template learning and comparison includes: 1) automatically learning and constructing template prototypes based on images of good products; 2) detecting template instances in the sample image to be inspected for each template prototype; 3) mapping and aligning the corresponding template prototype with the template instance region for each template instance; 4) comparing the mapped and aligned template instance with the template prototype for each template instance to obtain a feature difference map; 5) applying defect detection algorithm parameters to the feature difference map of each template instance to extract defect information within the template instance; 6) synthesizing the defect information of all template instances to output a product OK / NG determination and a defect information list. This invention integrates multiple technical approaches to achieve accurate, flexible, and efficient appearance defect detection for products with regular flat content.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision defect detection, and in particular relates to a thin surface appearance defect detection method based on template learning and comparison. Background Art

[0002] Modeling and comparison are common methods in machine vision defect detection, particularly suitable for detecting surface defects in products with regular, planar surfaces. Compared to methods based on manually designed feature extraction and analysis, or deep learning-based approaches, defect detection methods based on modeling and comparison are simple, efficient, and flexible. They don't require a large number of training samples, and through modeling, they can quickly support defect detection for new product models. Furthermore, defect detection can be quickly and easily switched between different product models.

[0003] In industrial inspection applications, current defect detection methods based on modeling and comparison have the following three prominent problems:

[0004] 1. The modeling process is difficult and cumbersome: Product modeling requires a series of selection and input operations. Without familiarity with the basic logic of the detection algorithm, errors or inappropriate operations are likely to occur, resulting in the inability to run the test or degraded detection performance.

[0005] 2. Product deformation leads to over-inspection and missed inspections: Current methods mostly use affine or perspective transformations to map product templates to the online product image space. Since thin products are prone to deformations such as warping and convexity, radial and perspective transformations of the entire piece cannot accurately align the template with the sample image to be inspected, resulting in over-inspection or missed inspections by the algorithm.

[0006] 3. Difficulty in meeting diverse needs: In some application scenarios, different inspection standards need to be adopted for different parts of the product, such as key areas, general areas, unimportant areas, and shielded areas. Even different areas of the same type have different requirements for defect detection. Existing methods often cannot meet these diverse needs, or the product needs to be divided into many small areas for separate modeling and inspection, which makes the modeling process more complicated and tedious.

[0007] These issues severely limit the scope of application of defect detection methods based on modeling and comparison, resulting in significant learning, usage, and management costs for personnel. Furthermore, they lead to unstable and unreliable defect detection performance and efficiency. Therefore, it is necessary to propose a new defect detection method based on modeling and comparison to address these three prominent issues. Summary of the Invention

[0008] In order to address the problems of current defect detection methods based on modeling and comparison, such as the difficult and cumbersome modeling process, product deformation leading to over-inspection and missed inspection, and difficulty in meeting diverse needs, the present invention provides a defect detection method based on template learning and comparison, which can achieve accurate, flexible and efficient appearance defect detection of products with planar regular content.

[0009] The present invention adopts the following technical solution. The thin surface appearance defect detection method based on template learning and comparison is used to detect appearance defects of flat surface regular content products, and specifically includes the following steps:

[0010] Step S1: Automatically learn and construct a template prototype based on good product images;

[0011] Step S2: for each template prototype, detecting the template instance in the sample image to be detected;

[0012] Step S3: For each template instance, mapping and aligning the corresponding template prototype with the template instance area;

[0013] Step S4: for each template instance, the template instance to be mapped and aligned is compared with the template prototype to obtain a feature difference map;

[0014] Step S5: applying the defect detection algorithm parameters to the feature difference graph of each template instance to extract the defect information within the template instance;

[0015] Step S6: Summarize the defect information of all template instances and output OK / NG judgment and defect information list.

[0016] According to the above technical solution, if Figure 1 As shown, the specific steps of automatically learning and constructing a template prototype based on good product images in step S1 are:

[0017] Step S1.1: Calculate and determine the image size of the product unit based on the good product image;

[0018] Step S1.2: extracting a series of positioning feature points from the good product image;

[0019] Step S1.3: Determine the area where each template is located based on the product unit image size, positioning feature points, and the number of templates;

[0020] Step S1.4: Divide each template area into grids and search for the nearest positioning feature point for each grid vertex as the template anchor point;

[0021] Step S1.5: Construct a template prototype based on the good product image, template area, template grid and anchor points. The template prototype includes the template original image, template feature map, template grid, template anchor points, template anchor point local images and several template detection areas.

[0022] In particular, the specific method of extracting a series of positioning feature points from the good product image in step S1.2 can be Harris, SIFT, SURF, FAST or ORB methods.

[0023] According to the above technical solution, the specific steps of detecting the template instance in the sample image to be detected for each template prototype in step S2 are:

[0024] Step S2.1: The template original image and the sample image to be detected in the template prototype are reduced step by step to obtain a template image pyramid and a sample image pyramid to be detected;

[0025] Step S2.2: Perform template matching and positioning on the template pyramid and the top image of the to-be-detected sample image pyramid to obtain the position and size of the template instance in the top image of the to-be-detected sample image pyramid at a small scale;

[0026] Step S2.3: Enlarge the position and size of the template instance in the upper layer of the pyramid of the sample image to be detected at a small scale and transfer it to the next layer of the pyramid of the sample image to be detected. Perform template matching and positioning within a local area centered on the position of the enlarged template instance to obtain the position and size of the template instance in the lower layer of the pyramid of the sample image to be detected at a higher scale.

[0027] Step S2.4: Repeat step S2.3 until the bottom layers of the template pyramid and the to-be-detected sample image pyramid are reached, and the position and size of the template instance at the maximum scale are obtained.

[0028] In particular, the specific method of establishing the image pyramid in step S1.2 is: using the original image as the bottom layer of the pyramid, reducing the width and height of the bottom layer of the pyramid by a specific ratio, and using the reduced image as the upper layer of the pyramid. Repeat the above method to reduce the image and add it to the upper layer of the pyramid until the number of pyramid layers reaches a specific number or the size of the top layer image is small enough.

[0029] According to the above technical solution, the specific steps of mapping and aligning the corresponding template prototype with the template instance area for each template instance in step S3 are as follows:

[0030] Step S3.1: For each positioning anchor point in the template prototype, determine the corresponding positioning point position within the template instance area using the template anchor point and the template anchor point local image;

[0031] Step S3.2: Figure 2 As shown, for each grid cell in the template grid in the template prototype, the template anchor points corresponding to the four vertices of the grid cell rectangle are determined according to the distance, and the coordinate mapping relationship from the template grid to the corresponding template instance is established through the four template anchor points and the corresponding positioning points;

[0032] Step S3.3: For each grid cell in the template grid in the template prototype, map the template feature map and template detection area in the template grid to the template instance plane, or map the template instance feature map to the template prototype plane according to the coordinate mapping relationship.

[0033] In particular, the specific method of using the template anchor point and the local image of the template anchor point in step S3.1 to determine the corresponding positioning point position within the template instance area is: determine the positioning point search area based on the template anchor point and the template instance position, and use the local image of the template anchor point as a template to perform template matching and positioning within the positioning point search area to obtain the positioning point position.

[0034] Preferably, the coordinate mapping relationship from the template grid to the corresponding template instance in step S3.2 is expressed as a perspective transformation.

[0035] In particular, the template feature map and template instance in step S3.3 may be or include images such as brightness map, color map, edge map, etc. that can reflect the characteristics of the product and its defects.

[0036] According to the above technical solution, the specific steps of extracting defect information in the template instance in step S5 are:

[0037] Step S5.1: for each template detection region of the template instance, extract a detection region feature difference map from the feature difference map;

[0038] Step S5.2: Apply the defect detection parameters corresponding to the inspection area to the inspection area feature difference map to extract defect information such as defect location, size, category, and confidence level.

[0039] According to the above technical solution, the specific steps of applying the defect detection parameters of the corresponding detection area to the detection area feature difference map in step S5.2 to extract defect information such as defect location, size, category, and confidence are as follows:

[0040] Step S5.2.1: Applying a feature difference threshold parameter to the feature difference map of the inspection area to obtain a defect binary map;

[0041] Step S5.2.2: extracting connected domains from the defect binary graph;

[0042] Step S5.2.3: Apply the merge distance parameter to the connected domains in the defect binary map to merge the connected domains. Combined with the edge map corresponding to the connected domains, apply the shield edge width parameter to filter defect pixels at the edges of the product. Apply the minimum defect size parameter to filter the connected domains.

[0043] Step S5.2.4: For each connected domain of the merged and filtered defect binary map, extract its position and size as the position and size of the defect, and estimate the defect category and confidence level by combining the feature map and feature difference map corresponding to the connected domain.

[0044] The present invention discloses a thin surface appearance defect detection method based on template learning and comparison, which has three beneficial effects:

[0045] 1) The automatic template learning method of this application can automatically construct template prototypes based on good product images, solving the problem of difficult and tedious modeling process, reducing the user's usage threshold and burden, and at the same time reducing the probability of defect detection failure or performance degradation due to improper manual operation;

[0046] 2) Compared with the current overall mapping method, the template grid division and mapping method of this application can more accurately align the template with the sample image to be inspected when thin products have warping, concave and convex deformation, solving the problem of missed inspections due to product deformation;

[0047] 3) The zoning detection method according to the template detection area of ​​the present application can apply different defect detection standards to different product parts, tighten the defect detection in key areas, and relax the defect detection in unimportant areas, which can meet the diverse detection needs of industrial inspection.

[0048] Based on the above technical means and their beneficial effects, the present invention realizes accurate, flexible and efficient appearance defect detection of planar regular content products. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the template prototype constructed by automatic learning based on good product images of the present invention.

[0050] Figure 2 It is a schematic diagram of the direct correspondence between the template anchor point and the template instance positioning point and the coordinate mapping relationship between the template grid and the template instance of the present invention.

[0051] Figure 3 An example of a translation similarity image. DETAILED DESCRIPTION

[0052] The present invention discloses a thin surface appearance defect detection algorithm based on template learning and comparison. The specific implementation of the present invention is further described below in conjunction with preferred embodiments.

[0053] The thin surface appearance defect detection method based on template learning and comparison of the present invention comprises the following steps:

[0054] Step S1: Figure 1 As shown, the template prototype is automatically learned and constructed based on the good product images;

[0055] Step S2: for each template prototype, detecting the template instance in the sample image to be detected;

[0056] Step S3: For each template instance, mapping and aligning the corresponding template prototype with the template instance area;

[0057] Step S4: for each template instance, the template instance to be mapped and aligned is compared with the template prototype to obtain a feature difference map;

[0058] Step S5: applying the defect detection algorithm parameters to the feature difference graph of each template instance to extract the defect information within the template instance;

[0059] Step S6: Summarize the defect information of all template instances and output OK / NG judgment and defect information list.

[0060] The specific steps of automatically learning and constructing a template prototype based on good product images in step S1 above are as follows:

[0061] Step S1.1: Calculate and determine the image size of the product unit based on the good product image. Figure 1 The good product image contains two complete product units. The image width of the product unit can be estimated by extracting and analyzing the edge image, and then the content of the good product image is translated and slid to compare and calculate the similarity of the overlapping parts with the original image to construct a translation similarity image (such as Figure 3 As shown), the image height of the product unit is estimated;

[0062] Step S1.2: extracting a series of positioning feature points from the good product image. Preferably, the Harris corner point extraction algorithm is used to extract the positioning feature points.

[0063] Step S1.3: appropriately enlarge the product unit image size to use as the template size, and determine the area where each template is located based on the distribution of positioning feature points and the number of templates;

[0064] Step S1.4: Divide each template area into grids according to the template size. Figure 1 The template area is divided into a grid of 2 rows and 3 columns, and the nearest positioning feature point is searched for each grid vertex as the anchor point of the template;

[0065] Step S1.5: Construct a template prototype based on the good product image, template area, template grid and anchor points. The template prototype includes the template original image, template feature map, template grid, template anchor points, template anchor point local images and several template detection areas.

[0066] One method for determining the template area in step S1.3 is to search the product image space with a template rectangular frame of a fixed size, and find the place with the most concentrated positioning feature points in the template rectangular frame as the template position.

[0067] The template detection area in step S1.4 can be set to categories such as key area, general area, unimportant area and shielded area according to the detection requirements. One or more detection areas can be set for each type. Each type of detection area has corresponding preset detection parameters, and the detection parameters of each detection area can be set and adjusted independently.

[0068] The specific steps of detecting the template instance in the sample image to be detected for each template prototype in step S2 are:

[0069] Step S2.1: The template original image and the sample image to be detected in the template prototype are reduced step by step to obtain a template image pyramid and a sample image pyramid to be detected;

[0070] Step S2.2: Perform template matching and positioning on the template pyramid and the top image of the to-be-detected sample image pyramid to obtain the position and size of the template instance in the top image of the to-be-detected sample image pyramid at a small scale;

[0071] Step S2.3: Enlarge the position and size of the template instance in the upper layer of the pyramid of the sample image to be detected at a small scale and transfer it to the next layer of the pyramid of the sample image to be detected. Perform template matching and positioning within a local area centered on the position of the enlarged template instance to obtain the position and size of the template instance in the lower layer of the pyramid of the sample image to be detected at a higher scale.

[0072] Step S2.4: Repeat step S2.3 until the bottom layers of the template pyramid and the to-be-detected sample image pyramid are reached, and the position and size of the template instance at the maximum scale are obtained.

[0073] The specific method of establishing the image pyramid in step S1.2 is: use the original image as the bottom layer of the pyramid, reduce the width and height of the bottom layer of the pyramid by a fixed ratio of 1 / 3, and use the reduced image as the upper layer of the pyramid. Repeat the above method to reduce the image and add it to the upper layer of the pyramid until the number of pyramid layers reaches 3 or the width and height of the top layer image are both less than 300 pixels.

[0074] The specific steps of mapping and aligning the corresponding template prototype with the template instance area for each template instance in step S3 are as follows:

[0075] Step S3.1: For each positioning anchor point in the template prototype, determine the corresponding positioning point position within the template instance area using the template anchor point and the template anchor point local image;

[0076] Step S3.2: Figure 2As shown, for each grid cell in the 3*2 template grid in the template prototype, find the template anchor points closest to the four vertices of the grid cell rectangle, and establish the coordinate mapping relationship from the template grid to the corresponding template instance through the four template anchor points and the corresponding positioning points;

[0077] Step S3.3: For each grid cell in the template grid in the template prototype, the template feature map and the template detection area in the template grid are mapped to the template instance plane according to the coordinate mapping relationship.

[0078] The specific method of using the template anchor point and the local image of the template anchor point in step S3.1 to determine the corresponding positioning point position within the template instance area is: determine the positioning point search area based on the template anchor point and the template instance position, and use the local image of the template anchor point as a template to perform template matching and positioning within the positioning point search area to obtain the positioning point position.

[0079] The coordinate mapping relationship from the template mesh to the corresponding template instance in step S3.2 is expressed as a perspective transformation.

[0080] The template feature map and template instance in step S3.3 may be or include images such as brightness map, color map, edge map, etc. that can reflect the characteristics of the product and its defects.

[0081] In step S4, for each template instance, the specific method of comparing the mapped aligned template instance with the template prototype to obtain a feature difference map is as follows: subtracting the template feature map in the template prototype from the corresponding template instance feature map to obtain a signed feature difference map.

[0082] The specific steps of applying the defect detection parameters of the corresponding detection area to the detection area feature difference map in step S5.2 to extract defect information such as defect location, size, category, and confidence level are as follows:

[0083] Step S5.2.1: Applying a feature difference threshold parameter to the feature difference map of the inspection area to obtain a defect binary map;

[0084] Step S5.2.2: extracting connected domains from the defect binary graph;

[0085] Step S5.2.3: Apply the merge distance parameter to the connected domains in the defect binary map to merge the connected domains. Combined with the edge map corresponding to the connected domains, apply the shield edge width parameter to filter defect pixels at the edges of the product. Apply the minimum defect size parameter to filter the connected domains.

[0086] Step S5.2.4: For each connected domain of the merged and filtered defect binary map, extract its position and size as the position and size of the defect, and estimate the defect category and confidence level by combining the feature map and feature difference map corresponding to the connected domain.

[0087] The specific method for synthesizing the defect information from all template instances and outputting an OK / NG judgment and a defect information list in step S6 is to summarize the defect information obtained from all template instances to obtain a defect information list, and then determine whether the product is OK / NG based on the number and category of defects in the defect information list. Preferably, if the defect information list contains one or more severe defects, the product is determined to be NG; if the defect information list contains no severe defects and the number of general defects exceeds 5, the product is determined to be NG; otherwise, the product is determined to be OK.

[0088] The present invention integrates multiple technical means to achieve accurate, flexible and efficient appearance defect detection of flat regular content products. Among them, automatic template learning improves the convenience of algorithm configuration and lowers the user threshold. Template grid division and mapping enhance the adaptability to warping and concave-convex deformation of thin-surface products. Zoning detection according to template detection areas supports the application of different defect detection standards to different product parts.

[0089] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A thin surface appearance defect detection method based on template learning and comparison includes the following steps: Step S1: Automatically learn and construct a template prototype based on good product images; Step S2: for each template prototype, detecting the template instance in the sample image to be detected; Step S3: For each template instance, mapping and aligning the corresponding template prototype with the template instance area; Step S4: for each template instance, the template instance to be mapped and aligned is compared with the template prototype to obtain a feature difference map; Step S5: applying the defect detection algorithm parameters to the feature difference graph of each template instance to extract the defect information within the template instance; Step S6: Summarize the defect information of all template instances and output OK / NG judgment and defect information list; The specific steps of step S1 are: Step S1.1: Calculate and determine the image size of the product unit based on the good product image; Step S1.2: extracting a series of positioning feature points from the good product image; Step S1.3: Determine the area where each template is located based on the product unit image size, positioning feature points, and the number of templates; Step S1.4: Divide each template area into grids and search for the nearest positioning feature point for each grid vertex as the template anchor point; Step S1.5: Construct a template prototype based on the good product image, template area, template grid, and anchor points. The template prototype includes the template original image, template feature map, template grid, template anchor points, template anchor point local images, and several template detection areas. The specific steps for mapping and aligning the template prototype with the template instance area in S3 are: Step S3.1: For each positioning anchor point in the template prototype, determine the corresponding positioning point position within the template instance area using the template anchor point and the template anchor point local image; Step S3.2: For each grid cell in the template grid of the template prototype, determine the template anchor points corresponding to the four vertices of the grid cell rectangle according to the distance, and establish a coordinate mapping relationship from the template grid to the corresponding template instance through the four template anchor points and the corresponding positioning points; Step S3.3: For each grid cell in the template grid of the template prototype, map the template feature map and template detection area in the template grid to the template instance plane, or map the template instance feature map to the template prototype plane according to the coordinate mapping relationship; the specific steps of extracting defect information in the template instance in S5 are as follows: Step S5.1: for each template detection region of the template instance, extract a detection region feature difference map from the feature difference map; Step S5.2: Apply the defect detection parameters corresponding to the inspection area to the inspection area feature difference map to extract defect information such as defect location, size, category, and confidence level.

Images