Screen printing defect detection method and device, electronic equipment and storage medium

Abstract

The application provides a silk screen defect detection method and device, electronic equipment and storage medium, and belongs to the technical field of intelligent manufacturing. The method comprises the following steps: performing global matching on a to-be-detected image of a to-be-detected product by using a global template; determining an initial matching position of a local template in the to-be-detected image based on the global matching result and a relative position relationship, determining a local search region according to the initial matching position, and performing accurate matching in the local search region to obtain a to-be-detected region image accurately aligned with the local template; and comparing the to-be-detected region image with the local template to generate a silk screen defect detection result. The application firstly performs global matching by using a global template, and then performs accurate alignment of each local template in a local region of the to-be-detected image based on the global matching result and the relative position relationship of the local template relative to the global template, thereby reducing false positives caused by inaccurate alignment and improving the efficiency and accuracy of silk screen defect detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting screen printing defects. Background Technology

[0002] During the manufacturing process, home appliances may have various minor defects on their surface, such as screen printing errors, ink leaks, dirt, black spots, and misalignments.

[0003] Currently, the main method for determining whether there are screen printing defects in the inspected product is to visually compare the standard product with the product to be inspected. However, manual work for long periods of time can lead to fatigue and loss of concentration, which may result in missed defects. Furthermore, some defects are very small and difficult to detect through manual visual inspection. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for detecting screen printing defects, in order to solve the problems of low detection efficiency and low accuracy that exist in the current manual inspection method.

[0005] This invention provides a method for detecting screen printing defects, comprising the following steps:

[0006] The global template is used to perform global matching on the image of the product to be inspected, and the global matching result is obtained.

[0007] Based on the global matching result and the relative position of a local template with respect to the global template, the initial matching position of the local template in the image to be inspected is determined;

[0008] The local search region corresponding to the initial matching position is determined in the image to be inspected;

[0009] Within the local search area, determine the precise matching result of the local template in the image to be inspected, so as to crop an image of the region to be inspected from the image to be inspected based on the precise matching result;

[0010] The image of the area to be inspected is compared with the local template to generate the screen printing defect detection result of the product to be inspected;

[0011] The global template, the local template, and the relative positional relationship of the local template relative to the global template are determined based on the template image of the standard product.

[0012] According to a screen printing defect detection method provided by the present invention, the local templates are multiple, and the step of comparing the image of the area to be inspected with the local templates to generate a screen printing defect detection result of the product to be inspected includes:

[0013] The image of the region to be inspected corresponding to each of the local templates is cropped from the image to be inspected;

[0014] Each local template is compared with its corresponding image of the region to be inspected to generate a comparison result;

[0015] Based on all the comparison results, the screen printing defect detection results of the product to be tested are generated.

[0016] According to a screen printing defect detection method provided by the present invention, the global template and the local template are determined based on the following steps:

[0017] Identify the silkscreened area in the template image that contains silkscreen traces;

[0018] Based on the pixel distance between the first center coordinates of each silkscreen region and the second center coordinates of the template image, multiple target regions are selected from all silkscreen regions.

[0019] Set the longest edge in all the target regions as the global template;

[0020] Each of the remaining silkscreen areas, excluding the global template, is set as a local template;

[0021] Extract the shape matching features of the global template and each of the local templates, and extract the relative positional relationship between the global template and each of the local templates.

[0022] According to the present invention, a method for detecting screen printing defects includes a global matching result comprising the matching position, matching rotation angle, and matching scaling ratio of the global template in the image to be inspected. The method for obtaining the global matching result by using the global template to perform global matching on the image to be inspected of the product includes:

[0023] Using the shape matching features of the global template, at least one set of candidate matching parameters is searched and obtained in the image to be inspected. Each set of candidate matching parameters includes a matching position, a matching rotation angle, a matching scaling ratio, and a corresponding matching degree.

[0024] Based on the matching degree of each set of candidate matching parameters, a set of optimal matching parameters is determined from the at least one set of candidate matching parameters;

[0025] The matching position, the matching rotation angle, and the matching scaling ratio included in the optimal matching parameters are determined as the global matching result.

[0026] According to a screen printing defect detection method provided by the present invention, determining the local search region corresponding to the initial matching position in the image to be inspected includes:

[0027] Using the center of the initial matching position as a reference, the initial matching position is expanded according to a preset scaling ratio to obtain the local search area.

[0028] According to a screen printing defect detection method provided by the present invention, the step of comparing the image of the area to be inspected with the local template to generate a screen printing defect detection result of the product to be inspected includes:

[0029] The image of the region to be inspected is compared with the local template in terms of structural similarity to generate a difference score map;

[0030] The difference score map is binarized to obtain a difference heatmap;

[0031] Connectivity analysis is performed on the difference heatmap to obtain the detection results of the screen printing defects of the product to be tested.

[0032] According to a screen printing defect detection method provided by the present invention, the step of comparing the image of the region to be inspected with the local template to generate a difference score map includes:

[0033] Using a preset sliding window, the image of the region to be inspected and the local template are traversed synchronously.

[0034] At each position during the sliding window traversal process, a brightness similarity measure, a structural similarity measure, and a contrast measure are determined between the first image region and the second image region; the first image region is the image on the region to be inspected covered by the sliding window, and the second image region is the image on the local template covered by the sliding window;

[0035] Based on the brightness similarity metric, the structural similarity metric, and the contrast metric, the local structural similarity score of the center pixel of each sliding window is determined;

[0036] The difference score map is obtained by combining the local structural similarity scores of all the center pixels during the traversal process.

[0037] According to the screen printing defect detection method provided by the present invention, the brightness similarity measure is determined based on the average pixel value of the first image region and the second image region, the structural similarity measure is determined based on the pixel covariance between the first image region and the second image region, and the contrast measure is determined based on the pixel variance of the first image region and the second image region.

[0038] According to a screen printing defect detection method provided by the present invention, the step of performing connected component analysis on the difference heatmap to obtain the screen printing defect detection result of the product to be inspected includes:

[0039] One or more connected domains are identified in the differential heatmap;

[0040] Calculate the area of ​​each of the connected components;

[0041] The area of ​​each of the connected components is compared with a first preset area threshold.

[0042] Connected regions with an area greater than the first preset area threshold are identified as potential defect regions.

[0043] The defect characteristics of each potential defect area are analyzed to generate the screen printing defect detection results.

[0044] According to a screen printing defect detection method provided by the present invention, the step of analyzing the defect features of each potential defect region to generate the screen printing defect detection result includes:

[0045] Deep learning features are extracted from each potential defect region and the corresponding region of each potential defect region in the local template.

[0046] Compare the deep learning features of the potential defect region with the deep learning features of the corresponding region to generate an anomaly score map for each potential defect region;

[0047] The screen printing defect detection results are determined based on all the aforementioned anomaly score maps.

[0048] According to the screen printing defect detection method provided by the present invention, the deep learning features are extracted based on a visual converter network model;

[0049] The step of extracting deep learning features for each potential defect region and the corresponding region of each potential defect region in the local template includes:

[0050] The potential defect region and the corresponding region are respectively input into the visual converter network model to obtain the deep learning features output by the visual converter network model.

[0051] According to the screen printing defect detection method provided by the present invention, the visual converter network model specifically performs the following operations:

[0052] After dividing the input into multiple image blocks, a set of multi-level local features is extracted from each image block from multiple preset levels;

[0053] At each of the preset levels, the extracted local features are subjected to adaptive pooling at various scales.

[0054] The local features processed by pooling at various scales are fused to form a set of multi-dimensional local features that contain both multi-level and multi-aggregation information, which are used as the deep learning features of each image patch.

[0055] According to the present invention, a method for detecting screen printing defects includes comparing the deep learning features of the potential defect region with the deep learning features of the corresponding region to generate an anomaly score map for each potential defect region, comprising:

[0056] Traverse each image block in the potential defect region and calculate the feature distance between the deep learning feature of any image block and the deep learning feature of all image blocks in the corresponding region.

[0057] The minimum feature distance is determined as the anomaly score for any image block;

[0058] The anomaly scores of all the image blocks are combined to form the anomaly score map.

[0059] According to a screen printing defect detection method provided by the present invention, when at least two reference templates are provided for the potential defect region, and each reference template contains a corresponding region of the potential defect region, determining the minimum feature distance as the anomaly score of any image block includes:

[0060] Calculate the feature distance between the deep learning features of any image patch and the deep learning features of all image patches in each corresponding region to obtain multiple minimum feature distances;

[0061] From the plurality of minimum feature distances, select a number of minimum feature distances with the smallest values ​​according to a preset ratio, and calculate the average feature distance.

[0062] The average value of the feature distance is determined as the anomaly score of any image patch.

[0063] According to a screen printing defect detection method provided by the present invention, determining the screen printing defect detection result based on all the anomaly score maps includes:

[0064] Each of the aforementioned anomaly score maps is thresholded to generate a defect binary map;

[0065] Perform connected component analysis on all the defect binary graphs. If it is determined that there is any connected component with an area greater than a second preset area threshold, then the screen printing defect detection result is determined to be that a screen printing defect exists.

[0066] The present invention also provides a screen printing defect detection device, comprising:

[0067] The global template matching unit is used to perform global matching on the image of the product to be inspected using a global template, and obtain the global matching result.

[0068] A local template localization unit is used to determine the initial matching position of the local template in the image to be inspected based on the global matching result and the relative positional relationship of a local template with respect to the global template.

[0069] The search region localization unit is used to determine the local search region corresponding to the initial matching position in the image to be inspected;

[0070] An image localization unit is used to determine the precise matching result of the local template in the image to be inspected within the local search area, so as to crop an image of the region to be inspected from the image to be inspected based on the precise matching result;

[0071] A defect detection processing unit is used to compare the image of the area to be inspected with the local template to generate the screen printing defect detection result of the product to be inspected;

[0072] The global template, the local template, and the relative positional relationship of the local template relative to the global template are determined based on the template image of the standard product.

[0073] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the screen printing defect detection method as described above.

[0074] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the screen printing defect detection method as described above.

[0075] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the screen printing defect detection method as described above.

[0076] The screen printing defect detection method, device, electronic device, and storage medium provided by the present invention first use a global template for global matching, and then, based on the global matching result and the relative positional relationship of local templates with respect to the global template, precisely align each local template in a local area of ​​the image to be inspected, thereby reducing false alarms caused by misalignment and improving the efficiency and accuracy of screen printing defect detection. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0078] Figure 1 This is a flowchart illustrating the screen printing defect detection method provided by the present invention.

[0079] Figure 2 This is a flowchart illustrating the process of determining global and local templates provided by the present invention.

[0080] Figure 3 This is a schematic diagram of the process of global matching of the image of the product to be inspected using a global template, provided by the present invention.

[0081] Figure 4 This is a schematic diagram of the process for generating screen printing defect detection results of a product to be inspected based on an image of the area to be inspected and a local template, as provided by the present invention.

[0082] Figure 5 This is a schematic diagram of the process provided by the present invention for comparing the structural similarity between the image of the region to be inspected and a local template.

[0083] Figure 6 This is a schematic diagram of the feature extraction process based on the visual converter network model provided by the present invention.

[0084] Figure 7 This is a schematic diagram of the process for generating an anomaly score map of a potential defect region based on deep learning features of the potential defect region and the corresponding region, as provided by the present invention.

[0085] Figure 8 This is a flowchart illustrating the process of determining the abnormal score of any image block provided by the present invention.

[0086] Figure 9 This is a schematic diagram of the automatic modeling process provided by the present invention.

[0087] Figure 10 This is a schematic diagram of the defect detection process provided by the present invention.

[0088] Figure 11 This is a schematic diagram illustrating the principle of the SSIM algorithm provided by this invention.

[0089] Figure 12 This is a schematic diagram of the screen printing defect detection device provided by the present invention.

[0090] Figure 13This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0092] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0093] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0094] The following is combined with Figures 1-13 This invention describes the screen printing defect detection method, apparatus, electronic device, and storage medium provided by the present invention.

[0095] Figure 1 This is a flowchart illustrating the screen printing defect detection method provided by the present invention, as shown below. Figure 1 As shown, the execution subject of the screen printing defect detection method provided by the present invention can be an industrial control computer, a server, a cloud computing platform, or a computer capable of executing the method of the present invention, etc. Unless otherwise specified, the following embodiments will be described using an industrial control computer as an example.

[0096] As an optional embodiment, the screen printing defect detection method includes, but is not limited to, the following steps:

[0097] Step 110: Use the global template to perform global matching on the image of the product to be inspected and obtain the global matching result.

[0098] A global template refers to an image region with significant and stable features extracted in advance from a template image of a standard product. This image region can serve as a reference for initial positioning in the image to be inspected. For example, a global template can be a pattern with rich edge or texture features selected from the silkscreen area of ​​a standard product image, such as a complex brand logo or function icon.

[0099] The product to be inspected refers to the physical product on the production line that needs to be inspected for silkscreen defects. Its surface is printed with text, brand logos or patterns, etc. For example, the product to be inspected can be the panel of a home appliance, such as a washing machine panel, microwave oven panel or air conditioner panel.

[0100] An image to be inspected refers to an image obtained by taking a picture of the product to be inspected placed in the inspection area under preset lighting conditions using an image acquisition device (such as an industrial camera), which can completely reflect the screen printing state of its surface. For example, the process of obtaining an image to be inspected may include: placing the product to be inspected on a tray, sending it to an inspection position equipped with a light source and a camera through a conveyor device, illuminating the surface of the product with the light source, and then taking a picture with the camera.

[0101] Global matching refers to searching for regions in the image to be inspected that are most similar to or consistent with the features of a global template. For example, global matching can be implemented based on shape matching algorithms, that is, searching for regions in the image to be inspected that match the shape features such as contours and edges of the global template.

[0102] The global matching result refers to a set of parameter information describing the position and orientation of the global template matched in the image to be inspected. The global matching result can include: the matching position of the global template in the image to be inspected (such as the center coordinates), the matching rotation angle, and the matching scaling ratio. For example, if a region in the image to be inspected that is most similar to or consistent with the features of the global template is found, and this region is rotated 1.5 degrees relative to the global template, scaled to 1.01 times its original size, and its center coordinates are translated to (x1, y1), then this set of parameters [(x1, y1), 1.5°, 1.01] constitutes the global matching result.

[0103] Step 120: Based on the global matching result and the relative position of a local template to the global template, determine the initial matching position of the local template in the image to be inspected; the global template and the local template, as well as the relative position of the local template to the global template, are determined based on the template image of the standard product.

[0104] A local template refers to an image block extracted from the template image of a standard product, representing other silkscreen areas to be inspected, excluding the global template. For example, in a template image of a home appliance panel, if a complex brand logo is selected as the global template, then other silkscreen areas on the appliance panel, such as the text of the "Start" button or the pattern of the "Power" gear, can each be used as an independent local template.

[0105] Relative positional relationship refers to the spatial position and orientation of each local template relative to the global template on a template image based on a standard product. For example, the relative positional relationship can be a two-dimensional displacement vector that describes the offset (Δx, Δy) of the center coordinates of the local template relative to the center coordinates of the global template. Furthermore, the relative positional relationship can also include the rotation angle and scaling information of each local template relative to the global template.

[0106] The initial matching position refers to the position of a local template initially calculated on the image to be inspected based on the global matching result and the relative positional relationship between the local template and the global template. For example, if the global matching result shows that the center coordinates of the global template are located at (x1, y1) in the image to be inspected, and the two-dimensional displacement vector of the relative positional relationship of a certain local template with respect to the global template is (Δx, Δy), then by applying this two-dimensional displacement vector to the center coordinates (x1, y1) of the global template, the initial matching position of the local template in the image to be inspected can be calculated. That is, the center coordinates of the initial matching position can be initially determined to be (x1+Δx, y1+Δy).

[0107] Step 130: Determine the local search region corresponding to the initial matching position in the image to be inspected.

[0108] The local search region refers to an image area centered on the initial matching position and slightly larger than that initial matching position.

[0109] Considering the slight pose deviations that may exist in the actual placement of the product under test, as well as the local nonlinear deformation caused by the curvature of the surface of the product under test, the present invention establishes a local search region to provide a sufficiently large and reliable search range for subsequent accurate matching, thereby ensuring that the silkscreen features to be matched can be completely contained within the local search region, avoiding matching errors caused by slight deviations in the initial matching position.

[0110] Step 140: Determine the precise matching result of the local template in the image to be inspected within the local search area, so as to crop an image of the region to be inspected from the image to be inspected based on the precise matching result.

[0111] A precise matching result refers to a set of high-precision parameters describing the exact position and orientation of a local template obtained by further searching and locating the local template within a local search area using a matching algorithm (such as a shape-feature-based matching algorithm). For example, a precise matching result could be a set of parameters containing precise center coordinates (x2, y2), precise rotation angles (e.g., 1.3°), and precise scaling ratios (e.g., 1.02). This set of parameters accurately describes the actual pose of a silkscreened area on the image to be inspected relative to a template image of a standard product.

[0112] The image of the region to be inspected refers to the image cropped from the local search region after geometric correction (such as translation, rotation, and scaling) based on the precise matching result. This image is strictly aligned in size and orientation with its corresponding local template. For example, based on the precise matching result, an inverse geometric transformation can be applied to the local search region to reverse the silkscreen area located at coordinates (x2, y2), rotated by 1.3°, and scaled by a factor of 1.02 to an orientation that is completely aligned with the local template. Then, an image with the exact same size as the local template is cropped from the geometrically transformed local search region. This image is the image of the region to be inspected.

[0113] Step 150: Compare the image of the area to be inspected with the local template to generate the screen printing defect detection results of the product to be inspected.

[0114] The screen printing defect detection result refers to the final conclusion on whether the screen printing quality of the product under inspection is qualified, which is based on the difference analysis between the image of the area to be inspected and the local template.

[0115] The comparison process between the image of the area to be inspected and the local template can be implemented using various image comparison algorithms. For example, it can be achieved by calculating the grayscale difference between corresponding pixels in the image of the area to be inspected and the local template, or by evaluating the similarity of the two images in terms of brightness, contrast, and structure, or by comparing the depth features extracted from the two images. Based on the degree of difference or difference features generated by the comparison, and compared with one or more preset judgment thresholds, the final screen printing defect detection result can be a judgment signal indicating "qualified" or "unqualified".

[0116] The screen printing defect detection method provided by this invention first uses a global template for global matching, and then, based on the global matching result and the relative positional relationship of local templates with respect to the global template, precisely aligns each local template in a local area of ​​the image to be inspected, thereby reducing false alarms caused by misalignment and improving the efficiency and accuracy of screen printing defect detection.

[0117] In another embodiment of the present invention, there are multiple local templates. The method of comparing the image of the area to be inspected with the local templates to generate the screen printing defect detection result of the product to be inspected includes: cropping the image of the area to be inspected corresponding to each local template from the image to be inspected; comparing each local template with its corresponding image of the area to be inspected to generate a comparison result; and generating the screen printing defect detection result of the product to be inspected based on all comparison results.

[0118] Specifically, the detection result of screen printing defects in the product under inspection can be determined by comparing each local template with its corresponding image of the area to be inspected. For example, the final inspection result of the product under inspection is judged as "qualified" only if the comparison result of all local templates with their corresponding images of the area to be inspected is "qualified". Conversely, if the comparison result of any one or more local templates with their corresponding images of the area to be inspected is "unqualified", the final inspection result of the product under inspection is judged as "unqualified".

[0119] The screen printing defect detection method provided by this invention can achieve comprehensive and complete detection of all screen printing areas of a product by independently locating and comparing multiple screen printing areas on a product and comprehensively judging all independent comparison results. This ensures the completeness of the detection of products with complex screen printing layouts and effectively avoids the situation where defects in other areas are missed due to focusing only on some areas.

[0120] Figure 2 This is a flowchart illustrating the process of determining global and local templates provided by the present invention, as shown below. Figure 2 As shown, as another optional embodiment provided by the present invention, the global template and the local template are determined based on the following steps:

[0121] Step 210: Determine the silkscreen area in the template image that contains silkscreen traces.

[0122] Considering that silkscreen traces on template images typically include both text and patterns, this invention employs a general text detection algorithm to process the template image for the text portion of the silkscreen (e.g., containing multiple languages ​​such as Chinese and English). This general text detection algorithm can automatically identify and locate the position of all text elements in the image, thereby determining the silkscreen area containing text.

[0123] For graphic elements in silkscreen printing (such as function key icons, brand logos, warning signs, etc.), a pre-trained rotating object detection network can be used for recognition. This rotating object detection network can accurately locate the position and orientation of the graphic elements, thereby determining the silkscreen area containing the graphic.

[0124] This invention, by combining the output results of a general text detection algorithm and a rotated target detection network, can automatically identify and select all independent silkscreen regions containing silkscreen traces on a template image, providing a foundation for the selection and creation of subsequent global and local templates.

[0125] Step 220: Select multiple target regions from all silkscreen regions based on the pixel distance between the first center coordinates of each silkscreen region and the second center coordinates of the template image.

[0126] Considering that the region located at the center of the image is least affected by lens distortion and has more stable features, it is more suitable as a benchmark for global localization. Therefore, this invention traverses each silkscreen region, calculates the pixel distance between its first center coordinate and the second center coordinate of the template image, then sorts all silkscreen regions in ascending order according to the calculated pixel distance, and selects a preset number of regions with the smallest pixel distance from the sorting results as the target region.

[0127] In a preferred embodiment, six silkscreened regions with the smallest pixel distance can be selected to form a target region set for the next step of screening.

[0128] Step 230: Set the longest edge in all target regions as the global template.

[0129] Considering that feature-rich templates can significantly improve the accuracy and robustness of subsequent global matching, and that edge length is an important indicator of image feature richness, this invention first traverses all target regions and performs bilateral filtering on the image of each target region. This bilateral filtering process can effectively remove image noise while preserving edge details, thus providing high-quality input for subsequent edge extraction.

[0130] Furthermore, this invention employs an edge extraction algorithm (such as the Canny edge detection algorithm) to extract all edge contours within each target region. Subsequently, by counting or accumulating the number of pixels on the edge contours, the total edge length of each target region can be calculated. Finally, the calculated edge lengths of all target regions are compared, and the target region with the longest edge length is determined as the global template.

[0131] Step 240: Set each remaining silkscreen area, excluding the global template, as a local template.

[0132] Specifically, after determining the global template, the global template can be excluded from the target area set, and then all the remaining silkscreen areas in the target area set, that is, each independent text or pattern area other than the global template (e.g., the text of the "Start" button, the pattern of the "Power" gear, etc.), are set as a local template.

[0133] Step 250: Extract the shape matching features of the global template and each local template, and extract the relative positional relationship between the global template and each local template.

[0134] Shape matching features refer to geometric contours, edge directions, gradient information, and spatial topological relationships extracted from global and local templates to describe the content of global and local template images. For example, for text regions, since they are mainly composed of strokes, their shape matching features could be obtained by extracting the contours or skeletons of the text strokes, decomposing these strokes into a series of point sets, and recording the gradient direction information of each point as well as the relative positions and topological relationships between the strokes. For pattern regions, their shape matching features could include the outer contour of the pattern region and internal holes, independent geometric shapes, etc.

[0135] The screen printing defect detection method provided by this invention can automatically identify and select all screen printing areas on the template image, and automatically assign global and local templates based on preset rules such as position and edge features. This enables the automated creation and modeling of detection templates, eliminating the need for tedious manual selection and template selection. This greatly simplifies the pre-detection preparation work for different product models, significantly shortens the modeling time, and reduces the dependence on operator experience.

[0136] Figure 3 This is a schematic diagram of the process provided by the present invention for global matching of the image of the product to be inspected using a global template, as shown in the figure. Figure 3 As shown, in another optional embodiment provided by the present invention, the global matching result includes the matching position, matching rotation angle, and matching scaling ratio of the global template in the image to be inspected. The global template is used to perform global matching on the image of the product to be inspected to obtain the global matching result, including:

[0137] Step 310: Using the shape matching features of the global template, search for and obtain at least one set of candidate matching parameters in the image to be inspected. Each set of candidate matching parameters includes a matching position, a matching rotation angle, a matching scaling ratio, and a corresponding matching degree.

[0138] Specifically, this invention utilizes the shape matching features of a global template to search across the entire area of ​​the image to be inspected. This search process considers not only position but also angles and scaling ranges to account for slight rotations or minor height changes in the product on the conveyor belt or distance from the camera that may occur. During the search, a matching algorithm calculates the similarity between regions in the image to be inspected and the shape matching features of the global template under different positions, angles, and scaling conditions, and provides a quantified score, which is the matching degree.

[0139] Each set of candidate matching parameters includes the center coordinates, rotation angle, scaling factor, and matching score of the candidate region. For example, the following three sets of candidate matching parameters may be obtained: [position (x1, y1), angle 1.2°, scaling factor 1.01, matching score 0.95], [position (x2, y2), angle 1.1°, scaling factor 1.03, matching score 0.88], and [position (x3, y3), angle 5.0°, scaling factor 0.80, matching score 0.75].

[0140] Step 320: Determine a set of optimal matching parameters from at least one set of candidate matching parameters based on the matching degree of each set of candidate matching parameters.

[0141] For example, the matching degrees of the three candidate matching parameters in step 310 are 0.95, 0.88, and 0.75, respectively. By comparing these three values, 0.95 can be determined as the highest score. Therefore, the complete set of parameters corresponding to this highest score of 0.95, namely [position (x1, y1), angle 1.2°, scaling 1.01, matching degree 0.95], is determined as the best matching parameters.

[0142] Step 330: The matching position, matching rotation angle, and matching scaling ratio included in the best matching parameters are determined as the global matching result.

[0143] For example, for the optimal matching parameters in step 320 [position (x1, y1), angle 1.2°, scaling 1.01, matching degree 0.95], the set consisting of the matching position (x1, y1), the matching rotation angle 1.2°, and the matching scaling ratio 1.01 [position (x1, y1), angle 1.2°, scaling 1.01] is determined as the global matching result.

[0144] The screen printing defect detection method provided by this invention searches and obtains multiple candidate matching parameters containing matching degree scores during global matching, and selects the one with the highest matching degree as the best matching parameter. This ensures that when there are multiple regions similar to the template in the image to be inspected, the most reliable and accurate matching result can always be selected, thereby significantly improving the robustness and reliability of global positioning and effectively avoiding subsequent detection failures or deviations caused by incorrect positioning.

[0145] In another embodiment of the present invention, determining the local search region corresponding to the initial matching position in the image to be inspected includes: taking the center of the initial matching position as a reference, expanding the initial matching position according to a preset scaling ratio to obtain the local search region.

[0146] The local search region can be determined by using the center coordinates of the initial matching position as a reference and scaling the initial matching position according to a preset scaling ratio. For example, the preset scaling ratio can be set to 1.2, that is, expanding the area of ​​the initial matching position along its center coordinates to 1.2 times its original size. The expanded area obtained in this way is the local search region used for subsequent precise matching.

[0147] The screen printing defect detection method provided by this invention determines the local search area by expanding it according to a preset ratio with the initial matching position as the center. This provides an appropriate redundancy range during subsequent precise matching, thereby effectively dealing with the offset between the actual local position and the theoretical initial position caused by factors such as slight deviations in product placement or surface deformation. This significantly improves the success rate and fault tolerance of local precise matching, ensuring that the target can be accurately found even in the presence of positioning disturbances.

[0148] Figure 4 This is a flowchart illustrating the process of generating screen printing defect detection results for a product based on an image of the area to be inspected and a local template, as provided by the present invention. Figure 4 As shown, as another optional embodiment provided by the present invention, comparing the image of the area to be inspected with a local template to generate the screen printing defect detection result of the product to be inspected includes:

[0149] Step 410: Compare the structural similarity of the image of the region to be inspected with the local template to generate a difference score map.

[0150] Structural similarity comparison can be achieved using the Structural Similarity (SSIM) algorithm. Unlike traditional image comparison algorithms that only compare pixel grayscale differences, SSIM comprehensively evaluates and compares the similarity of two images across three dimensions: brightness, contrast, and structure, to arrive at a more holistic similarity assessment that aligns with human visual perception. It's important to note that while SSIM is insensitive to changes in image lighting, it is highly sensitive to local structural changes such as ink leakage, misalignment, and dirt in screen printing.

[0151] The difference score map refers to a grayscale image of the same size as the image of the region to be inspected. The grayscale value of each pixel in the grayscale image represents the degree of structural difference between the image of the region to be inspected and the local template at the corresponding pixel position.

[0152] For example, the pixel value range of the difference score map can be from 0 to 255. In areas of the image to be inspected that are completely identical to the local template, the structural similarity is extremely high, so the corresponding pixel value in the difference score map will be very low (e.g., close to 0, appearing as black). Conversely, in areas of the image to be inspected that have defects (such as ink leakage or dirt), the structure is significantly different from the local template, and the structural similarity is low. Therefore, the corresponding pixel value in the difference score map will be very high (e.g., close to 255, appearing as a bright white dot or white area). In this way, the difference score map can visually highlight potential defective areas from the image to be inspected.

[0153] Step 420: Binarize the difference score map to obtain the difference heatmap.

[0154] Binarization refers to the process of converting a difference score map into a binary image containing only two pixel values ​​(e.g., black and white). This process can be achieved by setting a preset binarization threshold: for each pixel in the difference score map, if its grayscale value is greater than or equal to the binarization threshold, it is set to a pixel value representing a potential defect in the binary image (e.g., 255, represented as white); otherwise, it is set to a background pixel value (e.g., 0, represented as black).

[0155] Furthermore, to ensure that all possible minute defects can be filtered out, a low binarization threshold can be set. In one specific embodiment, this binarization threshold can be set to 20.

[0156] A difference heatmap is a binary image obtained by binarizing a difference score map. In a difference heatmap, potential defect areas are highlighted. For example, if there is a dirt spot on the image of the area to be inspected, the dirt spot will form a highlighted area on the difference score map. After binarization, the location of the dirt spot will appear as a clear white connected region on the final difference heatmap, while the rest of the image will be a black background, thus providing a basis for subsequent defect analysis and localization.

[0157] Step 430: Perform connected component analysis on the differential heatmap to obtain the detection results of screen printing defects in the product to be tested.

[0158] Connected Large Object (Blob) analysis refers to the process of finding and identifying all independent regions (i.e., connected regions) composed of interconnected pixels with the same pixel value in a difference heatmap. In this process, all white areas in the difference heatmap can be automatically identified, and the geometric parameters of each independent white area can be calculated, one key parameter being its area.

[0159] For example, by calculating the area of ​​each identified white connected region and comparing it to a preset area threshold, it can be determined whether the connected region corresponds to a real defect. If the area of ​​a connected region is greater than the area threshold, it is determined to be a real defect area; conversely, if its area is less than or equal to the area threshold, it can be regarded as a minor interference caused by non-defect factors such as image noise and ignored. The final screen printing defect detection result is generated based on this judgment: if there is at least one connected region determined to be a defect in the difference heatmap, the final detection result is "unqualified"; if the area of ​​all connected regions does not exceed the area threshold, the final detection result is "qualified".

[0160] The screen printing defect detection method provided by this invention identifies defects by using structural similarity comparison combined with binarization and connected component analysis. It can efficiently identify local structural changes caused by ink leakage, offset, dirt, etc. by using image processing algorithms with fast computing speed and insensitivity to overall illumination changes. Thus, without sacrificing detection speed, it achieves rapid and reliable preliminary screening of common screen printing defects.

[0161] Figure 5 This is a schematic diagram of the process provided by the present invention for comparing the structural similarity between an image of the region to be inspected and a local template, as shown in the figure. Figure 5 As shown, as another optional embodiment provided by the present invention, the image of the region to be inspected is compared with a local template for structural similarity to generate a difference score map, including:

[0162] Step 510: Using a preset sliding window, synchronously traverse the image of the region to be inspected and the local template.

[0163] A sliding window refers to a rectangular area of ​​a preset size; for example, a sliding window could be an 11x11 pixel window.

[0164] Synchronous traversal refers to the sliding window moving simultaneously over the region image and the local template with the same step size (e.g., moving pixel by pixel) and the same order (e.g., from left to right, from top to bottom). At any point during the traversal, when the sliding window covers a position in the region image, it also precisely covers the corresponding position in the local template. Therefore, at each position in the region image and the local template, the sliding window extracts a pair of image patches: one from the region image and the other from the local template.

[0165] Step 520: At each position during the sliding window traversal process, determine the brightness similarity measure, structural similarity measure, and contrast measure between the first image region and the second image region; the first image region is the image on the area to be inspected covered by the sliding window, and the second image region is the image on the local template covered by the sliding window.

[0166] Brightness similarity measure can assess the similarity of average illumination levels between a first image region and a second image region, and it measures whether the overall brightness of the image is consistent.

[0167] Contrast similarity measures the similarity of pixel intensity variation ranges within a first and second image region. It measures whether the range of variation from darkest to brightest in the two image regions is similar, that is, whether the "sharpness" of the two image regions is consistent.

[0168] Structural similarity measurement compares the similarity of structural information between a first image region and a second image region after removing the effects of brightness and contrast. For example, structural information may include the outline, texture, and other information of the silkscreen printing in the first and second image regions.

[0169] Step 530: Determine the local structural similarity score of the center pixel of each sliding window based on the brightness similarity measure, structural similarity measure, and contrast measure.

[0170] Specifically, the structural similarity score can be calculated as shown in formula (1):

[0171] (1)

[0172] in, This represents the first image region from the image of the region to be inspected. This represents the second image region derived from the local template. Representation of structural similarity algorithms, This represents the structural similarity score between the first image region and the second image region; This represents the average pixel value of the first image region. This represents the average pixel value of the second image region. and It can be used to calculate brightness similarity metrics; This represents the pixel variance of the first image region. This represents the pixel variance of the second image region. and It can be used to calculate contrast metrics; This represents the pixel covariance between the first and second image regions. It can be used to calculate structural similarity metrics; and These are two constants used to maintain computational stability, where , , It is the dynamic range of image pixel values. and It is a preset constant; in a preferred embodiment, , .

[0173] Formula (1) is used to measure the similarity between two images in three dimensions: brightness, contrast, and structure. Compared with traditional metrics such as mean square error or peak signal-to-noise ratio, structural similarity score can more effectively distinguish image distortion caused by structural deformation. By introducing the pixel mean, pixel standard deviation, and pixel covariance of the image for comprehensive measurement, structural similarity score significantly improves the robustness and accuracy of image similarity assessment while maintaining computational efficiency. Using structural similarity score as a measure of the difference between the image of the region to be inspected and the local template is beneficial to highlighting the local structural changes caused by screen printing defects and improving the reliability and accuracy of defect detection results.

[0174] Step 540: Collect the local structural similarity scores of all center pixels during the traversal process and obtain the difference score map.

[0175] After the sliding window completes the synchronous traversal of the entire image to be inspected and the local template, the structural similarity scores of all center pixels can be obtained. Then, a grayscale image with the same size as the image to be inspected can be created. Then, all the calculated structural similarity scores are traversed, and each structural similarity score is used as a pixel value to fill the center pixel position in the grayscale image corresponding to that structural similarity score. The grayscale image after filling is the difference score map.

[0176] The screen printing defect detection method provided by this invention utilizes a sliding window to perform localized analysis of the image and decomposes the similarity assessment of each locality into three independent dimensions: brightness, contrast, and structure. This allows for a more precise distinction between image changes caused by global, non-defect factors such as illumination and contrast, and real defects caused by changes in the shape of the screen printing itself. This provides a higher quality and more reliable difference score map for subsequent defect determination.

[0177] In another embodiment provided by the present invention, the brightness similarity measure is determined based on the average pixel value of the first image region and the second image region, the structural similarity measure is determined based on the pixel covariance between the first image region and the second image region, and the contrast measure is determined based on the pixel variance of the first image region and the second image region.

[0178] Since the average pixel value can directly reflect the overall brightness level of an image region, the brightness similarity measure is determined by comparing the closeness of the average pixel value of the first image region and the average pixel value of the second image region.

[0179] Since variance measures the dispersion of pixel values ​​relative to their average value and can directly reflect the contrast of an image region, the contrast similarity measure is determined by comparing the pixel variance of the first image region with that of the second image region.

[0180] Considering that covariance measures the overall trend of pixel values ​​in two image regions and can effectively represent the intrinsic structural information of an image, structural similarity measurement is determined based on the pixel covariance between the first and second image regions.

[0181] The screen printing defect detection method provided by this invention binds the three abstract similarity measures of brightness, contrast and structure to the directly calculable and physically meaningful pixel average value, pixel variance and pixel covariance, respectively. This provides a specific and reproducible mathematical implementation path for structural similarity comparison, thereby ensuring the objectivity and stability of the comparison results and eliminating the implementation differences that may be caused by the ambiguity of the algorithm definition.

[0182] In another embodiment of the present invention, connected component analysis is performed on the differential heat map to obtain the screen printing defect detection result of the product to be inspected, including: identifying one or more connected components in the differential heat map; calculating the area of ​​each connected component; comparing the area of ​​each connected component with a first preset area threshold; identifying connected components with an area greater than the first preset area threshold as potential defect areas; and analyzing the defect features of each potential defect area to generate the screen printing defect detection result.

[0183] Specifically, all connected regions in the differential heatmap are identified, their areas are calculated, and then the area of ​​each connected region is compared with a first preset area threshold.

[0184] For example, if the area of ​​a connected component is greater than the first preset area threshold, it will be marked as a potential defect region that requires further confirmation. If the area of ​​a connected component is less than or equal to the first preset area threshold, it will be ignored.

[0185] Furthermore, after comparing all connected components with a first preset area threshold, if the area of ​​all connected components does not exceed the first preset area threshold, meaning there are no potential defect areas, then the image of the area to be inspected can be determined to be qualified. If one or more potential defect areas are identified, then these potential defect areas and their corresponding areas on the local template are subjected to secondary comparison and feature analysis to ultimately confirm whether they are real defects.

[0186] The screen printing defect detection method provided by this invention introduces a two-stage defect judgment strategy. First, it uses a traditional visual algorithm with fast computing speed to initially screen out all potential defect areas. Then, it performs further precise feature analysis only on these potential defect areas. This can minimize the application scope of the computationally intensive precision analysis algorithm, thereby greatly improving the overall efficiency of the detection process while ensuring a high detection rate and a low false alarm rate, and achieving an effective balance between detection speed and detection accuracy.

[0187] In another embodiment of the present invention, defect feature analysis of each potential defect region is performed to generate screen printing defect detection results, including: extracting deep learning features of each potential defect region and the corresponding region of each potential defect region in the local template; comparing the deep learning features of the potential defect region and the deep learning features of the corresponding region to generate an anomaly score map of each potential defect region; and determining the screen printing defect detection results based on all anomaly score maps.

[0188] Deep learning feature extraction refers to inputting potential defect regions on the image to be inspected and corresponding regions in the local template into a pre-trained deep learning network model. This deep learning network model can learn and extract deep-level feature information from the image, which is usually represented as a set of high-dimensional feature vectors.

[0189] Furthermore, the two extracted sets of deep learning features can be compared to quantify the degree of difference between them. This comparison can be achieved by calculating the feature distance (e.g., Euclidean distance) between feature vectors; a larger feature distance indicates a significant difference in deep semantics between the region to be inspected and the local template. Based on this comparison result, an anomaly score map can be generated. The brightness value of each pixel in this anomaly score map represents the degree of difference between the potential defective region in the image to be inspected and the corresponding region in the local template. The greater the difference, the higher the score and the higher the brightness.

[0190] Next, threshold segmentation can be performed on the anomaly score map to filter out normal areas with low anomaly scores, retaining only the true anomaly areas with scores significantly higher than the threshold. The presence of a real defect in the image under inspection can be ultimately determined by analyzing the area parameters of these true anomaly areas. For example, if a real defect is confirmed in any potential defect area, the inspection result of the product is "unqualified"; only when no real defects are found in any potential defect areas is the inspection result of the product "qualified".

[0191] The screen printing defect detection method provided by this invention uses deep learning features for final defect confirmation. It can utilize the powerful feature representation capabilities of deep neural networks to capture pseudo-differences caused by non-defect factors such as minor textures or uneven reflection on the product surface, which are difficult to distinguish by traditional visual algorithms. This allows for more accurate identification of potential defects at a deep semantic level, significantly improving the accuracy and reliability of screen printing defect detection.

[0192] In another embodiment of the present invention, the deep learning features are extracted based on the visual converter network model; the deep learning features of each potential defect region and the corresponding region of each potential defect region in the local template are extracted respectively, including: inputting the potential defect region and the corresponding region into the visual converter network model respectively, and obtaining the deep learning features output by the visual converter network model.

[0193] The Vision Transformer (ViT) network model originates from the Transformer architecture in the field of natural language processing. It can accurately capture global dependencies and long-distance contextual information in images.

[0194] Specifically, during feature extraction, the ViT network model first segments the input image region (potential defect regions or corresponding regions on local templates) into a series of non-overlapping, fixed-size image patches, for example, each patch can be 14x14 pixels in size. Then, the ViT network model treats these image patches as sequential data, processes all image patches in parallel through a self-attention mechanism, and calculates the weights of their interrelationships. By stacking multiple Transformer modules, the ViT network model can progressively build increasingly abstract and robust feature representations of the image content. Finally, the ViT network model outputs a set of deep feature vectors (tokens) representing each image patch of the input image; these deep feature vectors are the deep learning features.

[0195] In one specific embodiment, a pre-trained ViT-L-14-336 network model can be used as the visual converter network model.

[0196] The screen printing defect detection method provided by this invention uses a visual converter network model as a feature extractor. It can leverage its powerful global context awareness to more effectively learn and understand the long-distance dependencies between various parts within an image region, thereby extracting deep features with a more global perspective and structural integrity than traditional convolutional networks. This is particularly effective for accurately identifying defects caused by large-area gradients, minor scratches, or overall offsets, further improving the accuracy of feature matching.

[0197] Figure 6 This is a schematic diagram of the feature extraction process based on the visual converter network model provided by the present invention, as shown below. Figure 6 As shown, as another optional embodiment provided by the present invention, the visual converter network model specifically performs the following operations:

[0198] Step 610: After dividing the input into multiple image blocks, extract a set of multi-level local features from multiple preset levels for each image block.

[0199] It should be noted that, in order to make the final extracted local features richer and more discriminative, this invention does not only use the final output layer of the ViT network model, but also extracts features from multiple layers of different depths in its backbone network. In deep neural networks, features at different levels typically correspond to different levels of semantic information: shallow networks focus more on low-level details such as color and edges, while deep networks can capture more abstract high-level semantics such as shape. By extracting features from multiple preset layers, it can be ensured that the final feature representation simultaneously contains multi-layered image information, from concrete to abstract.

[0200] For example, in the ViT network model, multiple preset layers can be selected, including four layers of features: [5, 11, 17, 23]. When an input image is divided into... M After generating image patches, for any one of these four preset levels, the ViT network model will generate this... M Image block output M There are corresponding feature vectors. Therefore, for each image patch, a set of feature vectors containing information at four different levels can be obtained, which constitutes a multi-level local feature.

[0201] Step 620: At each preset level, perform adaptive pooling processing on the extracted local features at multiple different scales.

[0202] Specifically, the output of a certain layer of the ViT network model will be taken first. M A length of C The feature vectors are transformed to obtain a tensor feature similar to the feature map of a convolutional network, with the shape of... Next, an adaptive average pooling operation is performed on this tensor feature. To achieve processing at various scales, multiple pooling kernels of different sizes can be used.

[0203] For example, three different pooling kernel sizes—1×1, 3×3, and 5×5—can be used to adaptively average pool the tensor features at each level. The pooled tensor features are then transformed back to their original form. Size. By deforming and pooling the feature vector in this way, three different aggregation degrees of feature representation can be obtained for the original feature vector extracted from a single layer of the ViT network model, thus greatly enriching the diversity of features.

[0204] Step 630: The local features processed by pooling at different scales are fused to form a set of multi-dimensional local features that contain both multi-level and multi-aggregation information, which are used as the deep learning features of each image patch.

[0205] For each local image patch of the input image, a set of feature vectors containing the number of levels (L) × aggregation degree (r) can be obtained. For example, for a certain image patch, all feature vectors obtained from processing it in 4 different ways (3 aggregation degrees per level, totaling 4 × 3 = 12) can be collected. This set of feature vectors is called the multidimensional local feature, which contains rich multi-level and multi-aggregation degree information, and can describe the content of the image patch in detail from multiple dimensions and multiple granularities. This multidimensional local feature is determined as the deep learning feature of the image patch.

[0206] The screen printing defect detection method provided by this invention can construct a multi-dimensional local feature with extremely rich information dimensions for each image block by fusing features from multiple layers of different depths from the visual converter network model and features processed by pooling at various scales. This multi-dimensional local feature contains comprehensive information from low-level details to high-level semantics, and from fine granularity to coarse generalization, thereby greatly enhancing the feature's representation ability and discriminative power, and providing high-quality feature input for subsequent high-precision anomaly comparison.

[0207] Figure 7 This is a schematic diagram of the process provided by the present invention for generating anomaly score maps of potential defect regions based on deep learning features of potential defect regions and corresponding regions, as shown in the figure. Figure 7 As shown, as another optional embodiment provided by the present invention, comparing the deep learning features of the potential defect region with the deep learning features of the corresponding region to generate an anomaly score map for each potential defect region includes:

[0208] Step 710: Traverse each image block in the potential defect region and calculate the feature distance between the deep learning features of any image block and the deep learning features of all image blocks in the corresponding region.

[0209] Feature distance can be the Euclidean distance between the deep learning features of two image patches. The larger the Euclidean distance, the greater the difference between the two image patches in the feature space of the deep learning network, that is, the less similar the two image patches are in terms of deep semantics such as content, texture, and structure.

[0210] Step 720: The minimum feature distance is determined as the anomaly score of any image block.

[0211] Considering that the minimum feature distance represents the region corresponding to the smallest difference that can be found in the local template for an image patch, if the difference score between the image patch and the region corresponding to the smallest difference that can be found in the local template is very large, it indicates that the image patch is likely an abnormal region with a silkscreen defect. Therefore, this invention determines the minimum feature distance as the abnormality score of any image patch.

[0212] Step 730: Combine the anomaly scores of all image blocks to form an anomaly score map.

[0213] After traversing all image blocks in the potential defect region and calculating an anomaly score for each image block, an anomaly score map of the same size as the potential defect region can be created. Then, according to the spatial arrangement of the image blocks in the potential defect region, the anomaly score of each image block is used as a pixel value to fill the corresponding image block position in the anomaly score map.

[0214] For example, if a potential defect area is divided into 256 image blocks of 16×16, then the anomaly scores corresponding to the 256 image blocks can be calculated. Then, an anomaly score map with the same size as the potential defect area can be created, and the 256 anomaly scores can be filled into this anomaly score map according to their spatial arrangement, with the anomaly score of each image block as the pixel value.

[0215] The screen printing defect detection method provided by this invention can effectively overcome the problem of local micro-displacement caused by factors such as product positioning and surface deformation by finding the minimum feature distance for each image block in the area to be inspected among all image blocks in the standard template and using this as its anomaly score. This greatly enhances the robustness of the deep learning comparison algorithm to pose perturbations and further reduces the false alarm rate.

[0216] Figure 8 This is a flowchart illustrating the process of determining the anomaly score of any image block provided by the present invention, as shown below. Figure 8 As shown, as another optional embodiment provided by the present invention, when at least two reference templates are provided for the potential defect region, and each reference template contains a corresponding region of the potential defect region, the minimum feature distance is determined as the anomaly score of any image block, including:

[0217] Step 810: Calculate the feature distance between the deep learning features of any image patch and the deep learning features of all image patches in each corresponding region to obtain multiple minimum feature distances.

[0218] The feature distance can be the Euclidean distance between the deep learning features of any image patch and the deep learning features of all image patches in each corresponding region. The minimum feature distance can be calculated as shown in formula (2):

[0219] (2)

[0220] in, Indicates the first The nth image to be detected The image patch in the first Layer and first At the degree of aggregation, compared with the first degree of aggregation... The minimum feature distance obtained by comparing all image patches in a reference template. Indicates the index of the image to be detected. Indicates the first A reference template, This represents the preset feature level in the visual converter network model. This indicates the aggregation degree used in adaptive pooling. Indicates the first element in the image to be detected. Image blocks; Indicates the first The nth image to be detected The image patch in the first Layer and first Deep learning feature vectors extracted under each degree of aggregation; Indicates the first The first reference template The image patch in the first Layer and first Deep learning feature vectors extracted under each degree of aggregation; This represents the Euclidean distance between two deep learning feature vectors.

[0221] Step 820: Select several minimum feature distances with the smallest values ​​from multiple minimum feature distances, based on a preset ratio, to calculate the average feature distance.

[0222] For example, there can be 10 minimum feature distances, and the preset ratio can be 20%. Then, the two smallest feature distance values ​​are selected from the 10 minimum feature distance values ​​to calculate the average feature distance.

[0223] Specifically, the average feature distance can be calculated as shown in formula (3):

[0224] (3)

[0225] in, Indicates the first The nth image to be detected The image patch in the first Layer and first The feature distance is obtained by comparing multiple reference templates and calculating the average at a certain aggregation degree. Indicates the index of the image to be detected. This represents the preset feature level in the visual converter network model. This indicates the aggregation degree used in adaptive pooling. Indicates the first element in the image to be detected. Image blocks; This indicates the number of reference templates with the minimum feature distance selected according to a preset ratio; Indicates the first A reference template image.

[0226] Step 830: The average feature distance is determined as the anomaly score for any image patch.

[0227] Specifically, the calculation method for the outlier score can be shown in formula (4):

[0228] (4)

[0229] in, Indicates the first The nth image to be detected Anomaly scores for each image patch, Indicates the index of the image to be detected. Indicates the first element in the image to be detected. Image blocks; This indicates the total number of feature levels selected; This indicates the three different aggregation degrees of the adaptive pooling used; Indicates the first The nth image to be detected The image patch in the first Layer and first The average feature distance at each degree of aggregation This represents the preset feature level in the visual converter network model. This indicates the degree of aggregation used in adaptive pooling.

[0230] Formula (4) is used to calculate the final anomaly score for each image patch. Its core idea is to perform a weighted average of the feature distances at different levels and with different aggregation degrees, thereby comprehensively reflecting the degree of anomaly of the image patch in the multi-scale and multi-level feature space. Specifically, the formula first obtains the feature distances of the image patch at three different scales (e.g., 1×1, 3×3, and 5×5 pooling kernels) at multiple preset feature levels extracted by the visual converter network model, and then calculates the average value of all combinations as the final anomaly score of the image patch. By introducing multi-level feature fusion, the anomaly score not only includes low-level detailed texture information, but also incorporates high-level abstract semantic features, which helps to enhance the recognition ability of complex screen printing defects (such as deformation, occlusion, blurring, etc.); thereby improving the robustness of the screen printing defect detection results.

[0231] The screen printing defect detection method provided by this invention introduces multiple reference templates and only averages the comparison results with a few most similar templates when calculating the anomaly score. This greatly improves the tolerance of the detection algorithm to process fluctuations within the allowable range of the standard product itself. As long as a certain shape on the product to be inspected is similar to a portion of the standard products, it can be considered normal. This effectively avoids false alarms caused by the randomness or insufficient representativeness of a single standard template, and significantly enhances the stability and adaptability in actual industrial production environments.

[0232] In another embodiment of the present invention, determining the screen printing defect detection result based on all abnormal score maps includes: performing threshold segmentation on each abnormal score map to generate a defect binary map; performing connected component analysis on all defect binary maps, and if it is determined that there is any connected component with an area greater than a second preset area threshold, then the screen printing defect detection result is determined to be that a screen printing defect exists.

[0233] Threshold segmentation refers to the practice in anomaly score maps where regions with scores above an anomaly score threshold are considered abnormal and are marked as foreground (e.g., white) on the defect binary map; regions with scores below or equal to the anomaly score threshold are considered normal and marked as background (e.g., black). In one specific embodiment, the anomaly score threshold can be set to 0.52.

[0234] Furthermore, connected component analysis can be performed on all generated defect binary graphs to identify all connected components in the defect binary graphs. Then, the area of ​​each connected component is calculated and compared with a second preset area threshold.

[0235] For example, if it is determined that any connected component exists whose calculated area is greater than the second preset area threshold, then the product under inspection is determined to have a screen printing defect, and the final screen printing defect detection result is "unqualified". Conversely, if no connected component with an area greater than the second preset area threshold is found in the defect binary graph, then the product under inspection is determined to have no screen printing defect, and the final screen printing defect detection result is "qualified".

[0236] The screen printing defect detection method provided by this invention performs threshold segmentation on the continuous anomaly score map output by the deep learning model, and makes a judgment based on whether the area of ​​the final connected region exceeds a preset threshold. This method can transform abstract, feature space-based differences into measurable defect regions with physical dimensions, thereby effectively filtering out small anomalies caused by random noise that have no practical significance, and ensuring the stability and reliability of the final detection results.

[0237] Figure 9 This is a flowchart illustrating the automatic modeling process provided by the present invention, such as... Figure 9 As shown, the process starts from the beginning node. First, the input template image is processed using a text localization algorithm and a rotation target detection algorithm. Through the combined effect of these two algorithms, the regions to be detected are automatically identified and bounded. These automatically bounded regions are the detection regions, avoiding the tedious work of manually defining regions in the traditional way and improving modeling efficiency.

[0238] Furthermore, after the automatic selection is completed, based on the aforementioned positioning results, the global template and each local template in the entire template are further determined. The global template refers to a representative region with rich edges selected from multiple detection regions, used for global positioning and coarse matching; the remaining detection regions are defined as local templates, which are used for fine matching of the corresponding regions.

[0239] Next, after determining the global and local templates, feature extraction is performed on both the global and local templates. Specifically, this involves using shape matching algorithms to extract shape-matching features corresponding to each template, while simultaneously recording the relative spatial relationships between the global and local templates. These shape-matching features and relative spatial relationships are stored on local storage media as crucial reference data in subsequent defect detection and comparison processes. Finally, the automated modeling process is completed and enters the final stage. The entire automated modeling process efficiently and automatically achieves accurate bounding selection of the area to be detected and the establishment of template features, providing a solid foundation for subsequent defect detection.

[0240] Figure 10 This is a schematic diagram of the defect detection process provided by the present invention, as shown below. Figure 10 As shown, the detection process starts from the beginning node. First, it reads the pre-established global template and local template and their corresponding shape matching features, and simultaneously reads the image of the product to be inspected. These shape matching features serve as key data for the templates, providing technical support for subsequent matching and localization.

[0241] Next, a global template is used to perform initial global matching and localization of the image under inspection, obtaining a rough overall pose information of the product. Based on this, and using the global localization result and the relative positional relationship of the local templates with respect to the global template as a reference, precise matching is performed on each local template in the image under inspection to achieve detailed localization, thereby obtaining accurate alignment of each local region. After completing the precise matching, image correction is performed on the image under inspection, including translation, rotation, and scaling operations, to ensure that each local position accurately coincides with the corresponding template.

[0242] The corrected image regions then proceed to the next step of processing. Similarity is calculated using the SSIM algorithm, and blob analysis is combined to locate and filter preliminary defect areas. Based on this, it is determined whether a silkscreen defect exists: if no defect is found, the detection process ends directly; if a potential defect exists, the process will call a deep learning-based defect detection module to generate a similarity score map.

[0243] Subsequently, threshold segmentation is performed on the similarity score map to obtain a binary defect map. Connectivity analysis is then performed using parameters such as area, and the presence of a defect is ultimately determined based on the area of ​​the connected components. This entire detection process effectively combines the advantages of traditional visual algorithms with deep learning techniques, ensuring both speed and real-time performance while improving the accuracy and robustness of defect identification.

[0244] Figure 11 This is a schematic diagram illustrating the principle of the SSIM algorithm provided by this invention, as shown below. Figure 11 As shown, firstly, the brightness of images x and y is measured separately, and their average brightness values ​​are extracted. Then, a brightness comparison calculation (i.e., brightness similarity measurement) is performed to measure the difference in overall brightness between the two images. Subsequently, a contrast measurement is calculated for each image, that is, the variance of pixel gray levels is statistically analyzed and contrast is compared (i.e., contrast measurement), reflecting the texture and detail changes of the images. Finally, combining the brightness and contrast results, the structural features in the images are evaluated using a structural comparison module (i.e., structural similarity measurement), measuring the similarity between the two images in terms of shape, edges, and texture patterns.

[0245] The similarity measurement module weighted and fused the results of brightness similarity measurement, contrast measurement, and structural similarity measurement to generate a final similarity index. This index can more accurately and comprehensively reflect the true similarity between two images, and is particularly suitable for identifying small but structurally significant defect areas in screen printing. By applying this SSIM algorithm, this invention can achieve high-precision defect detection of screen printing on the surface of household appliances, effectively distinguishing between real defects and image differences caused by environment or lighting, thereby improving the accuracy and robustness of detection.

[0246] Figure 12 This is a schematic diagram of the screen printing defect detection device provided by the present invention, as shown below. Figure 12 As shown, it mainly includes, but is not limited to:

[0247] The global template matching unit 1210 is used to perform global matching on the image of the product to be inspected using a global template, and obtain the global matching result.

[0248] The local template localization unit 1220 is used to determine the initial matching position of the local template in the image to be inspected based on the global matching result and the relative positional relationship of a local template with respect to the global template.

[0249] The search region localization unit 1230 is used to determine the local search region corresponding to the initial matching position in the image to be inspected.

[0250] The detection image localization unit 1240 is used to determine the precise matching result of the local template in the image to be inspected within the local search area, so as to crop an image of the region to be inspected from the image to be inspected based on the precise matching result.

[0251] The defect detection processing unit 1250 is used to compare the image of the area to be inspected with the local template to generate the screen printing defect detection result of the product to be inspected.

[0252] The global template, the local template, and the relative positional relationship of the local template relative to the global template are determined based on the template image of the standard product.

[0253] It should be noted that the screen printing defect detection device provided by the present invention can execute the screen printing defect detection method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0254] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 13 As shown, the electronic device may include: a processor 1310, a communications interface 1320, a memory 1330, and a communications bus 1340, wherein the processor 1310, the communications interface 1320, and the memory 1330 communicate with each other through the communications bus 1340. The processor 1310 can call logic instructions in the memory 1330 to execute a screen printing defect detection method. This method includes: performing global matching on an image of a product to be inspected using a global template to obtain a global matching result; determining an initial matching position of the local template in the image to be inspected based on the global matching result and the relative position of a local template relative to the global template; determining a local search region corresponding to the initial matching position in the image to be inspected; determining a precise matching result of the local template in the image to be inspected within the local search region, and cropping an image of an area to be inspected from the image to be inspected based on the precise matching result; comparing the image of the area to be inspected with the local template to generate a screen printing defect detection result for the product to be inspected; the global template, the local template, and the relative position of the local template relative to the global template are determined based on a template image of a standard product.

[0255] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0256] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the screen printing defect detection method provided in the above embodiments, the method including: using a global template to perform global matching on an image of a product to be inspected, and obtaining a global matching result; determining an initial matching position of the local template in the image to be inspected based on the global matching result and the relative positional relationship of a local template relative to the global template; determining a local search region corresponding to the initial matching position in the image to be inspected; determining a precise matching result of the local template in the image to be inspected within the local search region, so as to crop an image of an area to be inspected from the image to be inspected according to the precise matching result; comparing the image of the area to be inspected with the local template to generate a screen printing defect detection result of the product to be inspected; the global template and the local template, and the relative positional relationship of the local template relative to the global template, are determined based on a template image of a standard product.

[0257] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the screen printing defect detection method provided in the above embodiments. The method includes: performing global matching on an image of a product to be inspected using a global template to obtain a global matching result; determining an initial matching position of the local template in the image to be inspected based on the global matching result and the relative positional relationship of a local template relative to the global template; determining a local search region corresponding to the initial matching position in the image to be inspected; determining a precise matching result of the local template in the image to be inspected within the local search region, so as to crop an image of an area to be inspected from the image to be inspected according to the precise matching result; comparing the image of the area to be inspected with the local template to generate a screen printing defect detection result for the product to be inspected; the global template, the local template, and the relative positional relationship of the local template relative to the global template are determined based on a template image of a standard product.

[0258] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0259] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting defects in screen printing, characterized in that, include: The global template is used to perform global matching on the image of the product to be inspected, and the global matching result is obtained. The global template is an image region with significant and stable features extracted from a template image of a standard product in advance; the global matching result includes the matching position, matching rotation angle, and matching scaling ratio of the global template in the image to be inspected. Based on the global matching result and the relative position of a local template with respect to the global template, the initial matching position of the local template in the image to be inspected is determined; The local template is an image block extracted from the template image of the standard product, representing other screen printing areas to be inspected, excluding the global template; the relative positional relationship is the spatial position and orientation relationship of the local template relative to the global template on the template image of the standard product; the initial matching position is the position of the local template initially calculated on the image to be inspected based on the global matching result and the relative positional relationship of the local template relative to the global template. The local search region corresponding to the initial matching position is determined in the image to be inspected; Within the local search area, determine the precise matching result of the local template in the image to be inspected, so as to crop an image of the region to be inspected from the image to be inspected based on the precise matching result; The process of comparing the image of the region to be inspected with the local template to generate a screen printing defect detection result for the product to be inspected includes: performing a structural similarity comparison between the image of the region to be inspected and the local template to generate a difference score map; performing binarization processing on the difference score map to obtain a difference heatmap; and performing connected component analysis on the difference heatmap to obtain the screen printing defect detection result for the product to be inspected. The relative positional relationship between the global template and the local template, and between the local template and the global template, is determined based on the template image of the standard product.

2. The screen printing defect detection method according to claim 1, characterized in that, The local templates are multiple, and the comparison of the image of the area to be inspected with the local templates to generate the screen printing defect detection result of the product to be inspected includes: The image of the region to be inspected corresponding to each of the local templates is cropped from the image to be inspected; Each local template is compared with its corresponding image of the region to be inspected to generate a comparison result; Based on all the comparison results, the screen printing defect detection results of the product to be tested are generated.

3. The screen printing defect detection method according to claim 1, characterized in that, The global template and the local template are determined based on the following steps: Identify the silkscreened area in the template image that contains silkscreen traces; Based on the pixel distance between the first center coordinates of each silkscreen region and the second center coordinates of the template image, multiple target regions are selected from all silkscreen regions. Set the longest edge in all the target regions as the global template; Each of the remaining silkscreen areas, excluding the global template, is set as a local template; Extract the shape matching features of the global template and each of the local templates, and extract the relative positional relationship between the global template and each of the local templates.

4. The screen printing defect detection method according to claim 3, characterized in that, The step of using a global template to perform global matching on the image of the product to be inspected and obtaining the global matching result includes: Using the shape matching features of the global template, at least one set of candidate matching parameters is searched and obtained in the image to be inspected. Each set of candidate matching parameters includes a matching position, a matching rotation angle, a matching scaling ratio, and a corresponding matching degree. Based on the matching degree of each set of candidate matching parameters, a set of optimal matching parameters is determined from the at least one set of candidate matching parameters; The matching position, the matching rotation angle, and the matching scaling ratio included in the optimal matching parameters are determined as the global matching result.

5. The screen printing defect detection method according to claim 1, characterized in that, Determining the local search region corresponding to the initial matching position in the image to be inspected includes: Using the center of the initial matching position as a reference, the initial matching position is expanded according to a preset scaling ratio to obtain the local search area.

6. The screen printing defect detection method according to claim 1, characterized in that, The step of comparing the image of the region to be inspected with the local template to generate a difference score map includes: Using a preset sliding window, the image of the region to be inspected and the local template are traversed synchronously. At each position during the sliding window traversal process, a brightness similarity measure, a structural similarity measure, and a contrast measure are determined between the first image region and the second image region; the first image region is the image on the region to be inspected covered by the sliding window, and the second image region is the image on the local template covered by the sliding window; Based on the brightness similarity metric, the structural similarity metric, and the contrast metric, the local structural similarity score of the center pixel of each sliding window is determined; The difference score map is obtained by combining the local structural similarity scores of all the center pixels during the traversal process.

7. The screen printing defect detection method according to claim 6, characterized in that, The brightness similarity measure is determined based on the average pixel value of the first image region and the second image region, the structural similarity measure is determined based on the pixel covariance between the first image region and the second image region, and the contrast measure is determined based on the pixel variance of the first image region and the second image region.

8. The screen printing defect detection method according to claim 1, characterized in that, The connection component analysis of the differential heatmap is performed to obtain the detection results of the screen printing defects of the product to be tested, including: One or more connected domains are identified in the differential heatmap; Calculate the area of ​​each of the connected components; The area of ​​each of the connected components is compared with a first preset area threshold. Connected regions with an area greater than the first preset area threshold are identified as potential defect regions. The defect characteristics of each potential defect area are analyzed to generate the screen printing defect detection results.

9. The screen printing defect detection method according to claim 8, characterized in that, The defect feature analysis of each potential defect region, generating the screen printing defect detection result, includes: Deep learning features are extracted from each potential defect region and the corresponding region of each potential defect region in the local template. Compare the deep learning features of the potential defect region with the deep learning features of the corresponding region to generate an anomaly score map for each potential defect region; The screen printing defect detection results are determined based on all the aforementioned anomaly score maps.

10. The screen printing defect detection method according to claim 9, characterized in that, The deep learning features are extracted based on the visual converter network model; The step of extracting deep learning features for each potential defect region and the corresponding region of each potential defect region in the local template includes: The potential defect region and the corresponding region are respectively input into the visual converter network model to obtain the deep learning features output by the visual converter network model.

11. The screen printing defect detection method according to claim 10, characterized in that, The visual converter network model specifically performs the following operations: After dividing the input into multiple image blocks, a set of multi-level local features is extracted from each image block from multiple preset levels; At each of the preset levels, the extracted local features are subjected to adaptive pooling at various scales. The local features processed by pooling at various scales are fused to form a set of multi-dimensional local features that contain both multi-level and multi-aggregation information, which are used as the deep learning features of each image patch.

12. The screen printing defect detection method according to claim 9, characterized in that, The step of comparing the deep learning features of the potential defect region with the deep learning features of the corresponding region to generate an anomaly score map for each potential defect region includes: Traverse each image block in the potential defect region and calculate the feature distance between the deep learning feature of any image block and the deep learning feature of all image blocks in the corresponding region. The minimum feature distance is determined as the anomaly score for any image block; The anomaly scores of all the image blocks are combined to form the anomaly score map.

13. The screen printing defect detection method according to claim 12, characterized in that, When at least two reference templates are provided for the potential defect region, and each reference template contains a corresponding region of the potential defect region, determining the minimum feature distance as the anomaly score of any image patch includes: Calculate the feature distance between the deep learning features of any image patch and the deep learning features of all image patches in each corresponding region to obtain multiple minimum feature distances; From the plurality of minimum feature distances, select a number of minimum feature distances with the smallest values ​​according to a preset ratio, and calculate the average feature distance. The average value of the feature distance is determined as the anomaly score of any image patch.

14. The screen printing defect detection method according to claim 9, characterized in that, The step of determining the screen printing defect detection result based on all the aforementioned anomaly score maps includes: Each of the aforementioned anomaly score maps is thresholded to generate a defect binary map; Perform connected component analysis on all the defect binary graphs. If it is determined that there is any connected component with an area greater than a second preset area threshold, then the screen printing defect detection result is determined to be that a screen printing defect exists.

15. A screen printing defect detection device, characterized in that, include: The global template matching unit is used to perform global matching on the image of the product to be inspected using a global template, and obtain the global matching result. The global template is an image region with significant and stable features extracted from a template image of a standard product in advance; the global matching result includes the matching position, matching rotation angle, and matching scaling ratio of the global template in the image to be inspected. A local template localization unit is used to determine the initial matching position of the local template in the image to be inspected based on the global matching result and the relative positional relationship of a local template with respect to the global template. The local template is an image block extracted from the template image of the standard product, representing other screen printing areas to be inspected, excluding the global template; the relative positional relationship is the spatial position and orientation relationship of the local template relative to the global template on the template image of the standard product; the initial matching position is the position of the local template initially calculated on the image to be inspected based on the global matching result and the relative positional relationship of the local template relative to the global template. The search region localization unit is used to determine the local search region corresponding to the initial matching position in the image to be inspected; An image localization unit is used to determine the precise matching result of the local template in the image to be inspected within the local search area, so as to crop an image of the region to be inspected from the image to be inspected based on the precise matching result; A defect detection processing unit is used to compare the image of the region to be inspected with the local template to generate a screen printing defect detection result for the product to be inspected. This includes: performing a structural similarity comparison between the image of the region to be inspected and the local template to generate a difference score map; performing binarization processing on the difference score map to obtain a difference heatmap; and performing connected component analysis on the difference heatmap to obtain the screen printing defect detection result for the product to be inspected. The relative positional relationship between the global template and the local template, and between the local template and the global template, is determined based on the template image of the standard product.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the screen printing defect detection method as described in any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the screen printing defect detection method as described in any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the screen printing defect detection method as described in any one of claims 1 to 14.

Images