Screen printing defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a silk-screen defect detection method and device, electronic equipment and a storage medium, and belongs to the technical field of intelligent manufacturing, and the method comprises the steps: carrying out the global matching of a to-be-detected image of a to-be-detected product through a global template; determining an initial matching position of the local template in the to-be-detected image based on the global matching result and the relative position relationship, determining a local search area according to the initial matching position, and performing accurate matching in the local search area to obtain a to-be-detected area image accurately aligned with the local template; and comparing the to-be-detected area image with the local template to generate a silk-screen defect detection result. According to the invention, global matching is carried out by using the global template, and then each local template is accurately aligned in the local area of the to-be-detected image based on the global matching result and the relative position relation of the local template relative to the global template, so that false alarms caused by inaccurate alignment are reduced, and the efficiency and accuracy of silk-screen defect detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent manufacturing, and in particular to a screen printing defect detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the production process of home appliance products, various subtle defects such as screen printing errors, ink leakage, dirt, black spots and offset may exist on the product surface.

[0003] At present, whether a product to be detected has a screen printing defect is mainly determined by manually comparing a standard product with the product to be detected. However, long-time manual work is prone to fatigue and inattention, which may result in missed detection of defects, and some defects are very small and difficult to detect by manual inspection. SUMMARY

[0004] The present application provides a screen printing defect detection method, device, electronic equipment and storage medium to solve the problems of low detection efficiency and low precision caused by manual detection.

[0005] The present application provides a screen printing defect detection method, which comprises the following steps: A global template is used to globally match a to-be-detected image of a product to be detected, and a global matching result is obtained; Based on the global matching result and a relative position relationship of a local template relative to the global template, an initial matching position of the local template in the to-be-detected image is determined; A local search area corresponding to the initial matching position in the to-be-detected image is determined; An accurate matching result of the local template in the to-be-detected image is determined in the local search area, and a to-be-detected area image is cropped from the to-be-detected image according to the accurate matching result; The to-be-detected area image and the local template are compared to generate a screen printing defect detection result of the product to be detected; The global template, the local template and the relative position relationship of the local template relative to the global template are determined based on a template image of a standard product.

[0006] According to the screen printing defect detection method provided by the present application, the local template has multiple, and the comparison of the to-be-detected area image and the local template to generate the screen printing defect detection result of the product to be detected comprises: The to-be-detected area image corresponding to each local template is cropped from the to-be-detected image; Each local template and its corresponding to-be-detected area image are compared to generate a comparison result; According to all the comparison results, the screen printing defect detection result of the product to be detected is generated.

[0007] According to the screen printing defect detection method provided by the application, the global template and the local template are determined based on the following steps: Determine the screen printing area containing the screen printing trace in the template image; According to the pixel distance between the first center coordinates of each screen printing area and the second center coordinates of the template image, a plurality of target areas are selected from all screen printing areas; The one with the longest edge length in all the target areas is set as the global template; Each of the remaining screen printing areas except the global template is set as a local template; The shape matching features of the global template and each local template are extracted, and the relative position relationship between the global template and each local template is extracted.

[0008] According to the screen printing defect detection method provided by the application, the global matching result includes the matching position, the matching rotation angle and the matching scaling ratio of the global template in the image to be detected, and the global matching of the image to be detected of the product to be detected by using the global template to obtain the global matching result, including: Using the shape matching features of the global template, at least one set of candidate matching parameters is searched and obtained in the image to be detected, each set of candidate matching parameters includes a matching position, a matching rotation angle, a matching scaling ratio and a corresponding matching degree; According to the matching degree of each set of candidate matching parameters, a set of best matching parameters is determined from the at least one set of candidate matching parameters; The matching position, the matching rotation angle and the matching scaling ratio contained in the best matching parameter are determined as the global matching result.

[0009] According to the screen printing defect detection method provided by the application, the local search area corresponding to the initial matching position in the image to be detected is determined, including: Taking the center of the initial matching position as a reference, the initial matching position is expanded according to a preset scaling ratio to obtain the local search area.

[0010] According to the screen printing defect detection method provided by the application, the comparison between the image to be detected and the local template to generate the screen printing defect detection result of the product to be detected, including: The image to be detected and the local template are compared in structure similarity to generate a difference score map; binarize the difference score map to obtain a difference heat map; perform connected component analysis on the difference heat map to obtain the screen printing defect detection result of the product to be detected.

[0011] According to the screen printing defect detection method provided by the application, the image of the area to be detected and the local template are compared in structure similarity to generate a difference score map, which comprises: synchronously traverse the image of the area to be detected and the local template by using a preset sliding window; at each position in the sliding window traversal process, determine the brightness similarity measure, the structure similarity measure and the contrast measure between a first image region and a second image region; the first image region is an image on the image of the area to be detected covered by the sliding window, and the second image region is an image on the local template covered by the sliding window; determine the local structure similarity score of the center pixel point of each sliding window according to the brightness similarity measure, the structure similarity measure and the contrast measure; collect the local structure similarity scores of all the center pixel points in the traversal process to obtain the difference score map.

[0012] According to the screen printing defect detection method provided by the application, the brightness similarity measure is determined based on the pixel average values of the first image region and the second image region, the structure similarity measure is determined based on the pixel covariance between the first image region and the second image region, and the contrast measure is determined based on the pixel variances of the first image region and the second image region.

[0013] According to the screen printing defect detection method provided by the application, the connected component analysis on the difference heat map to obtain the screen printing defect detection result of the product to be detected comprises: identify one or more connected components in the difference heat map; respectively calculate the area of each connected component; compare the area of each connected component with a first preset area threshold; determine the connected component with an area greater than the first preset area threshold as a potential defect region; analyze the defect features of each potential defect region to generate the screen printing defect detection result.

[0014] According to the screen printing defect detection method provided by the application, the analysis of the defect features of each potential defect region to generate the screen printing defect detection result comprises: extracting a deep learning feature of each of the potential defect area and a corresponding area of the potential defect area in the local template respectively; comparing the deep learning feature of the potential defect area and the deep learning feature of the corresponding area to generate an anomaly score map of each of the potential defect area; determining the screen printing defect detection result according to all the anomaly score maps.

[0015] According to the screen printing defect detection method provided by the application, the deep learning feature is extracted based on a visual transformer network model; The extracting a deep learning feature of each of the potential defect area and a corresponding area of the potential defect area in the local template respectively comprises: inputting the potential defect area and the corresponding area into the visual transformer network model respectively to obtain the deep learning feature output by the visual transformer network model.

[0016] According to the screen printing defect detection method provided by the application, the visual transformer network model specifically performs the following operations: After dividing the input into a plurality of image blocks, a plurality of multi-level local features are extracted from each of the image blocks at a plurality of preset levels; At each of the preset levels, the extracted local features are subjected to adaptive pooling processing of a plurality of different scales; The local features subjected to the adaptive pooling processing of the plurality of different scales are fused to form a plurality of multi-dimensional local features containing multi-level information and multi-aggregation degree information, as the deep learning feature of each of the image blocks.

[0017] According to the screen printing defect detection method provided by the application, the comparing the deep learning feature of the potential defect area and the deep learning feature of the corresponding area to generate an anomaly score map of each of the potential defect area comprises: traversing each of the image blocks in the potential defect area, and calculating a feature distance between the deep learning feature of any image block and the deep learning feature of all image blocks in the corresponding area respectively; determining the smallest feature distance as an anomaly score of the any image block; combining the anomaly scores of all the image blocks to form the anomaly score map.

[0018] According to the screen printing defect detection method provided by the application, in the case that at least two reference templates are provided for the potential defect area, and each of the reference templates contains a corresponding area of the potential defect area, the determining the smallest feature distance as the anomaly score of the any image block comprises: Calculate feature distances between the deep learning feature of any image block and the deep learning features of all image blocks in each corresponding region, respectively, to obtain a plurality of minimum feature distances; From the plurality of minimum feature distances, select a plurality of minimum feature distances with the smallest values in a preset proportion to calculate a feature distance average value; Determine the feature distance average value as the anomaly score of any image block.

[0019] According to the present application, a silk screen defect detection method is provided, and the silk screen defect detection result is determined according to all the anomaly score maps, which comprises: Threshold segmentation is performed on each anomaly score map to generate a defect binary map; Connected domain analysis is performed on all the defect binary maps, and if it is determined that there is any connected domain with an area greater than a second preset area threshold, it is determined that the silk screen defect detection result is that there is a silk screen defect.

[0020] The present application also provides a silk screen defect detection device, which comprises: A global template matching unit is configured to perform global matching on a to-be-detected image of a to-be-detected product by using a global template to obtain a global matching result. A local template positioning unit is configured to determine an initial matching position of a local template in the to-be-detected image based on a relative position relationship of the local template with respect to the global template and the global matching result. A search area positioning unit is configured to determine a local search area corresponding to the initial matching position in the to-be-detected image. A detection image positioning unit is configured to determine an accurate matching result of the local template in the to-be-detected image in the local search area, and to crop a to-be-detected region image from the to-be-detected image according to the accurate matching result. A defect detection processing unit is configured to compare the to-be-detected region image with the local template to generate a silk screen defect detection result of the to-be-detected product. The global template, the local template, and the relative position relationship of the local template with respect to the global template are determined based on a template image of a standard product.

[0021] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-mentioned silk screen defect detection methods.

[0022] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the screen printing defect detection method.

[0023] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the screen printing defect detection method.

[0024] The screen printing defect detection method, device, electronic equipment and storage medium provided by the application first perform global matching by using a global template, and then perform accurate alignment on each local template in a local region of the image to be detected based on the global matching result and the relative position relationship of the local template relative to the global template, so that the false positives caused by inaccurate alignment are reduced, and the efficiency and accuracy of screen printing defect detection are improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 is a flowchart of the screen printing defect detection method provided by the application.

[0027] Figure 2 is a flowchart of determining the global template and the local template provided by the application.

[0028] Figure 3 is a flowchart of performing global matching on the image to be detected of the product to be detected by using the global template provided by the application.

[0029] Figure 4 is a flowchart of generating the screen printing defect detection result of the product to be detected based on the image of the region to be detected and the local template provided by the application.

[0030] Figure 5 is a flowchart of performing structure similarity comparison by using the image of the region to be detected and the local template provided by the application.

[0031] Figure 6 is a flowchart of extracting features based on the visual converter network model provided by the application.

[0032] Figure 7 is a flowchart of generating the anomaly score map of the potential defect region based on the deep learning features of the potential defect region and the corresponding region provided by the application.

[0033] Figure 8 is a flowchart of determining an abnormal score of any image block provided by the present application.

[0034] Figure 9 is a flowchart of automatic modeling provided by the present application.

[0035] Figure 10 is a flowchart of defect detection provided by the present application.

[0036] Figure 11 is a schematic diagram of the principle of SSIM algorithm provided by the present application.

[0037] Figure 12 is a structural schematic diagram of a silk screen defect detection device provided by the present application.

[0038] Figure 13 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0040] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitation, the element defined by the statement "comprising a" does not exclude the existence of another identical element in the process, method, article or device comprising the element. The above terms in the present application can be understood according to the specific meaning of the terms by those skilled in the art according to the specific circumstances.

[0041] The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" and the like are generally a class, and are not limited to the number of objects, for example, the first object can be one or more.

[0042] The following will be described in conjunction withFigures 1-13 The application provides a silk screen defect detection method, device, electronic equipment and storage medium.

[0043] Figure 1 FIG. 1 is a flowchart of the silk screen defect detection method provided by the application. Figure 1 As shown in FIG. 1, the execution subject of the silk screen defect detection method provided by the application can be an industrial control computer, a server, a cloud computing platform, or a computer capable of executing the method of the application, and the following embodiments are described by taking an industrial control computer as an example.

[0044] As an optional embodiment, the silk screen defect detection method comprises the following steps: In step 110, a global template is used to perform global matching on the to-be-detected image of the to-be-detected product, and a global matching result is obtained.

[0045] The global template refers to an image region with significant and stable features extracted from a template image of a standard product, which can be used as a reference for preliminary positioning in the to-be-detected image. For example, the global template can be a pattern logo with rich edge or texture features selected from the silk screen region of the standard product image, such as a complex brand logo or a functional icon.

[0046] The to-be-detected product refers to a product entity on a production line that needs to be detected for silk screen defects, and the surface of the product is printed with text, a brand logo or a pattern, etc. For example, the to-be-detected product can be a panel of a household appliance, such as a washing machine panel, a microwave oven panel or an air conditioner panel.

[0047] The to-be-detected image refers to an image that can fully reflect the silk screen state of the surface of the to-be-detected product, which is obtained by photographing the to-be-detected product placed in a detection area under a preset lighting condition by an image acquisition device (such as an industrial camera). For example, the process of obtaining the to-be-detected image can include: placing the to-be-detected product on a tray, sending it to a detection position configured with a light source and a camera by a conveying device, illuminating the surface of the product by the light source, and then photographing by the camera.

[0048] Global matching refers to searching for a region in the to-be-detected image that is most similar or consistent with the features of the global template. For example, global matching can be realized based on a shape matching algorithm, that is, searching for a region in the to-be-detected image that is consistent with the shape features such as contour and edge of the global template.

[0049] The global matching result refers to a set of parameter information describing the position and pose of the global template matched in the image to be detected. The global matching result can include the matching position (such as the center coordinates), the matching rotation angle, and the matching scaling ratio of the global template in the image to be detected. For example, if a region most similar or consistent with the features of the global template is found in the image to be detected, the region is rotated by 1.5 degrees relative to the global template, scaled to 1.01 times the original size, and the center coordinates are translated to (x1, y1), then the set of parameters [(x1, y1), 1.5°, 1.01] constitutes the global matching result.

[0050] In step 120, based on the global matching result and the relative position relationship of the local template relative to the global template, the initial matching position of the local template in the image to be detected is determined. The global template and the local template and the relative position relationship of the local template relative to the global template are determined based on the template image of the standard product.

[0051] The local template refers to an image block representing other regions to be detected in addition to the global template, which is extracted from the template image of the standard product. For example, in a template image of an electrical appliance panel, if a complex brand logo is selected as the global template, other silk-screen regions on the electrical appliance panel, such as the text of the "start" button and the pattern of the "power" gear, can be used as independent local templates.

[0052] The relative position relationship refers to the spatial position and pose relationship of each local template relative to the global template based on the template image of the standard product. For example, the relative position relationship can be a two-dimensional displacement vector, which describes the offset (Δx, Δy) of the center coordinates of the local template relative to the center coordinates of the global template. Further, the relative position relationship can also include the rotation angle and scaling ratio information of each local template relative to the global template.

[0053] The initial matching position refers to a position of a local template preliminarily calculated in the image to be detected based on the global matching result and the relative position relationship of the local template relative to the global template. For example, if the global matching result indicates that the center coordinates of the global template are located at (x1, y1) in the image to be detected, and the two-dimensional displacement vector of the relative position relationship of a certain local template relative to the global template is (Δx, Δy), then by applying the two-dimensional displacement vector to the center coordinates (x1, y1) of the global template, the initial matching position of the local template in the image to be detected can be calculated, that is, the center coordinates of the initial matching position can be preliminarily determined at (x1+Δx, y1+Δy).

[0054] In step 130, the local search region corresponding to the initial matching position in the image to be detected is determined.

[0055] The local search region refers to an image region centered at the initial matching position and having a larger range than the initial matching position.

[0056] Considering the possible slight pose deviation of the product to be detected in actual placement and the local nonlinear deformation caused by the surface curvature of the product to be detected, the application sets up a local search region to provide a large and reliable search range for subsequent accurate matching, so as to ensure that the silk print feature to be matched can be completely contained in the local search region, and avoid matching errors caused by slight deviation of the initial matching position.

[0057] Step 140, determining the accurate matching result of the local template in the detected image in the local search region, so as to crop a detected region image from the detected image according to the accurate matching result.

[0058] The accurate matching result refers to a set of high-precision parameters describing the accurate position and pose obtained by further searching and positioning the local template in the local search region through a matching algorithm (such as a shape feature-based matching algorithm). For example, the accurate matching result can be a parameter set containing an accurate center coordinate (x2, y2), an accurate rotation angle (such as 1.3°), and an accurate scaling ratio (such as 1.02), which accurately describes the actual pose of a silk print region on the detected image relative to the template image of the standard product.

[0059] The detected region image refers to an image strictly aligned in size and pose with the corresponding local template cropped from the local search region after geometric correction (such as translation, rotation, and scaling) according to the accurate matching result. For example, based on the accurate matching result, an inverse geometric transformation can be applied to the local search region to inversely restore the silk print region at coordinate (x2, y2) rotated by 1.3° and scaled by 1.02 times to a completely aligned pose with the local template, and then crop an image with the same size as the local template from the geometrically transformed local search region, which is the detected region image.

[0060] Step 150, comparing the detected region image with the local template to generate the silk print defect detection result of the product to be detected.

[0061] The silk print defect detection result refers to the final conclusion about whether the silk print quality of the detected product is qualified based on the difference analysis between the detected region image and the local template.

[0062] The comparison process between the inspection area image and the local template can be implemented by using various image comparison algorithms, for example, by calculating the gray difference value of the corresponding pixels of the inspection area image and the local template, or by evaluating the similarity of the two images in brightness, contrast and structure, or by comparing the depth features extracted from the two images. According to the difference degree or difference characteristics generated by the comparison, and compared with one or more preset judgment thresholds, the final silk screen defect detection result can be a judgment signal indicating "qualified" or "unqualified".

[0063] The silk screen defect detection method provided by the present application first performs global matching using the global template, and then performs accurate alignment of each local template in the local area of the inspection image based on the global matching result and the relative position relationship of the local template relative to the global template, thereby reducing false positives caused by inaccurate alignment and improving the efficiency and accuracy of silk screen defect detection.

[0064] In another embodiment provided by the present application, there are multiple local templates, and the comparison between the inspection area image and the local template is used to generate the silk screen defect detection result of the product under test, comprising: cutting out the inspection area image corresponding to each local template from the inspection image; comparing each local template with its corresponding inspection area image to generate a comparison result; and generating the silk screen defect detection result of the product under test according to all comparison results.

[0065] Specifically, the silk screen defect detection result of the product under test can be determined by the comparison result of each local template and its corresponding inspection area image. For example, the final detection result of the product under test is determined as "qualified" only when the comparison result of all local templates and their corresponding inspection area images is "qualified". On the contrary, as long as there is any one or more local templates and their corresponding inspection area images with a comparison result of "unqualified", the final detection result of the product under test is determined as "unqualified".

[0066] The silk screen defect detection method provided by the present application can realize comprehensive and non-missing detection of all silk screen areas of the entire product by performing independent positioning comparison on multiple silk screen areas existing on one product respectively and comprehensively judging all independent comparison results, thereby ensuring the detection integrity of the product with complex silk screen layout and effectively avoiding the situation that defects in other areas are missed due to only focusing on part of the area.

[0067] Figure 2 is a flowchart of determining the global template and the local template provided by the present application, as shown in Figure 2 As another optional embodiment provided by the present application, the global template and the local template are determined based on the following steps: Step 210, determine the silk printing area containing the silk printing trace in the template image.

[0068] Considering that the silk printing trace on the template image usually includes both text and pattern types, the present application adopts a general text detection algorithm for processing the template image for the text part in the silk printing (such as containing Chinese, English, and other languages), which can automatically identify and locate the position of all text elements in the image, thereby determining the silk printing area containing the text.

[0069] For the pattern part in the silk printing (such as function key icon, brand logo, warning sign, etc.), a pre-trained rotating target detection network can be used for identification. The rotating target detection network can accurately locate the position and direction of the pattern mark, thereby determining the silk printing area containing the pattern.

[0070] The present application can automatically identify and frame all independent silk printing areas containing silk printing traces on the template image by integrating the output results of the general text detection algorithm and the rotating target detection network, thereby providing a basis for the selection and creation of subsequent global templates and local templates.

[0071] Step 220, according to the pixel distance between the first center coordinates of each silk printing area and the second center coordinates of the template image, screen multiple target areas from all silk printing areas.

[0072] Considering that the area located in the center of the image is least affected by lens distortion and has more stable features, it is more suitable as a reference for global positioning. Therefore, the present application calculates the pixel distance between the first center coordinates of each silk printing area and the second center coordinates of the template image, then sorts all silk printing areas in ascending order according to the calculated pixel distance, and selects the preset number of areas with the smallest pixel distance from the sorting result as the target areas.

[0073] In a preferred embodiment, the six silk printing areas with the smallest pixel distance can be selected to form a target area set for the next step of screening.

[0074] Step 230, set the one with the longest edge length in all target areas as the global template.

[0075] Considering that a template with rich features can significantly improve the accuracy and robustness of subsequent global matching, and the edge length is an important indicator for measuring the richness of image features, the present application first traverses all target areas and performs bilateral filter (Bilateral filter) processing on the image of each target area. The bilateral filter processing can effectively remove image noise while preserving image edge details, thereby providing high-quality input for subsequent edge extraction.

[0076] Further, the present application adopts an edge extraction algorithm (for example, Canny edge detection algorithm) to extract all edge contours in each target region. Subsequently, by counting or accumulating the number of pixel points on the edge contours, the total edge length of each target region can be calculated. Finally, the edge lengths calculated for all target regions are compared, and the target region with the longest edge length is determined as the global template.

[0077] Step 240, set each of the remaining silk printing regions except the global template as a local template.

[0078] Specifically, after the global template is determined, the global template can be excluded from the target region set, and then all the silk printing regions remaining in the target region set, i.e. each independent character or pattern region (for example, the characters of the "start" button, the pattern of the "power" gear, etc.) except the global template, are set as a local template.

[0079] Step 250, extract the shape matching features of the global template and each local template, and extract the relative position relationship between the global template and each local template.

[0080] The shape matching features refer to a kind of geometric contours, edge directions, gradient information and their spatial topological relationships extracted from the global template and the local template, which are used to describe the image content of the global template and the local template. For example, for a character region, since it is mainly composed of strokes, its shape matching features can be obtained by extracting the contours or skeletons of the character strokes, and decomposing these strokes into a series of point sets, recording the gradient direction information of each point and the relative position and topological relationship between the strokes. For a pattern region, its shape matching features can be the external contour of the pattern region and the internal holes, independent geometric figures, etc.

[0081] The silk printing defect detection method provided by the present application can realize the automatic creation and modeling of the detection template by automatically identifying and framing all the silk printing regions on the template image and automatically assigning the global template and the local template based on the preset rules of position and edge features, so as to greatly simplify the preparation work before detection for different models of products, significantly shorten the modeling time and reduce the dependence on the experience of the operator.

[0082] Figure 3 is a flowchart of the global matching of the global template to the to-be-detected image of the to-be-detected product provided by the present application, as shown in Figure 3As shown, as another optional embodiment provided by the present application, the global matching result includes the matching position, the matching rotation angle and the matching scaling ratio of the global template in the image to be detected. The image to be detected of the product to be detected is globally matched by using the global template, and the global matching result is obtained, including: In step 310, at least one group of candidate matching parameters is searched and obtained in the image to be detected by using the shape matching feature of the global template. Each group of candidate matching parameters includes a matching position, a matching rotation angle, a matching scaling ratio and a corresponding matching degree.

[0083] Specifically, the present application can search in the range of the entire image to be detected by using the shape matching feature of the global template. The search process not only considers the position, but also considers a certain angle and scaling range to cope with the slight rotation or the slight height change of the product to be detected from the camera that may occur on the conveying belt. In the search process, the similarity between the region in the image to be detected and the shape matching feature of the global template under different positions, angles and scaling conditions is calculated by a matching algorithm, and a quantitative score, i.e., the matching degree, is given.

[0084] Each group of candidate matching parameters contains the center coordinates, the rotation angle, the scaling ratio and the matching degree score of the candidate region. For example, the following three groups of candidate matching parameters may be obtained: [position (x1, y1), angle 1.2°, scaling 1.01, matching degree 0.95], [position (x2, y2), angle 1.1°, scaling 1.03, matching degree 0.88] and [position (x3, y3), angle 5.0°, scaling 0.80, matching degree 0.75].

[0085] In step 320, a group of best matching parameters is determined from the at least one group of candidate matching parameters according to the matching degree of each group of candidate matching parameters.

[0086] For example, for the three groups of candidate matching parameters in step 310, the matching degrees are 0.95, 0.88 and 0.75 respectively. By comparing these three values, it can be determined that 0.95 is the highest score. Therefore, the complete parameters corresponding to the highest score 0.95, i.e., [position (x1, y1), angle 1.2°, scaling 1.01, matching degree 0.95], are determined as the best matching parameters.

[0087] In step 330, the matching position, the matching rotation angle and the matching scaling ratio contained in the best matching parameters are determined as the global matching result.

[0088] For example, for the best matching parameter [position (x1, y1), angle 1.2°, scaling 1.01, matching degree 0.95] in step 320, the set [position (x1, y1), angle 1.2°, scaling 1.01] of the matching position (x1, y1), the matching rotation angle 1.2° and the matching scaling ratio 1.01 is determined as the global matching result.

[0089] The silk screen defect detection method provided by the application can ensure that the most reliable and accurate matching result is screened out when there are multiple similar regions to the template in the image to be detected, thereby significantly improving the robustness and reliability of global positioning and effectively avoiding subsequent detection failure or deviation caused by false positioning.

[0090] In another embodiment provided by the application, the local search area corresponding to the initial matching position in the image to be detected is determined by: taking the center of the initial matching position as a reference, and expanding the initial matching position according to a preset scaling ratio to obtain the local search area.

[0091] The determination of the local search area can take the center coordinate of the initial matching position as a reference, and expand the initial matching position according to a preset scaling ratio, thereby forming the local search area. For example, the preset scaling ratio can be set to 1.2, that is, the range of the initial matching position is expanded to 1.2 times along the center coordinate, and the expanded area obtained is the local search area used for subsequent accurate matching.

[0092] The silk screen defect detection method provided by the application can provide an appropriate redundant range for subsequent accurate matching by expanding the initial matching position as a center according to a preset ratio to determine the local search area, thereby effectively dealing with the deviation between the actual local position and the theoretical initial position caused by factors such as slight deviation of product placement or curved deformation, significantly improving the success rate and fault tolerance of local accurate matching, and ensuring that the target can be accurately found even in the case of positioning disturbance.

[0093] Figure 4 is a flowchart of generating a silk screen defect detection result of a product to be detected based on an image of a region to be detected and a local template provided by the application, as shown in Figure 4 As another optional embodiment provided by the application, comparing the image of the region to be detected with the local template to generate a silk screen defect detection result of the product to be detected, comprising: Step 410, comparing the image of the region to be detected with the local template for structure similarity to generate a difference score map.

[0094] The structural similarity comparison can be implemented by a structural similarity (SSIM) algorithm. Unlike a traditional image comparison algorithm that only compares the gray value difference of a pixel, the SSIM algorithm evaluates and compares the similarity of two images in three dimensions of brightness, contrast, and structure to obtain a comprehensive similarity evaluation that is more consistent with human visual perception. It should be noted that the SSIM algorithm is not sensitive to changes in image illumination, but is very sensitive to local structural changes such as silk printing ink leakage, offset, and dirt.

[0095] The difference score map refers to a gray-scale image with the same size as the image of the region to be detected. The gray value of each pixel in the gray-scale image represents the structural difference between the region to be detected and the local template at the corresponding position of the pixel.

[0096] For example, the pixel value range of the difference score map can be 0 to 255. In the region where the image of the region to be detected is completely consistent with the local template, the structural similarity is very high, and therefore the corresponding pixel value in the difference score map will be very low (for example, close to 0, appearing as black). Conversely, in the region where the image of the region to be detected has defects (such as ink leakage or dirt), the structure is significantly different from the local template, the structural similarity is low, and therefore the corresponding pixel value in the difference score map will be very high (for example, close to 255, appearing as a highlighted white dot or white region). In this way, the difference score map can intuitively highlight potential defect regions from the image of the region to be detected.

[0097] At step 420, the difference score map is binarized to obtain a difference heat map.

[0098] Binarization refers to the process of converting the difference score map into a binary image containing only two pixel values (for example, black and white). This process can be achieved by setting a pre-set binarization threshold: for each pixel in the difference score map, if its gray value is greater than or equal to the binarization threshold, it is set to a pixel value representing a potential defect (for example, 255, appearing as white) in the binary image; otherwise, it is set to a background pixel value (for example, 0, appearing as black).

[0099] Further, in order to ensure that all possible small defects can be screened out, a lower binarization threshold can be set. In a specific embodiment, the binarization threshold can be set to 20.

[0100] The difference heat map refers to a binary image obtained by binarizing the difference score map, in which the potential defect area is highlighted. For example, if there is a dirty spot on the image of the region to be detected, the dirty spot will form a highlighted area on the difference score map, and after binarization, the position of the dirty spot will appear as a clear white connected region on the finally generated difference heat map, while the other parts of the image are black background, thereby providing a basis for subsequent defect analysis and positioning.

[0101] In step 430, the difference heat map is subjected to connected domain analysis to obtain the silk printing defect detection result of the product to be detected.

[0102] Connected domain (Binary Large Object, Blob) analysis refers to a process of finding and identifying all independent regions (i.e. connected domains) composed of pixels with the same pixel value and connected to each other in the difference heat map. In this process, all white regions in the difference heat map can be automatically identified, and the geometric parameters of each independent white region are calculated, one of which is the area.

[0103] For example, by calculating the area of each identified white connected domain and comparing the area with a preset area threshold, it can be determined whether the connected domain corresponds to a real defect. If the area of a connected domain is greater than the area threshold, it is determined to be a real defect region; otherwise, if the area is less than or equal to the area threshold, it can be regarded as a small disturbance caused by image noise and other non-defect factors and ignored. The final silk printing defect detection result is generated based on this judgment: if there is at least one connected domain determined to be a defect in the difference heat map, the final detection result is "unqualified"; if the areas of all connected domains do not exceed the area threshold, the final detection result is "qualified".

[0104] The silk printing defect detection method provided by the present application can efficiently identify local structural changes caused by ink leakage, offset, dirt, etc. by using an image processing algorithm with fast calculation speed and insensitivity to overall illumination changes, thereby realizing fast and reliable preliminary screening of common silk printing defects without sacrificing detection speed.

[0105] Figure 5 is a flowchart of the structure similarity comparison between the image of the region to be detected and the local template provided by the present application, as shown in Figure 5 As another optional embodiment provided by the present application, the structure similarity comparison between the image of the region to be detected and the local template is performed to generate a difference score map, which includes: Step 510, synchronously traverse the image of the region to be detected and the local template by using a preset sliding window.

[0106] The sliding window refers to a rectangular region of a preset size. For example, the sliding window can be a window of 11x11 pixels.

[0107] Synchronously traverse refers to the sliding window moving on the image of the region to be detected and the local template at the same step (e.g. pixel by pixel) and the same order (e.g. from left to right, from top to bottom). At any moment during the traversal, when the sliding window covers a certain position on the image of the region to be detected, the sliding window also covers the same corresponding position on the local template. Therefore, at each position of the image of the region to be detected and the local template, the sliding window extracts a pair of image blocks: one image block is from the image of the region to be detected, and the other image block is from the local template.

[0108] Step 520, at each position during the traversal of the sliding window, determine the brightness similarity measure, the structural similarity measure and the contrast measure between the first image region and the second image region; the first image region is the image on the image of the region to be detected covered by the sliding window, and the second image region is the image on the local template covered by the sliding window.

[0109] The brightness similarity measure can evaluate the closeness of the average illumination level between the first image region and the second image region, which measures whether the overall light and dark degree of the two image regions is consistent.

[0110] The contrast similarity measure can measure the similarity of the range of pixel intensity change within the first image region and the second image region, which measures whether the range of change from the darkest to the brightest in the two image regions is similar, that is, whether the "vividness" of the two image regions is consistent.

[0111] The structural similarity measure can compare the similarity of the structural information in the first image region and the second image region after removing the influence of brightness and contrast. For example, the structural information can include the outline of the silk screen, the texture and other information in the first image region and the second image region.

[0112] Step 530, according to the brightness similarity measure, the structural similarity measure and the contrast measure, determine the local structural similarity score of the center pixel point of each sliding window.

[0113] Specifically, the calculation method of the structural similarity score can be shown in formula (1): (1) Wherein, represents the first image region from the image of the region to be detected, denotes a second image region from the local template, denotes a structural similarity algorithm, denotes a structural similarity score of the first image region and the second image region; denotes a pixel mean value of the first image region, denotes a pixel mean value of the second image region, and can be used to calculate a luminance similarity measure; denotes a pixel variance of the first image region, denotes a pixel variance of the second image region, and can be used to calculate a contrast measure; denotes a pixel covariance of the first image region and the second image region, can be used to calculate a structural similarity measure; and are two constants for maintaining the stability of the calculation, where , , is a dynamic range of image pixel values, and is a preset constant, in a preferred embodiment, , .

[0114] Formula (1) is used to measure the similarity of two images in three dimensions of luminance, contrast and structure. Compared with traditional indicators such as mean square error or peak signal-to-noise ratio, the structural similarity score can more effectively distinguish image distortion caused by structural deformation. By introducing the pixel mean value, the pixel standard deviation and the pixel covariance of the image for comprehensive measurement, the structural similarity score significantly improves the robustness and accuracy of image similarity evaluation while maintaining the calculation efficiency. Taking the structural similarity score as the measurement standard of the difference between the image to be detected and the local template is conducive to highlighting the local structural changes caused by the silk screen defects, and improves the reliability and accuracy of the defect detection results.

[0115] Step 540, the local structural similarity scores of all center pixel points in the set traversal process are collected to obtain a difference score map.

[0116] After the sliding window completes the synchronous traversal of the entire image to be detected and the local template, the structural similarity scores of all center pixel points can be obtained. Then a gray-scale image with the same size as the image to be detected can be created. Then all the calculated structural similarity scores are traversed, and each structural similarity score is filled into the center pixel point position corresponding to the structural similarity score in the gray-scale image as a pixel value. The gray-scale image after the filling is the difference score map.

[0117] The silk screen defect detection method provided by the application can distinguish the image changes caused by global non-defect factors such as illumination and contrast from the real defects caused by the changes in the silk screen itself, thereby providing a difference score map with higher quality and higher reliability for subsequent defect determination.

[0118] In another embodiment provided by the application, the brightness similarity measure is determined based on the pixel average values of the first image region and the second image region, the structure similarity measure is determined based on the pixel covariance between the first image region and the second image region, and the contrast measure is determined based on the pixel variances of the first image region and the second image region.

[0119] Since the pixel average value can directly reflect the overall brightness level of the image region, the brightness similarity measure is determined by comparing the closeness of the pixel average value of the first image region and the pixel average value of the second image region.

[0120] Since the variance measures the dispersion degree of the pixel value relative to its average value and can directly reflect the contrast of the image region, the contrast similarity measure is determined by comparing the closeness of the pixel variance of the first image region and the pixel variance of the second image region.

[0121] Since the covariance measures the overall trend of the pixel values of the two image regions and can effectively represent the internal structure information of the image, the structure similarity measure is determined based on the pixel covariance between the first image region and the second image region.

[0122] The silk screen defect detection method provided by the application binds the three abstract similarity measures of brightness, contrast and structure with the pixel average value, the pixel variance and the pixel covariance which can be directly calculated and have clear physical meaning, thereby providing a specific and reproducible mathematical implementation path for the structure similarity comparison, ensuring the objectivity and stability of the comparison result and eliminating the implementation differences caused by the ambiguity of the algorithm definition.

[0123] In another embodiment provided by the application, the difference heat map is subjected to connected domain analysis to obtain the silk screen defect detection result of the product to be detected, including: identifying one or more connected domains in the difference heat map; calculating the area of each connected domain respectively; comparing the area of each connected domain with a first preset area threshold; determining the connected domain with an area greater than the first preset area threshold as a potential defect region; and performing defect feature analysis on each potential defect region to generate the silk screen defect detection result.

[0124] Specifically, all connected regions in the difference heat map are identified, and their areas are calculated, and then the area of each connected region is compared with a first preset area threshold.

[0125] For example, if the area of a connected region is greater than the first preset area threshold, it will be marked as a potential defect region that needs to be further confirmed. If the area of a connected region is less than or equal to the first preset area threshold, it is ignored.

[0126] Further, after the comparison of all connected regions with the first preset area threshold is completed, if the areas of all connected regions do not exceed the first preset area threshold, i.e., there is no any potential defect region, it can be determined that the image of the to-be-inspected region is qualified. If one or more potential defect regions are identified, secondary comparison and feature analysis are performed on these potential defect regions and their corresponding regions in the local template to finally confirm whether they are real defects.

[0127] The silk screen defect detection method provided by the present application can minimize the application range of the computationally intensive precise analysis algorithm by introducing a two-stage defect determination strategy, i.e., first using a traditional visual algorithm with fast calculation speed to preliminarily screen out all potential defect regions, and then only performing further precise feature analysis on these potential defect regions, thereby greatly improving the operation efficiency of the overall detection process while ensuring high detection rate and low false positive rate, and achieving effective balance between detection speed and detection accuracy.

[0128] In another embodiment provided by the present application, the defect feature analysis of each potential defect region to generate the silk screen defect detection result includes: extracting the deep learning features of each potential defect region and the corresponding region in the local template, respectively; comparing the deep learning features of the potential defect region and the corresponding region to generate an abnormal score map of each potential defect region; and determining the silk screen defect detection result according to all abnormal score maps.

[0129] Deep learning feature extraction refers to inputting the potential defect region on the to-be-inspected image and the corresponding region in the local template into a pre-trained deep learning network model, respectively. The deep learning network model can learn and extract deep features from the image. These feature information usually appears as a set of high-dimensional feature vectors.

[0130] Further, the two groups of extracted deep learning features can be compared to quantify the degree of difference between them. The comparison process can be realized by calculating the feature distance (e.g. Euclidean distance) between the feature vectors, and a larger feature distance means that the to-be-inspected region and the local template have significant differences in deep semantics. Based on the comparison result, an anomaly score map can be generated, and the brightness value of each pixel of the anomaly score map represents the degree of difference between the potential defect region on the to-be-inspected image and the corresponding region in the local template. The greater the difference, the higher the score and the brightness.

[0131] Then, the anomaly score map can be threshold segmented to filter out normal regions with low anomaly scores and only keep real abnormal regions with anomaly scores significantly higher than the threshold. The final determination of whether the to-be-inspected image has real defects can be made by analyzing the area parameters of these real abnormal regions. For example, if real defects are finally confirmed in any potential defect region, the detection result of the to-be-inspected product is “unqualified”; only when no real defects are found in all potential defect regions, the detection result of the to-be-inspected product is “qualified”.

[0132] The silk screen defect detection method provided by the present application can use the powerful feature representation capability of deep neural networks to capture pseudo differences caused by non-defect factors such as small surface texture or uneven reflection of the product, which are difficult to distinguish by traditional visual algorithms, thereby more accurately identifying potential defects at a deep semantic level, and significantly improving the accuracy and reliability of silk screen defect detection.

[0133] In another embodiment provided by the present application, the deep learning features are extracted based on a vision transformer network model; the deep learning features of each potential defect region and the corresponding region in the local template are extracted respectively, including: inputting the potential defect region and the corresponding region into the vision transformer network model respectively to obtain the deep learning features output by the vision transformer network model.

[0134] The vision transformer (ViT) network model is derived from the Transformer architecture in the natural language processing field, which can accurately capture global dependencies and long-distance context information in images.

[0135] Specifically, when extracting features, the ViT network model first divides the input image region (the potential defect region or the corresponding region on the local template) into a series of non-overlapping image patches of fixed size, for example, each image patch can be 14x14 pixels in size. Then, the ViT network model treats these image patches as sequence data, processes all image patches in parallel through a self-attention mechanism, and calculates the mutual relationship weights between them. By stacking multiple Transformer modules, the ViT network model can build more and more abstract and robust feature representations of image content layer by layer. Finally, the output of the ViT network model is a set of deep feature vectors (tokens) representing each image patch of the input image. These deep feature vectors are deep learning features.

[0136] In one specific embodiment, a pre-trained ViT-L-14-336 network model can be used as the visual transformer network model.

[0137] The silk screen defect detection method provided by the application can effectively learn and understand the long-distance dependency relationship between each part in the image region by using the powerful global context perception ability of the visual transformer network model as the feature extractor, thereby extracting deep features with a more global view and more complete structure than traditional convolutional networks. This is particularly effective for accurately identifying defects caused by large-area gradual changes, fine scratches, or overall shifts, and further improves the accuracy of feature comparison.

[0138] Figure 6 is a flowchart of the feature extraction based on the visual transformer network model provided by the application, as shown in Figure 6 As another optional embodiment of the application, the visual transformer network model specifically performs the following operations: Step 610: After dividing the input into multiple image patches, a set of multi-level local features is extracted from each image patch from multiple preset levels.

[0139] It should be noted that in order to make the final extracted local features more rich and have stronger distinguishability, the application does not only use the final output layer of the ViT network model, but extracts features from multiple different depth levels of the backbone network. In a deep neural network, features at different levels usually correspond to different levels of semantic information: shallow networks pay more attention to low-level details such as color and edge, while deep networks can capture more abstract high-level semantics such as shape. By extracting features from multiple preset levels, the final feature representation can contain multi-level image information from specific to abstract.

[0140] For example, for a ViT network model, four preset levels can select the features of the [5, 11, 17, 23] layers. When an input image is divided into M image blocks, for any one of the four preset levels, the ViT network model outputs M corresponding feature vectors for the M image blocks. Therefore, for each image block, a set of feature vectors containing information of four different levels can be obtained, that is, multi-level local features are formed.

[0141] At step 620, the extracted local features are subjected to adaptive pooling processing of multiple different scales at each preset level.

[0142] Specifically, first, the M feature vectors output from a certain level of the ViT network model are reshaped to obtain a tensor feature similar to a convolution network feature map, with a shape of C . Then, an adaptive average pooling operation is performed on the tensor feature. In order to realize processing of multiple different scales, multiple pooling kernels of different sizes can be used. For example, three different sizes of pooling kernels, 1x1, 3x3 and 5x5, can be used respectively to perform adaptive average pooling on the tensor feature of each level. The pooled tensor feature is then restored to the original

[0143] size by reshaping. In this way, for the original feature vector extracted from a single level of the ViT network model, three different aggregation degree feature representations can be obtained, greatly enriching the diversity of the features. At step 630, the local features subjected to multiple different scale pooling processing are fused to form a set of multi-dimensional local features containing both multi-level information and multi-aggregation degree information, as the deep learning features of each image block.

[0144] For each local image block of the input image, a set of feature vectors containing level number (L) x aggregation degree (r) can be obtained. For example, for a certain image block, the feature vectors obtained by processing in 4 levels, 3 aggregation degrees per level, a total of 4x3=12 different ways can be collected. This set of feature vectors, that is, the multi-dimensional local features, contains rich multi-level information and multi-aggregation degree information, and can describe the content of the image block in multiple dimensions and multiple granularities. The multi-dimensional local features are determined as the deep learning features of the image block.

[0145]

[0146] ​The silk screen defect detection method provided by the application can construct a multi-dimensional local feature with extremely rich information dimensions for each image block by fusing features from multiple different depth levels of a visual converter network model and multiple different scale pooling processing, and the multi-dimensional local feature simultaneously contains all-around information from low-level details to high-level semantics and from fine granularity to rough generalization, thereby greatly enhancing the representation ability and distinguishability of the feature and providing high-quality feature input for subsequent high-precision abnormal comparison.

[0147] Figure 7 is a flowchart of generating an abnormal score map of a potential defect area by a deep learning feature of the potential defect area and a corresponding area provided by the application, as shown in Figure 7 As another optional embodiment provided by the application, the deep learning feature of the potential defect area is compared with the deep learning feature of the corresponding area to generate an abnormal score map of each potential defect area, which includes: Step 710: Each image block in the potential defect area is traversed, and the feature distance between the deep learning feature of any image block and the deep learning feature of all image blocks in the corresponding area is calculated respectively.

[0148] The feature distance can be the Euclidean distance between the deep learning features of two image blocks, and the greater the Euclidean distance, the greater the difference between the two image blocks in the feature space of the deep learning network, that is, the two image blocks are more dissimilar in content, texture, structure and other deep semantics.

[0149] Step 720: The smallest feature distance is determined as the abnormal score of any image block.

[0150] Considering that the smallest feature distance represents the smallest difference of the corresponding area that can be found in the local template, if the difference score between the image block and the smallest difference of the corresponding area that can be found in the local template is large, it means that the image block is likely to be an abnormal area with silk screen defects. Therefore, the smallest feature distance is determined as the abnormal score of any image block.

[0151] Step 730: The abnormal scores of all image blocks are combined to form an abnormal score map.

[0152] After the traversal of all image blocks in the potential defect area is completed and an abnormal score of each image block is calculated, an abnormal score map with the same size as the potential defect area can be created, and then the abnormal score of each image block is filled into the corresponding image block position in the abnormal score map as a pixel value according to the spatial arrangement order of the image block in the potential defect area.

[0153] For example, if a potential defect area is divided into 256 image blocks of 16×16, the anomaly scores corresponding to the 256 image blocks can be calculated. Then, an anomaly score map with the same size as the potential defect area can be created, and these 256 anomaly scores can be filled into this anomaly score map according to their spatial arrangement order, with the anomaly score of each image block as the pixel value.

[0154] The silk screen defect detection method provided by the present invention can effectively overcome the problem of local micro-shift caused by factors such as product positioning and surface deformation by finding the minimum feature distance among all image blocks of the standard template for each image block of the inspected area and using this as its anomaly score, thereby greatly enhancing the robustness of the deep learning comparison algorithm to posture disturbances and further reducing the false alarm rate.

[0155] Figure 8 : is a flow chart of determining the abnormality score of any image block provided by the present invention, such as Figure 8 As shown, as another optional embodiment provided by the present invention, when at least two reference templates are provided for the potential defect area, and each reference template includes a corresponding area of ​​the potential defect area, the minimum feature distance is determined as the abnormality score of any image block, including: Step 810 , respectively calculating the feature distances between the deep learning features of any image block and the deep learning features of all image blocks in each corresponding region to obtain multiple minimum feature distances.

[0156] The feature distance can be the Euclidean distance between the deep learning features of any image block and the deep learning features of all image blocks in each corresponding area. The calculation method of the minimum feature distance can be shown in formula (2): (2) in, Indicates the The first The image block in Layer and Under the polymerization degree, The minimum feature distance obtained by comparing all image blocks in the reference template, Represents the index of the image to be detected, Indicates the Reference templates, represents the preset feature level in the visual transformer network model, represents the degree of aggregation used in the adaptive pooling process, Indicates the first image blocks; Indicates the The first The image block in Layer and Deep learning feature vectors extracted at different aggregation levels; Indicates the Reference template The image block in Layer and Deep learning feature vectors extracted at different aggregation levels; Represents the Euclidean distance between two deep learning feature vectors.

[0157] Step 820 : Selecting a plurality of minimum feature distances with the smallest values ​​in a preset ratio from the plurality of minimum feature distances to calculate an average feature distance.

[0158] For example, there may be 10 minimum feature distances, and the preset ratio may be 20%. Then, the two smallest values ​​are selected from the 10 minimum feature distance values ​​to calculate and obtain the average feature distance.

[0159] Specifically, the calculation method of the average feature distance can be shown as formula (3): (3) in, Indicates the The first The image block in Layer and At a certain degree of aggregation, the feature distance is obtained by comparing multiple reference templates and calculating the average. Represents the index of the image to be detected, represents the preset feature level in the visual transformer network model, represents the degree of aggregation used in the adaptive pooling process, Indicates the first image blocks; Indicates the number of reference templates with the minimum feature distance selected according to the preset ratio; Indicates the A reference template image.

[0160] Step 830: Determine the average value of the feature distances as the abnormality score of any image block.

[0161] Specifically, the calculation method of the abnormal score can be shown as formula (4): (4) in, Indicates the The first anomaly score of an image block, an index of an image to be detected, an index of an image block in the image to be detected, an index of an image block in the image to be detected, a total number of selected feature levels, three different aggregation degrees of adaptive pooling adopted, an index of an image to be detected, an index of an image block in the image to be detected, an index of an image block in the image to be detected, an index of an image block in the image to be detected, an index of an image block in the image to be detected, a preset feature level in a visual transformer network model, an aggregation degree used in adaptive pooling processing.

[0162] Formula (4) is used to calculate the final anomaly score of each image block, and the core idea is to perform weighted average on the feature distance average under different levels and different aggregation degrees, so as to comprehensively reflect the anomaly degree of the image block in the multi-scale multi-level feature space. Specifically, the formula first obtains the feature distance of the image block under three different scales (for example, 1x1, 3x3, 5x5 pooling kernel) on multiple preset feature levels extracted by the visual transformer network model, and then calculates the average value under all combinations as the final anomaly score of the image block. By introducing multi-level feature fusion, the anomaly score not only contains the low-level detail texture information, but also fuses the high-level abstract semantic features, which helps to enhance the recognition ability of complex silk screen defects (such as deformation, occlusion, blur, etc.), thereby improving the robustness of the silk screen defect detection result.

[0163] The silk screen defect detection method provided by the application can greatly improve the tolerance of the detection algorithm to the process fluctuations within the allowable range existing in the standard product itself by introducing multiple reference templates and only averaging the comparison results with a few most similar templates. As long as a certain form on the product to be detected is similar to a part of the standard product, it can be considered normal, thereby effectively avoiding false positives caused by the accidentalness or insufficient representativeness of a single standard template, and significantly enhancing the stability and adaptability in actual industrial production environment.

[0164] In another embodiment provided by the application, the silk screen defect detection result is determined according to all anomaly score maps, including: performing threshold segmentation on each anomaly score map to generate a defect binary map; performing connected domain analysis on all defect binary maps, and if it is determined that there is any connected domain with an area greater than a second preset area threshold, it is determined that the silk screen defect detection result is that there is a silk screen defect.

[0165] Threshold segmentation refers to that in the anomaly score map, the region with a score higher than the anomaly score threshold is considered as an anomaly, which is marked as foreground (e.g. white) on the defect binary map; the region with a score lower than or equal to the anomaly score threshold is considered as normal, which is marked as background (e.g. black). In one specific embodiment, the anomaly score threshold can be set to 0.52.

[0166] Further, a connected component analysis can be performed on all generated defect binary maps to identify all connected components in the defect binary map. Then, the area of each connected component is calculated and compared with a second preset area threshold.

[0167] For example, if it is determined that there is any connected component whose calculated area is greater than the second preset area threshold, it is determined that the product to be detected has a screen printing defect, and the final screen printing defect detection result is "unqualified". Conversely, if no connected component with an area greater than the second preset area threshold is found in the defect binary map, it is determined that the product to be detected has no screen printing defect, and the final screen printing defect detection result is "qualified".

[0168] The screen printing defect detection method provided by the present application can convert the abstract, feature space-based difference into a measurable defect region with a physical size by performing threshold segmentation on the continuous anomaly score map output by the deep learning model and determining whether the area of the final connected component exceeds the preset threshold, thereby effectively filtering out the tiny abnormal points caused by random noise without actual significance, and ensuring the stability and reliability of the final detection result.

[0169] Figure 9 is a schematic diagram of the automatic modeling process provided by the present application, as shown in Figure 9 The process is started from the start node. First, the input template image is processed by using the text positioning algorithm and the rotation target detection algorithm, and the regions to be detected are automatically identified and framed by the synergistic effect of the two algorithms. These automatically framed regions are the regions to be detected, which avoids the tedious work of manually framing the regions by traditional manual method and improves the modeling efficiency.

[0170] Further, after the automatic framing is completed, the global template and the local templates in the entire template are further determined according to the aforementioned positioning results. The global template refers to the representative and edge-rich region selected from the multiple detection regions, which is used for global positioning and coarse matching; the remaining detection regions are defined as local templates, which are respectively used for fine matching of the corresponding regions.

[0171] Then, after the determination of the global template and the local templates is completed, feature extraction is performed on the global template and each local template, specifically including extracting shape matching features corresponding to each template by using a shape matching algorithm, while recording the relative spatial position relationship between the global template and the local templates. These shape matching features and the relative position relationship are stored in a local storage medium as key reference data in the subsequent defect detection comparison process. Finally, the automatic modeling process is completed, entering the end node, and the entire automatic modeling process efficiently and automatically realizes accurate framing of the region to be detected and establishment of template features, providing a solid foundation for subsequent defect detection.

[0172] Figure 10 is a flowchart of the defect detection process provided by the present application, as shown in Figure 10 The detection process starts from the start node, first reads the pre-established global template and local template and their corresponding shape matching features, and reads the image to be detected of the product to be detected. These shape matching features are key data of the template, which provide technical support for subsequent matching positioning.

[0173] Then, the global template is used for global matching positioning of the image to be detected to obtain rough pose information of the product as a whole. On this basis, taking the global positioning result and the relative position relationship of the local template relative to the global template as a reference, accurate matching is performed in the image to be detected for each local template to realize detailed positioning, so as to obtain accurate alignment of each local region. After completing the accurate matching, image correction is performed on the image to be detected, including translation, rotation and scaling operations, to ensure that each local position accurately coincides with the corresponding template.

[0174] The corrected image region then enters the next step of processing, calculates the similarity by the SSIM algorithm, and combines blob analysis to position and screen the preliminary defect region. Based on this, it is judged whether there is a silk printing defect: if there is no defect, the detection process is directly ended; if there is a potential defect, the process will call the defect detection module based on deep learning to generate a similarity score map.

[0175] Then, threshold segmentation is performed on the similarity score map to obtain a defect binary image, and then connected component analysis is performed in combination with parameters such as area to finally determine whether there is a defect according to the area of the connected component. The entire detection process fully combines the advantages of traditional visual algorithms and deep learning techniques, which not only guarantees the speed and real-time performance of the detection, but also improves the accuracy and robustness of defect recognition.

[0176] Figure 11 is a schematic diagram of the principle of the SSIM algorithm provided by the present application, as shown in Figure 11As shown, first, the brightness of the image x and the image y is measured respectively, the average brightness value of each is extracted, and then the brightness contrast calculation (i.e. brightness similarity measure) is performed to measure the difference between the two images in overall brightness. Subsequently, the contrast measurement is calculated for the two images respectively, that is, the variance of the pixel gray scale is counted and the contrast contrast (i.e. contrast measure) is performed to reflect the texture and detail changes of the image. Finally, combining the results of brightness and contrast, the structural contrast module is used to evaluate the structural features in the image (i.e. structure similarity measure), which measures the similarity of the two images in shape, edge and texture pattern.

[0177] The similarity measurement module weights and fuses the results of the brightness similarity measure, the contrast measure and the structure similarity measure to generate the final similarity index, which can more accurately and comprehensively reflect the real similarity between the two images, and is particularly suitable for identifying small but structural defects in silk printing. By applying the SSIM algorithm, the present application can realize high-precision defect detection of silk printing on the surface of household appliances, effectively distinguish real defects from image differences caused by environment or lighting, thereby improving the accuracy and robustness of detection.

[0178] Figure 12 The structure of the silk printing defect detection device provided by the present application is shown in Figure 1. Figure 12 As shown, it mainly includes but is not limited to: The global template matching unit 1210 is used to perform global matching on the image to be detected of the product to be detected by using a global template, and obtain a global matching result.

[0179] The local template positioning unit 1220 is used to determine the initial matching position of the local template in the image to be detected based on the global matching result and the relative position relationship of the local template relative to the global template.

[0180] The search area positioning unit 1230 is used to determine the local search area corresponding to the initial matching position in the image to be detected.

[0181] The detection image positioning unit 1240 is used to determine the accurate matching result of the local template in the image to be detected in the local search area, so as to cut a detection area image from the image to be detected according to the accurate matching result.

[0182] The defect detection processing unit 1250 is used to compare the detection area image with the local template to generate the silk printing defect detection result of the product to be detected.

[0183] The global template and the local template and the relative position relationship of the local template relative to the global template are determined based on the template image of the standard product.

[0184] It should be noted that the silk screen defect detection device provided by the present application can execute the silk screen defect detection method described in any of the above embodiments during specific operation, and the present embodiment will not be described here.

[0185] Figure 13 is a structural schematic diagram of an electronic device provided by the present application, as Figure 13 shown, the electronic device can include a processor 1310, a communications interface 1320, a memory 1330, and a communications bus 1340, wherein the processor 1310, the communications interface 1320, and the memory 1330 complete mutual communication through the communications bus 1340. The processor 1310 can invoke a logical instruction in the memory 1330 to execute a silk screen defect detection method, which includes: performing global matching on a to-be-detected image of a to-be-detected product by using a global template to obtain a global matching result; determining an initial matching position of a local template in the to-be-detected image based on the global matching result and a relative position relationship of the local template relative to the global template; determining a local search area corresponding to the initial matching position in the to-be-detected image; determining an accurate matching result of the local template in the to-be-detected image in the local search area to crop a to-be-detected region image from the to-be-detected image according to the accurate matching result; comparing the to-be-detected region image with the local template to generate a silk screen defect detection result of the to-be-detected product; and the global template, the local template, and the relative position relationship of the local template relative to the global template are determined based on a template image of a standard product.

[0186] In addition, the logical instruction in the memory 1330 described above can be implemented in the form of a software function unit and sold or used as an independent product when the software function unit is used, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0187] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, which, when executed by a computer, enable the computer to perform the screen printing defect detection method provided by any of the above embodiments. The method comprises: performing global matching on a to-be-detected image of a to-be-detected product by using a global template to obtain a global matching result; determining an initial matching position of a local template in the to-be-detected image based on the global matching result and a relative position relationship of the local template relative to the global template; determining a local search region corresponding to the initial matching position in the to-be-detected image; determining an accurate matching result of the local template in the to-be-detected image in the local search region, so as to crop a to-be-detected region image from the to-be-detected image according to the accurate matching result; comparing the to-be-detected region image with the local template to generate a screen printing defect detection result of the to-be-detected product; and the global template, the local template, and the relative position relationship of the local template relative to the global template are determined based on a template image of a standard product.

[0188] In another aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement a screen printing defect detection method provided by any of the above embodiments. The method comprises: performing global matching on a to-be-detected image of a to-be-detected product by using a global template to obtain a global matching result; determining an initial matching position of a local template in the to-be-detected image based on the global matching result and a relative position relationship of the local template relative to the global template; determining a local search region corresponding to the initial matching position in the to-be-detected image; determining an accurate matching result of the local template in the to-be-detected image in the local search region, so as to crop a to-be-detected region image from the to-be-detected image according to the accurate matching result; comparing the to-be-detected region image with the local template to generate a screen printing defect detection result of the to-be-detected product; and the global template, the local template, and the relative position relationship of the local template relative to the global template are determined based on a template image of a standard product.

[0189] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0190] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of detecting a screen printing defect, characterized by, The method comprises the following steps: global matching is performed on a to-be-inspected image of a to-be-inspected product by using a global template to obtain a global matching result; an initial matching position of a local template in the to-be-inspected image is determined based on the global matching result and a relative position relationship of the local template relative to the global template; a local search region corresponding to the initial matching position in the to-be-inspected image is determined; an accurate matching result of the local template in the to-be-inspected image is determined in the local search region, so as to crop a to-be-inspected region image from the to-be-inspected image according to the accurate matching result; the to-be-inspected region image is compared with the local template to generate a silk-print defect detection result of the to-be-inspected product; the global template, the local template and the relative position relationship of the local template relative to the global template are determined based on a template image of a standard product.

2. The method of detecting a silk screen defect according to claim 1, wherein There are a plurality of local templates, and the comparison of the to-be-inspected region image and the local template to generate the silk-print defect detection result of the to-be-inspected product comprises the following steps: a to-be-inspected region image corresponding to each local template is cropped from the to-be-inspected image; each local template is compared with the to-be-inspected region image corresponding thereto to generate a comparison result; the silk-print defect detection result of the to-be-inspected product is generated according to all the comparison results.

3. The method of detecting a silk screen defect according to claim 1, wherein The global template and the local template are determined based on the following steps: a silk-print region containing a silk-print trace in the template image is determined; a plurality of target regions are selected from all the silk-print regions according to a pixel distance between a first center coordinate of each silk-print region and a second center coordinate of the template image; one of all the target regions with the longest edge length is set as the global template; each of the remaining silk-print regions except the global template is set as a local template; shape matching features of the global template and each local template are extracted, and the relative position relationship of the global template and each local template is extracted.

4. The method of detecting a silk screen defect according to claim 3, wherein The global matching result comprises a matching position, a matching rotation angle and a matching scaling ratio of the global template in the to-be-inspected image, and the global matching comprises the following steps: at least one set of candidate matching parameters is searched for and obtained in the to-be-inspected image by using the shape matching features of the global template, each set of candidate matching parameters comprising a matching position, a matching rotation angle, a matching scaling ratio and a corresponding matching degree; a set of optimal matching parameters is determined from the at least one set of candidate matching parameters according to the matching degree of each set of candidate matching parameters; the matching position, the matching rotation angle and the matching scaling ratio contained in the set of optimal matching parameters are determined as the global matching result.

5. The method of detecting a silk screen defect according to claim 1, wherein The local search region corresponding to the initial matching position in the to-be-inspected image is determined by the following steps: the initial matching position is expanded according to a preset scaling ratio with the center of the initial matching position as a reference to obtain the local search region.

6. The method of detecting a silk screen defect according to claim 1, wherein The comparing the to-be-detected region image and the local template to generate a silk print defect detection result of the to-be-detected product comprises: performing structural similarity comparison on the to-be-detected region image and the local template to generate a difference score map; performing binaryzation processing on the difference score map to obtain a difference heat map; performing connected domain analysis on the difference heat map to obtain the silk print defect detection result of the to-be-detected product.

7. The method of detecting a silk screen defect according to claim 6, wherein The performing structural similarity comparison on the to-be-detected region image and the local template to generate a difference score map comprises: synchronously traversing the to-be-detected region image and the local template by using a preset sliding window; determining, at each position in the traversal process of the sliding window, a brightness similarity measure, a structural similarity measure and a contrast measure between a first image region and a second image region; the first image region is an image on the to-be-detected region image covered by the sliding window, and the second image region is an image on the local template covered by the sliding window; determining, according to the brightness similarity measure, the structural similarity measure and the contrast measure, a local structural similarity score of a center pixel point of each sliding window; collecting the local structural similarity scores of all the center pixel points in the traversal process to obtain the difference score map.

8. The method of detecting a silk screen defect according to claim 7, wherein The brightness similarity measure is determined based on pixel average values of the first image region and the second image region, the structural similarity measure is determined based on pixel covariance between the first image region and the second image region, and the contrast measure is determined based on pixel variances of the first image region and the second image region.

9. The method of detecting a silk screen defect according to claim 6, wherein The performing connected domain analysis on the difference heat map to obtain the silk print defect detection result of the to-be-detected product comprises: identifying one or more connected domains in the difference heat map; respectively calculating areas of each of the connected domains; comparing the area of each of the connected domains with a first preset area threshold; determining, as a potential defect region, a connected domain whose area is greater than the first preset area threshold; performing defect feature analysis on each of the potential defect regions to generate the silk print defect detection result.

10. The method of detecting a silk screen defect according to claim 9, wherein The performing defect feature analysis on each of the potential defect regions to generate the silk print defect detection result comprises: respectively extracting a deep learning feature of each of the potential defect regions and a corresponding region of each of the potential defect regions in the local template; comparing the deep learning feature of each of the potential defect regions with the deep learning feature of the corresponding region to generate an anomaly score map of each of the potential defect regions; determining the silk print defect detection result according to all the anomaly score maps.

11. The method of detecting a silk screen defect according to claim 10, wherein The deep learning feature is extracted based on a visual transformer network model; The respectively extracting a deep learning feature of each of the potential defect regions and a corresponding region of each of the potential defect regions in the local template comprises: The potential defect area and the corresponding area are respectively input into the visual converter network model, and deep learning features output by the visual converter network model are obtained.

12. The method of detecting a silk screen defect according to claim 11, wherein, The visual converter network model specifically performs the following operations: After the input is divided into multiple image blocks, a set of multi-level local features is extracted from each image block at multiple preset levels; At each preset level, adaptive pooling processing of multiple different scales is performed on the extracted local features; The local features after the multiple different scale pooling processing are fused to form a set of multi-dimensional local features containing multi-level information and multi-aggregation degree information, as the deep learning features of each image block.

13. The method of detecting a silk screen defect according to claim 10, wherein The comparison of the deep learning features of the potential defect area and the deep learning features of the corresponding area generates an anomaly score map of each potential defect area, including: Traverse each image block in the potential defect area, and calculate the feature distance between the deep learning features of any image block and the deep learning features of all image blocks in the corresponding area respectively; The smallest feature distance is determined as the anomaly score of the any image block; Combine the anomaly scores of all image blocks to form the anomaly score map.

14. The method of detecting a silk screen defect according to claim 13, wherein, In the case that at least two reference templates are provided for the potential defect area, and each reference template contains a corresponding area of the potential defect area, the determination of the smallest feature distance as the anomaly score of the any image block includes: Calculate the feature distance between the deep learning features of the any image block and the deep learning features of all image blocks in each corresponding area respectively, to obtain multiple minimum feature distances; From the multiple minimum feature distances, select a preset proportion of the smallest minimum feature distances to calculate the average feature distance; The average feature distance is determined as the anomaly score of the any image block.

15. The method of detecting a silk screen defect according to claim 10, wherein The determination of the screen printing defect detection result according to all anomaly score maps includes: Threshold segmentation is performed on each anomaly score map to generate a defect binary image; If it is determined that there is any connected domain with an area greater than a second preset area threshold, it is determined that the screen printing defect detection result is that there is a screen printing defect.

16. A screen defect detection apparatus, characterized by comprising: It includes: A global template matching unit is configured to perform global matching on a to-be-detected image of a to-be-detected product by using a global template, and obtain a global matching result; A local template positioning unit is configured to determine an initial matching position of a local template in the to-be-detected image based on a relative position relationship of the local template with respect to the global template and the global matching result; A search region positioning unit is configured to determine a local search region corresponding to the initial matching position in the to-be-detected image; A detection image positioning unit is configured to determine an accurate matching result of the local template in the to-be-detected image in the local search region, and to crop a to-be-detected region image from the to-be-detected image according to the accurate matching result; a defect detection processing unit configured to compare the to-be-inspected area image with the local template to generate a silk-print defect detection result of the to-be-inspected product; the global template and the local template and the relative position relationship of the local template relative to the global template are determined based on a template image of a standard product.

17. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor, when executing the computer program, implements the silk-print defect detection method according to any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the silk-print defect detection method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the silk-print defect detection method according to any one of claims 1 to 15.

Images