Image processing-based needle mark detection method, device, equipment and medium

Abstract

The application discloses a needle mark detection method and device based on image processing, equipment and medium, and relates to the technical field of wafer testing. The method comprises the following steps: converting a to-be-processed image comprising a test contact point to be subjected to needle mark detection into a first gray-scale image; separating a single-color image corresponding to a predetermined color component from the to-be-processed image; generating a first binary image according to a first rule based on the first gray-scale image; wherein the first rule comprises: a pixel point with a pixel value not exceeding a corresponding pixel value threshold is white; generating a second binary image according to a second rule based on the single-color image; wherein the second rule comprises: a pixel point with a pixel value exceeding a corresponding pixel value threshold is white; merging the first binary image and the second binary image to obtain a merged binary image; and determining a needle mark detection result of the test contact point in the to-be-processed image based on the merged binary image. The embodiments of the present disclosure can be compatible with the detection of needle marks with different characteristics.

Description

Technical Field

[0001] This disclosure relates to the field of wafer testing technology, and in particular to a method, apparatus, device and medium for detecting pin marks based on image processing. Background Technology

[0002] After wafer manufacturing is completed, wafer testing is required. During testing, probes can be inserted into test pads on the wafer for positioning. After the test is completed, pin marks will be left on the test pads. Generally, pin mark detection methods are used to detect the position of the pin marks and to judge the wafer testing quality based on the detection results.

[0003] It should be noted that the quality of pin mark detection has a significant impact on the quality of wafer testing, and how to ensure the quality of pin mark detection is a problem worthy of attention for those skilled in the art. Summary of the Invention

[0004] This disclosure is made to ensure the quality of pin mark detection. Embodiments of this disclosure provide an image processing-based pin mark detection method, apparatus, device, and medium.

[0005] According to one aspect of the present disclosure, a needle mark detection method based on image processing is provided, comprising:

[0006] The image to be processed, including the test contact points to be detected by pin mark detection, is converted into a first grayscale image;

[0007] Separate a monochrome image corresponding to a predetermined color component from the image to be processed;

[0008] Based on the first grayscale image, a first binary image is generated according to a first rule; wherein, the first rule includes: pixels whose pixel values ​​do not exceed the corresponding pixel value threshold are white;

[0009] Based on the monochrome image, a second binary image is generated according to a second rule; wherein the second rule includes: pixels whose pixel values ​​exceed the corresponding pixel value threshold are white;

[0010] The first binary image and the second binary image are merged to obtain a merged binary image;

[0011] Based on the merged binary image, the needle mark detection result of the test joint in the image to be processed is determined.

[0012] According to another aspect of the present disclosure, an image processing-based needle mark detection device is provided, comprising:

[0013] A conversion module is used to convert the image to be processed, including the test contact to be detected by pin mark detection, into a first grayscale image;

[0014] A separation module is used to separate a monochrome image corresponding to a predetermined color component from the image to be processed;

[0015] A first generation module is configured to generate a first binary image based on the first grayscale image according to a first rule; wherein the first rule includes: pixels whose pixel values ​​do not exceed the corresponding pixel value threshold are white;

[0016] The second generation module is used to generate a second binary image based on the monochrome image according to a second rule; wherein the second rule includes: pixels whose pixel values ​​exceed the corresponding pixel value threshold are white;

[0017] The merging module is used to merge the first binary image and the second binary image to obtain a merged binary image;

[0018] The first determining module is used to determine the pin mark detection result of the test joint in the image to be processed based on the merged binary image.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the above-described image processing-based needle mark detection method.

[0020] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0021] processor;

[0022] Memory used to store the processor's executable instructions;

[0023] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described image processing-based needle mark detection method.

[0024] The image processing-based pin mark detection method, apparatus, device, and medium, for a to-be-processed image including test contacts to be detected for pin marks, can, on the one hand, obtain a first grayscale image that clearly shows dark pin marks through grayscale processing, and on the other hand, obtain a monochrome image that clearly shows bright pin marks through separation processing. For the first grayscale image, a first binary image for highlighting the dark component can be generated according to a first rule, and for the monochrome image, a second binary image for highlighting the bright component can be generated according to a second rule. By merging the first and second binary images, the dark and bright components can be merged into a single binary image (i.e., a merged binary image). The merged binary image is used to determine the pin mark detection result of the test contacts in the to-be-processed image. Whether the pin marks are dark or bright, they can be effectively detected. Thus, even if the test contacts have various materials, resulting in significant differences in the brightness of the pin marks, pin mark detection can be achieved efficiently and reliably. Therefore, the embodiments of this disclosure are compatible with the detection of pin marks with different characteristics, thereby improving the quality of pin mark detection.

[0025] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0026] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0027] Figure 1 This is a schematic flowchart of an image processing-based needle mark detection method provided in an exemplary embodiment of this disclosure.

[0028] Figure 2 This is a schematic diagram of a first grayscale image in an exemplary embodiment of this disclosure.

[0029] Figure 3 This is a schematic diagram of a monochrome image in an exemplary embodiment of this disclosure.

[0030] Figure 4 This is a schematic diagram of merging binary images in an exemplary embodiment of this disclosure.

[0031] Figure 5 This is a flowchart illustrating a needle mark detection method based on image processing provided in another exemplary embodiment of this disclosure.

[0032] Figure 6 This is a schematic diagram of a first filled image in an exemplary embodiment of this disclosure.

[0033] Figure 7 This is a schematic flowchart of an image processing-based needle mark detection method provided in another exemplary embodiment of this disclosure.

[0034] Figure 8 This is a schematic diagram of a second filled image in an exemplary embodiment of this disclosure.

[0035] Figure 9 This is a schematic diagram of a grayscale image corresponding to a grain surface image in an exemplary embodiment of this disclosure.

[0036] Figure 10 This is a schematic diagram of a grayscale image corresponding to a template image in an exemplary embodiment of this disclosure.

[0037] Figure 11 This is a flowchart illustrating the method for generating test contact positioning information in an exemplary embodiment of this disclosure.

[0038] Figure 12 This is a schematic diagram of a cropped image extracted from a first grayscale image in an exemplary embodiment of this disclosure.

[0039] Figure 13 This is a schematic diagram of a third binary image before the opening operation is performed in an exemplary embodiment of this disclosure.

[0040] Figure 14 This is a schematic diagram of a third binary image after the opening operation is performed in an exemplary embodiment of this disclosure.

[0041] Figure 15 This is a schematic diagram of a first maximum connected region in an exemplary embodiment of this disclosure.

[0042] Figure 16 This is a schematic diagram of a first minimum rectangle in an exemplary embodiment of this disclosure.

[0043] Figure 17 This is a schematic diagram of a water-filled image in an exemplary embodiment of this disclosure.

[0044] Figure 18 This is a schematic diagram of a test contact positioning mask in an exemplary embodiment of this disclosure.

[0045] Figure 19 This is a schematic flowchart illustrating the method of generating a template image in an exemplary embodiment of this disclosure.

[0046] Figure 20 This is a schematic diagram of a second grayscale image in an exemplary embodiment of this disclosure.

[0047] Figure 21This is a schematic diagram of a fourth binary image before deburring in an exemplary embodiment of this disclosure.

[0048] Figure 22 This is a schematic diagram of a fourth binary image after deburring in an exemplary embodiment of this disclosure.

[0049] Figure 23 This is a schematic diagram of the convex hull in an exemplary embodiment of this disclosure.

[0050] Figure 24 This is a schematic diagram of the second largest connected region in an exemplary embodiment of this disclosure.

[0051] Figure 25 This is a schematic diagram of the Pad convex hull-filled image in an exemplary embodiment of this disclosure.

[0052] Figure 26 This is a schematic diagram of the Pad outline filling image in an exemplary embodiment of this disclosure.

[0053] Figure 27 This is a schematic diagram of a binary image with edge defects in an exemplary embodiment of this disclosure.

[0054] Figure 28 This is a schematic diagram of a grayscale image corresponding to a reference image after the edge defect area has been repaired, in an exemplary embodiment of this disclosure.

[0055] Figure 29 This is a schematic diagram of the expansion region corresponding to the convex hull in an exemplary embodiment of this disclosure.

[0056] Figure 30 This is a schematic flowchart of an image processing-based needle mark detection method provided in another exemplary embodiment of this disclosure.

[0057] Figure 31 This is a flowchart illustrating a needle-pricking risk warning method in an exemplary embodiment of this disclosure.

[0058] Figure 32 This is a schematic diagram of the grayscale image corresponding to the needle mark detection result in an exemplary embodiment of this disclosure.

[0059] Figure 33 This is a schematic diagram of the Pad template registration stage in an exemplary embodiment of this disclosure.

[0060] Figure 34 This is a flowchart illustrating the needle mark detection stage in an exemplary embodiment of this disclosure.

[0061] Figure 35 This is a schematic diagram of the structure of an image processing-based needle mark detection device provided in an exemplary embodiment of the present disclosure.

[0062] Figure 36 This is a schematic diagram of the modules involved in generating a template image in an exemplary embodiment of this disclosure.

[0063] Figure 37 This is a schematic diagram of the module involved in the needle puncture risk warning in an exemplary embodiment of this disclosure.

[0064] Figure 38 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0065] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0066] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0067] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent specific technical meanings or necessary logical sequences.

[0068] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0069] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0070] In this disclosure, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " in this disclosure indicates that the preceding and following related objects have an "or" relationship.

[0071] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0072] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0073] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0074] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0075] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0076] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0077] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0078] Exemplary methods

[0079] Figure 1 This is a schematic flowchart of an image processing-based needle mark detection method provided in an exemplary embodiment of this disclosure. Figure 1 The method shown may include steps 110, 120, 130, 140, 150 and 160, each of which will be explained below.

[0080] Step 110: Convert the image to be processed, including the test contact to be detected by pin mark detection, into a first grayscale image.

[0081] Optionally, the image to be processed can be a color image, including but not limited to RGB images, YUV images, etc.; where R in RGB represents red, G represents green, and B represents blue; and Y in YUV represents luminance, and U and V represent chrominance.

[0082] Optionally, the number of test points to be detected in the image to be processed can be one, two, or more, and will not be listed here.

[0083] For ease of understanding, the embodiments of this disclosure are all illustrated using the case where the image to be processed is an RGB image and the number of test points in the image to be detected for pin marks is one. This test point in the image to be detected for pin marks may also be referred to as the test point to be detected below.

[0084] In step 110, the image to be processed can be converted to grayscale to obtain a first grayscale image; wherein, the first grayscale image and the image to be processed can have the same image size. In one example, the first grayscale image can be as follows: Figure 2 As shown.

[0085] Step 120: Separate the monochrome image corresponding to the predetermined color component from the image to be processed.

[0086] In step 120, the three channels of the image to be processed can be separated to obtain a monochrome image corresponding to a predetermined color component; wherein, the monochrome image and the image to be processed can have the same image size.

[0087] Optionally, the predetermined color component may include a blue component. Thus, the monochrome image corresponding to the predetermined color component can also be called a blue image. In one example, the monochrome image corresponding to the predetermined color component may be as follows: Figure 3 As shown.

[0088] Of course, the predefined color component is not limited to the blue component; the predefined color component can also include the red component or the green component.

[0089] Step 130: Based on the first grayscale image, generate a first binary image according to a first rule; wherein the first rule includes: pixels whose pixel values ​​do not exceed the corresponding pixel value threshold are white.

[0090] In step 130, a first image to be binarized can be determined based on the first grayscale image, and the first image to be binarized can be converted into a first binary image according to a first rule; wherein, the image size of the first image to be binarized and the first binary image can be the same. Here, the pixels in the first image to be binarized can correspond one-to-one with the pixels in the first binary image. For example, the pixel located in the i-th row and j-th column of the first image to be binarized can correspond to the pixel located in the i-th row and j-th column of the first binary image.

[0091] Optionally, the first grayscale image can be directly used as the first image to be binarized; or, the first grayscale image can be subjected to predetermined processing, and the result of the predetermined processing can be used as the first image to be binarized. For clarity, examples of the predetermined processing will be given below.

[0092] Optionally, a first threshold segmentation algorithm can be used to determine the pixel value threshold corresponding to each pixel in the first image to be binarized. For example, a sliding window can be used to slide across the first image to be binarized, and the pixel values ​​of the pixels in the area covered by the sliding window can be used to determine the pixel value threshold corresponding to the respective pixel. In one example, the size of the sliding window is 3×3. When the sliding window slides to a certain position, the average pixel value of the 8 pixels (excluding the central pixel) covered by the sliding window can be used as the pixel value threshold corresponding to the central pixel. For pixels in the first image to be binarized whose pixel values ​​do not exceed the corresponding pixel value threshold, the pixel value of the corresponding pixel in the first binary image can be set to 255. For pixels in the first image to be binarized whose pixel values ​​exceed the corresponding pixel value threshold, the pixel value of the corresponding pixel in the first binary image can be set to 0. Thus, the first binary image can be generated according to the first rule.

[0093] As described above, the first threshold segmentation algorithm processes the dark components in the image as white and the bright components as black. Therefore, the first threshold segmentation algorithm can be an inverse binary threshold segmentation algorithm, such as a local adaptive inverse binary threshold segmentation algorithm.

[0094] Step 140: Based on the monochrome image, generate a second binary image according to the second rule; wherein the second rule includes: pixels whose pixel values ​​exceed the corresponding pixel value threshold are white.

[0095] In step 140, a second image to be binarized can be determined based on the monochrome image, and the second image to be binarized can be converted into a second binary image according to the second rule; wherein the image size of the second image to be binarized and the second binary image can be the same. Here, the pixels in the second image to be binarized can correspond one-to-one with the pixels in the second binary image. For example, the pixel located in the i-th row and j-th column of the second image to be binarized can correspond to the pixel located in the i-th row and j-th column of the second binary image.

[0096] Optionally, the monochrome image can be directly used as the second image to be binarized; or, the monochrome image can be subjected to predetermined processing, and the result of the predetermined processing can be used as the second image to be binarized.

[0097] Optionally, a second threshold segmentation algorithm can be used to determine the pixel value threshold corresponding to each pixel in the second image to be binarized. The specific determination method is the same as described above regarding the method for determining the pixel value threshold corresponding to each pixel in the first image to be binarized, and will not be repeated here. For pixels in the second image to be binarized whose pixel value exceeds the corresponding pixel value threshold, the pixel value of the corresponding pixel in the second binary image can be set to 255. For pixels in the second image to be binarized whose pixel value does not exceed the corresponding pixel value threshold, the pixel value of the corresponding pixel in the second binary image can be set to 0. Thus, the second binary image can be generated according to the second rule.

[0098] As described above, the function of the second threshold segmentation algorithm is to process the bright components in the image as white and the dark components in the image as black. Therefore, the second threshold segmentation algorithm can be a binary threshold segmentation algorithm, such as a local adaptive binary threshold segmentation algorithm.

[0099] Step 150: Merge the first binary image and the second binary image to obtain a merged binary image.

[0100] It should be noted that the image sizes of the first binary image, the second binary image, and the merged binary image can be the same. The pixels in the first binary image can correspond one-to-one with the pixels in the second binary image, the pixels in the first binary image can correspond one-to-one with the pixels in the merged binary image, and the pixels in the second binary image can correspond one-to-one with the pixels in the merged binary image.

[0101] Assume that the pixel located in the i-th row and j-th column of the first binary image is pixel O1, and the pixel value of O1 is S1; the pixel located in the i-th row and j-th column of the second binary image is pixel O2, and the pixel value of O2 is S2; and the pixel located in the i-th row and j-th column of the merged binary image is O3, and the pixel value of O3 is S3. Then S3 can be calculated according to the following rules:

[0102]

[0103] In this way, by adding the corresponding pixel values ​​of the first binary image and the second binary image, and referring to the relationship between the sum and 255, the pixel values ​​of the pixels in the merged binary image can be determined efficiently and reliably. Therefore, by merging binary images, a merged binary image can be generated. In one example, merging binary images can be done as follows: Figure 4 As shown, by Figure 4 It can be seen that merging binary images can reflect information related to needle marks, such as the shape, size, and position of the needle marks in the image to be processed. Merging binary images can also be called needle mark binary images.

[0104] Step 160: Based on the merged binary image, determine the pin mark detection result of the test joint in the image to be processed.

[0105] In step 160, by referring to the pin mark information reflected in the merged binary image, the pin mark detection result of the test point in the image to be processed can be determined. The pin mark detection result may include the shape, size, and position of the pin mark on the test point to be detected.

[0106] In the embodiments of this disclosure, for a to-be-processed image including test contacts to be detected by pin marks, on the one hand, a first grayscale image that clearly shows dark pin marks can be obtained through grayscale processing; on the other hand, a monochrome image that clearly shows bright pin marks can be obtained through separation processing. For the first grayscale image, a first binary image for highlighting the dark component can be generated according to a first rule; for the monochrome image, a second binary image for highlighting the bright component can be generated according to a second rule. By merging the first binary image and the second binary image, the dark component and the bright component can be merged into a single binary image (i.e., merged binary image). The merged binary image is used to determine the pin mark detection result of the test contacts in the to-be-processed image. Whether the pin marks are dark or bright, they can be effectively detected. In this way, even if the test contacts have various materials, resulting in significant differences in the brightness of the pin marks, pin mark detection can be achieved efficiently and reliably. Therefore, the embodiments of this disclosure are compatible with the detection of pin marks with different characteristics, thereby improving the quality of pin mark detection.

[0107] It should be noted that monochrome images, including blue images, are best at rendering shiny needle marks. Therefore, in this case, they can more effectively detect needle marks with different features.

[0108] exist Figure 1 Based on the illustrated embodiments, as Figure 5 As shown, the method provided in the embodiments of this disclosure further includes step 125.

[0109] Step 125: Determine the test contact location information corresponding to the image to be processed.

[0110] It should be noted that the test node location information corresponding to the image to be processed can be used to indicate the area occupied by the test node to be detected in the image to be processed. Optionally, the test node location information can be in the form of coordinates or in the form of a location mask.

[0111] Optionally, step 125 may be performed after steps 110 and 120, or before steps 110 and 120, or between steps 110 and 120.

[0112] Step 130 includes steps 1301, 1303, 1305 and 1307.

[0113] Step 1301: Based on the test contact location information, determine the test contact area in the first grayscale image.

[0114] In step 1301, the area indicated by the test contact location information in the image to be processed can be mapped to the first grayscale image to determine the first mapped area in the first grayscale image. Based on the first mapped area, the test contact area in the first grayscale image can be determined. The determined test contact area can be considered as the area occupied by the test contact to be detected in the first grayscale image.

[0115] Optionally, the mapping between a region (e.g., region A) in the first grayscale image and another region (e.g., region B) in the image to be processed can be understood as follows: region A includes pixels that correspond one-to-one with the pixels in region B. In one example, if region B includes the pixels of the 3×3 region in the upper left corner of the image to be processed, then region A may include the pixels of the 3×3 region in the upper left corner of the first grayscale image. It should be noted that the region mapping methods between different images can all refer to the explanation in this paragraph, and will not be elaborated on further below.

[0116] Optionally, the first mapping region can be directly used as the test node region; or, based on experience, the first mapping region can be modified (e.g., by slightly scaling the first mapping region) and the modified result of the first mapping region can be used as the test node region.

[0117] Step 1303: Determine the first fill pixel value based on the pixel values ​​of the pixels in the test junction region of the first grayscale image.

[0118] In step 1303, the pixel values ​​of all pixels (assuming N pixels) in the test junction area of ​​the first grayscale image can be determined to obtain N pixel values, and the mean value is calculated based on the N pixel values ​​to obtain the first fill pixel value.

[0119] Optionally, the average of the N pixel values ​​can be directly calculated, and the calculated average pixel value can be used as the first fill pixel value; or, some pixel values ​​can be filtered out from the N pixel values ​​(for example, the largest and smallest pixel values ​​can be filtered out), the average of the remaining pixel values ​​can be calculated, and the calculated average pixel value can be used as the first fill pixel value.

[0120] Step 1305: Using the first fill pixel value, fill the area in the first grayscale image except for the test junction area to obtain the first filled image.

[0121] Optionally, the test contact area in the first grayscale image can also be referred to as the Pad area in the first grayscale image, and the area in the first grayscale image other than the test contact can also be referred to as the outside of the Pad area in the first grayscale image.

[0122] In step 1305, the pixel values ​​of all pixels in the region of the first grayscale image excluding the test nodes can be updated to the first fill pixel values, so as to obtain the first fill image corresponding to the first grayscale image through mean fill processing. In one example, the first fill image can be as follows: Figure 6 As shown.

[0123] Step 1307: According to the first rule, convert the first filled image into a first binary image.

[0124] It should be noted that mean filling can be considered as a predetermined process as described above, and the first filled image can be considered as a predetermined processing result of the first grayscale image as described above. For a detailed implementation of converting the first filled image into a first binary image according to the first rule, please refer to [reference needed]. Figure 1 The relevant descriptions in the illustrated embodiments are sufficient and will not be repeated here. Of course, the predetermined processing is not limited to this; for example, the predetermined processing may also include filtering processing, etc.

[0125] In the embodiments of this disclosure, the Pad region in the first grayscale image can be determined through a simple mapping operation using the test node positioning information corresponding to the image to be processed. Using the pixel values ​​of the pixels in the Pad region, and through simple calculation logic such as averaging, the first filling pixel value can be determined efficiently and reliably for filling the outside of the Pad region in the first grayscale image. Thus, when determining the pixel value threshold using the first threshold segmentation algorithm, if the sliding window slides to a position close to the edge of the Pad region, resulting in some pixels in the sliding window being located in the Pad region and others outside, since the outside of the Pad region has already been filled with the first filling pixel value, the pixel values ​​outside the Pad region will not differ too much from the pixel values ​​in the Pad region. This helps to avoid the darkened edges of the Pad region negatively impacting the determination of the pixel value threshold, thereby ensuring the accuracy and reliability of the determined pixel value threshold and improving the image segmentation effect.

[0126] exist Figure 1 Based on the illustrated embodiments, as Figure 7 As shown, the method provided in the embodiments of this disclosure further includes step 125.

[0127] Step 125: Determine the test contact location information corresponding to the image to be processed.

[0128] Step 140 includes:

[0129] Step 1401: Based on the test contact location information, determine the test contact area in the monochrome image;

[0130] Step 1403: Determine the second fill pixel value based on the pixel value of the pixel in the test junction region of the monochrome image;

[0131] Step 1405: Using the second fill pixel value, fill the area in the monochrome image except for the test junction area to obtain the second fill image;

[0132] Step 1407: According to the second rule, convert the second filled image into a second binary image.

[0133] It should be noted that, Figure 7 The specific implementation methods of the illustrated embodiments are as follows: Figure 5 The specific implementation methods of the illustrated embodiments are similar, with the main difference being that... Figure 5 The illustrated embodiment fills the area outside the Pad region in the first grayscale image, and uses the resulting filled image to generate the first binary image. Figure 7 The illustrated embodiment fills the area outside the pad region in a monochrome image, and uses the resulting second filled image to generate a second binary image. Further details are omitted here. Figure 7 The specific implementation of the illustrated embodiment will be described in detail below. In one example, the second filling image can be as follows: Figure 8 As shown.

[0134] and Figure 5 The embodiments shown are similar, employing, as Figure 7 The embodiment shown can ensure the accuracy and reliability of the determined pixel value threshold when using the second threshold segmentation algorithm to determine the pixel value threshold, thereby improving the image segmentation effect.

[0135] In one optional example, the method provided by embodiments of this disclosure further includes:

[0136] The image to be processed is extracted from the first region of the grain surface image.

[0137] In one example, the grain surface image can be Figure 9 The grayscale image shown corresponds to the color image.

[0138] Step 125 includes:

[0139] Based on the template image corresponding to the first region, the test contact positioning information corresponding to the image to be processed is generated; wherein, the template image includes: an image used to present the size of the template test contact corresponding to the first region and the background.

[0140] The test junctions of different dies on a wafer are usually arranged in the same way. Therefore, a certain die can be used as a template die in advance. Based on the surface image of the template die, the test junctions included in it can be determined. Each of these test junctions can be used as a template test junction. For each template test junction, the distribution area of ​​the template test junction in the surface image of the template die can be manually selected. When selecting, a certain amount of die background can be left at the four edges of the template test junction. In this way, a correspondence between multiple (e.g., M) template test junctions and multiple (e.g., M) distribution areas can be formed.

[0141] In one example, M is 4, and the IDs of the four template test nodes are ID1, ID2, ID3, and ID4, respectively. The resulting correspondence can be represented as follows:

[0142] ID Distribution area ID1 P1 ID2 P2 ID3 P3 ID4 P4

[0143] Table 1

[0144] For each of the M distribution regions, a corresponding template image can be generated through Pad template registration. The template image is used to present the size of the template test point and the background corresponding to that distribution region.

[0145] Similar to the surface image of the template grain, the grain surface image in the embodiments of this disclosure may include four distribution regions, namely P1', P2', P3', and P4'; wherein, P1' can be mapped to P1, P2' can be mapped to P2, P3' can be mapped to P3, and P4' can be mapped to P4. Optionally, the first region in the grain surface image can be any one of the distribution regions P1', P2', P3', and P4'.

[0146] Assuming the first region is P1', since P1' maps to P1, and P1 corresponds to ID1, the template image corresponding to the first region can be considered the template image corresponding to P1. In one example, the template image corresponding to the first region could be... Figure 10 The grayscale image shown corresponds to the color image.

[0147] In one optional implementation, based on the template image corresponding to the first region, test contact positioning information corresponding to the image to be processed is generated, including... Figure 11 Steps 1110, 1120, 1130, 1140, 1150, and 1160 are shown below, and each step will be explained separately.

[0148] Step 1110: Determine the second region in the image to be processed that matches the template image (the template image mentioned here and below refers to the template image corresponding to the first region).

[0149] Optionally, the image size of the image to be processed can be larger than the image size of the template image. In step 1110, template matching can be performed between the image to be processed and the template image. The matching method includes, but is not limited to, shape-based matching methods, correlation-based matching methods, gray-scale-based matching methods, etc. Through template matching, the region in the image to be processed that is most similar to the template image can be found, and the found region can be used as the second region in step 1110.

[0150] Step 1120: Based on the second region, extract a cropped image from the first grayscale image, including the test contact to be detected by pin mark detection.

[0151] In step 1120, the second region can be mapped onto the first grayscale image to determine the second mapped region in the first grayscale image. Based on the second mapped region, the first grayscale image can be cropped to obtain a cropped image including the test contact to be detected by pin mark detection.

[0152] Optionally, a partial image of the second mapped region can be directly cropped as the cropped image; alternatively, the second mapped region can be modified empirically (e.g., by slightly scaling the second mapped region), and a partial image of the modified second mapped region can be cropped as the cropped image. In one example, the cropped image can be as follows: Figure 12 As shown.

[0153] Step 1130: According to the first rule, convert the cropped image into a third binary image.

[0154] It should be noted that the specific implementation of step 1130 can be referred to the above description of the method of converting the first image to be binarized into a first binary image, and will not be repeated here.

[0155] Step 1140: Determine the first largest connected component in the third binary image.

[0156] It is understandable that a connected component typically refers to a region in a binary image where white pixels are connected, and the pixel value of each pixel in the connected component can be 255. In step 1140, the number of white pixels in the third binary image can be counted, and the largest connected component can be calculated based on the statistical results, thus obtaining the first largest connected component.

[0157] Optionally, an opening operation (in morphology, opening refers to erosion followed by dilation) can be performed on the third binary image to eliminate some noise, and deburring can be performed to smooth the test junction edges (for example, if only one pixel in a continuous set of eight pixels has a value of 0, and the remaining pixels have a value of 255, then the pixel with a value of 0 is updated to 255). Then, the first largest connected component in the third binary image can be determined. In one example, the third binary image before the opening operation can be as follows: Figure 13 As shown, the third binary image after the opening operation can be obtained as follows: Figure 14 As shown, the first maximum connected component determined for the third binary image after opening and descrambling can be as follows: Figure 15 The largest white area is shown in the middle.

[0158] Step 1150: Determine the first minimum rectangle that can enclose the first maximum connected region.

[0159] In step 1150, the minimum rotational bounding moment of the first maximum connected component can be determined, and this minimum rotational bounding moment can be used as the first minimum rectangle. Optionally, each of the four sides of the first minimum rectangle may or may not be parallel to the corresponding side of the four sides of the third binary image. In one example, the first minimum rectangle can be... Figure 16 A rectangle located inside the image.

[0160] Step 1160: Based on the first minimum rectangle, generate the test node positioning information corresponding to the image to be processed.

[0161] In one alternative implementation, step 1160 includes:

[0162] The edges of the region enclosed by the first minimum rectangle are clipped, and the un-clipped areas within the region enclosed by the first minimum rectangle are determined.

[0163] The uncropped region is mapped to an image with the same size as the image to be processed to generate a test node positioning mask map corresponding to the image to be processed.

[0164] Optionally, any pixel within the region enclosed by the first minimum rectangle can be selected as the seed pixel for water immersion filling to obtain... Figure 17 The image shown is filled with water, and then... Figure 17 The white areas in the water-filled image shown are subjected to erosion to crop the edges. Alternatively, a pure black image with the same size as the image to be processed can be prepared beforehand, and the uncropped areas within the region enclosed by the first minimum rectangle can be mapped to the pure black image to determine a third mapped region in the pure black image. The pixel values ​​of the pixels in the third mapped region can all be set to 255, thereby forming... Figure 18 The image shown is a test contact positioning mask corresponding to the image to be processed. The test contact positioning mask can also be called a Pad positioning mask.

[0165] In this implementation, combining edge cropping and mapping operations enables efficient and reliable generation of a test contact location mask. This mask effectively indicates the area occupied by the test contact to be detected in the image being processed, and can serve as the test contact location information mentioned above. Furthermore, the edge cropping operation helps reduce the size of the area indicated by the test contact location mask, preventing black borders at the test contact edges from negatively impacting pin mark detection.

[0166] Of course, the implementation of step 1160 is not limited to this. For example, after determining the first minimum rectangle, the first minimum rectangle can be reduced by a certain proportion (which can be predetermined based on experience) to obtain a reduced rectangle, and the reduced rectangle can be mapped to an image with the same image size as the image to be processed to realize the generation of the test contact positioning mask image.

[0167] Furthermore, the specific implementation method for generating test contact location information corresponding to the image to be processed based on the template image corresponding to the first region is not limited to... Figure 11In the illustrated implementation, for example, a second region in the image to be processed that matches the template image can be determined. Based on the second region, a cropped image including the test contact point to be detected for pin marks is extracted from the monochrome image. By using the obtained cropped image, a test contact point positioning mask map corresponding to the image to be processed is determined.

[0168] In the embodiments of this disclosure, by cropping a certain area of ​​the grain surface image, the image to be processed can be obtained efficiently and reliably. Then, using the corresponding template image generated in the Pad template registration stage as guiding information, test contact positioning information in the form of a positioning mask map can be generated efficiently and reliably.

[0169] In an optional example, such as Figure 19 As shown, the method provided in the embodiments of this disclosure further includes steps 1910, 1920, 1930 and 1940, which are described below.

[0170] Step 1910: Obtain a reference image including the template test junction corresponding to the first region.

[0171] Optionally, a local image of the distribution area corresponding to the first region in the surface image of the template grain can be used as a reference image including the template test junction corresponding to the first region. For example, if the first region is P1', then a local image of P1 in the surface image of the template grain can be used as a reference image.

[0172] Step 1920: Determine the edge defect area of ​​the template test joint in the reference image.

[0173] In one alternative implementation, step 1920 includes:

[0174] Convert the reference image to a second grayscale image;

[0175] According to the first rule, the second grayscale image is converted into the fourth binary image;

[0176] Determine the second largest connected component in the fourth binary image;

[0177] Determine the convex hull corresponding to the second largest connected component;

[0178] Based on the second largest connected component and the convex hull, the edge defect region of the template test node in the reference image is determined.

[0179] Optionally, the reference image can be converted to grayscale to obtain a second grayscale image; wherein the second grayscale image and the reference image can have the same image size. In one example, the second grayscale image is as follows: Figure 20 As shown.

[0180] It should be noted that, according to the first rule, the specific implementation method for converting the second grayscale image into the fourth binary image can refer to the above description of the method for converting the first image to be binarized into the first binary image. The specific implementation method for determining the second largest connected component in the fourth binary image can refer to the above description of the method for determining the first largest connected component in the third binary image, and will not be repeated here.

[0181] Optionally, the fourth binary image can be deburred first to smooth the test node edges, and then the deburred fourth binary image can be used to determine the second maximum connected component. In one example, the fourth binary image before deburring is as follows: Figure 21 As shown, the fourth binary image after deburring is as follows: Figure 22 As shown.

[0182] It should be noted that, as a concept in computer graphics, the convex hull can be defined as follows: Assuming S is any subset of Euclidean space, the smallest convex set including S can be called the convex hull of S. Following this definition, the smallest convex set including the second largest connected component can be considered the convex hull corresponding to that component.

[0183] In one alternative implementation, the edge defect region of the template test node in the reference image is determined based on the second maximum connected component and the convex hull (the convex hull mentioned here and below refers to the convex hull corresponding to the second maximum connected component), including:

[0184] Determine the difference between the region enclosed by the convex hull and the region enclosed by the outer contour of the second largest connected region;

[0185] The difference set is mapped to the reference image to determine the edge defect area of ​​the template test junction in the reference image.

[0186] In one example, the region enclosed by the convex hull can be as follows: Figure 23 As shown in the image, the region enclosed by the white border inside the image can be represented as follows: Figure 24 The area enclosed by a white border inside the image is shown.

[0187] Optionally, the pixel values ​​of all pixels in the region enclosed by the convex hull can be set to 255 to obtain Figure 25 The Pad convex hull-filled image shown can be further modified by setting the pixel values ​​of all pixels in the region enclosed by the outer contour of the second largest connected component to 255 to obtain... Figure 26 The image shown is the Pad outline filled image. By subtracting the Pad outline filled image from the Pad convex hull filled image, we can obtain... Figure 27The set of white regions in the edge-damped binary image shown can be considered as the difference between the region enclosed by the convex hull and the region enclosed by the outer contour of the second largest connected region. By... Figure 27 By mapping the set of white regions in the image to the reference image, the edge defect areas in the reference image can be determined.

[0188] In this implementation, by combining the computational and mapping operations of the regional difference set, the determination of edge-damaged regions can be achieved efficiently and reliably.

[0189] Of course, when determining the edge defect region, it is also possible to not determine the Pad contour filling image and the Pad convex hull filling image, or to subtract the Pad contour filling image from the Pad convex hull filling image. Instead, the bounding region of the convex hull can be mapped to the image to be processed to determine the fourth mapping region in the image to be processed, and the bounding region of the outer contour of the second largest connected region can be mapped to the image to be processed to determine the fifth mapping region in the image to be processed. The difference between the fourth mapping region and the fifth mapping region is taken as the edge defect region.

[0190] In the specific implementation of step 1920 above, by combining image grayscale processing, the application of the first threshold segmentation algorithm, and the determination of the maximum connected component and the application of the convex hull, the region of edge-damaged areas can be efficiently and reliably identified. Of course, the specific implementation of step 1920 is not limited to this; for example, edge-damaged areas can be manually marked.

[0191] Step 1930: Repair the edge defect area.

[0192] In step 1930, for each pixel in the edge-damaged region, a set of neighboring pixels can be determined in the reference image. The original pixel value of the pixel is replaced with the average pixel value of the neighboring pixel set, thereby repairing the edge-damaged region. This is equivalent to propagating and blending the color and structure of the undamaged area of ​​the test junction edge into the reference image with the test junction edge defect, thus achieving test junction edge repair. In one example, the reference image after edge-damaged region repair can be... Figure 28 The grayscale image shown corresponds to the color image.

[0193] Step 1940: Generate a template image based on the reference image after the edge defect area has been repaired.

[0194] In one alternative implementation, step 1940 includes:

[0195] Expand the convex hull and determine the expansion region corresponding to the convex hull;

[0196] From the reference image after repairing the edge defect area, a local image of the third region mapped to the dilated region is extracted;

[0197] Generate a template image based on a local image.

[0198] Understandably, the bulging process of a convex hull allows its boundaries to expand outwards. In one example, the enclosed region of the convex hull is as follows: Figure 23 As shown in the image, the area enclosed by the white border inside the image represents the expansion region corresponding to the convex hull. Figure 29 The white area in the image is shown.

[0199] By mapping the dilated region to a reference image after the edge defect region has been repaired, a sixth mapped region can be determined in the repaired reference image. This sixth mapped region can serve as the aforementioned third region. A local image of the third region can be directly used as a template image, or the local image of the third region can be smoothed or otherwise processed, and the smoothed result can be used as the template image. Optionally, the template image can be stored locally or in another predetermined storage location.

[0200] In this implementation, by combining dilation processing and image cropping processing, the template image can be determined efficiently and reliably. The template image includes not only the edge of the template test junction itself, but also a certain grain background around the edge of the template test junction. In this way, the template image can effectively reflect the size of the template test junction and the background, which can effectively guide the subsequent template matching.

[0201] Of course, the implementation of step 1940 is not limited to this. For example, the reference image after the edge defect area has been repaired can be manually selected by man, and the local image of the selected area can be used as a template image.

[0202] In the embodiments of this disclosure, a reference image after repairing the edge defect area can be used to generate a template image. This helps to ensure the quality of the template image, thereby improving the accuracy of subsequent template matching and thus improving the accuracy of needle mark detection.

[0203] In an optional example, such as Figure 30 As shown, step 160 includes steps 1601, 1603, 1605 and 1607.

[0204] Step 1601: Determine multiple connected components in the merged binary image.

[0205] Optionally, the number of connected components in the merged binary image can be 2, 3, 4, 5 or more, which will not be listed here.

[0206] Step 1603: Determine the connected components whose dimensions meet the first preset constraint conditions among the multiple connected components.

[0207] Optionally, the size of a connected component can be characterized by the number of pixel values ​​it includes. If the number of pixel values ​​included in the connected component is greater than a preset number, the size of the connected component is determined to meet the first preset constraint condition. If the number of pixel values ​​included in the connected component is less than or equal to the preset number, the size of the connected component is determined to not meet the first preset constraint condition. In this way, the connected component whose size meets the first preset constraint condition can be determined efficiently and reliably from multiple connected components.

[0208] Of course, the size of a connected component can also be represented by its area. In this way, if the area of ​​the connected component is greater than the preset area, it can be determined that the size of the connected component meets the first preset constraint condition. If the area of ​​the connected component is less than or equal to the preset area, it can be determined that the size of the connected component does not meet the first preset constraint condition.

[0209] Step 1605: Merge connected components whose dimensions meet the first preset constraint condition and whose distances between them meet the second preset constraint condition to obtain merged connected components.

[0210] For any two connected components that meet the first preset constraint, the distance between these two connected components can be determined. If the distance between these two connected components is less than or equal to the first preset distance, it can be determined that the distance between these two connected components meets the second preset constraint. If the distance between these two connected components is greater than the first preset distance, it can be determined that the distance between these two connected components does not meet the second preset constraint. Thus, all connected components whose distances to each other meet the second preset constraint can be merged to obtain a merged connected component.

[0211] Step 1607: Based on merging connected components, determine the pin mark detection results of the test nodes in the image to be processed.

[0212] In one alternative implementation, step 1607 includes:

[0213] Determine the second smallest rectangle that can enclose the merged connected components;

[0214] Based on the length and width of the second minimum rectangle, the region attributes of the merged connected components are determined; whereby the region attributes are used to characterize whether the merged connected components belong to the pinhole region.

[0215] Based on the regional properties of merged connected components, the pin mark detection results of test nodes in the image to be processed are determined.

[0216] It should be noted that the specific implementation method for determining the second minimum rectangle that can enclose and merge connected regions can be referred to the above description of the method for determining the first minimum rectangle, and will not be repeated here.

[0217] In one alternative implementation, the region attributes of the merged connected components are determined based on the length and width of the second minimum rectangle, including:

[0218] Determine the first numerical relationship between the length of the second minimum rectangle and the preset length;

[0219] Determine a second numerical relationship between the width of the second minimum rectangle and the preset width;

[0220] Determine a third numerical relationship between the ratio of the length to the width of the second minimum rectangle and a preset ratio;

[0221] Based on the first, second, and third numerical relationships, the regional attributes of the merged connected components are determined.

[0222] Optionally, the first numerical relationship can be used to characterize the size relationship between the length of the second minimum rectangle and the preset length, the second numerical relationship can be used to characterize the size relationship between the width of the second minimum rectangle and the preset width, and the third numerical relationship can be used to characterize the size relationship between the ratio of the length and width of the second minimum rectangle and the preset ratio.

[0223] If the following three conditions are met: the length of the second minimum rectangle is greater than the preset length, the width of the second minimum rectangle is greater than the preset width, and the ratio of the length to the width of the second minimum rectangle is less than the preset ratio, then the size of the merged connected region can be considered to conform to the size characteristics of a pin mark. In this case, the regional attributes of the merged connected region can be used to characterize that the merged connected region belongs to the pin mark region.

[0224] If at least one of the following three conditions is not met: the length of the second minimum rectangle is greater than the preset length, the width of the second minimum rectangle is greater than the preset width, and the ratio of the length to the width of the second minimum rectangle is less than the preset ratio, it can be considered that the size of the merged connected region does not conform to the size characteristics of the pin mark. In this case, the regional attribute of the merged connected region can be used to characterize that the merged connected region does not belong to the pin mark region.

[0225] In this way, by combining the length, width, and aspect ratio of the merged connected region, it is possible to efficiently and reliably determine whether the merged connected region belongs to the pin mark region based on its shape.

[0226] Of course, the methods for determining the regional attributes of merged connected components are not limited to this. For example, the regional attributes of merged connected components can be determined by referring to two of the first, second, and third numerical relationships. Or, the regional attributes of merged connected components can be determined by referring to one of the first, second, and third numerical relationships and the area of ​​the merged connected components.

[0227] If the region attribute of the merged connected component is used to characterize that the merged connected component belongs to the pin mark region, the pin mark detection result may include at least one of the merged connected component and the second minimum rectangle. If the region attribute of the merged connected component is used to characterize that the merged connected component does not belong to the pin mark region, the pin mark detection result may exclude either the merged connected component or the second minimum rectangle.

[0228] In this implementation, by determining the second smallest rectangle that can enclose the merged connected region, and combining the size information of the second smallest rectangle, it is possible to efficiently and reliably determine whether the merged connected region belongs to the pin mark region. Thus, the pin mark detection result of the test node in the image to be processed can be determined efficiently and reliably. In this way, pin marks can be identified efficiently and reliably through a simple discrimination criterion.

[0229] Of course, the implementation of step 1607 is not limited to this. For example, after determining the second minimum rectangle, the second minimum rectangle can be directly used as a component of the needle mark detection result.

[0230] In the embodiments of this disclosure, for multiple connected components in a merged binary image, connected components whose size does not meet the requirements can be filtered out, connected components whose size meets the requirements can be retained, and connected components whose distance between each other meets the requirements can be merged. The merged connected components are then used to determine the pin mark detection results of test points in the image to be processed. This helps to avoid the influence of invalid connected components on pin mark detection, thereby improving the accuracy and reliability of pin mark detection, and also improving the efficiency of pin mark detection.

[0231] In an optional example, such as Figure 31 As shown, the method provided in the embodiments of this disclosure further includes steps 3110 and 3120.

[0232] Step 3110: Based on the pin mark detection results of the test joints in the image to be processed, determine the relative distribution information of the pin mark region and the test joint region in the image to be processed.

[0233] Optionally, the pin mark detection result can be used to indicate the pin mark region and the test contact region in the image to be processed; wherein, the pin mark region can be the region enclosed by a second minimum rectangle, and the test contact region can be the region enclosed by a first minimum rectangle. In one example, the pin mark detection result can be... Figure 32The grayscale image shown corresponds to the color image, where the area enclosed by rectangle 3210 can be considered as the test contact area, and the area enclosed by rectangle 3220 can be considered as the pin mark area.

[0234] Optionally, the relative distribution information of the needle mark area and the test contact area in the image to be processed may include: a first vertical distance between the top edge of rectangle 3210 and the top edge of rectangle 3220, a second vertical distance between the bottom edge of rectangle 3210 and the bottom edge of rectangle 3220, a third vertical distance between the left side of rectangle 3210 and the left side of rectangle 3220, and a fourth vertical distance between the right side of rectangle 3210 and the right side of rectangle 3220.

[0235] Step 3120: In response to the relative distribution information meeting the preset abnormal conditions, an abnormal alarm signal is output.

[0236] In step 3120, the vertical distance with the largest value can be selected from the first vertical distance, the second vertical distance, the third vertical distance, and the fourth vertical distance.

[0237] If the vertical distance with the largest value is greater than the second preset distance, it can be considered that the needle is too off-center on the test point to be tested. Then, the relative distribution information can be determined to meet the preset abnormal conditions, and abnormal alarm information can be output in the form of text, voice, light, etc., to prompt the human to detect that the needle is too off-center on the test point to be tested, and to warn of the needle risk, so that the human can deal with the situation in a timely manner.

[0238] If the vertical distance with the largest value is less than or equal to the second preset distance, it can be considered that the needle position on the test contact is normal. In this case, it can be determined that the relative distribution information does not meet the preset abnormal conditions, and there is no need to output an abnormal alarm signal.

[0239] In the embodiments of this disclosure, the relative distribution information of the pin mark area and the test contact area in the image to be processed can be used to efficiently and reliably determine whether the test contact to be tested is too off-center. In such cases, risk warnings can be issued in a timely manner, which is beneficial to ensuring the quality of wafer testing.

[0240] It should be noted that the embodiments of this disclosure can be divided into two stages: the Pad template registration stage and the pin mark detection stage.

[0241] Optionally, the process for the Pad template registration phase can be as follows: Figure 33As shown. In the Pad template registration stage, the reference image can be grayscaled to obtain a second grayscale image. Using an image thresholding algorithm, the second grayscale image is converted into a fourth binary image. For the fourth binary image, opening operations can be used to denoise and smooth the Pad edges. By searching the maximum connected component, the Pad contour (corresponding to the outer contour of the second maximum connected component) can be extracted, and the Pad contour filling image can be calculated based on this. For the second maximum connected component, the Pad convex hull can be extracted, and the Pad convex hull filling image can be calculated based on this. By subtracting the Pad convex hull filling image from the Pad contour filling image, an edge defect binary image can be obtained. Based on the edge defect binary image, the Pad edge defect area in the reference image can be repaired. In the Pad template registration stage, the edge definition domain (equivalent to the dilated region mentioned above) can also be obtained by dilating the Pad convex hull. Using the edge definition domain, a Pad template can be created on the reference image after the edge defect area has been repaired to obtain the required template image. This completes the Pad template registration for a template image. This Pad template registration method can improve template image quality by repairing edge defects, ensuring template matching accuracy and subsequent needle mark detection accuracy.

[0242] Alternatively, the process for the pin mark detection stage can be as follows: Figure 34 As shown. Figure 34 As shown, in the pin mark detection stage, a camera used to image the grain surface can be moved to an initial position (e.g., the position of the test contact to be detected) by controlling a motor. Using the initialized camera and light source, an image is captured to obtain an image of the grain surface. To improve pin mark detection efficiency, the number of test contacts included in the grain surface image can be as large as possible.

[0243] For the grain surface image, a local detection region of the Pad can be cropped (equivalent to the image to be processed above). This local detection region is then converted to grayscale to obtain a grayscale image of the Pad local detection region (equivalent to the first grayscale image above). Template matching is then performed on the Pad local detection region using a registered Pad template (equivalent to the template image above). Based on the matching position, the actual Pad region can be cropped from the first grayscale image (equivalent to the cropped image involved in step 1120 above). Image thresholding segmentation can be used to process the actual Pad region. Opening operations can be used to remove noise from the actual Pad region, and deburring can smooth the Pad edges. The matched Pad connected region (equivalent to the first maximum connected region above) can be obtained by acquiring the maximum connected region. For the matched Pad connected region, the minimum rotation bounding moment (equivalent to the first minimum rectangle above) can be extracted. This minimum rotation bounding moment is then filled with water and cropped using erosion operations. Combined with mapping operations, the Pad positioning mask image above can be generated. For the local detection area of ​​Pad, a blue image of the local detection area of ​​Pad (equivalent to the monochrome image mentioned above) can also be obtained through separation processing.

[0244] Additionally, the average pixel value of the Pad region in the grayscale image of the Pad local detection region (equivalent to the first fill pixel value mentioned above) and the average pixel value of the Pad region in the blue image of the Pad local detection region (equivalent to the second fill pixel value mentioned above) can be calculated. The outer edges of the Pad regions in the grayscale and blue images of the Pad local detection region are then filled with the corresponding fill pixel values ​​(thus obtaining the first and second filled images mentioned above). The first and second filled images can be segmented using corresponding thresholding algorithms to separate dark connected components in the grayscale image of the Pad local detection region and bright connected components in the blue image of the Pad local detection region. The dark and bright connected component binary images can then be merged (equivalent to obtaining the merged binary image mentioned above through merging). For the merged binary image, connected components with areas larger than a set area can be traversed, and the minimum rotational bounding moment of the connected components can be calculated (equivalent to the second minimum rectangle mentioned above).

[0245] For the second minimum rectangle, its length, width, and aspect ratio can be determined. If the determined length is greater than the preset length (equivalent to the preset length mentioned above), the determined width is greater than the set value (equivalent to the preset width mentioned above), and the determined aspect ratio is less than the set value (equivalent to the preset ratio mentioned above), then the area enclosed by the second minimum rectangle can be determined to be a needle mark area. The perpendicular distance between each of the four sides of the second minimum rectangle and the corresponding side of the first minimum rectangle is calculated, and it is determined whether the maximum perpendicular distance among the four obtained perpendicular distances is greater than the set value (equivalent to the second preset distance mentioned above). If the maximum perpendicular distance is greater than the set value, an abnormal alarm message needs to be output to warn of the risk of needle puncture, thereby prompting the operator to confirm and adjust accordingly. If the maximum vertical distance is less than or equal to the set value, the pin mark detection can be terminated (in cases where there are no other test points that need to be detected by pin marks). Alternatively, the camera can be moved to the next detection field of view by controlling the motor to capture an image of another grain surface. Subsequently, the image of the local detection area of ​​the Pad can be captured again. The operation after capturing the image of the local detection area of ​​the Pad can be referred to the above description in this paragraph, and will not be repeated here.

[0246] This pin mark detection method extracts dark connected components from a grayscale image and bright connected components from a bluescale image. By merging these connected components, a binary pin mark image can be obtained. Both bright and dark pin marks can be detected in this binary image, facilitating the detection of both bright and dark pin marks. Processing techniques such as opening operations, deburring, and mean filling improve adaptability to interference from lighting, dirt, and cracks, minimizing missed or incorrect pin mark extraction and thus enhancing the accuracy and reliability of pin mark detection. Furthermore, combining pin mark recognition with connected component shape also helps avoid missed or incorrect pin mark extraction, further improving the accuracy and reliability of pin mark detection.

[0247] Any of the image processing-based pin mark detection methods provided in the embodiments of this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing-based pin mark detection methods provided in the embodiments of this disclosure can be executed by a processor, such as by a processor executing any of the image processing-based pin mark detection methods mentioned in the embodiments of this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0248] Exemplary device

[0249] Figure 35 This is a schematic diagram of the structure of an image processing-based needle mark detection device provided in an exemplary embodiment of the present disclosure. Figure 35 The apparatus shown includes:

[0250] The conversion module 3510 is used to convert the image to be processed, including the test contact to be detected by pin mark, into a first grayscale image;

[0251] The separation module 3520 is used to separate a monochrome image corresponding to a predetermined color component from the image to be processed;

[0252] The first generation module 3530 is used to generate a first binary image based on a first grayscale image according to a first rule; wherein the first rule includes: pixels whose pixel values ​​do not exceed the corresponding pixel value threshold are white;

[0253] The second generation module 3540 is used to generate a second binary image based on a monochrome image according to a second rule; wherein the second rule includes: pixels whose pixel values ​​exceed the corresponding pixel value threshold are white;

[0254] The merging module 3550 is used to merge the first binary image and the second binary image to obtain a merged binary image;

[0255] The first determining module 3560 is used to determine the pin mark detection result of the test joint in the image to be processed based on the merged binary image.

[0256] In an optional example, the apparatus provided by embodiments of this disclosure further includes:

[0257] The second determining module is used to determine the test contact positioning information corresponding to the image to be processed;

[0258] The first generation module 3530 is specifically used to determine the test contact area in the first grayscale image based on the test contact location information; determine the first fill pixel value based on the pixel value of the pixel in the test contact area in the first grayscale image; fill the area in the first grayscale image other than the test contact area using the first fill pixel value to obtain the first fill image; and convert the first fill image into a first binary image according to the first rule.

[0259] In an optional example, the apparatus provided by embodiments of this disclosure further includes:

[0260] The second determining module is used to determine the test contact positioning information corresponding to the image to be processed;

[0261] The second generation module 3540 is specifically used to determine the test node region in the monochrome image based on the test node positioning information; determine the second filling pixel value based on the pixel value of the pixel in the test node region in the monochrome image; fill the region in the monochrome image other than the test node region using the second filling pixel value to obtain the second filled image; and convert the second filled image into a second binary image according to the second rule.

[0262] In an optional example, the apparatus provided by embodiments of this disclosure further includes:

[0263] The cropping module is used to crop the image to be processed from a first region in the grain surface image;

[0264] The second determining module is specifically used to generate test contact positioning information corresponding to the image to be processed based on the template image corresponding to the first region; wherein, the template image includes: an image used to present the size of the template test contact corresponding to the first region and the background.

[0265] In an optional example, such as Figure 36 As shown, the apparatus provided in the embodiments of this disclosure further includes:

[0266] The acquisition module 3610 is used to acquire a reference image including the template test contact point corresponding to the first region;

[0267] The third determining module 3620 is used to determine the edge defect area of ​​the template test joint in the reference image;

[0268] Repair module 3630 is used to repair edge-damaged areas;

[0269] The third generation module 3640 is used to generate a template image based on the reference image after the edge defect area has been repaired.

[0270] In an optional example, such as Figure 37 As shown, the apparatus provided in the embodiments of this disclosure further includes:

[0271] The fourth determining module 3710 is used to determine the relative distribution information of the pin mark region and the test contact region in the image to be processed based on the pin mark detection results of the test contact in the image to be processed;

[0272] The output module 3720 is used to output an abnormal alarm signal in response to the relative distribution information meeting the preset abnormal conditions.

[0273] In one optional example, the predefined color component includes: the blue component.

[0274] In the apparatus disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve the corresponding functions and effects, and this disclosure does not list them all.

[0275] Exemplary electronic devices

[0276] Figure 38 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure, including at least one processor 3810 and a memory 3820.

[0277] The processor 3810 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 3800 to perform desired functions.

[0278] The memory 3820 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 3810 may execute the program instructions to implement the image processing-based needle mark detection method and / or other desired functions described in the various embodiments of this disclosure above.

[0279] In one example, the electronic device 3800 may also include an input device 3830 and an output device 3840, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0280] The input device 3830 may also include, for example, a keyboard, a mouse, etc.

[0281] The output device 3840 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0282] Of course, for the sake of simplicity, Figure 38 Only some of the components of the electronic device 3800 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 3800 may include any other suitable components depending on the specific application.

[0283] Exemplary computer program products and computer-readable storage media

[0284] In addition to the methods and devices described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the image processing-based pin mark detection methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0285] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0286] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image processing-based needle mark detection methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0287] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0288] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0289] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A needle mark detection method based on image processing, comprising: The image to be processed, including the test contact points to be detected by pin mark detection, is converted into a first grayscale image; Separate a monochrome image corresponding to a predetermined color component from the image to be processed; Based on the first grayscale image, a first binary image is generated according to a first rule; wherein, the first rule includes: pixels whose pixel values ​​do not exceed the corresponding pixel value threshold are white; Based on the monochrome image, a second binary image is generated according to a second rule; wherein the second rule includes: pixels whose pixel values ​​exceed the corresponding pixel value threshold are white; The first binary image and the second binary image are merged to obtain a merged binary image; Based on the merged binary image, the pin mark detection result of the test point in the image to be processed is determined, including: determining multiple connected components in the merged binary image; determining connected components whose size meets a first preset constraint condition; merging connected components whose size meets the first preset constraint condition and whose distance between them meets a second preset constraint condition to obtain a merged connected component; determining the pin mark detection result of the test point in the image to be processed based on the region attributes of the merged connected component; the region attributes of the merged connected component are obtained based on a first numerical relationship, a second numerical relationship, and a third numerical relationship, wherein the first numerical relationship represents the relationship between the length of the second smallest rectangle that can enclose the merged connected component and a preset length, the second numerical relationship represents the relationship between the width of the second smallest rectangle and a preset width, and the third numerical relationship represents the relationship between the ratio of the length to the width of the second smallest rectangle and a preset ratio.

2. The method according to claim 1, wherein, The method further includes: Determine the test node positioning information corresponding to the image to be processed; The step of generating a first binary image based on the first grayscale image according to a first rule includes: Based on the test contact location information, the test contact area in the first grayscale image is determined. The first fill pixel value is determined based on the pixel value of the pixel in the test junction region of the first grayscale image; Using the first fill pixel value, the area in the first grayscale image other than the test junction area is filled to obtain the first filled image; According to the first rule, the first filled image is converted into the first binary image.

3. The method according to claim 1, wherein, The method further includes: Determine the test node positioning information corresponding to the image to be processed; The step of generating a second binary image based on the monochrome image according to the second rule includes: Based on the test contact location information, the test contact area in the monochrome image is determined; The second fill pixel value is determined based on the pixel values ​​of the pixels in the test junction region of the monochrome image; Using the second fill pixel value, the area in the monochrome image other than the test junction area is filled to obtain a second filled image; According to the second rule, the second filled image is converted into the second binary image.

4. The method according to claim 2 or 3, wherein, The method further includes: The image to be processed is extracted from a first region in the grain surface image; The step of determining the test node positioning information corresponding to the image to be processed includes: Based on the template image corresponding to the first region, test contact positioning information corresponding to the image to be processed is generated; wherein, the template image includes: an image used to present the size of the template test contact corresponding to the first region and the background.

5. The method according to claim 4, wherein, The step of generating test contact location information corresponding to the image to be processed based on the template image corresponding to the first region includes: Determine a second region in the image to be processed that matches the template image; Based on the second region, a cropped image including the test contact point to be detected by pin mark is extracted from the first grayscale image; According to the first rule, the captured image is converted into a third binary image; Determine the first largest connected component in the third binary image; Determine the first minimum rectangle that can enclose the first maximum connected region; Based on the first minimum rectangle, the test node positioning information corresponding to the image to be processed is generated.

6. The method according to claim 5, wherein, The step of generating test node positioning information corresponding to the image to be processed based on the first minimum rectangle includes: The region enclosed by the first minimum rectangle is cropped at the edges, and the uncropped region within the region enclosed by the first minimum rectangle is determined. The uncropped region is mapped to an image with the same size as the image to be processed to generate a test node positioning mask map corresponding to the image to be processed.

7. The method according to claim 4, wherein, The method further includes: Obtain a reference image including the template test points corresponding to the first region; Determine the edge defect area of ​​the template test joint in the reference image; Repair the edge defect area; The template image is generated based on the reference image after the edge defect area has been repaired.

8. The method according to claim 7, wherein, Determining the edge defect region of the template test joint in the reference image includes: Convert the reference image into a second grayscale image; According to the first rule, the second grayscale image is converted into a fourth binary image; Determine the second largest connected component in the fourth binary image; Determine the convex hull corresponding to the second largest connected component; Based on the second maximum connected component and the convex hull, the edge defect region of the template test node in the reference image is determined.

9. The method according to claim 8, wherein, The step of determining the edge defect region of the template test point in the reference image based on the second maximum connected component and the convex hull includes: Determine the difference between the region enclosed by the convex hull and the region enclosed by the outer contour of the second maximum connected region; The difference set is mapped onto the reference image to determine the edge defect region of the template test junction in the reference image.

10. The method according to claim 8, wherein, The process of generating the template image based on the reference image after repairing the edge defect area includes: The convex hull is expanded, and the expansion region corresponding to the convex hull is determined; From the reference image after repairing the edge defect area, a partial image of the third region mapped to the dilated region is extracted; The template image is generated based on the local image.

11. The method according to claim 1, wherein, The step of determining the pin mark detection result of the test node in the image to be processed based on the merged connected components includes: Determine the second smallest rectangle that can enclose the merged connected region; Based on the length and width of the second minimum rectangle, the region attribute of the merged connected region is determined; wherein, the region attribute is used to characterize whether the merged connected region belongs to the pin mark region; Based on the regional attributes of the merged connected components, the pin mark detection results of the test nodes in the image to be processed are determined.

12. The method according to claim 1, wherein, The determination of the region attributes of the merged connected components based on the length and width of the second minimum rectangle includes: Determine a first numerical relationship between the length of the second minimum rectangle and the preset length; Determine a second numerical relationship between the width of the second minimum rectangle and the preset width; Determine a third numerical relationship between the ratio of the length to the width of the second minimum rectangle and a preset ratio; Based on the first numerical relationship, the second numerical relationship, and the third numerical relationship, the regional attributes of the merged connected domain are determined.

13. The method according to any one of claims 1-3 and 5-12, wherein, The method further includes: Based on the pin mark detection results of the test joints in the image to be processed, the relative distribution information of the pin mark region and the test joint region in the image to be processed is determined. In response to the relative distribution information meeting preset abnormal conditions, an abnormal alarm signal is output.

14. The method according to any one of claims 1-3 and 5-12, wherein, The predetermined color component includes: the blue component.

15. A needle mark detection device based on image processing, comprising: A conversion module is used to convert the image to be processed, including the test contact to be detected by pin mark detection, into a first grayscale image; A separation module is used to separate a monochrome image corresponding to a predetermined color component from the image to be processed; A first generation module is configured to generate a first binary image based on the first grayscale image according to a first rule; wherein the first rule includes: pixels whose pixel values ​​do not exceed the corresponding pixel value threshold are white; The second generation module is used to generate a second binary image based on the monochrome image according to a second rule; wherein the second rule includes: pixels whose pixel values ​​exceed the corresponding pixel value threshold are white; The merging module is used to merge the first binary image and the second binary image to obtain a merged binary image; The first determining module is used to determine the pin mark detection result of the test point in the image to be processed based on the merged binary image, and is further used to: determine multiple connected components in the merged binary image; determine the connected components whose size meets the first preset constraint condition; merge the connected components whose size meets the first preset constraint condition and whose distance between them meets the second preset constraint condition to obtain a merged connected component; determine the pin mark detection result of the test point in the image to be processed based on the region attributes of the merged connected component; the region attributes of the merged connected component are obtained based on a first numerical relationship, a second numerical relationship and a third numerical relationship, wherein the first numerical relationship represents the size relationship between the length of the second minimum rectangle that can surround the merged connected component and a preset length, the second numerical relationship represents the size relationship between the width of the second minimum rectangle and a preset width, and the third numerical relationship represents the size relationship between the ratio of the length and width of the second minimum rectangle and a preset ratio.

16. An electronic device comprising: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the image processing-based needle mark detection method according to any one of claims 1 to 14.

17. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the image processing-based needle mark detection method according to any one of claims 1 to 14.

Images