A SAM-based SEM image processing method

Abstract

The present invention provides a SAM-based SEM image processing method, which relates to the technical field of SEM image processing. The processing method includes performing model inference on the SEM image using SAM to obtain multiple sets of segmentation masks, determining whether the second mask of the segmentation mask belongs to the first mask. If so, calculating the white pixels of the first mask of the segmentation mask and the first intersection over union of the first union, and determining whether the first intersection over union is greater than the first threshold; if so, retaining the target mask in each second mask set and deleting the corresponding first mask; if not, retaining the first mask and deleting the target mask in the corresponding second mask set; filling the white pixel area of the retained target mask or the first mask with color to obtain the SEM segmentation image. The SEM image processing method of the present invention can achieve accurate segmentation of SEM images by SAM with zero-shot training.

Description

Technical Field

[0001] The present invention relates to the technical field of SEM image processing, and particularly to a SAM-based SEM image processing method. Background Art

[0002] The core function of SAM (Segment - Anything Model) is to automatically segment images. It can perform accurate pixel-level segmentation and has strong generalization ability through zero-shot learning. That is, SAM uses a large-scale general image dataset during pre-training, and these data cover a variety of scenes, objects, and image features.

[0003] However, when applied to a specific field or task, the actual data distribution may be very different from the pre-training data. That is to say, when applying SAM to SEM (scanning electron microscope) image segmentation, since SEM images are generated by a scanning electron microscope and are mainly used to observe the surface morphology and composition of samples, their image features (such as gray values, textures, shapes, etc.) are very different from general images.

[0004] In the prior art, when applying SAM to SEM image segmentation, SAM is usually pre-trained based on a large number of SEM image samples to improve the segmentation accuracy of SAM for SEM images. However, since there are significant differences in SEM for each material, that is, the pre-training method in the prior art often has specificity, that is, SAM pre-trained based on a certain material cannot be applied to the SEM image segmentation of multiple other materials. This will cause SAM to require re-pre-training when performing graphic segmentation on SEM images of different materials, which is time-consuming and laborious, and it is difficult to obtain SEM image samples of different materials. Therefore, SAM cannot be directly applied to accurate and simple segmentation operations of SEM images. Summary of the Invention

[0005] An object of the present invention is to provide a SAM-based SEM image processing method to solve the technical problem of low segmentation accuracy caused by directly applying SAM to SEM image segmentation in the prior art.

[0006] Another object of the present invention is to further improve the automatic segmentation accuracy of SEM images.

[0007] According to the object of the present invention, the present invention provides a SAM-based SEM image processing method, including:

[0008] Perform model inference on SEM images using SAM and obtain multiple sets of segmentation masks. Each set of the segmentation masks corresponds to an image concentration area and includes a first mask and at least one second mask. The first mask is the mask with the largest number of white pixels in each set of segmentation masks, and the second mask is the mask with the number of white pixels within a preset range in each set of segmentation masks;

[0009] Preprocess all the segmentation masks;

[0010] Determine whether each of the second masks in each set of the segmentation masks belongs to the first mask;

[0011] If so, define the second mask as the target mask, define the set of all the target masks as the second mask set, and define the union of the white pixels of all the target masks as the first union;

[0012] Calculate the first intersection over union of the white pixels of the first mask and the first union in each set of the segmentation masks, and determine whether the first intersection over union is greater than the first threshold;

[0013] If so, retain all the target masks in each second mask set and delete the corresponding first mask;

[0014] If not, retain the first mask and delete all the target masks in the corresponding second mask set;

[0015] Perform color filling on the white pixel areas of all the retained target masks or the first mask to obtain the particle color filling map of the SEM image.

[0016] Optionally, the first threshold is any value in the range of 0.7 - 0.9.

[0017] Optionally, the step of determining whether each of the second masks in each set of the segmentation masks belongs to the first mask further includes:

[0018] Calculate the number of the white pixels of all the masks in each set of the segmentation masks, and define the mask with the largest number of white pixels as the first mask;

[0019] Perform intersection operations on the first mask and all the second masks in sequence to obtain multiple first intersections of the first mask and each second mask;

[0020] Calculate the second intersection over union of each first intersection and the corresponding second mask, and determine whether the second intersection over union is greater than the second threshold, where the second threshold is any value in the range of 0.9 - 1;

[0021] If so, determine that the second mask belongs to the first mask.

[0022] Optionally, the intersection operation is performed as a bitwise AND operation.

[0023] Optionally, after the step of using the second mask as the target mask, defining the set of all the target masks as the second mask set, and defining the union of the white pixels of all the target masks as the first union, the following steps are further included:

[0024] Store the first mask and the second mask set of each group of the segmented masks into a mask dictionary.

[0025] Optionally, each of the first unions is obtained through a bitwise OR operation.

[0026] Optionally, the step of preprocessing all the segmented masks includes:

[0027] Determine whether there are black holes in the white pixel region of each mask in each group of the segmented masks;

[0028] If so, calculate the ratio of the number of black pixels in the black hole to the number of white pixels in the corresponding white pixel region;

[0029] Determine whether the ratio is less than a third threshold;

[0030] If so, determine that the black hole is a pre-filled hole, and perform white filling on the pre-filled hole.

[0031] Optionally, the step of preprocessing all the segmented masks further includes:

[0032] Calculate the area of all the white pixel regions in each mask of each group of the segmented masks, and obtain the first contour with the largest area of the white pixel region in each mask;

[0033] Determine whether the area of the white pixel region outside the first contour in each mask is less than a fourth threshold;

[0034] If so, determine that the corresponding contour is an outlier region, and delete the outlier region.

[0035] Optionally, the step of performing color filling on the white pixel regions of all the reserved target masks or the first mask to obtain the particle color filling map of the SEM image further includes:

[0036] Perform contour detection on the white pixel regions of each of the said target photomasks, and calculate the minimum bounding rectangle.

[0037] Optionally, the step of color-filling the white pixel regions of all the said target photomasks or the first photomask that are retained to obtain the particle color-filled map of the SEM image further includes:

[0038] Perform edge detection on the white pixel regions of each of the said target photomasks or the first photomask to obtain the particle segmentation edge map of the SEM image.

[0039] The present invention preprocesses, determines the inclusion relationship, calculates and compares the first intersection over union for the first photomask and multiple second photomasks in each set of segmented photomasks obtained through SAM processing in sequence. Based on the comparison between the first intersection over union and the first threshold, all the target photomasks or the first photomask in each second photomask set are retained, and the white pixel regions of all the retained target photomasks or the first photomask are color-filled. That is, through the above steps, multi-level analysis of the spatial relationship and geometric morphology between particles in the SEM image is carried out, and through the automated processing of the merging and deletion between photomasks, not only is the manual intervention reduced and the processing efficiency improved, but also a SEM segmentation image with a higher visualization effect, clearer and more accurate can be obtained, improving the accuracy of applying zero-shot trained SAM to SEM image segmentation.

[0040] Furthermore, the present invention determines the black holes within the white pixel regions of each photomask, and performs white filling on the black holes smaller than the third threshold to eliminate small defects or holes in the photomask, improving the quality of SEM image segmentation and the accuracy of subsequent analysis. By performing contour detection and area comparison of the white pixel regions on each photomask after pre-filling or without black holes, the outlier regions with an area smaller than the fourth threshold in each photomask are determined and deleted, which can effectively remove the noise and irrelevant regions in the segmented regions, simplify the subsequent image analysis, and thus further improve the segmentation accuracy and efficiency of the SEM image.

[0041] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly and implement it in accordance with the content of the specification, the following takes the preferred embodiments of the present invention and describes them in detail with the accompanying drawings as follows. Brief Description of the Drawings

[0042] Some specific embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings in an exemplary but non-limiting manner. The same reference numerals in the drawings denote the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0043] Figure 1 is a schematic partial flowchart of an SEM image processing method according to an embodiment of the present invention;

[0044] Figure 2 is a schematic overall flowchart of an SEM image processing method according to an embodiment of the present invention;

[0045] Figure 3 is a schematic SEM image of a multi-particle target material according to an embodiment of the present invention;

[0046] Figure 4 is Figure 3 a schematic structural diagram of the first mask after the SEM image shown is processed by SAM segmentation;

[0047] Figure 5 is Figure 3 a schematic structural diagram of the second mask after the SEM image shown is processed by SAM segmentation;

[0048] Figure 6 is a schematic particle color filling diagram and minimum circumscribed rectangle diagram of the first mask according to an embodiment of the present invention;

[0049] Figure 7 is a schematic particle segmentation edge diagram and minimum circumscribed rectangle diagram of the first mask according to an embodiment of the present invention;

[0050] Figure 8 is a schematic SEM image of a single-particle target material according to another embodiment of the present invention;

[0051] Figure 9 is Figure 8 a schematic structural diagram of identifying black holes in the SEM image shown;

[0052] Figure 10 is Figure 8 a schematic structural diagram after filling the black holes in the SEM image shown;

[0053] Figure 11 is Figure 8 a schematic structural diagram of identifying outlier regions in the SEM image shown;

[0054] Figure 12 is Figure 8 a schematic structural diagram after deleting the outlier regions in the SEM image shown;

[0055] Figure 13 is Figure 8 a schematic structural diagram of the first mask after the SEM image shown is processed by SAM segmentation;

[0056] Figure 14 isFigure 8 Schematic structural diagram of the second mask after SAM segmentation processing of the shown SEM image. Detailed implementation manners

[0057] The following will further describe in detail the specific implementation manners of the present invention in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0058] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will describe in detail the specific implementation manners of the present application in conjunction with the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of convenience of description, only the parts related to the present application are shown in the accompanying drawings rather than all the structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0059] The terms "comprising" and "having" in the present application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0060] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various positions in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0061] Figure 1 is a schematic partial flowchart of an SEM image processing method according to an embodiment of the present invention, Figure 2 is a schematic overall flowchart of an SEM image processing method according to an embodiment of the present invention, Figure 3 is a schematic partial SEM image of a target material according to an embodiment of the present invention, Figure 4 is Figure 3 schematic structural diagram of the first mask after SAM segmentation processing of the shown SEM image, Figure 5 is Figure 3 schematic structural diagram of the second mask after SAM segmentation processing of the shown SEM image, Figure 6 is a schematic diagram of the particle color filling and the minimum circumscribed rectangle of the contour of the first mask according to an embodiment of the present invention,Figure 7 is a schematic particle segmentation edge map and minimum bounding rectangle map of the first mask according to an embodiment of the present invention, Figure 8 is a schematic SEM image of a single-particle target material according to another embodiment of the present invention, Figure 9 is Figure 8 a schematic structural diagram for black hole identification of the SEM image shown, Figure 10 is Figure 8 a schematic structural diagram after black hole filling of the SEM image shown, Figure 11 is Figure 8 a schematic structural diagram for outlier region identification of the SEM image shown, Figure 12 is Figure 8 a schematic structural diagram after deletion of the outlier region of the SEM image shown, Figure 13 is Figure 8 a schematic structural diagram of the first mask after SAM segmentation processing of the SEM image shown, Figure 14 is Figure 8 a schematic structural diagram of the second mask after SAM segmentation processing of the SEM image shown.

[0062] As Figure 1 shown, the present invention provides a SAM-based SEM image processing method, including:

[0063] Step S100: Use SAM to perform model inference on the SEM image and obtain multiple sets of segmentation masks. Each set of segmentation masks corresponds to an image set region and includes a first mask and at least one second mask. The first mask is the mask with the largest number of white pixels in each set of segmentation masks, and the second mask is the mask with the number of white pixels within a preset range in each set of segmentation masks;

[0064] Step S200: Preprocess all the segmentation masks;

[0065] Step S300: Determine whether each second mask in each set of segmentation masks belongs to the first mask;

[0066] Step S350: If so, define the second mask as the target mask, define the set of all target masks as the second mask set, and define the union of the white pixels of all target masks as the first union;

[0067] Step S400: Calculate the first intersection over union of the white pixels of the first mask and the first union of each set of segmentation masks, and determine whether the first intersection over union is greater than the first threshold;

[0068] Step S410: If so, retain all the target masks in each second mask set and delete the corresponding first mask;

[0069] Step S510: Perform color filling on the white pixel regions of all the remaining target masks to obtain a particle color filling map of the SEM image;

[0070] Step S420: If not, retain the first mask and delete all the target masks in the corresponding second mask set;

[0071] Step S520: Perform color filling on the white pixel region of the retained first mask to obtain a particle color filling map of the SEM image.

[0072] In this embodiment, first, the SEM image is subjected to model inference using SAM, and multiple sets of segmentation masks are obtained. Each set of segmentation masks corresponds to an image concentration area and includes a first mask and at least one second mask. All the segmentation masks are preprocessed. Then, it is determined whether each second mask in each set of preprocessed segmentation masks belongs to the first mask. If it is determined that the second mask belongs to the first mask, the second mask is defined as a target mask, and the set of all target masks is defined as the second mask set. The union of the white pixels of all the target masks is defined as the first union. Next, the first intersection-over-union of the white pixels of the first mask and the first union of each set of segmentation masks is calculated, and it is determined whether the first intersection-over-union is greater than the first threshold. If the first intersection-over-union is greater than the first threshold, all the target masks in each second mask set are retained, and the corresponding first mask is deleted. Otherwise, the first mask is retained, all the target masks in the corresponding second mask set are deleted, and color filling is performed on the white pixel region of all the retained target masks or the first mask to obtain a SEM segmentation image. Here, the image concentration area is a particle distribution area in the SEM image, that is, SAM divides the SEM image into multiple particle distribution areas, and the number of second masks can be one, two, or more.

[0073] In this embodiment, by sequentially preprocessing, judging the inclusion relationship, calculating and comparing the first intersection over union (IoU) for the first mask and multiple second masks in each set of segmentation masks obtained through SAM processing, based on the comparison between the first IoU and the first threshold, if the first IoU is greater than the first threshold, all target masks in each second mask set are retained, and the corresponding first mask is deleted; otherwise, the first mask is retained, all target masks in the corresponding second mask set are deleted, and the white pixel regions of all the retained target masks or the first mask are filled with color. That is, through the above steps, a multi-level analysis of the spatial relationship and geometric morphology between particles in the SEM image is carried out, and through the automated processing of the merging and deletion between masks, not only is the manual intervention reduced and the processing efficiency improved, but also a particle color filling map of the SEM image with a higher visualization effect, clearer and more accurate can be obtained, improving the accuracy of applying zero-shot trained SAM to SEM image segmentation.

[0074] In step S100, the second mask is a mask in each set of segmentation masks whose white pixel count is within a preset range. Here, the preset range is the ratio range of the white pixel count of the second mask to the white pixel count of the first mask, and specifically, the preset range is any value in [0.5 - 1), that is, the ratio of the white pixel count of the second mask to the white pixel count of the first mask is 0.5, 0.6, 0.7, 0.8, 0.9, or 0.95, or it can also be any value in 0.5 - 1. That is to say, masks with white pixel counts set to be more than half of the white pixel count of the first mask and less than the white pixel count of the first mask are all second masks, so as to include all images in the SEM image that are close to the target particles in the segmentation masks, preventing omissions during SAM's graphic segmentation or subsequent image processing, thereby initially improving the accuracy of applying SAM to SEM image segmentation.

[0075] In a further embodiment, the first threshold is any value within the range of 0.7 - 0.9, that is, the first threshold can be 0.7, 0.75, 0.8, 0.85, or 0.9, or any value within 0.7 - 0.9. In this embodiment, the first threshold is set to any value within the range of 0.7 - 0.9. By setting the first threshold to be adjustable, the particle precision and region merging degree of the segmentation result of the SEM image are controlled. That is, when the first threshold approaches 0.9, by judging the size of the first intersection over union and the first threshold, more image details and small particles are retained to avoid misdeleting small particles. When the first threshold approaches 0.7, even when the overlapping degree of the first mask and the second mask is low, the second mask will still be regarded as a partial area of the first mask, that is, more small particle areas are merged and deleted, thereby reducing the number of segmented regions and generating a simpler and more concise segmentation result. That is to say, the first threshold in this embodiment can be selected according to the detail degree requirement of the SEM image.

[0076] In a preferred embodiment, the first threshold is set to 0.8. That is, when the first threshold is set to 0.8, the overlapping degree between the first mask and the second mask enables the obtained mask to retain certain particle details while merging some small adjacent particle regions, which can effectively avoid the existence of over-segmentation and noise regions, and is applicable to most cases where most particle shapes and boundaries need to be retained during segmentation. It can not only retain the integrity of the particles but also avoid the influence of excessive small particles and noise, and is applicable to most SEM image analysis tasks.

[0077] As Figure 2 shown, in a further embodiment, step S200 includes:

[0078] Step S210: Determine whether there is a black hole in the white pixel area of each mask in each group of segmentation masks. If so, enter step S211; if not, enter step S250;

[0079] Step S211: Calculate the ratio of the number of black pixels in the black hole to the number of white pixels in the corresponding white pixel area;

[0080] After step S211, it further includes:

[0081] Step S230: Determine whether the ratio is less than the third threshold. If so, enter step S231; if not, enter step S232;

[0082] Step S231: Determine that the black hole is a pre-filled hole and perform white filling on the pre-filled hole;

[0083] Step S232: Retain the black hole;

[0084] After step S231 or step S232, it further includes:

[0085] Step S250: Calculate the area within all white pixel regions in each mask of each group of segmented masks, and obtain the first contour with the largest area of the white pixel region in each mask;

[0086] Step S260: Determine whether the area of the white pixel region outside the first contour in each mask is less than the fourth threshold. If so, go to step S262; if not, go to step S264;

[0087] Step S262: Determine the corresponding white pixel region as an outlier region and delete the outlier region;

[0088] Step S264: Retain the corresponding white pixel region;

[0089] After step S262 or step S264, it further includes:

[0090] Step S310: Calculate the number of white pixels in all masks of each group of segmented masks, and define the mask with the largest number of white pixels as the first mask;

[0091] Step S320: Perform an intersection operation on the first mask with all second masks in sequence to obtain multiple first intersections of the first mask with each second mask;

[0092] Step S330: Calculate the second intersection-over-union of each first intersection with the corresponding second mask, and determine whether the second intersection-over-union is greater than the second threshold. The second threshold is any value in the range of 0.9 - 1. If so, go to step S350; if not, go to step S360;

[0093] Step S360: Determine that there is no inclusion relationship between the first mask and the corresponding second mask, and retain the corresponding second mask;

[0094] After step S350, it further includes:

[0095] Step S370: Store the first mask and the second mask set of each group of segmented masks into the mask dictionary;

[0096] After step S370, it further includes:

[0097] Step S400: Calculate the first intersection-over-union of the white pixels of the first mask of each group of segmented masks and the first union, and determine whether the first intersection-over-union is greater than the first threshold. If so, go to step S410; if not, go to step S420;

[0098] Step S410: Retain all target masks in each second mask set and delete the corresponding first mask;

[0099] Step S420: Retain the first mask in each set of segmented masks and delete the corresponding second mask set;

[0100] After step S410, it further includes:

[0101] Step S510: Fill the white pixel regions of all the retained target masks to obtain a SEM segmentation image;

[0102] After step S420, it further includes:

[0103] Step S520: Fill the white pixel regions of the retained first mask to obtain a SEM segmentation image.

[0104] In this embodiment, after using SAM to segment the SEM image, first, determine the black holes and outlier regions of each mask in each set of segmented masks, and sequentially fill the black holes with white and delete the outlier regions. Then, perform intersection operations between the first mask in each set of segmented masks and all the second masks in turn to obtain multiple first intersections of the first mask and each second mask, calculate the second intersection over union of each first intersection and the corresponding second mask, and compare the second intersection over union with the second threshold to determine whether the second mask belongs to the first mask. Define the second masks that belong to the first mask as target masks, define the union of the white pixels of all the target masks as the first union, calculate the intersection over union of the first union and the white pixels in the corresponding first mask, and compare the obtained first intersection over union with the first threshold to determine whether to retain or delete the first mask and all the target masks in the second mask set in each set of segmented masks, and selectively perform contour detection, edge detection, or color filling on the confirmed retained masks to meet different requirements of the user for the SEM segmentation image. Here, color filling, contour detection, and edge detection are three independent SEM image processing means, and the user can select one or more processing means according to the requirements.

[0105] In step S200, the preprocessing methods for all segmented masks include black hole judgment and filling, and outlier region judgment and deletion. That is, by judging the black holes in the white pixel regions of each mask and filling the black holes smaller than the third threshold with white to eliminate small defects or holes in the mask, improving the quality of SEM image segmentation and the accuracy of subsequent analysis. By performing contour detection and comparing the areas of white pixel regions for each mask after pre-filling or without black holes, outlier regions with an area smaller than the fourth threshold in each mask are determined and deleted, which can effectively remove the noise and irrelevant regions in the segmented regions, simplify the subsequent image analysis, and further improve the segmentation accuracy and efficiency of SEM images.

[0106] As Figure 9 shown, in this embodiment, since the surface of the particles in the SEM image is usually irregular, there will be black holes in the output mask during the segmentation of the SEM image by SAM (refer to Figure 9 ). The black holes in the mask can be the irregular parts of the particle surface in the SEM image or the noise generated during the SAM image segmentation. By setting the third threshold to identify and screen the black holes, the black holes smaller than the third threshold are retained and filled with white (refer to Figure 10 ), so that the white region pixels in the mask output by SAM are complete, and the black holes larger than the third threshold are retained, making the white pixel region in the mask output by SAM closer to the original image of the SEM image, thereby further improving the segmentation accuracy of the SEM image. Here, the third threshold can be 0.1, and the third threshold can be set according to the SEM image of the corresponding material and the requirements of image segmentation accuracy. Regarding the third threshold, it is not limited here.

[0107] As Figure 11 shown, in this embodiment, during the process of identifying the outlier regions in each mask of each group of segmented masks and determining whether to delete the outlier regions, the fourth threshold can be set to 0.5. That is, for each white pixel region in each mask with an area smaller than the first contour, when its area is less than half of the area of the first contour, it is determined as an outlier region and deleted. The largest contour is obtained, the areas of the remaining regions are judged, and finally the outlier regions are deleted (refer to Figure 12 ), to remove the noise and irrelevant regions, improving the accuracy and efficiency of SEM image segmentation. Here, the fourth threshold can also be 0.4 or 0.6, or any other value that meets the requirements of SEM image segmentation accuracy. Regarding the fourth threshold, it is not limited here.

[0108] In step S300, by calculating and sorting the number of white pixels of all the masks in each group of preprocessed segmented masks, and defining that the first mask has the largest number of white pixels, the first mask is sequentially subjected to an intersection operation with all the second masks in the corresponding group of segmented masks to obtain a plurality of first intersections. By calculating whether the second intersection over union of each first intersection and the corresponding second mask is greater than a second threshold, it is determined whether each second mask in each group of segmented masks belongs to the first mask. That is, when the second intersection over union is greater than the second threshold, it is determined that the corresponding second mask belongs to the first mask and is defined as a target mask. After that, the set of all target masks is defined as the second mask set, that is, all the second masks belonging to the first mask in each group of segmented masks are grouped into the second mask set, so as to facilitate the subsequent judgment of retaining or deleting the first mask or the second mask set, and avoid misdeleting the masks not included in the first mask when deleting the second mask set later, that is, improve the integrity of the SEM image after SAM segmentation, thereby further improving the segmentation accuracy of the SEM image. Here, the second threshold is any value in the range of 0.9-1, that is, the second threshold can be 0.9, 0.92, 0.94, 0.96, 0.98 or 1, or any value in 0.9-1.

[0109] In step S370, the mask dictionary includes the first mask and the second mask set of each group of segmented masks. That is, all the target masks and the first mask in the second mask set are included in the mask dictionary, which can manage the first mask and all the target masks included in the first mask in an orderly manner, facilitate key access and quick search, and support efficient subsequent operations, such as deletion, update, merging, etc., providing a good data basis for the subsequent processing of the segmentation result. At the same time, the setting of the mask dictionary can avoid repeatedly calculating the intersection over union of the same region in the subsequent processing. Only need to check whether the small masks it contains meet the deletion conditions, instead of repeatedly traversing the region, reducing the redundancy of the calculation and improving the overall calculation efficiency. Especially when processing a large number of segmented masks, the calculation time can be significantly shortened.

[0110] In step S510, by filling the color of the white pixel region in the retained mask, the visualization effect of the segmentation result is improved, making each particle easier to distinguish and analyze in the image, facilitating manual or machine observation and analysis of the segmentation effect. And different particles can be distinguished by color, helping to understand the relationship between particles and the distribution of particles, and facilitating the distinction of the single-particle distribution in the multi-particle mask.

[0111] After step S410, it further includes:

[0112] Perform contour detection on the white pixel regions of each target mask and calculate the minimum bounding rectangle.

[0113] In this embodiment, to locate the shape, position, and external boundary of the particles in the SEM image, that is, through contour detection and the calculation of the minimum bounding rectangle, an accurate bounding box can be provided for each particle, facilitating subsequent region recognition, visualization of segmentation results, and analysis of the relationships between particles. Here, the contour detection method can be the contour detection algorithm in OpenCV (Open Source Computer Vision Library, open-source model) or other computer vision libraries.

[0114] After step S410, it further includes:

[0115] Perform edge detection on each target mask to obtain the segmentation edge map of the SEM image.

[0116] In this embodiment, perform edge detection on each target mask to obtain the segmentation edge map of the SEM image, that is, by detecting the edges of the particles in the SEM image, a clearer particle contour can be obtained, thereby helping to analyze the shape, size, and spatial relationship of the particles, avoiding blurred boundaries that may occur in segmentation, and at the same time assisting in further analysis of morphological features such as the geometric shape, area, and perimeter of the particles. Here, Figure 3 、 Figure 6 and Figure 7 KYKY-EM8000F in Figure 3 、 Figure 6 and Figure 7 are the product models of the field emission scanning electron microscopes of KYKY Technology Co., Ltd. SE is secondary electron, which is a signal collected in the scanning electron microscope and is mainly used to reflect the morphological features of the sample surface. HV is the acceleration voltage, Mag is the image magnification, and WD is the working distance from the sample surface to the lower end of the objective lens. That is to say,

[0117] In step S420, if it is determined that the first intersection over union is less than the first threshold, then retain the first mask and delete all target masks in the second mask set. That is, at this time, the target masks in the second mask set are determined to be the attached particles of the first mask. For example, in the SEM image of a single-particle material (refer to Figure 8 ), the second mask completely belongs to the first mask (refer to Figure 8 ), that is, the first mask (refer to Figure 13) has a much larger white pixel area than the second mask (refer to Figure 14 ), at this time, it is determined that the second mask (refer to Figure 14 ) is an attached particle and is deleted to retain the first mask closer to the morphological characteristics of the target material. That is to say, the SEM image processing method in the present invention can perform multi-level screening and processing according to the morphological characteristics of the target material to retain the SEM image closest to the target material, making the segmentation result more accurate. Here, the single-particle material can be hydroxyl iron powder, Figure 8 In Figure 8 , KYKY-EM6900LV is the product model of the field emission scanning electron microscope of Zhongke Keyi, HV is the acceleration voltage, WD is the working distance from the sample surface to the lower end of the objective lens, Mag is the image magnification, DET:SE means Figure 8 is detected using a secondary electron detector (SE), Vac:High Vac indicates that the vacuum state in the sample chamber is the high vacuum mode, and KYKY Sample is the sample observed by the scanning electron microscope of Zhongke Keyi.

[0118] In one embodiment, after step S420, it further includes:

[0119] Detect the contour of the white pixel area of the first mask and calculate the minimum bounding rectangle.

[0120] In another embodiment, after step S420, it further includes:

[0121] Edge detection is performed on the first mask to obtain the segmentation edge map of the SEM image. In a further embodiment, the intersection operation method is bitwise AND operation. In this embodiment, the bitwise AND operation compares the corresponding pixels of the second mask and the first mask. When the pixel values at the same position of the two masks are both 1, the position where the corresponding pixel is located is the overlapping area between the first mask and the second mask, so as to ensure that the calculated is the actual intersection part of the two masks, and at the same time remove irrelevant areas to ensure that only the overlapping part is retained, which helps to clearly define and distinguish the subordinate relationship of each mask to the same particle or structure. And, because the bitwise AND operation is a very efficient bit-level operation in the computer, usually without complex loops and conditional judgments, it can directly operate on the binary bits of each pixel, with a fast calculation speed and low calculation overhead, improving the determination efficiency of the subordinate relationship between the first mask and the second mask in each group of segmentation masks of the SEM image.

[0122] In a further embodiment, each first union is obtained through a bitwise OR operation. In this embodiment, the union of all target masks in each second mask set, that is, the first union, is obtained by means of a bitwise OR operation, so as to integrate all target masks in all second mask sets into the first union, that is, all second masks included in the first mask in each group of segmented masks are retained in the first union, so that after subsequent judgment on the retention or deletion of the first mask or the second mask set, redundant parts that duplicate the retained masks can be accurately deleted, reducing the workload of subsequent image analysis and improving the segmentation accuracy of SEM images at the same time.

[0123] The present application will be further described in detail below in conjunction with specific embodiments.

[0124] First, obtain Figure 3 the SEM image of the target material shown, and use SAM to perform image segmentation processing on the SEM image to obtain a segmented mask. The segmented mask includes a first mask (refer to Figure 4 ), and multiple second masks (refer to Figure 5 ). By sequentially performing the operations of identifying and filling black holes and identifying and deleting outlier regions on the first mask and the multiple second masks, and then judging the subordinate relationship between the first mask and the multiple second masks. After determining to retain the second mask, the first mask is deleted, and color filling, contour detection or edge detection is selectively performed on the retained target mask, thereby realizing the automatic segmentation of the SEM image.

[0125] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0126] The above-described embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent should be subject to the appended claims.

Claims

1. A SEM image processing method based on SAM, characterized in that: include: Using SAM to perform model inference on the SEM image, and obtain multiple groups of segmentation masks, each group of the segmentation masks corresponds to an image concentration area and includes a first mask and at least one second mask, the first mask is a mask with the largest number of white pixels in each group of segmentation masks, and the second mask is a mask with the number of white pixels in each group of segmentation masks within a preset range; Preprocessing all the segmentation masks; Determining whether each of the second masks in each group of the segmented masks belongs to the first mask; If yes, define the second mask as a target mask, define a set of all the target masks as a second mask set, and define a union of white pixels of all the target masks as a first union; Calculating a first intersection-and-union ratio of white pixels of the first mask of each group of the segmented mask and the first union, and determining whether the first intersection-and-union ratio is greater than a first threshold; If yes, retain all the target masks in each of the second mask sets and delete the corresponding first mask; If not, keep the first mask and delete all target masks in the corresponding second mask set; Filling the retained white pixel areas of all the target masks or the first mask with colors to obtain a particle color-filled image of the SEM image; The first threshold is any value in the range of 0.7-0.9, and each of the first unions is obtained by a bitwise OR operation; The step of judging whether each of the second mask plates in each group of the segmented mask plates belongs to the first mask plate also includes: Calculating the number of white pixels of all the masks in each group of segmented masks, and defining the mask with the largest number of white pixels as the first mask; Performing intersection operations on the first mask plate and all the second mask plates in sequence to obtain a plurality of first intersections of the first mask plate and each of the second mask plates; Calculating a second intersection-and-union ratio between each of the first intersections and the corresponding second mask, and determining whether the second intersection-and-union ratio is greater than a second threshold, where the second threshold is any value in the range of 0.9-1; If so, determining that the second mask belongs to the first mask; The intersection operation is a bitwise AND operation.

2. The SEM image processing method according to claim 1, characterized in that: After the step of using the second mask as a target mask, defining a set of all the target masks as a second mask set, and defining a union of white pixels of all the target masks as a first union, the method further includes: The first mask and the second mask set of each group of the segmented mask are stored in a mask dictionary.

3. The SEM image processing method according to claim 2, characterized in that: The step of preprocessing all the segmentation masks comprises: Determining whether there is a black hole in the white pixel area of ​​each mask in each group of segmented masks; If yes, calculate the ratio of the number of black pixels in the black hole to the number of white pixels in the corresponding white pixel area; Determining whether the ratio is less than a third threshold; If so, the black hole is determined to be a pre-filled hole, and the pre-filled hole is filled with white.

4. The SEM image processing method according to claim 3, characterized in that: The step of preprocessing all the segmentation masks further comprises: Calculating the area of ​​all the white pixel regions in each mask of each group of the segmented mask, and obtaining a first contour of the white pixel region with the largest area in each mask; Determining whether the area of ​​the white pixel region outside the first outline in each of the mask plates is less than a fourth threshold; If so, the corresponding contour is determined to be an outlier area, and the outlier area is deleted.

5. The SEM image processing method according to claim 1, characterized in that: The step of color-filling all the retained white pixel areas of the target mask or the first mask to obtain a particle color-filled map of the SEM image also includes: Contour detection is performed on the white pixel area of ​​each target mask or the first mask, and a minimum circumscribed rectangle is calculated.

6. The SEM image processing method according to claim 1, characterized in that: The step of color-filling all the retained white pixel areas of the target mask or the first mask to obtain a particle color-filled map of the SEM image also includes: Edge detection is performed on the white pixel region of each of the target mask or the first mask to obtain a particle segmentation edge map of the SEM image.

Images