SAM-based SEM image processing method

Abstract

The invention provides an SAM-based SEM image processing method, and relates to the technical field of SEM image processing. The processing method comprises the following steps: performing model reasoning on an SEM image by using SAM to obtain a plurality of groups of segmented masks, judging whether a second mask of the segmented masks belongs to a first mask, if so, calculating a first intersection-union ratio of white pixels of the first mask of the segmented masks and a first union set, and judging whether the first intersection-union ratio is greater than a first threshold; if yes, the target mask in each second mask set is reserved, and the corresponding first mask is deleted; if not, the first mask is reserved, and the target mask in the corresponding second mask set is deleted; and performing color filling on the reserved white pixel region of the target mask or the first mask to obtain an SEM segmented image. According to the SEM image processing method provided by the invention, accurate segmentation of the SAM of zero sample training on the SEM image can be realized.

Description

Technical Field

[0001] The invention relates to the technical field of SEM image processing, and in particular to a SEM image processing method based on SAM. Background Art

[0002] The core function of SAM (Segment-Anything Model) is to automatically segment images. It can perform accurate pixel-level segmentation and achieve strong generalization capabilities through zero-sample learning. That is, SAM uses a large-scale general image dataset for pre-training, which covers a wide variety of scenes, objects, and image features.

[0003] However, when applied to specific fields or tasks, the actual data distribution may be very different from the pre-training data. That is, when SAM is applied to SEM (scanning electron microscope) image segmentation, since SEM images are images generated by scanning electron microscopes and are mainly used to observe the surface morphology and composition of samples, their image features (such as grayscale value, texture, shape, etc.) are very different from general images.

[0004] In the prior art, when SAM is applied to SEM image segmentation, SAM is usually pre-trained on the basis of a large number of SEM image samples to improve the segmentation accuracy of SAM applied to SEM images. However, since the SEM of each material is quite different, the pre-training method of the prior art is often specific, that is, it is impossible to apply SAM pre-trained on a certain material to the SEM image segmentation of multiple other materials. This will cause SAM to be re-pre-trained when performing graphic segmentation on SEM images of different materials, which is time-consuming and labor-intensive, and it is difficult to obtain SEM image samples of different materials, and it is impossible to directly apply SAM to the accurate and simple segmentation operation of SEM images. Summary of the invention

[0005] An object of the present invention is to provide a SEM image processing method based on SAM to solve the technical problem in the prior art that when SAM is directly applied to SEM image segmentation, the segmentation accuracy is low.

[0006] Another object of the present invention is to further improve the automatic segmentation accuracy of SEM images.

[0007] According to the purpose of the present invention, the present invention provides a SEM image processing method based on SAM, comprising: Using SAM to perform model inference on the SEM image, and obtain multiple groups of segmentation masks, each group of the segmentation masks corresponds to an image concentration area and includes a first mask and at least one second mask, the first mask is a mask with the largest number of white pixels in each group of segmentation masks, and the second mask is a mask with the number of white pixels in each group of segmentation masks within a preset range; Preprocessing all the segmentation masks; Determining whether each of the second masks in each group of the segmented masks belongs to the first mask; If yes, define the second mask as a target mask, define a set of all the target masks as a second mask set, and define a union of white pixels of all the target masks as a first union; Calculating a first intersection-and-union ratio of white pixels of the first mask of each group of the segmented mask and the first union, and determining whether the first intersection-and-union ratio is greater than a first threshold; If yes, retain all the target masks in each of the second mask sets and delete the corresponding first mask; If not, keep the first mask and delete all target masks in the corresponding second mask set; Color filling is performed on all the retained white pixel areas of the target mask or the first mask to obtain a particle color filling map of the SEM image.

[0008] Optionally, the first threshold is any value in the range of 0.7-0.9.

[0009] Optionally, the step of determining whether each of the second masks in each group of the segmented masks belongs to the first mask further includes: Calculating the number of white pixels of all the masks in each group of segmented masks, and defining the mask with the largest number of white pixels as the first mask; Performing intersection operations on the first mask plate and all the second mask plates in sequence to obtain a plurality of first intersections of the first mask plate and each of the second mask plates; Calculating a second intersection-and-union ratio between each of the first intersections and the corresponding second mask, and determining whether the second intersection-and-union ratio is greater than a second threshold, where the second threshold is any value in the range of 0.9-1; If so, it is determined that the second mask belongs to the first mask.

[0010] Optionally, the intersection operation is a bitwise AND operation.

[0011] Optionally, after the step of using the second mask as a target mask, defining a set of all the target masks as a second mask set, and defining a union of white pixels of all the target masks as a first union, the method further includes: The first mask and the second mask set of each group of the segmented mask are stored in a mask dictionary.

[0012] Optionally, each of the first unions is obtained by a bitwise OR operation.

[0013] Optionally, the step of preprocessing all the segmentation masks comprises: Determining whether there is a black hole in the white pixel area of ​​each mask in each group of segmented masks; If yes, calculate the ratio of the number of black pixels in the black hole to the number of white pixels in the corresponding white pixel area; Determining whether the ratio is less than a third threshold; If so, the black hole is determined to be a pre-filled hole, and the pre-filled hole is filled with white.

[0014] Optionally, the step of preprocessing all the segmentation masks further includes: Calculating the area of ​​all the white pixel regions in each mask of each group of the segmented mask, and obtaining a first contour of the white pixel region with the largest area in each mask; Determining whether the area of ​​the white pixel region outside the first outline in each of the mask plates is less than a fourth threshold; If so, the corresponding contour is determined to be an outlier area, and the outlier area is deleted.

[0015] Optionally, the step of color-filling all the retained white pixel areas of the target mask or the first mask to obtain a particle color-filled map of the SEM image further includes: Contour detection is performed on the white pixel area of ​​each target mask, and a minimum circumscribed rectangle is calculated.

[0016] Optionally, the step of color-filling all the retained white pixel areas of the target mask or the first mask to obtain a particle color-filled map of the SEM image further includes: Edge detection is performed on the white pixel region of each of the target mask or the first mask to obtain a particle segmentation edge map of the SEM image.

[0017] The present invention sequentially performs preprocessing, inclusion relationship judgment, calculation and comparison of a first intersection-and-union ratio on a first mask plate and a plurality of second masks in each group of segmented masks obtained through SAM processing, so as to retain all target masks or first masks in each second mask plate set according to the comparison of the first intersection-and-union ratio with a first threshold value, and performs color filling on white pixel areas of all retained target masks or first masks, that is, multi-level analysis of the spatial relationship and geometric morphology between particles in the SEM image is performed through the above steps, and merging and deletion between masks are processed automatically, which not only reduces manual intervention and improves processing efficiency, but also can obtain a SEM segmentation image with higher visualization effect, clearer and more accurate, thereby improving the accuracy of zero-sample trained SAM in SEM image segmentation.

[0018] Furthermore, the present invention determines the black holes in the white pixel area in each mask and fills the black holes smaller than the third threshold with white to eliminate small defects or holes in the mask, thereby improving the quality of SEM image segmentation and the accuracy of subsequent analysis. By performing contour detection and area comparison of the white pixel area on each mask after pre-filling or without black holes, outlier areas with an area smaller than the fourth threshold in each mask are determined and deleted, which can effectively remove noise and irrelevant areas in the segmented area, simplify subsequent image analysis, and further improve the segmentation accuracy and efficiency of the SEM image.

[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings: Figure 1 is a schematic partial flow chart of a SEM image processing method according to one embodiment of the present invention; Figure 2 is a schematic overall flow chart of a SEM image processing method according to an embodiment of the present invention; Figure 3 is a schematic SEM image of a multi-particle target material according to one embodiment of the present invention; Figure 4 yes Figure 3 The SEM image shown is a schematic structural diagram of the first mask after SAM segmentation processing; Figure 5 yes Figure 3 The SEM image shown is a schematic structural diagram of the second mask after SAM segmentation processing; Figure 6 is a schematic particle color filling diagram and a minimum circumscribed rectangle diagram of a first mask according to an embodiment of the present invention; Figure 7 is a schematic grain segmentation edge diagram and a minimum bounding rectangle diagram of a first mask according to an embodiment of the present invention; Figure 8 is a schematic SEM image of a single particle target material according to another embodiment of the present invention; Fig. 9 yes Figure 8 The schematic structure diagram of the black hole identification using the SEM image shown; Fig.10 yes Figure 8 The schematic structure diagram of the SEM image after black hole filling is shown; Fig.11 yes Figure 8 Schematic diagram of the structure of outlier region identification using SEM images shown; Fig.12 yes Figure 8 The SEM image shown is a schematic structural diagram of the outlier region removal; Fig.13 yes Figure 8 The SEM image shown is a schematic structural diagram of the first mask after SAM segmentation processing; Fig.14 yes Figure 8 The SEM image shown is a schematic structural diagram of the second mask after SAM segmentation processing. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0022] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0024] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] Figure 1 is a schematic partial flow chart of a SEM image processing method according to one embodiment of the present invention, Figure 2 is a schematic overall flow chart of a SEM image processing method according to an embodiment of the present invention, Figure 3 is a schematic partial SEM image of a target material according to one embodiment of the present invention, Figure 4 yes Figure 3 The SEM image shown is a schematic diagram of the structure of the first mask after SAM segmentation processing. Figure 5 yes Figure 3 The SEM image shown is a schematic diagram of the structure of the second mask after SAM segmentation processing. Figure 6 is a schematic particle color filling diagram and a minimum circumscribed rectangle diagram of a first mask according to an embodiment of the present invention, Figure 7 is a schematic particle segmentation edge diagram and a minimum bounding rectangle diagram of a first mask according to an embodiment of the present invention, Figure 8 is a schematic SEM image of a single particle target material according to another embodiment of the present invention, Fig. 9 yes Figure 8 The schematic structure diagram of black hole identification using SEM images is shown. Fig.10 yes Figure 8 The schematic structure diagram of the SEM image after black hole filling is shown. Fig.11 yes Figure 8 Schematic diagram of the structure of outlier region identification using SEM images. Fig.12 yes Figure 8 The SEM image shown is a schematic structural diagram of the outlier area removal. Fig.13 yes Figure 8 The schematic structure diagram of the first mask after the SEM image is processed by SAM segmentation. Fig.14 yes Figure 8The SEM image shown is a schematic structural diagram of the second mask after SAM segmentation processing.

[0026] like Figure 1 As shown, the present invention provides a SEM image processing method based on SAM, comprising: Step S100: using SAM to perform model inference on the SEM image, and obtaining multiple groups of segmentation masks, each group of segmentation masks corresponds to an image concentration area and includes a first mask and at least one second mask, the first mask is a mask with the largest number of white pixels in each group of segmentation masks, and the second mask is a mask with the number of white pixels in each group of segmentation masks within a preset range; Step S200: pre-processing all segmentation masks; Step S300: determining whether each second mask of each group of segmented masks belongs to the first mask; Step S350: If yes, define the second mask as the target mask, define the set of all target masks as the second mask set, and define the union of white pixels of all target masks as the first union; Step S400: calculating a first intersection-and-union ratio of white pixels of a first mask of each group of segmented masks and a first union, and determining whether the first intersection-and-union ratio is greater than a first threshold; Step S410: If yes, retain all target masks in each second mask set and delete the corresponding first mask; Step S510: color-filling the white pixel areas of all the retained target masks to obtain a particle color-filled image of the SEM image; Step S420: If not, retain the first mask and delete all target masks in the corresponding second mask set; Step S520: color-filling the reserved white pixel area of ​​the first mask to obtain a particle color-filled image of the SEM image.

[0027] In this embodiment, SAM is first used to perform model inference on the SEM image, and multiple groups of segmented masks are obtained, each group of segmented masks corresponds to an image concentration area and includes a first mask and at least one second mask, all the segmented masks are preprocessed, and then it is determined whether each second mask of each group of segmented masks after preprocessing belongs to the first mask. If it is determined that the second mask belongs to the first mask, the second mask is defined as a target mask, and the set of all target masks is defined as a second mask set, and the white The union of color pixels is defined as the first union, and then the first intersection-and-union ratio of the white pixels of the first mask of each group of segmented masks and the first union is calculated, and it is determined whether the first intersection-and-union ratio is greater than the first threshold. If the first intersection-and-union ratio is greater than the first threshold, all target masks in each second mask set are retained, and the corresponding first mask is deleted. Otherwise, the first mask is retained, and all target masks in the corresponding second mask set are deleted, and the white pixel area of ​​all retained target masks or the first mask is filled with color to obtain a SEM segmented image. Here, the image concentration area is a particle distribution area in the SEM image, that is, SAM divides the SEM image into multiple particle distribution areas, and the number of second masks can be one, two or more.

[0028] In this embodiment, the first mask and multiple second masks in each group of segmented masks obtained by SAM processing are preprocessed, the inclusion relationship is judged, and the first intersection-and-union ratio is calculated and compared in sequence. If the first intersection-and-union ratio is compared with the first threshold, all target masks in each second mask set are retained and the corresponding first mask is deleted. Otherwise, the first mask is retained and all target masks in the corresponding second mask set are deleted, and the white pixel areas of all retained target masks or first masks are filled with color. That is, the spatial relationship and geometric morphology between particles in the SEM image are analyzed at multiple levels through the above steps, and the merging and deletion between masks are processed automatically, which not only reduces manual intervention and improves processing efficiency, but also can obtain a particle color filling map of the SEM image with higher visualization effect, clearer and more accurate, thereby improving the accuracy of zero-sample trained SAM in SEM image segmentation.

[0029] In step S100, the second mask is a mask whose number of white pixels in each group of segmented masks is within a preset range, wherein the preset range is the ratio range of the number of white pixels of the second mask to the number of white pixels of the first mask, and that is, the preset range is any value in [0.5-1), that is, the ratio of the number of white pixels of the second mask to the number of white pixels of the first mask is 0.5, 0.6, 0.7, 0.8, 0.9 or 0.95, and can also be any value in 0.5-1. In other words, the masks whose number of white pixels is set to be more than half of the number of white pixels of the first mask and less than the number of white pixels of the first mask are all second masks, so as to include all images close to the target particles in the SEM image in the segmented mask, prevent omissions when the SAM performs graphic segmentation or subsequent image processing, thereby preliminarily improving the accuracy of the SAM applied to the SEM image segmentation.

[0030] In a further embodiment, the first threshold is any value in the range of 0.7-0.9, that is, the first threshold can be 0.7, 0.75, 0.8, 0.85 or 0.9, or any value in the range of 0.7-0.9. In this embodiment, the first threshold is set to any value in the range of 0.7-0.9, and the first threshold is set to be an adjustable threshold to control the particle accuracy and region merging degree of the segmentation result of the SEM image. That is, when the first threshold approaches 0.9, by judging the size of the first intersection ratio and the first threshold, more image details and small particles are retained to avoid mistaken deletion of small particles, and when the first threshold approaches 0.7, the overlap between the first mask and the second mask is low, and the second mask is still regarded as a partial area of ​​the first mask, that is, more small particle areas are merged and deleted, thereby reducing the number of segmented areas and generating a simpler and more concise segmentation result. In other words, the first threshold in this embodiment can be selected according to the detail level requirements of the SEM image.

[0031] In a preferred embodiment, the first threshold is set to 0.8, that is, when the first threshold is set to 0.8, the degree of overlap between the first mask and the second mask enables the acquired mask to retain certain particle details while merging some small adjacent particle areas, which can effectively avoid excessive splitting and the existence of noise areas. It is suitable for most situations where it is necessary to retain most of the particle shapes and boundaries in the segmentation. It can retain the integrity of the particles and avoid the influence of too many small particles and noise. It is suitable for most SEM image analysis tasks.

[0032] like Figure 2 As shown, in a further embodiment, step S200 includes: Step S210: Determine whether there is a black hole in the white pixel area of ​​each mask in each group of segmented masks, if yes, proceed to step S211, if not, proceed to step S250; Step S211: calculating the ratio of the number of black pixels in the black hole to the number of white pixels in the corresponding white pixel area; After step S211, the following steps are also included: Step S230: determine whether the ratio is less than a third threshold value, if so, proceed to step S231, if not, proceed to step S232; Step S231: determining that the black hole is a pre-filled hole, and filling the pre-filled hole with white; Step S232: retaining black holes; After step S231 or step S232, the method further includes: Step S250: calculating the area of ​​all white pixel regions in each mask of each group of segmented masks, and obtaining the first contour of the largest white pixel region in each mask; Step S260: Determine whether the area of ​​the white pixel region outside the first outline in each mask is less than a fourth threshold value, if so, proceed to step S262, if not, proceed to step S264; Step S262: determining the corresponding white pixel area as an outlier area, and deleting the outlier area; Step S264: retaining the corresponding white pixel area; After step S262 or step S264, the following steps are further included: Step S310: Calculate the number of white pixels of all masks in each group of segmented masks, and define the mask with the largest number of white pixels as the first mask; Step S320: performing intersection operations on the first mask and all the second masks in sequence to obtain a plurality of first intersections of the first mask and each of the second masks; Step S330: Calculate the second intersection-over-union ratio of each first intersection and the corresponding second mask, and determine whether the second intersection-over-union ratio is greater than a second threshold value, where the second threshold value is any value in the range of 0.9-1. If so, proceed to step S350; if not, proceed to step S360. Step S360: determining that there is no inclusion relationship between the first mask and the corresponding second mask, and retaining the corresponding second mask; After step S350, the following steps are also included: Step S370: storing the first mask and the second mask set of each group of segmented masks in a mask dictionary; After step S370, the following steps are also included: Step S400: Calculate the first intersection-over-union ratio of the white pixels of the first mask of each group of segmented masks and the first union, and determine whether the first intersection-over-union ratio is greater than a first threshold, if so, proceed to step S410, if not, proceed to step S420; Step S410: retaining all target masks in each second mask set and deleting the corresponding first mask; Step S420: retaining the first mask in each group of segmented masks, and deleting the corresponding second mask set; After step S410, the following steps are also included: Step S510: color-filling the white pixel areas of all retained target masks to obtain a SEM segmentation image; After step S420, the following steps are also included: Step S520: color-filling the reserved white pixel area of ​​the first mask to obtain a SEM segmentation image.

[0033] In this embodiment, after the SEM image is segmented by using SAM, the black holes and outlier regions of each mask in each group of segmented masks are first determined, and the black holes are filled with white and the outlier regions are deleted in turn. Then, the first mask in each group of segmented masks is sequentially intersected with all the second masks to obtain multiple first intersections of the first mask and each second mask, and the second intersection-and-union ratio of each first intersection with the corresponding second mask is calculated and the second intersection-and-union ratio is compared with the second threshold to determine whether the second mask belongs to the first mask. The second mask plate belonging to the first mask plate is defined as the target mask plate, the union of the white pixels of all the target mask plates is defined as the first union, and the intersection and union ratio of the first union and the white pixels in the corresponding first mask plate is calculated, and the first intersection and union ratio obtained is compared with the first threshold value to determine whether to retain or delete all the target mask plates in the first mask plate and the second mask plate set in each group of segmented mask plates, and the mask plates confirmed to be retained are selectively detected by contour detection, edge detection or color filling to meet the different needs of users for SEM segmented images. Here, color filling, contour detection and edge detection are three independent SEM image processing methods, and users can choose single or multiple processing methods according to their needs.

[0034] In step S200, the preprocessing method for all segmented masks includes black hole judgment and filling and outlier area judgment and deletion, that is, by judging the black holes in the white pixel area of ​​each mask and filling the black holes smaller than the third threshold with white, small defects or holes in the mask are eliminated, the quality of SEM image segmentation and the accuracy of subsequent analysis are improved, and the contour detection and area comparison of the white pixel area of ​​each mask after pre-filling or without black holes are performed to determine the outlier area with an area smaller than the fourth threshold in each mask and delete it, which can effectively remove noise and irrelevant areas in the segmented area, simplify subsequent image analysis, and further improve the segmentation accuracy and efficiency of the SEM image.

[0035] like Fig. 9 As shown, in this embodiment, since the particle surface in the SEM image is usually irregular, black holes may appear in the output mask during the SAM segmentation of the SEM image (refer to Fig. 9 ), the black holes in the mask can be the irregular surface morphology of the particles in the SEM image, or the noise generated when the SAM segmented the image. By setting the third threshold, the black holes are identified and screened, so that the black holes smaller than the third threshold are retained and filled with white (refer to Fig.10 ), so that the white area pixels in the mask output by SAM are complete, and black holes larger than the third threshold are retained, so that the white pixel area in the mask output by SAM is closer to the original image of the SEM image, thereby further improving the segmentation accuracy of the SEM image. Here, the third threshold can be 0.1, and the third threshold can be set according to the SEM image of the corresponding material and the image segmentation accuracy requirements. There is no limitation on the third threshold here.

[0036] like Fig.11 As shown, in this embodiment, in the process of identifying the outlier area in each mask of each group of segmented masks and determining whether to delete the outlier area, the fourth threshold can be set to 0.5, that is, when each white pixel area in each mask whose area is smaller than the first contour is smaller than half of the area of ​​the first contour, it is determined to be an outlier area and deleted, the largest contour is obtained, the area of ​​the remaining areas is determined, and finally the outlier area is deleted (refer to Fig.12 ), to remove noise and irrelevant areas, thereby improving the accuracy and efficiency of SEM image segmentation. Here, the fourth threshold value may also be 0.4 or 0.6, or any other value that meets the SEM image segmentation accuracy requirement. The fourth threshold value is not limited here.

[0037] In step S300, the number of white pixels of all masks in each group of segmented masks after preprocessing is calculated and sorted, and it is defined that the first mask has the largest number of white pixels. The first mask and all the second masks in each corresponding group of segmented masks are sequentially subjected to intersection operations to obtain a plurality of first intersections. The second intersection-and-union ratio of each first intersection with the corresponding second mask is calculated to be greater than a second threshold value to determine whether each second mask in each group of segmented masks belongs to the first mask. That is, when the second intersection-and-union ratio is greater than the second threshold value, it is determined that The corresponding second mask belongs to the first mask and is defined as the target mask, and then the set of all target masks is defined as the second mask set, that is, all the second masks belonging to the first mask in each group of segmented masks are classified as the second mask set, so as to facilitate the subsequent judgment of the retention or deletion of the first mask or the second mask set, and avoid the accidental deletion of the mask not included in the first mask when the second mask set is deleted later, that is, improve the integrity of the SEM image after SAM segmentation, thereby further improving the segmentation accuracy of the SEM image. Here, the second threshold is any value in the range of 0.9-1, that is, the second threshold can be 0.9, 0.92, 0.94, 0.96, 0.98 or 1, or any value in 0.9-1.

[0038] In step S370, the mask dictionary includes the first mask and the second mask set of each group of segmented masks, that is, all target masks in the second mask set and the first mask are included in the mask dictionary, which can manage the first mask and all target masks included in the first mask in an orderly manner, facilitate key access and quick search, and support efficient subsequent operations, such as deletion, update, merging, etc., providing a good data basis for subsequent processing of segmentation results. At the same time, the setting of the mask dictionary can avoid repeated calculation of the intersection and union ratio of the same area in the subsequent processing process, and only needs to check whether the small mask contained therein meets the deletion condition, without the need to repeatedly traverse the area, reducing the redundancy of the calculation and improving the overall calculation efficiency. In particular, when processing a large number of segmented masks, the calculation time can be significantly shortened.

[0039] In step S510, by color filling the white pixel area in the reserved mask, the visualization effect of the segmentation result is improved, so that each particle is easier to distinguish and analyze in the image, which is convenient for manual or machine observation and analysis of the segmentation effect. In addition, different particles can be distinguished by color, which helps to understand the relationship between particles and the distribution of particles, and is convenient for distinguishing the distribution of single particles in a multi-particle mask.

[0040] After step S410, the following steps are also included: Perform contour detection on the white pixel area of ​​each target mask and calculate the minimum bounding rectangle.

[0041] In this embodiment, the shape, position and external boundary of the particles in the SEM image are located, that is, through contour detection and calculation of the minimum circumscribed rectangle, an accurate boundary box can be provided for each particle, which is convenient for subsequent region identification, visualization of segmentation results, and relationship analysis between particles. Here, the contour detection method can be a contour detection algorithm in OpenCV (Open Source Computer Vision Library, open source model) or other computer vision libraries.

[0042] After step S410, the following steps are also included: Perform edge detection on each target mask to obtain a segmented edge map of the SEM image.

[0043] In this embodiment, edge detection is performed on each target mask to obtain a segmented edge map of the SEM image, that is, by detecting the edges of the particles in the SEM image, a clearer particle outline is obtained, thereby helping to analyze the shape, size and spatial relationship of the particles, avoiding fuzzy boundaries that may appear in segmentation, and assisting in further analyzing the morphological characteristics of the particles, such as the geometric shape, area, and perimeter. Here, Figure 3 , Figure 6 and Figure 7 KYKY-EM8000F is the product model of the field emission scanning electron microscope of Zhongke Instruments, SE is the secondary electron, which is a signal collected in the scanning electron microscope, mainly used to reflect the morphological characteristics of the sample surface, HV is the acceleration voltage, Mag is the image magnification, and WD is the working distance from the sample surface to the lower end of the objective lens. In other words, Figure 3 , Figure 6 and Figure 7 All of them were obtained by scanning using a KYKY-EM8000F field emission scanning electron microscope produced by Zhongke Instruments. The signals obtained during the scanning process were secondary electrons, the acceleration voltage was 7kV, the image magnification was 5000 times, and the working distance from the sample surface to the lower end of the objective lens was 9.285mm.

[0044] In step S420, if it is determined that the first intersection-over-union ratio is less than the first threshold, the first mask is retained and all target masks in the second mask set are deleted, that is, the target masks in the second mask set are determined to be attached particles of the first mask. For example, in the SEM image of a single particle material (refer to Figure 8 ), the second mask completely belongs to the first mask (refer to Figure 8 ), that is, the first mask (refer to Fig.13 ) is much larger than the second mask (refer to Fig.14 ) white pixel area, then determine the second mask (refer to Fig.14) are attached particles and are deleted to retain the first mask that is closer to the morphological features of the target material. That is to say, the SEM image processing method in the present invention can perform multi-level screening and processing according to the morphological features of the target material to retain the SEM image closest to the target material, making the segmentation result more accurate. Here, the single particle material can be hydroxy iron powder, Figure 8 KYKY-EM6900LV is the product model of the field emission scanning electron microscope of Zhongke Instruments, HV is the accelerating voltage, WD is the working distance from the sample surface to the lower end of the objective lens, Mag is the image magnification, and DET:SE stands for Figure 8 For detection using a secondary electron detector (SE), Vac:High Vac means that the vacuum state in the sample chamber is in high vacuum mode, and KYKY Sample is the sample observed by a scanning electron microscope at the China Science and Technology Instrument.

[0045] In one embodiment, after step S420, the method further includes: The contour of the white pixel area of ​​the first mask is detected, and the minimum bounding rectangle is calculated.

[0046] In another embodiment, step S420 further includes: The first mask is edge detected to obtain a segmented edge map of the SEM image. In a further embodiment, the intersection operation is a bitwise AND operation. In this embodiment, the bitwise AND operation is to compare the corresponding pixels of the second mask with the first mask. When the pixel values ​​at the same position of the two masks are both 1, the position of the corresponding pixel is the overlapping area between the first mask and the second mask, so as to ensure that the actual intersection of the two masks is calculated, and at the same time, irrelevant areas are removed to ensure that only the overlapping parts are retained, which helps to clearly define and distinguish the subordinate relationship between each mask and the same particle or structure. In addition, since the bitwise AND operation is a very efficient bit-level operation in a computer, it usually does not require complex loops and conditional judgments, and can directly operate the binary bit of each pixel, with fast calculation speed and low calculation overhead, thereby improving the efficiency of determining the subordinate relationship between the first mask and the second mask in each group of segmented masks of the SEM image.

[0047] In a further embodiment, each first union is obtained by a bitwise OR operation. In this embodiment, the union of all target masks in each second mask set, i.e., the first union, is obtained by a bitwise OR operation, so that all target masks in all second mask sets are integrated into the first union, that is, all second masks included in the first mask in each group of segmented masks are retained in the first union, so that after the subsequent judgment of the retention or deletion of the first mask or the second mask set, the redundant parts repeated with the retained mask can be accurately deleted, thereby reducing the workload of subsequent image analysis and improving the segmentation accuracy of the SEM image.

[0048] The present application will be further described in detail below with reference to specific embodiments.

[0049] First get Figure 3 The SEM image of the target material shown in FIG. 1 is segmented by using SAM to obtain a segmentation mask, wherein the segmentation mask includes a first mask (refer to Figure 4 ) and multiple second masks (refer to Figure 5 ), by sequentially performing black hole identification and filling operations on a first mask and a plurality of second masks and identifying and deleting outlier regions, and then judging the subordinate relationship between the first mask and the plurality of second masks, after determining to retain the second mask, deleting the first mask, and selectively performing color filling, contour detection or edge detection on the retained target mask, thereby realizing automatic segmentation of the SEM image.

[0050] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A SEM image processing method based on SAM, characterized in that: include: Using SAM to perform model inference on the SEM image, and obtain multiple groups of segmentation masks, each group of the segmentation masks corresponds to an image concentration area and includes a first mask and at least one second mask, the first mask is a mask with the largest number of white pixels in each group of segmentation masks, and the second mask is a mask with the number of white pixels in each group of segmentation masks within a preset range; Preprocessing all the segmentation masks; Determining whether each of the second masks in each group of the segmented masks belongs to the first mask; If yes, define the second mask as a target mask, define a set of all the target masks as a second mask set, and define a union of white pixels of all the target masks as a first union; Calculating a first intersection-and-union ratio of white pixels of the first mask of each group of the segmented mask and the first union, and determining whether the first intersection-and-union ratio is greater than a first threshold; If yes, retain all the target masks in each of the second mask sets and delete the corresponding first mask; If not, keep the first mask and delete all target masks in the corresponding second mask set; Color filling is performed on all the retained white pixel areas of the target mask or the first mask to obtain a particle color filling map of the SEM image.

2. The SEM image processing method according to claim 1, characterized in that: The first threshold is any value in the range of 0.7-0.

9.

3. The SEM image processing method according to claim 2, characterized in that: The step of judging whether each of the second mask plates in each group of the segmented mask plates belongs to the first mask plate also includes: Calculating the number of white pixels of all the masks in each group of segmented masks, and defining the mask with the largest number of white pixels as the first mask; Performing intersection operations on the first mask plate and all the second mask plates in sequence to obtain a plurality of first intersections of the first mask plate and each of the second mask plates; Calculating a second intersection-and-union ratio between each of the first intersections and the corresponding second mask, and determining whether the second intersection-and-union ratio is greater than a second threshold, where the second threshold is any value in the range of 0.9-1; If so, it is determined that the second mask belongs to the first mask.

4. The SEM image processing method according to claim 3, characterized in that: The intersection operation is a bitwise AND operation.

5. The SEM image processing method according to claim 4, characterized in that: After the step of using the second mask as a target mask, defining a set of all the target masks as a second mask set, and defining a union of white pixels of all the target masks as a first union, the method further includes: The first mask and the second mask set of each group of the segmented mask are stored in a mask dictionary.

6. The SEM image processing method according to any one of claims 1 to 5, characterized in that: Each of the first unions is obtained by a bitwise OR operation.

7. The SEM image processing method according to claim 6, characterized in that: The step of preprocessing all the segmentation masks comprises: Determining whether there is a black hole in the white pixel area of ​​each mask in each group of segmented masks; If yes, calculate the ratio of the number of black pixels in the black hole to the number of white pixels in the corresponding white pixel area; Determining whether the ratio is less than a third threshold; If so, the black hole is determined to be a pre-filled hole, and the pre-filled hole is filled with white.

8. The SEM image processing method according to claim 7, characterized in that: The step of preprocessing all the segmentation masks further comprises: Calculating the area of ​​all the white pixel regions in each mask of each group of the segmented mask, and obtaining a first contour of the white pixel region with the largest area in each mask; Determining whether the area of ​​the white pixel region outside the first outline in each of the mask plates is less than a fourth threshold; If so, the corresponding contour is determined to be an outlier area, and the outlier area is deleted.

9. The SEM image processing method according to claim 1, characterized in that: The step of color-filling all the retained white pixel areas of the target mask or the first mask to obtain a particle color-filled map of the SEM image also includes: Contour detection is performed on the white pixel area of ​​each target mask or the first mask, and a minimum circumscribed rectangle is calculated.

10. The SEM image processing method according to claim 1, characterized in that: The step of color-filling all the retained white pixel areas of the target mask or the first mask to obtain a particle color-filled map of the SEM image also includes: Edge detection is performed on the white pixel region of each of the target mask or the first mask to obtain a particle segmentation edge map of the SEM image.

Images