Scanning electron microscope stacked particle image interaction instance segmentation method

Abstract

The invention provides a scanning electron microscope stacked particle image interaction instance segmentation method. The method comprises the following steps: step (1), constructing an image data set and dividing the image data set into a training set, a verification set and a test set; step (2), on the basis of the YOLOv8n backbone network, adding a continuous feed-in cascade self-attention module, and constructing the backbone network; step (3), constructing a neck network; step (4), constructing an efficient series-parallel connection type segmentation head; (5) establishing a segmentation network according to the constructed backbone network, neck network and segmentation head; training the segmentation network by using the training set and the verification set to obtain a segmentation model; and (6) inputting test set data into the segmentation model to obtain a segmentation result. The technical problems that in the prior art, a manual recognition method is low in man-machine interaction degree, low in efficiency, high in consumed cost and high in error rate can be solved.

Description

Technical Field

[0001] The invention relates to the technical field of image segmentation computer vision, and in particular to a method for interactive instance segmentation of stacked particle images using a scanning electron microscope. Background Art

[0002] With the development of scanning electron microscopes, the distribution state, density, quantity, radius and other properties of nano- and micro-level granular microscopic objects have become the key research direction of new energy materials. The distribution, radius and quantity of particles of different microscopic materials will determine the macroscopic properties and application scenarios of the material to varying degrees. Nowadays, the qualitative and quantitative analysis of scanning electron microscope images relies heavily on the experience of researchers, and it is difficult to efficiently and accurately identify stacked particle images of scanning electron microscopes.

[0003] It is difficult to directly extract key information from the images obtained by electron microscope without processing, so it is necessary to use image processing technology to segment the stacked particle images to obtain information such as particle area and quantity. In the existing technology, the method of manual counting and statistical time cost has problems such as high cost, low efficiency and high error rate; most of the existing segmentation methods are based on traditional digital image processing methods, which have problems such as high false detection rate and missed detection rate, and low human-computer interaction. Summary of the invention

[0004] The purpose of the present invention is to provide a method for interactive instance segmentation of stacked particle images of a scanning electron microscope to solve the above technical problems.

[0005] A scanning electron microscope stacked particle image interactive instance segmentation method of the present invention comprises the following steps:

[0006] Step (i), constructing a data set of scanning electron microscope stacked particle images, dividing the data set into a training set, a validation set, and a test set, and performing instance-level annotation on the scanning electron microscope stacked particle images;

[0007] Step (ii), based on the YOLOv8n backbone network, add a continuous feed cascade self-attention module to build the backbone network;

[0008] Step (iii), constructing the neck network;

[0009] Step (iv), constructing a high-efficiency serial-parallel segmentation head for stacked particles of a scanning electron microscope, including three branches: generating mask initialization results, generating mask matrices, and generating mask coefficients:

[0010] (1) The branch that generates the mask initialization result is to first input the scanning electron microscope stack particle image feature X inThe dimension is C×H×W for mask initialization, and 3×3 convolution, 2×2 transposed convolution, 3×3 convolution are performed in sequence for upsampling, and then 1×1 convolution is performed for dimensionality reduction to obtain the mask initialization result X proto ;

[0011] (2) The branch that generates the mask matrix first processes the input SEM stacked particle image features, and then generates a mask matrix based on the layer i Perform i serial-parallel convolution operations, i∈{1,2,…,n}; the serial convolution operation includes inputting two parallel 3×3 convolutions and two sets of parallel 1×1 convolutions to obtain the mask matrix X mask ;

[0012] (3) The branch for generating mask coefficients first calculates the mask coefficients of the input SEM stacked particle image features; i Perform i times of 3×3 convolution, 3×3 convolution and 1×1 convolution to obtain the mask coefficient X of the i-layer feature map ratio ;

[0013] Step (V), constructing a scanning electron microscope stacked particle instance segmentation network based on the constructed backbone network, neck network and segmentation head; training the scanning electron microscope stacked particle instance segmentation network using the training set and validation set obtained in step (I) to obtain a scanning electron microscope stacked particle instance segmentation model;

[0014] Step (six): using the test set data, inputting it into the segmentation model to obtain the segmentation result.

[0015] As a further preferred embodiment of the above scheme:

[0016] In the step (i), the scanning electron microscope stacked particle image is segmented to construct a scanning electron microscope stacked particle image segmentation data set.

[0017] As a further preferred embodiment of the above scheme:

[0018] The segmentation method is as follows: (1) selecting the grid size: for an n*m image, determining the size of each sub-image to be p*q; (2) calculating the number of grids: number of rows = n / p, number of columns = m / q; (3) segmenting the image: segmenting the image from left to right and from top to bottom according to the grid; the sub-image size of each grid area is p*q.

[0019] As a further preferred embodiment of the above scheme:

[0020] The structure of the backbone network in step (i) includes a group of 3×3 convolutions, 3×3 convolutions, C2f modules, three groups of 3×3 convolutions, C2f modules, an SPPF module, and a continuous feed cascade self-attention module.

[0021] As a further preferred embodiment of the above scheme:

[0022] The calculation formula for the continuous feed into the cascaded self-attention module is:

[0023]

[0024] Among them, W i Q , W i K , W i V are the weight factors of Q, K, and V vectors respectively, Attention() is the self-attention calculation, X i represents the feature of input self-attention, Represents the output result of self-attention;

[0025]

[0026] Among them, W i P represents the linear mapping weight, j represents the number of split heads, and there are h heads in total; Represents the output of the cascaded self-attention module.

[0027] As a further preferred embodiment of the above scheme:

[0028] The neck network of step (iii) adopts a dual-stream FPN structure, including an upsampling module, a splicing operation, a C2f module and a 3×3 convolution.

[0029] As a further preferred embodiment of the above scheme:

[0030] The mask initialization result X in step (iv) proto The calculation formula is:

[0031] X proto =Conv 1×1 (Conv 3×3 (TransConv 2×2 (Conv 3×3 (X in )))).

[0032] As a further preferred embodiment of the above scheme:

[0033] The mask matrix X in step (iv) mask The calculation formula is:

[0034] X mask =Concat(2×||i×Conv 1×1 ((i×Conv 1×1 (i×||Conv 3×3 (Conv 3×3 (X in )||))||).

[0035] As a further preferred embodiment of the above scheme:

[0036] The calculation formula of the mask coefficient in step (iv) is:

[0037] X ratio =Concat(i×||Conv 1×1 (Conv 3×3 (Conv 3×3 (X in )))||);

[0038] Among them, X in represents the input image, X ratio represents the mask coefficient, X mask represents the mask matrix, X proto represents mask initialization, |‖‖| represents parallel structure, TransConv 2×2 Represents a transposed convolution with 2 kernels, Conv 1×1 and Conv 3×3 They represent convolutions with 1 and 3 kernels respectively, Concat represents a concatenation operation; C′ refers to the number of regression output channels for each bounding box.

[0039] The beneficial effects of the present invention are as follows: it can solve the technical problems of low human-computer interaction, low efficiency, high cost and high error rate in the manual recognition method in the prior art, and achieve the technical effect of high-precision human-computer interactive scanning electron microscope particle image instance-level segmentation, which is as follows:

[0040] 1. In order to solve the problems of low segmentation accuracy and poor segmentation effect of stacked occluded particles in traditional methods, and easy false detection and missed detection, the present invention adds an attention mechanism and an improved series-parallel segmentation head to the existing deep network, and further adds a model prompt mechanism through human-computer interaction to the segmentation results of deep learning to improve the segmentation accuracy of the model.

[0041] 2. The present invention first adds an efficient serial-parallel segmentation head to the YOLOv8n benchmark network, and the mask accuracy (mAP50, mAP75, mAP50-95) of the model is improved by 1.3%, 9.1%, and 5.5% respectively; after adding a continuous feed-in cascade self-attention module on the above basis, the model mask-level mAP50, mAP75, and mAP50-95 are significantly improved, and the mask-level mAP is improved by 2.2%, 10.9%, and 7.1% respectively. And further add a model prompt mechanism through human-computer interaction to the segmentation result of YOLOv8n, increase the manual prompt information (points, boxes) of the model, provide auxiliary information for nanoparticle targets that are difficult to segment correctly, make up for the segmentation mask accuracy of the deep model, and then improve the segmentation accuracy of the nano-positioning recognition task.

[0042] 3. The innovation of this patent is mainly reflected in: (1) Adding a continuously fed cascade self-attention module after the fast spatial pyramid module of the backbone network to capture the effective features of the spatial pyramid module. (2) In the head structure, the traditional segmentation head is improved to an efficient serial-parallel segmentation head to quickly enhance the feature extraction process and improve the segmentation accuracy. (3) Adding a human-computer interactive post-processing part to add manual prompts to the output results of the segmentation model, thereby improving the model segmentation results. Adding a continuously fed cascade self-attention module and an efficient serial-parallel segmentation head after the backbone extraction network. Aggregating the global features of stacked particles in the scanning electron microscope to optimize the segmentation effect of adhered particles and occluded particles. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0044] Figure 1 This is a scanning electron microscope image of nanoparticles in Example 1 of the present invention;

[0045] Figure 2 A 200×200 nanoparticle image cut in Example 1 of the present invention;

[0046] Figure 3 It is a structural diagram of the continuous feed cascade self-attention module of the present invention;

[0047] Figure 4 This is a structural diagram of the efficient series-parallel splitter head of the present invention;

[0048] Figure 5 This is a structural diagram of the cascaded self-attention serial-parallel segmentation network of the present invention;

[0049] Figure 6 This is a human-computer interactive flow chart of Embodiment 2 of the present invention;

[0050] Figure 7 This is the segmentation result of the traditional YOLOv8n in the comparative example;

[0051] Figure 8 is the segmentation result of the segmentation method in Example 1 of the present invention;

[0052] Fig. 9 for Figure 8 A local enlarged schematic diagram in FIG.

[0053] Fig.10 The method of embodiment 2 of the present invention is Fig. 9 Schematic diagram after splitting again;

[0054] Fig.11 for Figure 8 Another partial enlarged schematic diagram in;

[0055] Fig.12 The method of embodiment 2 of the present invention is Fig.11 Schematic diagram after re-division. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with specific examples. These examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0057] Example 1

[0058] In this embodiment 1, a method for interactive instance segmentation of a stacked particle image of a scanning electron microscope is provided, comprising the following steps:

[0059] Step 1: Image preprocessing

[0060] See also Figure 1 and Figure 2 , the SEM stacked particle image is segmented to construct a SEM stacked particle image segmentation dataset; specifically, the segmentation process of an n*m image is as follows: (1) Select the grid size: determine the size of each sub-image to be p*q; (2) Calculate the number of grids: number of rows = n / p, number of columns = m / q; (3) Segment the image: divide the image into grids from left to right and from top to bottom; the sub-image size of each grid area is p*q, if the remaining part at the edge of the image is less than p*q, zero padding or other processing can be performed;

[0061] In this embodiment, a scanning electron microscope is used to obtain a scanning electron microscope stacked particle image with a resolution of 800×600 and a particle size range of 20 to 50 microns;

[0062] In this embodiment, the aforementioned scanning electron microscope stacked particle image with a resolution of 800×600 is cut into 12 sub-images with a resolution of 200×200;

[0063] In this embodiment, the purpose of using the segmentation method to preprocess the image is: in view of the problem that the instances of the stacked particles of the scanning electron microscope are densely distributed and the instance pixels account for a small proportion, the detailed features are lost through operations such as model compression and convolution. Therefore, the scanning electron microscope stacked particle image with a resolution of 800×600 is segmented into sub-images with a resolution of 200×200, so as to provide a more balanced training set for the neural network training, so as to obtain a more accurate instance segmentation model; that is, an image of 800×600 can be segmented into 12 sub-images of 200×200, which can introduce less repeated parts of the image and retain the detailed features in the image;

[0064] Step 2: Dataset annotation and division

[0065] After the coordinates of the scanning electron microscope stacked particle image data set are normalized, the data set is divided into a training set, a validation set, and a test set, and the scanning electron microscope stacked particle images are annotated at the instance level. In this embodiment, the instance level annotation is preferably performed in the labelme software;

[0066] In this embodiment, the data ratio of the training set, validation set and test set is 4:1:1, with 100 images as the training set, 25 images as the validation set and 25 images as the test set, wherein the training set, validation set and test set contain 17802, 5684 and 4490 particle instances respectively.

[0067] In this embodiment, the image data of the test set is used to evaluate the network performance; the image data of the training set is used to train the network model; the image data of the validation set is used to evaluate the model in the network training process;

[0068] Step 3: Build the backbone network

[0069] Based on the YOLOv8n backbone network, a continuous feed-in cascaded self-attention module is added to form an improved backbone network; see Figure 5 ,The structure of the backbone network is as follows: it includes a set of 3×3 convolution, 3×3 convolution, C2f modules, three sets of 3×3 convolution, C2f modules, and the SPPF module and the continuous feed cascaded self-attention module in parallel;

[0070] The backbone network constructed in this embodiment can improve the computational efficiency of the model by reducing the spatial resolution of the feature map, and enhance the network's ability to capture high-level semantic information; in this embodiment, the purpose of adding a continuous feed cascade self-attention module after the SPPF module is: compared with visible light images, scanning electron microscope images lack color information, and the spatial and texture detail information is not rich enough. Embedding a continuous feed cascade self-attention module effectively utilizes context-related semantic information and enriches the feature space extracted by the network. The stacked particles of the scanning electron microscope are densely distributed and the occlusion phenomenon is serious. The continuous feed cascade self-attention can establish long-distance dependencies between features. Although there is a certain feature loss in the occluded target, the occluded target can be approximately expressed by the surrounding target features, which can significantly improve the recognition accuracy of the occluded target;

[0071] See also Figure 3 , the calculation formula for continuously feeding the cascaded self-attention is:

[0072]

[0073] In formula (1), W i Q , W i K , W i V are the weight factors of Q, K, and V vectors respectively, Attention() is the self-attention calculation, X i represents the feature of input self-attention, Represents the output result of self-attention;

[0074]

[0075] In formula (2), W i P represents the linear mapping weight, j represents the number of split heads, and there are h heads in total; Represents the output of the cascaded self-attention module continuously fed into it;

[0076] Furthermore, the specific steps of adding a continuous feed-in cascaded self-attention module are:

[0077] (1) The continuous feeding cascade self-attention module first splits the input SEM stacked particle features into channels according to the number of heads. During the training process, the model automatically adjusts the parameters, and the heads containing important information will obtain more channels.

[0078] (2) Perform self-attention calculation on the sub-features of each branch respectively, and generate Q, K, and V vectors through linear mapping; further, the self-attention calculation formula is:

[0079]

[0080] In formula (3), d k is the vector dimension, scaled by the factor Avoid the problem of gradient vanishing or gradient exploding during Softmax calculation;

[0081] Specifically, the self-attention calculation establishes global dependencies in the SEM stacked particle feature sequence by calculating the degree of association between each element and all other elements in the SEM stacked particle feature sequence; each element in the self-attention will establish a dynamically determined association with other elements; the dynamic association is achieved through the weights learned during the training process, which can effectively capture the importance and correlation between different elements in the SEM stacked particle input features;

[0082] (3) The output of each self-attention branch that is continuously fed into the cascaded self-attention will be added to the input sub-feature of the next branch;

[0083] (4) Use the Concat channel to concatenate multiple branch outputs, and then use a linear layer to map the output granular aggregation features;

[0084] Specifically, the strategy of reallocating the number of channels by the number of heads allocates more channels to important modules, which can better learn representations in high-dimensional space, thereby minimizing the loss of feature information. Through multi-head attention and cascade processing, continuously feeding cascaded self-attention can better capture the relationship between SEM stacked particle instances, help distinguish between adherent particles and overlapping particle boundaries, and improve the performance and generalization ability of the model in image segmentation tasks;

[0085] Step 4: Build the neck network

[0086] The neck network adopts the dual-stream FPN structure in the existing technology, which includes an upsampling module, a splicing operation, a C2f module, and a 3×3 convolution. Its main function is to realize the fusion of low-resolution deep semantic features and high-resolution shallow detail features of the multi-scale information output in the backbone network, and restore or increase the spatial resolution of the feature map;

[0087] Step 5: Build the segmentation header

[0088] An efficient serial-parallel segmentation head for stacked particles of a scanning electron microscope is constructed, comprising three branches. The first branch is used to generate a mask initialization result, the purpose of which is to improve the performance of the model through a pre-set or initialized prototype vector during the learning process of the model; the second branch generates a mask matrix, which is used to represent the probability or category label of each pixel belonging to a specific category; the third branch generates a mask coefficient, which is used to balance the coefficient weight of the loss function in the particle image instance segmentation task; the segmentation head in the method of this embodiment is three efficient serial-parallel segmentation heads, which respectively process the three different scale feature maps output by the neck network in step (iv), map the intermediate features of the network to pixel-level category predictions, and thus generate a segmentation result of the same size as the input image;

[0089] See also Figure 4 ,The specific steps of constructing the segmentation head include:

[0090] (1) The branch that generates the mask initialization result is to first input the scanning electron microscope stack particle image feature X in The dimension is C×H×W for mask initialization, and 3×3 convolution, 2×2 transposed convolution, 3×3 convolution are performed in sequence for upsampling, and then 1×1 convolution is performed for dimensionality reduction to obtain the mask initialization result X proto , whose feature dimension is C×2H×2W; as shown below:

[0091] X proto =Conv 1×1 (Conv 3×3 (TransConv 2×2 (Conv 3×3 (X in )))) Formula (4)

[0092] (2) The branch that generates the mask matrix first processes the input SEM stacked particle image features, and then generates a mask matrix based on the layer i Perform i serial-parallel convolution operations, where i∈{1,2,…,n}. The serial convolution operation includes inputting two parallel 3×3 convolutions and two sets of parallel 1×1 convolutions to obtain the mask matrix X mask , whose feature dimension is C′×H×W; as shown below:

[0093] X mask =Concat(2×||i×Conv 1×1 ((i×Conv 1×1 (i×||Conv 3×3 (Conv 3×3 (X in )||))||)Formula (5)

[0094] (3) The branch for generating mask coefficients first calculates the mask coefficients of the input SEM stacked particle image features to adjust the contribution of each instance in the final synthesized SEM stacked particle image. i Perform i times of 3×3 convolution, 3×3 convolution and 1×1 convolution to obtain the mask coefficient X of the i-layer feature map ratio ; as shown below:

[0095] X ratio =Concat(i×||Conv 1×1 (Conv 3×3 (Conv 3×3 (X in )))||) Formula (6)

[0096] Among them, X in represents the input image, X ratio represents the mask coefficient, X mask represents the mask matrix, X proto represents mask initialization, ||·|| represents parallel structure, TransConv 2×2 Represents a transposed convolution with 2 kernels, Conv 1×1 and Conv 3×3 They represent convolutions with 1 and 3 kernels respectively, Concat represents a concatenation operation; C′ refers to the number of regression output channels for each bounding box.

[0097] (4) In the training mode, the efficient serial-parallel convolution segmentation head can output the aforementioned mask coefficients, the SEM stacked particle feature processing results, and the mask initialization results to calculate the loss function;

[0098] The segmentation head constructed in this embodiment can efficiently fuse the local feature information of the SEM stacked particles, realize the information interaction between the feature channels of the SEM stacked particles, and improve the segmentation accuracy of the SEM stacked particles;

[0099] Step 6: Build a segmentation network

[0100] See also Figure 5 According to the backbone network constructed in step (iii), the neck network constructed in step (iv) and the segmentation head constructed in step (v), a scanning electron microscope stacked particle instance segmentation network is constructed.

[0101] Step 7: Training the segmentation model

[0102] Using the training set and the validation set obtained in step (ii) to train the SEM stacked particle instance segmentation network in step (vi) to obtain a SEM stacked particle instance segmentation model;

[0103] Step 8: Test the segmentation model

[0104] The test set data obtained in step (ii) is input into the segmentation model in step (vii) to obtain the segmentation result; the segmentation result obtained in this embodiment 1 is as follows: Figure 8 shown.

[0105] Example 2

[0106] See also Figure 6 Based on Example 1, a scanning electron microscope stacked particle image interactive instance segmentation method of Example 2 further includes:

[0107] Step 9: Prompt segmentation model

[0108] The SAM model is used to further segment the segmentation results obtained in step (eight), combined with Figure 6 , using human-computer interaction combined with the SAM model to predict the image segmentation mask, and performing human-computer interactive refinement based on the preliminary segmentation results; the aforementioned prompt segmentation model is a complete end-to-end neural network architecture, mainly composed of an image encoder, a prompt encoder, and an embedded information decoder, and predicts the image segmentation mask by combining the scanning electron microscope stacked particle image embedding and the artificial prompt embedding; specifically including:

[0109] User interface a: Create a human-computer interactive instance segmentation application interface for SEM stacked particle images, allowing users to upload SEM stacked particle images of any resolution and use the function of providing points or prompt boxes to save instance segmentation results;

[0110] Hint encoder b: The scanning electron microscope stacked particle image human-computer interactive segmentation system will allow the user to upload an image and then use the weights of the scanning electron microscope stacked particle instance segmentation model to predict the image to be segmented to obtain a preliminary segmentation result and provide points or hint boxes by the user. The hint encoder is responsible for converting the points or hint boxes provided by the user into an embedding vector; this embedding vector captures the points or hint boxes, and is used to provide a refined area to guide the prediction of the scanning electron microscope stacked particle instance segmentation model;

[0111] Embedding information decoder c: receives the embedding vector from the hint encoder, generates a corresponding refined region according to the embedding vector, predicts the refined region again using the SEM stacked particle instance segmentation model weight, and obtains a segmentation mask that matches the missed SEM particles in the refined region;

[0112] Real-time interaction and feedback d: Real-time interaction and feedback are implemented in the scanning electron microscope stacked particle image interactive instance segmentation system, allowing the user to provide feedback or further prompts based on the instance segmentation results generated by the scanning electron microscope stacked particle instance segmentation model. The user can independently adjust and save the prompt results and feed back to the model to improve the performance of the scanning electron microscope stacked particle image interactive instance segmentation system.

[0113] In this embodiment 2, first, the input image is encoded using the scanning electron microscope stacked particle instance segmentation model to obtain a feature representation of the scanning electron microscope stacked particle image; then, the prompt encoder receives the point and box prompt information provided by the user, which is also converted into a feature representation after being encoded by the encoder; finally, the embedded information decoder inputs the feature representation output by the prompt encoder into the scanning electron microscope stacked particle instance segmentation model for regional refinement, and combines the result with the preliminary segmentation result to generate a more accurate segmentation mask, see Fig. 9 and Fig.11 After the stacked particle images are processed in sequence by the scanning electron microscope stacked particle image interactive instance segmentation method described in Example 2, the following are obtained: Fig.10 , Fig.11 and Fig.12 The segmentation result is shown.

[0114] Comparative Example

[0115] The present invention solves the problems of high false positive rate and missed positive rate and low human-computer interaction in the existing segmentation methods, and achieves the technical effect of high-precision and high-quality segmentation of stacked particles by scanning electron microscope. The module effect in Example 1 is quantitatively analyzed by a specific ablation experiment and compared with the effect of the prior art (YOLOv8n). The experimental results and experimental description are as follows:

[0116] Table 1 Ablation experiment of improved module

[0117]

[0118] As shown in Table 1 above, P represents precision, R represents recall, and mAP represents mean average precision. mAP50 and mAP75 refer to the test recognition accuracy evaluation indicators when the IoU threshold is set greater than 0.5 and 0.75, respectively. mAP50-95 refers to the average value of 10 mAP values ​​with an interval of 5% from the mAP threshold of 0.5 to the mAP threshold of 0.95. Inference represents the model inference time, and Postprogress represents the model post-processing time. Table 1 above shows that after adding efficient serial-parallel segmentation heads and continuous-feed cascade self-attention in turn to the baseline network YOLOv8n, the model's mask-level mAP50, mAP75, and mAP50-95 have all been significantly improved, with the mask-level mAP increasing by 2.2%, 10.9%, and 7.1%, respectively. Among them, when only the efficient serial-parallel segmentation head is added, the model's mask-level mAP50, mAP75, and mAP50-95 increase by 1.3%, 9.1%, and 5.5%, respectively. The inference time is reduced from 11.6ms to 8.8ms, a reduction of 24%, and the post-processing time is reduced from 1.1ms to 0.9ms, a reduction of 18%. After further adding the continuous feed cascade self-attention, the model's mask-level mAP50, mAP75, and mAP50-95 increased by 0.9%, 1.8%, and 1.6%, respectively. Since this module introduces a certain amount of computation, the model's inference time increased to 11.7ms, which was only 0.1ms longer than the baseline model. As the accuracy continued to improve, the post-processing time was further reduced to 0.8ms, a 27% reduction compared to the baseline model.

[0119] Furthermore, the inference model inference time listed in Table 1 is to illustrate the "efficiency" of "efficient series-parallel segmentation"; the Postprogress post-processing time is to illustrate that the model further reduces the post-processing time in the process of improving the mAP accuracy. These two indicators are inversely proportional. Listing the model inference time and post-processing time can reflect the corresponding relationship between the model time and the model accuracy. From the overall evaluation index data, it can be seen that with the addition of two modules, the mask accuracy has been improved, the post-processing time has been reduced, and the inference time is not much different from the original basic model.

[0120] The present invention has been described in detail with reference to the aforementioned embodiments. For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for interactive instance segmentation of stacked particle images of scanning electron microscope, characterized in that: The following steps are involved: Step (i), constructing a data set of scanning electron microscope stacked particle images, dividing the data set into a training set, a validation set, and a test set, and performing instance-level annotation on the scanning electron microscope stacked particle images; Step (ii), based on the YOLOv8n backbone network, add a continuous feed cascade self-attention module to build the backbone network; Step (iii), constructing the neck network; Step (iv), constructing a high-efficiency serial-parallel segmentation head for stacked particles of a scanning electron microscope, including three branches: generating mask initialization results, generating mask matrices, and generating mask coefficients: (1) The branch that generates the mask initialization result first converts the input scanning electron microscope stacked particle image feature X in The dimension is C×H×W for mask initialization, and 3×3 convolution, 2×2 transposed convolution, 3×3 convolution are performed in sequence for upsampling, and then 1×1 convolution is performed for dimensionality reduction to obtain the mask initialization result X proto ; (2) The branch that generates the mask matrix first processes the input SEM stacked particle image features, and then generates a mask matrix based on the layer i Perform i serial-parallel convolution operations, i∈{1,2,…,n}; the serial convolution operation includes inputting two parallel 3×3 convolutions and two sets of parallel 1×1 convolutions to obtain the mask matrix X mask ; (3) The branch for generating mask coefficients first calculates the mask coefficients of the input SEM stacked particle image features; i Perform i times of 3×3 convolution, 3×3 convolution and 1×1 convolution to obtain the mask coefficient X of the i-layer feature map ratio ; Step (V), constructing a scanning electron microscope stacked particle instance segmentation network based on the constructed backbone network, neck network and segmentation head; training the scanning electron microscope stacked particle instance segmentation network using the training set and validation set obtained in step (I) to obtain a scanning electron microscope stacked particle instance segmentation model; Step (six): using the test set data, inputting it into the segmentation model to obtain the segmentation result.

2. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: In the step (i), the scanning electron microscope stacked particle image is segmented to construct a scanning electron microscope stacked particle image segmentation data set.

3. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 2, characterized in that: The segmentation method is as follows: (1) selecting the grid size: for an n*m image, determining the size of each sub-image to be p*q; (2) calculating the number of grids: number of rows = n / p, number of columns = m / q; (3) segmenting the image: segmenting the image from left to right and from top to bottom according to the grid; the sub-image size of each grid area is p*q.

4. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: The structure of the backbone network in step (i) includes a group of 3×3 convolutions, 3×3 convolutions, C2f modules, three groups of 3×3 convolutions, C2f modules, an SPPF module, and a continuous feed cascade self-attention module.

5. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: The calculation formula for the continuous feed into the cascaded self-attention module is: Among them, W i Q , W i K , W i V are the weight factors of Q, K, and V vectors respectively, Attention() is the self-attention calculation, X i represents the feature of input self-attention, Represents the output result of self-attention; Among them, W i P represents the linear mapping weight, j represents the number of split heads, and there are h heads in total; Represents the output of the cascaded self-attention module.

6. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: The neck network of step (iii) adopts a dual-stream FPN structure, including an upsampling module, a splicing operation, a C2f module and a 3×3 convolution.

7. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: The mask initialization result X in step (iv) proto The calculation formula is: X proto =Conv 1×1 (Conv 3×3 (TransConv 2×2 (Conv 3×3 (X in ))))。 8. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: The mask matrix X in step (iv) mask The calculation formula is: X mask =Concat(2×||i×Conv 1×1 ((i×Conv 1×1 (i×||Conv 3×3 (Conv 3×3 (X in )||))||)。 9. The method for interactive instance segmentation of stacked particle images of a scanning electron microscope according to claim 1, characterized in that: The calculation formula of the mask coefficient in step (iv) is: X ratio =Concat(i×||Conv 1×1 (Conv 3×3 (Conv 3×3 (X in )))||); Among them, X in represents the input image, X ratio represents the mask coefficient, X mask represents the mask matrix, X proto represents mask initialization, ||·|| represents parallel structure, TransConv 2×2 Represents a transposed convolution with 2 kernels, Conv 1×1 and Conv 3×3 They represent convolutions with 1 and 3 kernels respectively, Concat represents a concatenation operation; C′ refers to the number of regression output channels for each bounding box.

Images