A pyrite SEM image segmentation method and device

Abstract

The invention relates to the technical field of electron microscope image processing, and discloses a pyrite SEM image segmentation method and device. The method comprises the following steps: collecting and arranging SEM images to be processed, wherein the SEM images to be processed have image information of strawberry-shaped pyrite; processing the SEM images to be processed by using a target SEM image segmentation model to obtain a SEM image segmentation result; wherein the target SEM image segmentation model is constructed based on a SegBoundary model, the SegBoundary model is constituted by a segformer model to which a SAM boundary perception module is added, and the image segmentation processing of the target SEM image segmentation model comprises an encoder stage and a decoder stage, wherein in the encoder stage, multi-level features of an input image are extracted by a Transformer layer, and boundary features are extracted by using the SAM boundary perception module to obtain multi-layer characteristic information; and in the decoder stage, multi-scale feature fusion and mask segmentation are performed on the multi-layer characteristic information to obtain a SEM image segmentation result.

Description

Technical Field

[0001] The invention relates to the technical field of electron microscope image processing, and in particular to a pyrite SEM image segmentation method and device. Background Art

[0002] Image classification and image segmentation are two important tasks in the field of computer vision. In today's era of continuous progress in sedimentary geology, advanced image classification and image segmentation methods continue to emerge.

[0003] A large amount of geological data information has always been an important material in geological research. Specifically, in the analysis of strawberry pyrite, scholars have been exploring how to quickly and accurately obtain strawberry SEM image datasets based on strawberry SEM images. Among them, strawberry SEM image datasets can include image data, annotation data, geological information, chemical composition, physical properties, etc.

[0004] However, in the related art, the method of obtaining a strawberry-shaped SEM image dataset based on a strawberry-shaped SEM image still mostly adopts manual and semi-automatic tracking methods. The manual method has the problem that manual detection is time-consuming and labor-intensive; and if the strawberry-shaped SEM image dataset is small, it may not fully represent all possible morphological changes and redox conditions, affecting the deposition analysis; in addition, human bias (such as a preference for "better" samples) will also affect the collection of strawberry-shaped SEM image datasets. In addition, the selection of thresholding parameters in the semi-automatic image processing method will also lead to detection differences in the strawberry-shaped SEM image dataset due to subjective bias.

[0005] As far as image segmentation is concerned, accurate segmentation of scanning electron microscope (SEM) images is crucial for fields such as materials science, medicine, and geological sedimentation. However, traditional image segmentation methods rely on complex algorithms or manual processing, which are inefficient and easily affected by subjective factors. In recent years, the development of deep learning technology has provided new solutions for image segmentation. However, existing strawberry-shaped SEM images still face challenges in segmentation, especially in the boundary areas of the image, where the effect is poor.

[0006] In summary, we propose a pyrite SEM image segmentation method and device, aiming to improve the segmentation accuracy. Summary of the invention

[0007] Based on the above, the object of the present invention is to provide a pyrite SEM image segmentation method and device, aiming to improve the segmentation accuracy.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] The first aspect of the present application provides a pyrite SEM image segmentation method, which comprises:

[0010] Collecting and arranging the SEM images to be processed, wherein the SEM images to be processed have image information of strawberry-shaped pyrite;

[0011] Processing the SEM image to be processed using the target SEM image segmentation model to obtain a SEM image segmentation result;

[0012] Among them, the target SEM image segmentation model is constructed based on the SegBoundary model, and the SegBoundary model is composed of a segformer model with a SAM boundary perception module added. The image segmentation processing of the target SEM image segmentation model includes an encoder stage and a decoder stage. In the encoder stage, the multi-level features of the input image are extracted through the Transformer layer, and the boundary features are extracted using the SAM boundary perception module to obtain multi-layer feature information; in the decoder stage, the multi-layer feature information is subjected to multi-scale feature fusion and mask segmentation to obtain the SEM image segmentation result.

[0013] In some embodiments, before the step of processing the SEM image to be processed using the target SEM image segmentation model to obtain the SEM image segmentation result, it also includes training and obtaining the target SEM image segmentation model, and the training and obtaining the target SEM image segmentation model includes:

[0014] Collect and organize the SEM images to be tested, and obtain segmentation labels corresponding to the SEM images to be tested;

[0015] Using the SEM image to be tested and the segmentation labels to create a training data set, a verification data set and a test data set;

[0016] Constructing the SegBoundary model as an initial SEM image segmentation model;

[0017] Based on the training data set and the validation data set, optimizing the initial SEM image segmentation model using a stochastic gradient descent method to obtain an optimized SEM image segmentation model;

[0018] Using the optimized SEM image segmentation model to perform test segmentation on the strawberry-shaped pyrite SEM image in the test data set to obtain a prediction result;

[0019] The prediction result is compared and analyzed with the labeled strawberry pyrite segmentation data corresponding to the test data set to determine the target SEM image segmentation model.

[0020] In some embodiments, the step of using the SEM image to be tested and the segmentation labels to prepare a training data set, a verification data set, and a test data set comprises:

[0021] Preprocessing the SEM image to be tested, scaling it to a preset resolution, and obtaining a preprocessed SEM image;

[0022] The preprocessed SEM images are divided into a training data set, a verification data set and a test data set according to a preset ratio.

[0023] In some embodiments, the step of optimizing the initial SEM image segmentation model based on the training data set and the validation data set using a stochastic gradient descent method to obtain an optimized SEM image segmentation model comprises:

[0024] The initial SEM image segmentation model is trained using an AdamW optimizer and a cross entropy loss; wherein an initial value of the learning rate of the initial SEM image segmentation model is set to 0.00006;

[0025] The initial SEM image segmentation model is optimized using a data enhancement strategy, wherein the data enhancement strategy includes random scaling, random horizontal flipping, and random cropping.

[0026] In some embodiments, comparing and analyzing the prediction result with the labeled strawberry pyrite segmentation data corresponding to the test data set to determine the target SEM image segmentation model includes:

[0027] The first prediction result of the optimized SEM image segmentation model with the SAM boundary perception module added is compared with the second prediction result of the segformer model corresponding to the first prediction structure without the SAM boundary perception module added to analyze the improvement of intersection over union (IoU), recall, precision, accuracy and F1 score, where the F1 score is the harmonic mean of precision and recall.

[0028] In some implementations, in the encoder stage, extracting multi-level features of the input image through the Transformer layer, extracting boundary features using the SAM boundary perception module, and obtaining multi-layer feature information includes:

[0029] Given an input image of size H×W, split the input image into 4×4 patches and embed each patch into the dimensions of H×W×C;

[0030] The patch is passed to the hierarchical Transformer layer. After the self-attention mechanism calculation and feedforward neural network processing of the hierarchical Transformer layer, the overlapping patch merging operation is performed to obtain the hierarchical feature map F i ,in, In the formula, i represents each layer, belonging to the set {1,2,3,4}, and C represents the dimension;

[0031] Input an input image of a given H×W size into an image encoder, extract image features to obtain a preliminary feature map, wherein the image encoder is constructed according to a pre-trained Vit visual transformer, which includes a windowed attention and four equally spaced global attention blocks;

[0032] Scaling and filling the preliminary feature map to obtain a processed feature map;

[0033] Perform convolution and normalization operations on the processed feature map to obtain a reduced-dimensional feature map;

[0034] The reduced dimension feature map is input into the mask decoder, and the self-attention and cross-attention mechanisms are used in the mask decoder to obtain an updated reduced dimension feature map;

[0035] Upsampling the updated reduced-dimensionality feature map to restore the updated reduced-dimensionality feature map to the resolution of the input image, thereby obtaining a feature map after resolution restoration;

[0036] The feature map after resolution restoration is operated by an MLP multi-layer perceptron, and a segmentation mask M of the target in the feature map after resolution restoration is output. b , the target is image information of strawberry-shaped pyrite;

[0037] The boundary feature map F is obtained by using the Canny edge detection algorithm b , the multi-layer feature information includes the hierarchical feature map F i and the boundary feature map F b .

[0038] In some embodiments, the boundary feature map F is obtained by using the Canny edge detection algorithm. b include:

[0039] The segmentation mask M of the target is calculated using the Sobel operator through the Canny algorithm b Gradients in the x and y directions;

[0040] Calculate the magnitude and direction of the gradient to determine the location and direction of the edge; where,

[0041] G x =Sobel(M b,x);

[0042] G y =Sobel(M b ,y);

[0043]

[0044] θ=arctan(G y / G x );

[0045] F b =Canny(G,θ,100,200);

[0046] In the formula, G x and G y The Sobel operator calculates the segmentation mask M of the target in the x and y directions respectively. b The gradient of , G and θ are the gradient magnitude and direction.

[0047] In some embodiments, the patch is passed to the hierarchical Transformer layer, and after the self-attention mechanism calculation and feedforward neural network processing of the hierarchical Transformer layer, an overlapping patch merging operation is performed to obtain a hierarchical feature map F i ,include:

[0048] The self-attention mechanism is used to calculate the self-attention between the input patches to generate the output features. In the self-attention mechanism calculation, each query vector Q, key vector K, and value vector V have the same dimension H×W×C, and the self-attention Attention(Q,K,V) is:

[0049]

[0050] Where, d head represents the dimension of the key, and uses a sequence reduction process for patches in the calculation of the self-attention mechanism, using a reduction ratio R to reduce the sequence length, and the final dimension is for in:

[0051]

[0052] Where K represents the key vector to be reduced;

[0053] The output features are input into a feedforward neural network, which calculates the features by mixing a 3×3 convolution and an MLP multi-layer perceptron module and outputs the final output, where:

[0054] X out =MLP(GELU(Conv 3×3(MLP(x in ))))+x in ;

[0055] In the formula, x in Output features from the self-attention mechanism, X out Represents the final output;

[0056] For the final output of each Transformer layer, perform overlapping patch merging operations to obtain the hierarchical feature map F i , wherein the overlapping patch merging operation includes defining K1, S, and P, wherein K1 is the patch size, S is the step size between adjacent patches, and P is the padding size; setting K1=9, S=5, P=4 or K1=3, S=2, P=1, and performing overlapping patch merging.

[0057] In some embodiments, performing multi-scale feature fusion and mask segmentation in the decoder stage to obtain a SEM image segmentation result includes:

[0058] The multi-layer feature information obtained from the encoder stage is unified in channel dimension through the first MLP layer to obtain unified dimension feature information

[0059] By upsampling the uniform dimensional feature information, the resolution is increased to The size of the upsampling result is obtained Then perform splicing to obtain the spliced ​​features;

[0060] The second MLP layer is used to fuse the concatenated features to obtain the fused features;

[0061] The fused features are input into the third MLP layer to predict the segmentation mask and obtain the final predicted segmentation mask M; where the resolution is in,

[0062]

[0063] M=Linear(C,N class )(F);

[0064] Where M represents the predicted mask, C represents the unified channel dimension, and N class is the number of categories, It represents the concatenation of the upsampling results, F represents the fused features, and M represents the final predicted segmentation mask.

[0065] A second aspect of the present application provides a pyrite SEM image segmentation device, comprising:

[0066] at least one processor;

[0067] a memory communicatively coupled to the at least one processor;

[0068] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the pyrite SEM image segmentation method of any of the above embodiments.

[0069] The device further includes an input unit for receiving a pyrite SEM image, a display unit for displaying a segmentation result, and a storage unit for storing segmented image data and related processing information. The processor is configured to receive a pyrite SEM image from the input unit; execute an image segmentation algorithm according to instructions in the memory to perform segmentation processing on the received pyrite SEM image; transmit the segmented image result to the display unit for display; and store the processed image data and related information in the storage unit for subsequent analysis and processing.

[0070] The beneficial effects of the present invention are:

[0071] The present invention processes the SEM image to be processed by a target SEM image segmentation model. The target SEM image segmentation model combines the efficient segmentation capability of the SegFormer model and the excellent zero-sample segmentation capability of the SAM boundary perception module. By feeding the SEM image to be processed into the target SEM image segmentation model designed by the present invention, a segmentation mask of strawberry pyrite can be automatically generated, which has efficient segmentation capability, and improves the accuracy of strawberry pyrite segmentation through the SAM boundary perception module.

[0072] Compared with other models, the present invention can achieve good results in image segmentation by only performing simple resolution adaptation. Therefore, the present invention has great potential in geological image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the contents of the embodiments of the present invention and these drawings without paying any creative work.

[0074] Figure 1 A schematic flow chart of a pyrite SEM image segmentation method provided by an embodiment of the present invention;

[0075] Figure 2 A schematic diagram of the training process of the SEM image segmentation model provided by an embodiment of the present invention;

[0076] Figure 3 This is the image segmentation result of strawberry-shaped pyrite on the test data set during the training of the SEM image segmentation model of the present invention.

[0077] Figure 4 A schematic diagram of a SEM image segmentation model provided for an embodiment of the present invention; DETAILED DESCRIPTION

[0078] In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0079] In the description of the present invention, unless otherwise clearly specified and limited, the terms "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0080] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0081] In the description of this embodiment, the terms "upper", "lower", "left", "right", etc., and the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of description and simplified operation, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.

[0082] refer to Figure 1-4The first aspect of the present application provides a pyrite SEM image segmentation method, which comprises:

[0083] Step 100, collecting and arranging the SEM images to be processed, wherein the SEM images to be processed have image information of strawberry-shaped pyrite;

[0084] Step 200, using the target SEM image segmentation model to process the SEM image to be processed to obtain a SEM image segmentation result; wherein the target SEM image segmentation model is constructed based on the SegBoundary model, the SegBoundary model is composed of a segformer model with a SAM boundary perception module added, and the image segmentation processing of the target SEM image segmentation model includes an encoder stage and a decoder stage, in which the multi-level features of the input image are extracted through the Transformer layer in the encoder stage, and the boundary features are extracted using the SAM boundary perception module to obtain multi-layer feature information; in the decoder stage, the multi-layer feature information is subjected to multi-scale feature fusion and mask segmentation to obtain the SEM image segmentation result. wherein the SegBoundary model can be written in the Pytorch framework (version 1.10.0).

[0085] In the embodiment of the present application, the SEM image to be processed is processed by a target SEM image segmentation model, and its target SEM image segmentation model combines the efficient segmentation capability of the SegFormer model and the excellent zero-sample segmentation capability of the SAM boundary perception module. By feeding the SEM image to be processed into the target SEM image segmentation model, a segmentation mask of strawberry-shaped pyrite can be automatically generated, which has efficient segmentation capability, and improves the accuracy of strawberry-shaped pyrite segmentation through the SAM boundary perception module. Compared with other models, it only needs to perform simple resolution adaptation to achieve good results in image segmentation. Therefore, it has great potential in geological image segmentation.

[0086] In some embodiments, reference Figure 1-4 , before the step 100 of processing the SEM image to be processed using the target SEM image segmentation model to obtain the SEM image segmentation result, it also includes step 000 of training and obtaining the target SEM image segmentation model. The step 000 of training and obtaining the target SEM image segmentation model includes:

[0087] Step 010, collect and organize the SEM images to be tested, and obtain the segmentation labels corresponding to the SEM images to be tested; interpretatively, the strawberry-shaped pyrite particles can be imaged through an electron microscope to obtain the SEM images to be tested, and then the SEM images to be tested are annotated and interpreted, and the annotated objects are the strawberry-shaped pyrite in the SEM images to be tested.

[0088] Step 020: using the SEM image to be tested and the segmentation labels to create a training data set, a verification data set and a test data set;

[0089] Step 030, constructing the SegBoundary model as an initial SEM image segmentation model;

[0090] Step 040: Based on the training data set and the verification data set, the initial SEM image segmentation model is optimized by using a stochastic gradient descent method to obtain an optimized SEM image segmentation model;

[0091] Step 050, using the optimized SEM image segmentation model to perform test segmentation on the strawberry-shaped pyrite SEM image in the test data set to obtain a prediction result;

[0092] Step 060, compare and analyze the predicted results with the labeled strawberry pyrite segmentation data corresponding to the test data set to determine the target SEM image segmentation model. After experiments, the test results were analyzed and it was confirmed that the IoU (intersection over union) effect was improved from 75.81 to 81.01.

[0093] in, Figure 3 A column corresponding to the original image represents the SEM image to be tested, a column corresponding to the label represents the labeled strawberry pyrite segmentation data corresponding to the test data set, that is, the real segmentation mask of the test data set, a column corresponding to the Boundary can represent the predicted segmentation mask processing result of the traditional Boundary model, a column corresponding to the SegFormer can represent the predicted segmentation mask processing result of the traditional SegFormer model, and a column corresponding to the SegBoundary can represent the predicted segmentation mask processing result of the optimized SEM image segmentation model of the implementation mode of the present application.

[0094] refer to Figure 1-4 In some embodiments, the step 020, using the SEM image to be tested and the segmentation labels to prepare a training data set, a verification data set and a test data set, comprises:

[0095] Step 021, preprocess the SEM image to be tested, scale it to a preset resolution, and obtain a preprocessed SEM image; specifically, in some embodiments, the preset resolution is 512×512.

[0096] Step 022: Divide the pre-processed SEM images into a training data set, a validation data set, and a test data set according to a preset ratio. Specifically, in some embodiments, the preset ratio is 6:1:1.

[0097] In some embodiments, reference Figure 1-4 The step 050, based on the training data set and the verification data set, optimizing the initial SEM image segmentation model using a stochastic gradient descent method to obtain an optimized SEM image segmentation model comprises:

[0098] The initial SEM image segmentation model was trained and optimized using the AdamW optimizer and cross entropy loss, and the best segmentation effect was obtained through continuous optimization; wherein the initial value of the learning rate of the initial SEM image segmentation model was set to 0.00006. After experiments, the initial SEM image segmentation model can achieve an intersection-over-union (IoU) of 81.01% in the test data set after 100 iterations, which can further improve the automatic segmentation capability of strawberry pyrite SEM images.

[0099] refer to Figure 1-4 In some embodiments, the step 050, based on the training data set and the validation data set, optimizing the initial SEM image segmentation model using a stochastic gradient descent method to obtain the optimized SEM image segmentation model includes or may also include optimizing the initial SEM image segmentation model using a data enhancement strategy, wherein the data enhancement strategy includes random scaling, random horizontal flipping, and random cropping.

[0100] In some embodiments, reference Figure 1-4 The step 060, comparing and analyzing the prediction result with the labeled strawberry pyrite segmentation data corresponding to the test data set to determine the target SEM image segmentation model includes:

[0101] The first prediction result of the optimized SEM image segmentation model with the SAM boundary perception module added is compared with the second prediction result of the segformer model corresponding to the first prediction structure without the SAM boundary perception module added to analyze the improvement of intersection over union (IoU), recall, precision, accuracy and F1 score, where the F1 score is the harmonic mean of precision and recall.

[0102] In some embodiments, reference Figure 1-4In step 200, in the encoder stage, the multi-level features of the input image are extracted through the Transformer layer, and the boundary features are extracted using the SAM boundary perception module to obtain multi-layer feature information, including:

[0103] Step 210: according to a given H×W size input image, the input image is divided into 4×4 size patches, and each patch is embedded into the dimensions of H×W×C;

[0104] Step 220: pass the patch to the hierarchical Transformer layer, calculate it through the self-attention mechanism of the hierarchical Transformer layer and process it through the feedforward neural network, and then perform the overlapping patch merging operation to obtain the hierarchical feature map F i ,in, In the formula, i represents each layer, belonging to the set {1,2,3,4}, and C represents the dimension;

[0105] Explanatory, obtain the hierarchical feature map F i Corresponding references Figure 4 Input-Overlap patchEmbeddings-Transformer block1-Transformer block2-Transformer block3-..... Among them, the processing process in the Transformer block can refer to Figure 4 The Transformer block in the lower left corner. The processing in the MLP Layer can be referred to Figure 4 The MLP Layer in the lower left corner.

[0106] Step 230: Input the input image of a given H×W size into the image encoder, extract the image features to obtain a preliminary feature map, wherein the image encoder is constructed according to a pre-trained Vit visual transformer, and the pre-trained Vit visual transformer includes a windowed attention and four equally spaced global attention blocks; specifically, the windowed attention is a 14×14 windowed attention; the resolution of the input image of a given H×W size can be 1024×1024.

[0107] Step 240: scaling and filling the preliminary feature map to obtain a processed feature map;

[0108] Step 250: Perform convolution and normalization operations on the processed feature map to obtain a reduced-dimensional feature map; for explanation, the image encoder can refer to Figure 4Image encoder, i.e., Vit; in some embodiments, 1×1 and 3×3 convolutions are used on the processed feature map to reduce the number of channels to 256, and each convolution is followed by a layer of normalization, and the final resolution of the reduced feature map can be 64×64.

[0109] Step 260: input the reduced dimension feature map into the mask decoder, and use the self-attention and cross-attention mechanisms in the mask decoder to obtain an updated reduced dimension feature map;

[0110] Step 270: up-sample the updated reduced-dimensionality feature map to restore the updated reduced-dimensionality feature map to the resolution of the input image, thereby obtaining a feature map after resolution restoration;

[0111] Step 280: operate the feature map after the resolution is restored through the MLP multi-layer perceptron, and output the segmentation mask M of the target in the feature map after the resolution is restored. b , the target is the image information of strawberry pyrite; for explanation, the mask decoder can refer to Figure 4 Mask decoder in .

[0112] Step 290: Obtain a boundary feature map F by using the Canny edge detection algorithm b , the multi-layer feature information includes the hierarchical feature map F i and the boundary feature map F b .

[0113] refer to Figure 1-4 In some embodiments, the step 290, obtaining the boundary feature map F by using the Canny edge detection algorithm b include:

[0114] Step 291: Calculate the segmentation mask M of the target using the Sobel operator through the Canny algorithm. b Gradients in the x and y directions;

[0115] Step 292: Calculate the magnitude and direction of the gradient to determine the position and direction of the edge; wherein,

[0116] G x =Sobel(M b ,x);

[0117] G y =Sobel(M b ,y);

[0118]

[0119] θ=arctan(G y / G x);

[0120] F b =Canny(G,θ,100,200);

[0121] In the formula, G x and G y The Sobel operator calculates the segmentation mask M of the target in the x and y directions respectively. b The gradient of , G and θ are the gradient magnitude and direction.

[0122] In some embodiments, reference Figure 1-4 In step 220, the patch is passed to the hierarchical Transformer layer, and after the self-attention mechanism calculation and feedforward neural network processing of the hierarchical Transformer layer, the overlapping patch merging operation is performed to obtain the hierarchical feature map F i ,include:

[0123] Step 221, calculate the self-attention between each patch of the input through the self-attention mechanism to generate output features; wherein, in the self-attention mechanism calculation, each query vector Q, key vector K, and value vector V have the same dimension H×W×C, and the self-attention Attention(Q,K,V) is:

[0124]

[0125] Where, d head represents the dimension of the key, and uses a sequence reduction process for patches in the calculation of the self-attention mechanism, using a reduction ratio R to reduce the sequence length, and the final dimension is for in:

[0126]

[0127] Where K represents the key vector to be reduced;

[0128] Step 222: Input the output features into a feedforward neural network, which calculates the features by mixing 3×3 convolution and an MLP multi-layer perceptron module and outputs the final output, wherein:

[0129] X out =MLP(GELU(Conv 3×3 (MLP(x in ))))+x in ;

[0130] In the formula, x in Output features from the self-attention mechanism, X out Represents the final output;

[0131] Step 223: For the final output of each Transformer layer, perform overlapping patch merging operation to obtain a hierarchical feature map F i , wherein the overlapping patch merging operation includes defining K1, S, P, wherein K1 is the patch size, S is the step size between adjacent patches, and P is the padding size; setting K1=9, S=5, P=4 or K1=3, S=2, P=1, and performing overlapping patch merging. Performing overlapping patch merging can ensure local continuity around the patch.

[0132] refer to Figure 1-4 In some embodiments, the step 200 of performing multi-scale feature fusion and mask segmentation at the decoder stage to obtain a SEM image segmentation result includes:

[0133] Step 2100: The multi-layer feature information obtained from the encoder stage is unified in channel dimension through the first MLP layer to obtain unified dimension feature information. The multi-layer feature information includes the hierarchical feature map F i and the boundary feature map F b .

[0134] Step 2200: Upsampling the uniform dimension feature information to increase the resolution to The size of the upsampling result is obtained Then perform splicing to obtain the spliced ​​features;

[0135] Step 2300: Use the second MLP layer to fuse the spliced ​​features to obtain fused features;

[0136] Step 2400: Input the fused features into the third MLP layer, predict the segmentation mask, and obtain the final predicted segmentation mask M; wherein the resolution is in,

[0137]

[0138]

[0139] M=Linear(C,N class )(F);

[0140] Where M represents the predicted mask, C represents the unified channel dimension, and N class is the number of categories, It represents the concatenation of the upsampling results, F represents the fused features, and M represents the final predicted segmentation mask.

[0141] In addition, the embodiment of the present application also provides a pyrite SEM image segmentation device, which includes at least one processor and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the pyrite SEM image segmentation method of any of the above embodiments. The image segmentation algorithm is executed according to the instructions in the memory to segment the received pyrite SEM image; the segmented image result is transmitted to the display unit for display; the processed image data and related information are stored in the storage unit for subsequent analysis and processing.

[0142] In addition, in the embodiment of the present application, the device further includes an input unit for receiving a pyrite SEM image, a display unit for displaying a segmentation result, and a storage unit for storing segmented image data and related processing information. The processor is configured to receive a pyrite SEM image from the input unit; execute an image segmentation algorithm according to instructions in the memory to perform segmentation processing on the received pyrite SEM image; transmit the segmented image result to the display unit for display; and store the processed image data and related information in the storage unit for subsequent analysis and processing.

[0143] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A pyrite SEM image segmentation method, characterized in that: include: Collecting and arranging the SEM images to be processed, wherein the SEM images to be processed have image information of strawberry-shaped pyrite; Processing the SEM image to be processed using the target SEM image segmentation model to obtain a SEM image segmentation result; The target SEM image segmentation model is constructed based on the SegBoundary model, which is composed of a segformer model with a SAM boundary perception module added thereto. The image segmentation processing of the target SEM image segmentation model includes an encoder stage and a decoder stage. In the encoder stage, the multi-level features of the input image are extracted through the Transformer layer, and the boundary features are extracted using the SAM boundary perception module to obtain multi-layer feature information; in the decoder stage, the multi-layer feature information is subjected to multi-scale feature fusion and mask segmentation to obtain the SEM image segmentation result. In the encoder stage, the multi-level features of the input image are extracted through the Transformer layer, and the boundary features are extracted using the SAM boundary perception module to obtain multi-layer feature information, including: Given an input image of size H×W, split the input image into 4×4 patches and embed each patch into the dimensions of H×W×C; The patch is passed to the hierarchical Transformer layer. After the self-attention mechanism calculation and feedforward neural network processing of the hierarchical Transformer layer, the overlapping patch merging operation is performed to obtain the hierarchical feature map F i ,in, i∈{1,2,3,4}, where i represents each layer, belonging to the set {1,2,3,4}, and C represents the dimension; Input an input image of a given H×W size into an image encoder, extract image features to obtain a preliminary feature map, wherein the image encoder is constructed according to a pre-trained Vit visual transformer, which includes a windowed attention and four equally spaced global attention blocks; Scaling and filling the preliminary feature map to obtain a processed feature map; Perform convolution and normalization operations on the processed feature map to obtain a reduced-dimensional feature map; The reduced dimension feature map is input into the mask decoder, and the self-attention and cross-attention mechanisms are used in the mask decoder to obtain an updated reduced dimension feature map; Upsampling the updated reduced-dimensionality feature map to restore the updated reduced-dimensionality feature map to the resolution of the input image, thereby obtaining a feature map after resolution restoration; The feature map after resolution restoration is operated by an MLP multi-layer perceptron, and a segmentation mask M of the target in the feature map after resolution restoration is output. b , the target is image information of strawberry-shaped pyrite; The boundary feature map F is obtained by using the Canny edge detection algorithm b , the multi-layer feature information includes the hierarchical feature map F i and the boundary feature map F b .

2. The pyrite SEM image segmentation method according to claim 1, characterized in that: Before the step of processing the SEM image to be processed by using the target SEM image segmentation model to obtain the SEM image segmentation result, it also includes training and obtaining the target SEM image segmentation model, and the training and obtaining the target SEM image segmentation model includes: Collect and organize the SEM images to be tested, and obtain segmentation labels corresponding to the SEM images to be tested; Using the SEM image to be tested and the segmentation labels to create a training data set, a verification data set and a test data set; Constructing the SegBoundary model as an initial SEM image segmentation model; Based on the training data set and the validation data set, optimizing the initial SEM image segmentation model using a stochastic gradient descent method to obtain an optimized SEM image segmentation model; Using the optimized SEM image segmentation model to perform test segmentation on the strawberry-shaped pyrite SEM image in the test data set to obtain a prediction result; The prediction result is compared and analyzed with the labeled strawberry pyrite segmentation data corresponding to the test data set to determine the target SEM image segmentation model.

3. The pyrite SEM image segmentation method according to claim 2, characterized in that, The step of using the SEM image to be tested and the segmentation labels to prepare a training data set, a verification data set and a test data set comprises: Preprocessing the SEM image to be tested, scaling it to a preset resolution, and obtaining a preprocessed SEM image; The preprocessed SEM images are divided into a training data set, a verification data set and a test data set according to a preset ratio.

4. The pyrite SEM image segmentation method according to claim 2, characterized in that: The optimizing the initial SEM image segmentation model by using a stochastic gradient descent method based on the training data set and the verification data set to obtain the optimized SEM image segmentation model comprises: The initial SEM image segmentation model is trained using an AdamW optimizer and a cross entropy loss; wherein an initial value of the learning rate of the initial SEM image segmentation model is set to 0.00006; The initial SEM image segmentation model is optimized using a data enhancement strategy, wherein the data enhancement strategy includes random scaling, random horizontal flipping, and random cropping.

5. The pyrite SEM image segmentation method according to claim 2, characterized in that: The comparing and analyzing the prediction result with the labeled strawberry pyrite segmentation data corresponding to the test data set to determine the target SEM image segmentation model includes: The first prediction result of the optimized SEM image segmentation model with the SAM boundary perception module added is compared with the second prediction result corresponding to the first prediction result of the segformer model without the SAM boundary perception module to analyze the improvement of intersection over union (IoU), recall, precision, accuracy and F1 score, where the F1 score is the harmonic mean of precision and recall.

6. The pyrite SEM image segmentation method according to claim 1, characterized in that: The boundary feature map F is obtained by the Canny edge detection algorithm. b include: The segmentation mask M of the target is calculated using the Sobel operator through the Canny algorithm b Gradients in the x and y directions; Calculate the magnitude and direction of the gradient to determine the location and direction of the edge; where, G x =Sobel(M b ,x); G y =Sobel(M b ,y); θ=arctane(G y / G x ); F b =Canny(G,θ,100,200); In the formula, G x and G y The Sobel operator calculates the segmentation mask M of the target in the x and y directions respectively. b The gradient of , G and θ are the gradient magnitude and direction.

7. The pyrite SEM image segmentation method according to claim 1, characterized in that: The patch is passed to the hierarchical Transformer layer, and after being calculated by the self-attention mechanism of the hierarchical Transformer layer and processed by the feedforward neural network, an overlapping patch merging operation is performed to obtain a hierarchical feature map F i ,include: The self-attention mechanism is used to calculate the self-attention between the input patches and generate the output features. In the self-attention mechanism calculation, each query vector Q, key vector K, and value vector V have the same dimension H×W×C, and the self-attention Attention(Q,K,V) is: Where, d head represents the dimension of the key, and uses a sequence reduction process for patches in the calculation of the self-attention mechanism, using a reduction ratio R to reduce the sequence length, and the final dimension is for in: Where K represents the key vector to be reduced; The output features are input into a feedforward neural network, which calculates the features by mixing a 3×3 convolution and an MLP multi-layer perceptron module and outputs the final output, where: X out =MLP(GELU(Conv 3×3 (MLP(x in ))))+x in ; In the formula, x in Output features from the self-attention mechanism, X out Represents the final output; For the final output of each Transformer layer, perform overlapping patch merging operations to obtain the hierarchical feature map F i , wherein the overlapping patch merging operation includes defining K1, S, and P, wherein K1 is the patch size, S is the step size between adjacent patches, and P is the padding size; setting K1=9, S=5, P=4 or K1=3, S=2, P=1, and performing overlapping patch merging.

8. The pyrite SEM image segmentation method according to any one of claims 1 to 7, characterized in that: The multi-scale feature fusion and mask segmentation are performed in the decoder stage to obtain the SEM image segmentation result, including: The multi-layer feature information obtained from the encoder stage is unified in channel dimension through the first MLP layer to obtain unified dimension feature information By upsampling the uniform dimensional feature information, the resolution is increased to The size of the upsampling result is obtained Then perform splicing to obtain the spliced ​​features; The second MLP layer is used to fuse the concatenated features to obtain the fused features; The fused features are input into the third MLP layer to predict the segmentation mask and obtain the final predicted segmentation mask M; where the resolution is in, M=Linear(C,N class )(F); Where M represents the predicted mask, C represents the unified channel dimension, and N class is the number of categories, It represents the concatenation of the upsampling results, F represents the fused features, and M represents the final predicted segmentation mask.

9. A pyrite SEM image segmentation device, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the pyrite SEM image segmentation method according to any one of claims 1 to 8; The device further comprises an input unit for receiving a pyrite SEM image, a display unit for displaying a segmentation result, and a storage unit for storing segmented image data and related processing information. The processor is configured to receive the pyrite SEM image from the input unit; execute an image segmentation algorithm according to instructions in the memory to perform segmentation processing on the received pyrite SEM image; and transmit the segmented image result to the display unit for display; The processed image data and related information are stored in a storage unit for subsequent analysis and processing.

Images