Electron microscope microscopic image dislocation instance segmentation method based on visual large model

Abstract

The invention relates to a dislocation instance segmentation method of an electron microscope microscopic image, and belongs to the technical field of material defect detection and image processing. Aiming at the problems of low efficiency and poor precision of traditional manual analysis, the invention provides a multi-scale dislocation instance segmentation method based on a visual large model, the method comprises an image coding module, a multi-scale feature generation module and a mask decoding module, a multi-scale feature pyramid is generated by fusing global and local features, model training is optimized in combination with a parameter efficient fine tuning technology, and a multi-scale dislocation instance segmentation result is obtained. And bounding box positioning, category classification and pixel-level mask segmentation of dislocation instances are realized. The method remarkably improves the detection efficiency and precision, supports automatic material quality evaluation, reduces personal errors, and can be widely applied to dislocation defect analysis in the fields of semiconductors, aerospace and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of material defect detection and image processing, and relates to a method for segmenting dislocation instances in electron microscope microscopic images based on a vision large model. Background Art

[0002] Materials play a crucial role in modern industries such as aerospace, electron microscope devices, and energy. Their performance directly affects the quality and reliability of products. Dislocations, as common line defects, are usually caused by external stress, temperature changes, or non-uniform deformation during the processing. The configuration and migration behavior of dislocations have an important impact on the mechanical properties of structural materials. In typical materials such as single-crystalline silicon and diamond, the magnitude of the dislocation density directly determines the performance of semiconductor devices.

[0003] Therefore, accurately detecting and characterizing dislocations is crucial for optimizing the preparation process and improving material performance. The distribution, density, and type of dislocations are important indicators for evaluating material quality. Traditional detection methods mainly rely on manual microscopic image analysis, which is not only time-consuming and laborious, but also the detection results are easily affected by human factors, and it is difficult to guarantee accuracy and consistency. In addition, with the improvement of material application requirements and the growth of detection data volume, traditional methods are no longer able to meet the high-throughput and high-precision requirements in modern material research. The vision large model uses its own powerful feature extraction ability to perform dislocation instance segmentation operations on electron microscope microscopic images. This model can accurately identify complex dislocation morphologies in microscopic images, segment each dislocation instance, and achieve automated analysis of dislocation types and distributions. Compared with traditional manual methods, this method has the advantages of high efficiency, accuracy, and automation, can significantly improve the detection speed and accuracy, and at the same time greatly reduce human errors, providing strong technical support for the automatic evaluation and optimization of material quality.

[0004] The instance segmentation task is used to simultaneously identify the category of the target in the image, locate the boundary of each target, and generate an accurate pixel-level segmentation mask for each target. In contrast, object detection only provides the location and category of the target, while semantic segmentation only classifies all pixels in the image and cannot distinguish different targets of the same category. Compared with the two, instance segmentation includes both the boundary localization function of object detection and the pixel classification ability of semantic segmentation. The dislocation instance segmentation task based on electron microscope microscopic images depends on the appropriate selection and extraction of image features. There are various different dislocation types in electron microscope microscopic images. The dislocation instance segmentation model based on the vision large model realizes end-to-end learning. After training, it can better learn the features in the microscopic image, and complete the extraction and classification of image features, and finally realize the dislocation instance segmentation of electron microscope microscopic images. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a dislocation instance segmentation method for electron microscope microscopic images based on a large vision model. The detection results vary in terms of the average precision (AP) under different intersection over union (IoU) threshold settings. When the IoU is from 0.50 to 0.95, the AP is 0.597; when the IoU is 0.50, the AP reaches 0.824; when the IoU is 0.75, the AP is 0.688. Compared with traditional methods, the method of the present invention has the advantages of high efficiency, precision, and automation, can significantly improve the detection speed and accuracy, and at the same time greatly reduce human errors, providing strong technical support for the automatic evaluation and optimization of material quality.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A dislocation instance segmentation method for electron microscope microscopic images based on a large vision model, comprising the following steps:

[0008] S1: Prepare the dataset: Collect electron microscope microscopic images and perform dislocation annotation to generate label data including the bounding boxes, categories, and pixel-level masks of the dislocation regions, and perform size normalization and enhancement processing on the data;

[0009] S2: Construct a multi-scale dislocation instance segmentation network based on the large vision model, and the network includes:

[0010] An image encoding module for extracting the global features and local features of the input image;

[0011] A multi-scale feature generation module for fusing features at different levels and generating a multi-scale feature pyramid;

[0012] A mask decoding module for generating the bounding boxes, categories, and segmentation masks of dislocation instances based on the multi-scale feature pyramid;

[0013] S3: Train the network model: Optimize the model parameters by using pre-trained weights combined with parameter-efficient fine-tuning techniques, and jointly optimize the classification, bounding box regression, and mask segmentation results through a multi-task loss function;

[0014] S4: Test the network model: Input the electron microscope microscopic image to be tested, output the segmentation results of dislocation instances, and screen out valid detections through post-processing.

[0015] Further, the data enhancement in S1 includes: randomly scaling and sampling the image, with the scaling range being 0.1 to 2.0, and then cropping it to a fixed size.

[0016] Furthermore, the image encoding module is based on the Vision Transformer architecture, uses the multi-head self-attention mechanism to extract features, and introduces the global self-attention mechanism in some attention layers; the image encoding module also fine-tunes the pre-trained weights through the LoRA technique for low-rank matrix factorization.

[0017] Furthermore, the LoRA technique decomposes the original weight matrix into the product of two low-rank matrices, and the rank of the low-rank matrices is set to be below 16.

[0018] Furthermore, the multi-scale feature generation module first generates aggregated features through the fusion encoding module, and then generates a feature pyramid containing scales of 1 / 4, 1 / 8, 1 / 32, and 1 / 64 through upsampling and downsampling.

[0019] Furthermore, the mask decoding module includes a region proposal network and a mask generation network; the region proposal network outputs candidate regions through anchor generation and bounding box decoding, and the mask generation network generates instance masks through RoIAlign and convolutional operations.

[0020] Furthermore, during the training process of S3, the optimizer uses AdamW, the learning rate scheduling strategy includes a linear growth stage and a cosine annealing stage, and mixed precision acceleration is enabled during training.

[0021] Furthermore, the multi-task loss function includes mask cross-entropy loss, bounding box regression L1 loss, and classification cross-entropy loss.

[0022] Furthermore, the post-processing of S4 includes: filtering low-confidence detection results based on a score threshold, and eliminating duplicate bounding boxes through non-maximum suppression.

[0023] Furthermore, the dislocation categories include edge dislocations, screw dislocations, loop dislocations, and background.

[0024] The beneficial effects of the present invention are as follows:

[0025] (1) The method of the present invention automatically identifies and segments dislocation instances in electron microscope micrographs through a vision large model, without the need for manual analysis one by one, greatly improving the detection speed and efficiency, especially when dealing with a large amount of image data. The vision large model can process multiple images in parallel, further improving the detection efficiency and meeting the high-throughput requirements in modern material research.

[0026] (2) The method of the present invention can accurately identify the positions and boundaries of dislocation instances, generate pixel-level segmentation masks, avoiding the errors and subjectivity of manual analysis, and improving the accuracy and consistency of detection results. The model can capture dislocation features at different scales through multi-level feature fusion, thereby more accurately identifying dislocation instances of different types and sizes.

[0027] (3) The method of the present invention does not require manual cumbersome annotation work. Only a small amount of annotated data is needed to train the model, reducing labor costs and improving work efficiency. The model can automatically analyze the type, distribution, and density of dislocations, providing strong technical support for material property evaluation and optimization.

[0028] (4) Automated detection avoids the influence of human factors and improves the objectivity and reliability of detection results. The method of the present invention can be applied to different types of electron microscope micrographs and can be extended to the detection and analysis of other material defects.

[0029] Other advantages, objectives, and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the following specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail preferably with reference to the accompanying drawings, where:

[0031] Figure 1 is a flowchart of an instance segmentation method for detecting dislocations in electron microscope micrographs based on a vision large model;

[0032] Figure 2 is a general diagram of the model structure of an instance segmentation method for dislocations in electron microscope micrographs based on a vision large model;

[0033] Figure 3 is a structural diagram of an image encoding module;

[0034] Figure 4 is a structural diagram of a multi-scale feature generation module;

[0035] Figure 5 is a structural diagram of a mask decoding module;

[0036] Figure 6 is a diagram of the instance segmentation result of dislocations in electron microscope micrographs. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0038] Among them, the drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation on the present invention; in order to better illustrate the embodiments of the present invention, some components in the drawings will be omitted, enlarged or reduced, which do not represent the dimensions of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0039] In the drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the drawings. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes and should not be construed as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0040] Figure 1 It is a flowchart of an instance segmentation method for detecting dislocations in electron microscope micrographs based on a vision large model. To make the technical solutions of the present invention clearer, the present invention will be further described below with reference to the drawings. The present invention is specifically implemented according to the following steps:

[0041] The first step is to prepare the dataset.

[0042] (1) Prepare image data and label data.

[0043] Collect the images with dislocation defects captured by instruments such as electron microscopes and perform manual annotation on them. Record all the dislocation regions in each image. For each dislocation region, first, similar to object detection, use a rectangular box for annotation and record the dislocation category. At the same time, draw its detailed contour with a polygon and record the coordinates of each vertex of the polygon. During the annotation process, the dislocation types are divided into 4 categories, namely line, circle, screw, and background. To unify the input of the model and enhance the learning effect, normalize the images with a size of 2048×2048 to obtain a high-definition dataset of 1024×1024 for training a dislocation instance segmentation model based on a large vision model. In the experiment, the dataset after data augmentation is randomly split into three parts: the training set, the test set, and the validation set. The training set contains 165 images, and both the test set and the validation set contain 20 images.

[0044] (2) Data augmentation.

[0045] The data augmentation operation used in this experiment is Large Scale Jitter (LSJ). First, randomly sample the data within the range of 0.1 to 2.0 for scaling. This step can greatly enrich the diversity of the data, enabling the model to access image information at different scales during training, thereby enhancing the recognition ability for objects of different sizes. For example, for some small target objects, after appropriate enlarged sampling, the model can better learn their features; for large target objects, reduced sampling helps the model grasp the relationship between their overall structure and detailed features. After completing the random scaling, then perform a fixed-size cropping operation to crop the image back to 1024×1024 to conform to the input of the model. In this way, the model can better adapt to image data with different proportions and scale changes, improving the generalization ability and robustness of the model, and maintaining good performance when facing image data in various actual scenarios.

[0046] The second step is to design a multi-scale dislocation instance segmentation network based on a large vision model.

[0047] We designed a multi-scale dislocation instance segmentation network based on a large vision model, which is optimized for the instance segmentation task in electron microscopy images. The model structure is shown in the appendix Figure 2 . Through multi-level feature fusion and segmentation, this model can make full use of the output features at different levels from the image encoding module to accurately identify dislocation instances at different scales. The model structure consists of three major parts: the image encoding module, the multi-scale feature generation module, and the mask decoding module. The structure diagrams of each module correspond to Figure 3 、 Figure 4 and Figure 5, Next, the design and functions of each part will be introduced separately.

[0048] (1) The structure of the image encoding module is as Figure 3 , and the backbone of the network is designed based on the Vision Transformer layer. On this basis, the attention module is fine-tuned with LoRA (Low-Rank Adaptation, LoRA). The encoding module contains a total of 12 Vision Transformer layers, and each layer contains a multi-head attention module, which has 12 self-attention heads. The global self-attention mechanism is adopted in the 2nd, 5th, 8th, and 11th layers of the encoding module. This design can take into account the global information in the image and adapt to the complex structure of dislocations in microscopic images. Through this multi-level attention mechanism, the network can extract deep features in the image, thus providing richer feature information for subsequent segmentation tasks. The encoding module receives 768-dimensional image patch embeddings and absolute position embeddings, and can efficiently process the detail information and important features in electron microscopy images while considering the position information of the image patches. The input image is first cut into image patches (Patches) of size 16*16, and then each image patch is converted into a one-dimensional embedding vector of length 768 through a convolution operation. The specific operation is as follows:

[0049] F PE =W 16 (l)

[0050] where, represents the input electron microscopy image, and H, W, and C represent the height, width, and number of channels of the input image respectively. W 16 (·) represents a convolution operation of size 16×16, and F PE represents the 768-dimensional image patch embedding (Patch Embedding) obtained by conversion. In addition, the model also uses absolute position embeddings. Based on dividing the image into Patches, the position coordinates of each Patch in the original image are obtained by dividing the width and height of the image by the Patch size, so as to create a trainable position embedding based on the image size and Patch size, adding position information to the Patch sequence to improve the model's understanding of the image spatial structure and feature learning ability. By dividing the input image into Patches and encoding and absolute position embeddings, the input of the model is obtained as:

[0051] F I =F PE +PE abs

[0052] where, PE absRepresents the absolute positional embedding corresponding to the Patch block.

[0053] Next, F I will be input into the first encoding layer. First, it will go through a normalization operation to make the distribution of the input data more stable. Immediately afterwards, a window partitioning operation is performed on the normalized features. The input is partitioned according to the window size, and at the same time, necessary padding is added to the input features according to the calculation results. The input is reshaped into a windowed tensor, and the padding shape is recorded. Subsequently, the features after window partitioning enter the multi-head attention module, which contains 12 attention heads. Attention calculations will be performed according to the window size to enrich local feature extraction and interaction, obtaining new features and attention weights. The attention operation can assign different weights to different features of the image, thereby better capturing important information related to dislocations in the image. After that, according to the padding shape of the previous step, the output features are restored to the original shape using the window restoration operation and added to the input F I to get F' I , and then it goes through normalization again. Using residual connections can avoid the problem of gradient disappearance in deep networks. Finally, the normalized features enter the multi-layer perceptron (MLP) to extract more complex features and add them to F' I to obtain the output features of the first encoding layer The MLP uses 2 fully connected layers and 1 non-linear activation function to perform non-linear transformations on the features, thereby enhancing the expressive power of the features. The operation process of the visual layer in the encoder is expressed by the following formula:

[0054] F′ I = WU(A(WP(LN(F I )))) + F I

[0055]

[0056] where WP(·) and WU(·) represent the window partitioning and window restoration operations respectively, A(·) represents the multi-head attention calculation, and LN(·) is the normalization operation.

[0057] In the multi-head attention module, we use the LoRA method in the Parameter-Efficient Fine-Tuning (PEFT) technique to fine-tune the pre-trained weights W. LoRA updates the parameters by decomposing the original full-rank matrix W into the product of two low-rank matrices A and B, thereby reducing the number of parameters and computational complexity. At the same time, LoRA introduces a scaling factor α, and the model scales the parameter update amount ΔW by α / r to adjust the influence degree of the low-rank matrices A and B on the original weight matrix W. The update process of W can be expressed as follows:

[0058] W′ = W + ΔW = W + A·B

[0059] where ΔW represents the parameter update value of the corresponding matrix, r is the rank of the low-rank matrix, and the size of the rank will determine the performance of fine-tuning and the training time.

[0060] Specifically, the multi-head attention module first obtains query, key, and value vectors by passing the input features through a linear layer. Then, it projects them into multiple subspaces through multiple different linear transformations respectively. Next, it independently applies the ScaledDot-ProductAttention mechanism in each subspace, calculates the dot product of the query and the key, and multiplies it by the scaling factor to obtain the initial attention weights. At the same time, using the pre-set relative position encoding matrix, according to the sizes of the query and the key input to the subspace currently, it extracts the matching position information from the relative position encoding matrix and adds it to the initial attention weights, so that the attention mechanism can take into account the relative positions between elements. Then, it performs softmax normalization on the attention weights after adding the position information, and uses dropout to prevent overfitting. Subsequently, it multiplies the obtained attention weights by the value vectors to get the attention output of this subspace. Finally, it concatenates the results of these subspaces together and obtains the final attention output through a linear transformation. Through the attention mechanism that includes relative position information, information aggregation and feature extraction are realized, so as to better grasp the associations between elements at different positions in the input. The above process can be expressed as:

[0061]

[0062] MultiHead(Q,K,V) = FC(Concat(Head1,Head2,…,Head 12 ))

[0063] where, PE rel represents Relative Positional Embedding, d kis the scaling factor, and FC(·) represents the fully connected layer.

[0064] (2) The structure of the multi-scale feature generation module is as Figure 4 shown. Its main task is to further process the features from the encoding module while generating multi-scale feature maps suitable for the instance segmentation task. The multi-scale feature generation module consists of two parts: a feature aggregation model and a feature segmentation model. The role of the feature aggregation model is to fuse the features from different vision Transformer layers in the encoding module, including the 2nd, 4th, 6th, 8th, 10th, and 12th layers. First, downsample and perform hidden layer operations on the input features of the 6 selected layers, then perform residual addition and residual connection of the hidden layers, and finally obtain the final aggregated features through a fusion operation. Through operations such as multiple convolutions and residual connections, the features of different encoding layers can be effectively screened, transformed, and fused, integrating information at multiple levels into a unified feature representation to extract richer information and enhance the expressive power of the features. The specific process is as follows:

[0065]

[0066] Among them, it should be noted that F down , F hidden and F fusion are operations composed of multiple convolutions, batch normalizations, and activation functions, and F agg represents the aggregated features.

[0067] The construction of the multi-scale feature pyramid then uses the fused features to perform upsampling through learnable transposed convolutions and downsampling through max pooling, thereby generating a multi-scale feature pyramid. These feature representations can provide rich multi-scale information for the dislocation segmentation task to adapt to targets and features of different sizes, thereby improving the segmentation accuracy of the model for dislocation instances. Perform upsampling operations on the feature map upwards, using two transposed convolutions with a kernel size of 2 and a stride of 2 to obtain a feature map of 1 / 4 size, and similarly obtain a 1 / 8 size. Downwards, downsample it to 1 / 32 size through multiple max poolings and then continue to pool to 1 / 64 size. Finally, adjust the output channels of the feature maps of 1 / 4, 1 / 8, 1 / 32, and 1 / 64 sizes to match the subsequent operations. The detailed process is as follows:

[0068] F 1 / 4 = W T (GELU(LN(W T (F agg ))))

[0069] F 1 / 8 = W T (F agg )

[0070] F 1 / 32 = MaxPool(F agg )

[0071] F 1 / 64 = MaxPool(MaxPool(F agg ))

[0072] where, W T (·) represents transposed convolution, and MaxPool(·) is max pooling.

[0073] (3) The mask decoding module structure is as Figure 5 shown, responsible for converting the multi-scale feature map in the previous step into the final prediction results, including bounding boxes, classes, and instance masks. The mask decoding module includes two key networks: the Region Proposal Network (RPN) and the mask generation network. The Region Proposal Network mainly relies on the anchor generation module and the bounding box encoding module to work together to complete tasks such as loss calculation, prediction information processing, anchor generation, and encoding and decoding of bounding boxes, thereby converting the input feature map into available region proposals, providing basic candidate regions for subsequent dislocation instance segmentation; while the mask generation module generates masks for each instance through a convolutional network. The fully connected layer outputs the confidence of each instance and the position parameters of the region box by regressing the coordinates of the target.

[0074] The RPN receives the feature map for initialization and processes it to obtain classification and bounding box prediction information. The classification result is processed through an activation function, and at the same time, combined with the real instance information and image metadata, the losses of classification and bounding box prediction are calculated. During prediction, the classification information is first screened, combined with the prior boxes and decoded by the bounding box encoder to obtain the bounding boxes, then size adjustment, filtering of small bounding boxes, and Non-Maximum Suppression (NMS) operations are performed to obtain the predicted bounding boxes, corresponding scores, and class labels. The anchor generator generates anchors of different scales, first calculates the basic anchors and then generates grid offsets and adds them to obtain the final grid anchors, providing a basis for region proposals. The bounding box encoding module is responsible for encoding the source and target bounding boxes into offsets and decoding the predicted offsets into actual bounding boxes.

[0075] C i , B i = RPNHead(F 1 / i )

[0076]

[0077] F 1 / i comes from the feature pyramid, f RPN according to the classification prediction result Ci , the predicted result B of the bounding box i , the prior box A i and the image information I i are post - processed to finally output the predicted result R.

[0078] In the design of the mask generation network, the Region of Interest Align (RoIAlign) operation is used to process the features in the previous - step region proposals. It is pooled into a fixed size through bilinear interpolation. After the features of these regions are decoded by convolution, the mask of each instance is obtained, and the classification and bounding box are output through the fully - connected layer. The mask generation network contains two processing flows for the bounding box and the mask. In the bounding - box processing, first, the proposed regions after pooling are further precisely adjusted by the bounding - box region - of - interest extraction module, and then the target category and the bounding - box position are predicted by the bounding - box head, and the classification and bounding - box regression losses are calculated; for the mask processing, the mask region - of - interest extraction module is used to extract features, and then the mask head performs convolution operations on the features to predict the mask of the target and calculate the mask loss.

[0079] C box , B box = BoxHead(F 1 / i , ROIs)

[0080] M mask = MaskHead(F 1 / i , ROIs)

[0081] where ROIs represents the set of regions of interest, C box , B box represent the classification result and the bounding - box regression result respectively, and M mask is the mask prediction result.

[0082] (4) Loss calculation: The total loss of this model includes the mask loss, the bounding - box regression loss, and the bounding - box classification loss. The mask loss and the bounding - box classification loss are calculated using the cross - entropy loss, and the bounding - box regression loss is calculated using the L1 loss. The specific calculation formulas are as follows:

[0083] Loss = L mask + L box + L cls

[0084]

[0085] where, and respectively represent the true mask and the true category. The model evaluation metric uses mAP (mean Average Precision), which is widely used in fields such as object detection and image classification. It comprehensively considers Precision and Recall. Precision refers to the proportion of samples that are truly positive among all samples predicted as positive, while Recall refers to the proportion of samples that are actually positive and are correctly predicted as positive. The calculation formulas are as follows:

[0086]

[0087] TP (True Positive, TP) refers to the true positive cases, which are the number of samples that are actually positive and are correctly predicted as positive. FP (False Positive, FP) refers to the false positive cases, which are the number of samples that are actually negative but are wrongly predicted as positive. FN (False Negative, FN) refers to the false negative cases, which are the number of samples that are actually positive but are wrongly predicted as negative.

[0088] In the third step, train a multi-scale dislocation instance segmentation model based on the vision large model.

[0089] In LoRA fine-tuning, the rank of the low-rank matrix is set to 16. The matrix A is initialized with a random Gaussian distribution, and the matrix B is initialized with a zero matrix. During the training process, the pre-trained parameters of the image encoding model are fixed, and only the two address matrices A and B are trained; after the training is completed, matrix A and B are multiplied, and then the result of the multiplication is combined with the pre-trained model parameters to obtain the fine-tuned model parameters. The fine-tuning process not only effectively retains the rich knowledge and strong generalization ability learned by the pre-trained model on large-scale general datasets, but also enables the model to quickly learn the unique features and patterns in electron microscopy images, enhancing its performance in the dislocation instance segmentation task.

[0090] During the training process, 4 GPUs are used, and the batch size is set to 4 to ensure that each GPU is allocated 4 images. The optimizer uses AdamW, the initial learning rate is set to 0.0002, and the weight decay is 0.05. Mixed precision is enabled during the training process to improve the training efficiency. The learning rate scheduling strategy includes two stages: First, use a linear learning rate scheduler to increase from 0 to the initial learning rate within 50 epochs; then use a cosine annealing strategy to gradually reduce the learning rate within 250 epochs. According to the pre-annotated dataset, the dislocation types are set to 4 categories during the training process, namely line, circle, screw, and background.

[0091] The maximum number of epochs in the training process is 250. Validation is performed every 5 epochs, and the current best model is saved. During the training process, the standard MaxIoUAssigner strategy is used for positive and negative sample matching, appropriate IoU thresholds are set, and random cropping within the batch and negative sample filtering are performed at each iteration to improve the generalization ability of the model.

[0092] In the fourth step, test the detection effect of this system

[0093] During testing, the electron microscope microscopic image to be tested is input into the detection model. After passing through the image encoding module, the feature representation of the image is obtained. Then, the hierarchical feature processing module is used to aggregate the features and generate a five-stage feature pyramid. Finally, through the mask decoding module, mask segmentation, bounding box regression, and class classification are performed to finally obtain the best dislocation segmentation result.

[0094] Specifically, during the testing process, necessary preprocessing is first performed on the input data to ensure the consistency and applicability of the data. The input image will be uniformly adjusted to a size of 1024×1024 through scaling and padding operations. The preprocessing process ensures the standardization of the input data, enabling subsequent inference detection to be carried out under stable input conditions. In the inference stage, the preprocessed image will extract features through encoding, and then dislocation target detection and instance segmentation will be performed. Each detected target will generate a bounding box and a corresponding segmentation mask. To filter out effective detection results, the model will filter the prediction results according to a preset score threshold, eliminating low-score detections, thereby reducing noise and improving detection accuracy.

[0095] Subsequently, for the results of multi-target detection, algorithms such as NMS are used for post-processing to eliminate duplicate detections and ensure that each target is detected only once. This process can effectively avoid duplicate labeling and misclassification of targets. In the evaluation stage, the detection boxes and segmentation masks output by the model will be quantitatively analyzed according to standard evaluation metrics. The evaluation metrics include the average localization accuracy of the target boxes and the quality assessment of the segmentation masks. Finally, the test results are visualized and saved to implement the testing and evaluation of the model performance. The detection results of the model are as Figure 6 shown.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A method for instance segmentation of dislocation in electron microscopy micrographs based on a large vision model, characterized in that: It includes the following steps: S1: Prepare the dataset: Collect electron microscopy images and perform dislocation annotation to generate label data including the bounding boxes, categories, and pixel-level masks of dislocation regions, and perform size normalization and enhancement processing on the data; S2: Construct a multi-scale dislocation instance segmentation network based on a large vision model, and the network includes: An image encoding module for extracting global features and local features of the input image; A multi-scale feature generation module for fusing features at different levels and generating a multi-scale feature pyramid; A mask decoding module for generating the bounding boxes, categories, and segmentation masks of dislocation instances based on the multi-scale feature pyramid; S3: Train the network model: Optimize the model parameters using pre-trained weights combined with the parameter-efficient fine-tuning technique, and jointly optimize the classification, bounding box regression, and mask segmentation results through a multi-task loss function; S4: Test the network model: Input the electron microscopy image to be tested, output the segmentation results of dislocation instances, and screen for valid detections through post-processing.

2. The method for segmenting dislocation examples in an electron microscope microscopic image according to claim 1, wherein: In the above S1, the data enhancement includes: randomly scaling and sampling the image, with the scaling range being 0.1 - 2.0, and then cropping it to a fixed size.

3. The method for segmenting dislocation examples in electron microscope microscopic images according to claim 1, wherein: The above image encoding module is based on the Vision Transformer architecture, uses the multi-head self-attention mechanism to extract features, and introduces the global self-attention mechanism in some attention layers; the image encoding module also fine-tunes the pre-trained weights through the LoRA technique.

4. The method for segmenting dislocation instances in an electron microscope microscopic image according to claim 3, wherein: The LoRA technique decomposes the original weight matrix into the product of two low-rank matrices, and the rank of the low-rank matrices is set to be below 16.

5. The method for segmenting dislocation examples in an electron microscope microscopic image according to claim 1, characterized in that: The above multi-scale feature generation module first generates aggregated features through a fusion encoding module, and then generates a feature pyramid including scales of 1 / 4, 1 / 8, 1 / 32, and 1 / 64 through upsampling and downsampling.

6. The method for segmenting dislocation instances in an electron microscope microscopic image according to claim 1, wherein: The above mask decoding module includes a region proposal network and a mask generation network; the region proposal network outputs candidate regions through anchor generation and bounding box decoding, and the mask generation network generates instance masks through RoIAlign and convolutional operations.

7. The method for segmenting dislocation examples in electron microscope microscopic images according to claim 1, wherein: In the above S3, during the training process, the optimizer uses AdamW, the learning rate scheduling strategy includes a linear growth stage and a cosine annealing stage, and mixed precision acceleration is enabled during training.

8. The method for segmenting dislocation instances of an electron microscope microscopic image according to claim 1, wherein: The above multi-task loss function includes mask cross-entropy loss, bounding box regression L1 loss, and classification cross-entropy loss.

9. The method for segmenting dislocation examples in an electron microscope microscopic image according to claim 1, characterized in that: In the above S4, the post-processing includes: filtering out low-confidence detection results based on a score threshold, and eliminating duplicate bounding boxes through non-maximum suppression.

10. The method for segmenting dislocation examples in electron microscope microscopic images according to claim 1, characterized in that: The above dislocations include edge dislocations, screw dislocations, loop dislocations, and backgrounds.

Images