An online adaptive interactive segmentation method and system for electron microscope image sequences

Abstract

The present application relates to the technical field of electron microscope image segmentation, and provides an online adaptive interactive segmentation method and system for electron microscope image sequences.The method comprises the following steps: obtaining an electron microscope image dataset for training and testing, constructing an interactive encoder and a basic segmentation network, and fusing an original electron microscope image and an interactive prior feature map; establishing a short-time adaptive branch and a long-time memory branch; processing the images according to the electron microscope image sequences, generating an initial segmentation result of the current image based on a previous sequence update model; outputting the final segmentation mask of each electron microscope image, and saving the segmentation result, a click log and an updated model weight.The present application expands the user interaction from a single spatial prompt to an online learning signal in the test stage by constructing an online adaptive interactive segmentation network, and introduces a double-time-scale online adaptive mechanism combining a short-time adaptive branch and a long-time memory branch, so that the local missegmentation area in the current electron microscope image can be quickly corrected.

Description

Technical Field

[0001] This invention relates to the field of electron microscopy image segmentation technology, and in particular to an online adaptive interactive segmentation method and system for electron microscopy image sequences. Background Technology

[0002] Transmission electron microscopy (TEM) is a commonly used microscopic characterization tool in materials science, biomedicine, and nanostructure analysis. TEM images provide quantitative information such as particle size, boundary morphology, structural connectivity, cell morphology, and pore distribution. For subsequent statistical analysis and structural characterization, accurate segmentation of the target region in the TEM image is usually required.

[0003] Traditional image segmentation methods include thresholding, edge detection, region growing, and watershed algorithms. These methods typically rely on fixed parameters and human experience, and their segmentation accuracy and robustness are limited when faced with weak boundaries, low contrast, high noise, complex textures, and structural adhesion common in electron microscopy images. In recent years, methods such as convolutional neural networks (CNNs) have been widely used for electron microscopy image segmentation tasks. However, these methods usually rely on a large amount of high-quality pixel-level annotation, and fine annotation of electron microscopy images is time-consuming, labor-intensive, and highly dependent on the experience of domain experts.

[0004] Interactive image segmentation methods can reduce the cost of manual annotation to some extent by introducing user feedback such as positive clicks, negative clicks, doodles, or bounding boxes. Existing click-based interactive segmentation methods typically encode user clicks as additional input and refine the segmentation result of the current image through multiple forward inferences. However, most of these methods keep the model parameters fixed during the testing phase, and the error correction information provided by the user in the current image cannot be retained by the model and passed to subsequent images. When the object to be processed is a sequence of continuous slice images, images sampled in the same batch, or electron microscopy images with similar structures, the user still needs to repeatedly provide similar interactive feedback in multiple images, resulting in low overall segmentation efficiency.

[0005] Therefore, there is a need for an online adaptive interactive segmentation method that can improve the segmentation accuracy and interaction efficiency of consecutive or batch electron microscope images. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an online adaptive interactive segmentation method and system for electron microscopy image sequences. By constructing an online adaptive interactive segmentation network, the user interaction is expanded from single spatial prompts to online learning signals during the testing phase. A dual-timescale online adaptive mechanism combining short-time adaptive branches and long-time memory branches is introduced to achieve rapid correction of locally missegmented regions in the current electron microscopy image and cross-image inheritance of reliable segmentation experience from consecutive or batched electron microscopy images. This invention significantly improves the segmentation accuracy and interactive efficiency of complex structures and boundary details in electron microscopy images, providing an efficient and reliable solution for materials science and microstructure analysis, and laying the foundation for further research and application.

[0007] The technical means employed in this invention are as follows: An online adaptive interactive segmentation method for electron microscope image sequences includes: acquiring an electron microscope image dataset for training and testing, and preprocessing the electron microscope images and their corresponding segmentation annotations; constructing an interactive encoder to generate an interactive prior feature map based on user interaction information; constructing a basic segmentation network to fuse the original electron microscope image and the interactive prior feature map, and obtaining the segmentation prediction result of the current electron microscope image through a backbone encoder, decoder, and decoupled output head; establishing a short-time adaptive branch to update the decoder and decoupled output head online based on the click feedback when the user clicks to correct the current segmentation prediction result, obtaining the corrected segmentation result of the current image; establishing a long-time memory branch to consolidate the model online using the final segmentation mask confirmed by the user after the current image interaction is completed, and writing reliable structural information in the current image into the backbone encoder representation; processing each image sequentially according to the electron microscope image sequence, and generating an initial segmentation result using the model updated by the interaction of the previous image when processing the current image, outputting the final segmentation mask after the current image interaction is completed, and saving the segmentation result, click log, and updated model weights.

[0008] Furthermore, the electron microscopy image dataset includes at least one of cell structure images and nanoparticle images; the electron microscopy images are subjected to size unification, grayscale normalization, and training / test segmentation processing to adapt to the input requirements of the network model.

[0009] Furthermore, the user interaction information is composed of positive clicks and negative clicks, wherein positive clicks represent target regions and negative clicks represent background regions or mispredicted regions; a distance map is generated based on the click position, which describes the spatial distance relationship between each pixel in the image and the user's click point; the distance map is then sequentially processed through 1×1 convolution, LeakyReLU activation, 3×3 convolution, and Sigmoid mapping to obtain an interactive prior feature map with the same spatial size as the original electron microscope image.

[0010] Furthermore, the basic segmentation network includes a backbone encoder, a decoder, and a decoupled output head; the backbone encoder adopts a ResNet50 encoder, which concatenates the original electron microscope image with the interactive prior feature map in the channel dimension and inputs it into the backbone encoder to form a joint representation of visual information and interactive prior. The decoder adopts a U-Net-style stepwise upsampling structure. Each decoding block includes an upsampling layer, a feature concatenation layer, a convolutional layer, a batch normalization layer, and a ReLU activation layer. Features of different scales from the backbone encoder are passed to the decoder through skip connections and concatenated with the features at the decoding end to supplement high-resolution spatial information and restore the boundary details of the target region. The decoupled output head uses an independent 1×1 convolutional structure to generate pixel-level segmentation prediction results.

[0011] Furthermore, the short-time adaptive branch is used to handle real-time error correction during the interaction of a single electron microscope image; when the user adds a positive or negative click on the current segmentation prediction result, a local pseudo-label region is generated according to the click position, and an interaction-aware weight mask is constructed; the short-time adaptive branch freezes the backbone encoder during the update process and only updates the decoder and decoupled output head, enabling the model to quickly correct boundary offsets, region omissions, and local missegmentation with low computational cost; The interactive perception weight mask considers both the spatial proximity of the click point and the local grayscale consistency. The spatial term uses Gaussian attenuation centered on the click point, and the grayscale term is suppressed based on the difference between the pixel grayscale and the median grayscale of the click point's neighborhood, so that the click feedback mainly acts on the local error area indicated by the user interaction. The short-time adaptive branch employs interactively-aware cross-entropy loss, weighted anchor L2 regularization, and softprobability distillation loss to jointly constrain the online update process. The interactively-aware cross-entropy loss ensures that the gradient is primarily concentrated in the local region corresponding to the user's click. The soft probability distillation loss maintains the original prediction distribution in non-interactive regions. The weighted anchor L2 regularization limits the parameters after online updates from deviating too far from the initial model, thus improving the stability of single-graph interactive correction.

[0012] Furthermore, the long-term memory branch is used for experience accumulation between electron microscope images. After the current image interaction is completed, the long-term memory branch uses the segmentation mask of the current image confirmed by the user as a supervision signal to consolidate and update the model online. The long-term memory branch uses a mask-supervised decoder and a decoupled output head to enable the model to fit the reliable segmentation result of the current image. Error correction clicks are automatically generated based on the error regions between the predicted mask and the confirmed mask, and the local error regions are strengthened and corrected by combining interactively perceived constraint cross-entropy loss. The decoder and decoupled output head are frozen, and the backbone encoder is updated only with a small learning rate, writing the interactively confirmed structural information in the current image into the long-term representation. The long-term memory branch employs a final segmentation mask supervision loss and a weight anchor L2 regularization term to jointly constrain the model update during the encoder memory update process. The final segmentation mask supervision loss is used to constrain the model to maintain its ability to represent the current image's reliable structure even in the absence of explicit click cues. The weight anchor L2 regularization term is used to limit the offset of encoder parameters relative to the initial parameters, reducing the risk of forgetting during continuous online adaptation.

[0013] Furthermore, when processing each image sequentially according to the electron microscope image sequence, the electron microscope images to be processed are input into the model one by one according to the order of continuous slicing, the order of acquisition in the same batch, the order of file number, or the order of structural similarity; each image generates an initial segmentation result from the current model, and the user adds positive or negative clicks to correct it according to the segmentation result, and the system performs short-term adaptive updates; after the current image interaction is completed, the system performs long-term memory updates and uses the updated model for the next image.

[0014] This invention also provides an online adaptive interactive segmentation system for electron microscope image sequences, comprising: a model loading module, an image sequence reading module, a positive / negative click interaction module, an interactive encoding module, a basic segmentation prediction module, a real-time mask display module, a short-term adaptive update module, a long-term memory update module, a result storage module, and a model weight storage module, wherein: the model loading module is used to load a trained basic segmentation model; the image sequence reading module is used to read the electron microscope image sequence to be processed; the positive / negative click interaction module is used to receive positive and negative clicks input by the user; the interactive encoding module is used to generate an interactive prior feature map; the basic segmentation prediction module is used to obtain the segmentation prediction result of the current electron microscope image; the real-time mask display module is used to display the original image, the user click position, and the current prediction mask; the short-term adaptive update module is used to update the decoder and decoupled output head according to the user click feedback; the long-term memory update module is used to update the backbone encoder representation after the current image interaction is completed; the result storage module is used to store the segmentation result and click log; and the model weight storage module is used to store the online updated model weights.

[0015] Compared with the prior art, the present invention has the following advantages: The online adaptive interactive segmentation method for electron microscope image sequences of this invention demonstrates excellent performance in terms of segmentation accuracy and interactive efficiency. By introducing short-time adaptive branches and long-time memory branches into the interactive segmentation network, the complementary effect of the structural correlation between user interaction feedback and source TEM images is fully utilized, significantly improving the segmentation accuracy of TEM image sequences. Especially in segmentation tasks with complex textures, weak boundaries, region adhesion, and local omissions, the short-time adaptive branch can promptly correct boundary offsets, missegmented regions, and missed segments in the current image by rapidly updating the decoder and decoupled output head online; the long-time memory branch gradually integrates reliable segmentation results confirmed by the user into the backbone encoder representation, overcoming the shortcomings of traditional interactive segmentation methods that rely solely on single-image click prompts and have difficulty inheriting interaction experience from previous images.

[0016] The dual-timescale online adaptive mechanism also exhibits strong robustness. Whether handling complex boundaries and structural changes in cell structure images, nanoparticle images, or continuous slice images, its segmentation performance outperforms static interactive segmentation methods without online parameter updates. Through an interactively perceptive weight mask, this invention enables user click feedback to primarily affect local error regions, reducing the impact on prediction results for non-clicked areas. Through soft probability distillation and L2 regularization of weight anchors, this invention limits parameter drift during online updates, reduces the risk of local overfitting and catastrophic forgetting, and ensures stable adaptability of the model during continuous TEM image sequence processing. Furthermore, this invention uses a decoupled output head as the main update object in the short-time adaptive phase, reducing the number of parameters updated online and improving segmentation efficiency while maintaining interactive response speed.

[0017] In summary, the method of this invention is specifically designed for interactive segmentation of TEM image sequences, enabling efficient and accurate target region segmentation. It is particularly suitable for TEM image analysis of continuous slices, batch sampling, cellular structures, and nanoparticles, which possess structural correlations or complex boundary features. This method reduces the repetitive interactive burden on domain experts during continuous image annotation, providing crucial technical support for quantitative microstructure research, morphological statistics, and automated annotation in materials science, biomedicine, and nanostructure analysis. It also lays the foundation for subsequent intelligent analysis and performance optimization of electron microscopy images. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a framework diagram of the online adaptive interactive segmentation method for electron microscope image sequences in this invention.

[0020] Figure 2 This is a schematic diagram of the dual-timescale online adaptive mechanism in this invention.

[0021] Figure 3 This is a schematic diagram illustrating the actual interactive segmentation effect of the TEM dataset of cell structure in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram illustrating the actual interactive segmentation effect of the nanoparticle TEM dataset in an embodiment of the present invention.

[0023] Figure 5This is a schematic diagram of the interactive interface of the online adaptive interactive segmentation system for electron microscope image sequences in an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0028] like Figure 1As shown, this invention provides an online adaptive interactive segmentation method for electron microscope image sequences, comprising: acquiring an electron microscope image dataset for training and testing, and preprocessing the electron microscope images and their corresponding segmentation annotations; in a preferred embodiment of this invention, the electron microscope image dataset includes at least one of cell structure images and nanoparticle images; and performing size unification, grayscale normalization, and training / test segmentation processing on the electron microscope images to adapt to the input requirements of the network model.

[0029] An interactive encoder is constructed to generate an interactive prior feature map based on user interaction information. In a preferred embodiment of the invention, the user interaction information consists of positive and negative clicks, where positive clicks represent target regions and negative clicks represent background regions or mispredicted regions. A distance map is generated based on the click positions, which describes the spatial distance relationship between each pixel in the image and the user's click point. The distance map is then sequentially processed through 1×1 convolution, LeakyReLU activation, 3×3 convolution, and Sigmoid mapping to obtain an interactive prior feature map with the same spatial dimensions as the original electron microscope image.

[0030] A basic segmentation network is constructed to fuse the original electron microscope image and the interactive prior feature map. The segmentation prediction result of the current electron microscope image is obtained through a backbone encoder, decoder, and decoupled output head. Specifically, in a preferred embodiment of this invention, the basic segmentation network includes a backbone encoder, decoder, and decoupled output head. The backbone encoder uses a ResNet50 encoder, preferably a ResNet50 pre-trained on a large-scale electron microscope image dataset, as the initial weights to obtain low-level texture, boundary, and structural representations suitable for TEM images. The original electron microscope image and the interactive prior feature map are concatenated along the channel dimension and then input into the backbone encoder to form a joint representation of visual information and interactive prior. The decoder adopts a U-Net-style step-by-step upsampling structure. Each decoding block includes an upsampling layer, a feature concatenation layer, a convolutional layer, a batch normalization layer, and a ReLU activation layer. Features of different scales from the backbone encoder are passed to the decoder through skip connections, concatenated with the features at the decoding end, to supplement high-resolution spatial information and restore the boundary details of the target region. The decoupled output head uses an independent 1×1 convolutional structure to generate pixel-level segmentation prediction results. With fewer parameters and a simpler structure, the decoupled output head is suitable as the primary update target during the online interaction phase, thereby reducing the risk of large-scale parameter perturbations caused by single-sample interaction feedback.

[0031] A short-time adaptive branch is established. When the user clicks to correct the current segmentation prediction result, the decoder and decoupled output head are updated online based on the click feedback to obtain the corrected segmentation result of the current image. In a preferred embodiment of the invention, the short-time adaptive branch is used to handle real-time error correction during the interaction of a single electron microscope image. When the user adds a positive or negative click to the current segmentation prediction result, a local pseudo-label region is generated based on the click position, and an interactive perception weight mask is constructed. During the update process, the short-time adaptive branch freezes the backbone encoder and only updates the decoder and decoupled output head, enabling the model to quickly correct boundary offsets, region omissions, and local missegmentation with low computational cost. The interactive perception weight mask considers both the spatial proximity of the click point and the local grayscale consistency. The spatial term uses Gaussian attenuation centered on the click point, while the grayscale term is suppressed based on the difference between the pixel grayscale and the median grayscale of the click point's neighborhood. This ensures that the click feedback mainly affects the local error area indicated by the user interaction. As the number of interactions increases, the radius of the click's influence gradually decreases, allowing early clicks to focus on the rapid correction of large-scale errors, while later clicks focus on the fine adjustment of boundaries and detail areas.

[0032] The short-time adaptive branch employs interactive-aware cross-entropy loss, weight anchor L2 regularization, and softprobability distillation loss to jointly constrain the online update process. Interactive-aware cross-entropy loss is used to ensure that the gradient is mainly concentrated in the local region corresponding to the user's click. Soft probability distillation loss is used to maintain the original prediction distribution in non-interactive regions. Weight anchor L2 regularization is used to limit the parameters after online update from deviating too far from the initial model, thereby improving the stability of single-graph interactive correction.

[0033] A long-term memory branch is established. After the current image interaction is completed, the model is consolidated online using the final segmentation mask confirmed by the user, and the reliable structural information in the current image is written into the backbone encoder representation. In a preferred embodiment of the invention, the long-term memory branch is used for experience accumulation between electron microscopy images. After the current image interaction is completed, the long-term memory branch uses the segmentation mask of the current image confirmed by the user as a supervision signal to consolidate and update the model online. The long-term memory branch uses a mask-supervised decoder and a decoupled output head to enable the model to fit the reliable segmentation result of the current image. Error correction clicks are automatically generated based on the error regions between the predicted mask and the confirmed mask, and local error regions are strengthened and corrected by combining interactively perceived constraint cross-entropy loss. The decoder and decoupled output head are frozen, and the backbone encoder is updated only with a small learning rate, and the interactively confirmed structural information in the current image is written into the long-term representation. The long-term memory branch employs a final segmentation mask supervision loss and a weight anchor L2 regularization term to jointly constrain the model update during the encoder memory update process. The final segmentation mask supervision loss is used to constrain the model to maintain its ability to represent the current image's reliable structure even in the absence of explicit click cues. The weight anchor L2 regularization term is used to limit the offset of encoder parameters relative to the initial parameters, reducing the risk of forgetting during continuous online adaptation.

[0034] The system processes each electron microscope image sequentially according to the image sequence. When processing the current image, it uses a model updated through interactions with previous images to generate an initial segmentation result. After the current image interaction is complete, the final segmentation mask is output. Specifically, in a preferred embodiment, the electron microscope images are processed sequentially by inputting them into the model one by one according to the order of consecutive slices, batch acquisition, file number, or structural similarity. Each image generates an initial segmentation result from the currently updated model. The user adds positive or negative clicks to correct the segmentation result, and the system performs short-term adaptive updates. After the current image interaction is complete, the system performs long-term memory updates and uses the updated model for the next image. Through this process, the model can gradually inherit effective interaction experience from previous images in consecutive or batch TEM images, reducing the number of interactions required for subsequent images to achieve the target segmentation accuracy.

[0035] Output the final segmentation mask for each electron microscope image, and save the segmentation results, click logs, and online updated model weights.

[0036] This invention also provides an online adaptive interactive segmentation system for electron microscope image sequences, comprising: a model loading module, an image sequence reading module, a positive / negative click interaction module, an interactive encoding module, a basic segmentation prediction module, a real-time mask display module, a short-term adaptive update module, a long-term memory update module, a result storage module, and a model weight storage module, wherein: the model loading module is used to load the trained basic segmentation model; the image sequence reading module is used to read the electron microscope image sequence to be processed; the positive / negative click interaction module is used to receive positive and negative clicks input by the user; the interactive encoding module is used to generate interactive prior feature maps; the basic segmentation prediction module is used to obtain the segmentation prediction result of the current electron microscope image; the real-time mask display module is used to display the original image, the user click position, and the current prediction mask; the short-term adaptive update module is used to update the decoder and decoupled output head according to the user click feedback; the long-term memory update module is used to update the backbone encoder representation after the current image interaction is completed; the result storage module is used to store the segmentation result and click log; and the model weight storage module is used to store the model weights after online updates.

[0037] like Figure 5As shown, after the user loads the pre-trained model and the sequence of electron microscope images to be processed, the system first displays the current original electron microscope image and its prediction mask. The user selects a positive or negative click based on the current segmentation result to correct the missed and mis-segmented regions. The system performs short-term adaptive updates based on the user's clicks and performs long-term memory updates after the current image interaction is completed. After completing the processing of the current image, the system uses the updated model for the initial segmentation of the next electron microscope image until the segmentation of the entire electron microscope image sequence is completed, and saves the segmentation results, click logs, and updated model weights.

[0038] Example 1 This embodiment provides an online adaptive interactive segmentation method for electron microscope image sequences, including the following steps: Step 1: Obtain TEM image datasets for model training and testing. In this embodiment, two representative electron microscopy image datasets are used for validation: the HeLa cell image dataset and the nanoparticle TEM image dataset. It should be noted that the two datasets are used for independent training and testing. Specifically, a corresponding basic segmentation model is trained for each dataset, and then online adaptive interactive segmentation validation is performed on the test sequences of that dataset. For each dataset, image size unification, grayscale normalization, annotation mask preparation, and training and testing set partitioning are performed separately.

[0039] Step 2: Construct an interactive encoder based on user-input positive and negative clicks to generate an interactive prior feature map with the same spatial dimensions as the original TEM image. User interaction information is composed of both positive and negative clicks, where positive clicks represent target regions and negative clicks represent background regions or mispredicted regions. The system generates a distance map based on the click positions, which describes the spatial distance relationship between each pixel in the image and the user's click point. Subsequently, the distance map is sequentially processed through 1×1 convolution, LeakyReLU activation, 3×3 convolution, and Sigmoid mapping to obtain the interactive prior feature map with the same spatial dimensions as the original TEM image. This interactive prior feature map is used to guide the model to perceive the user's intention to correct the foreground and background.

[0040] Step 3: The original TEM image from Step 1 is concatenated with the interactive prior feature map obtained in Step 2 along the channel dimension. This concatenation is then fed into the backbone encoder to extract multi-scale visual features. The segmentation prediction result of the current TEM image is obtained through the decoder and decoupled output head. Specifically, an online adaptive interactive segmentation network is constructed. This network includes an interactive encoder, a ResNet50 backbone encoder, a U-Net decoder, and a decoupled output head. The backbone encoder uses a ResNet50 pre-trained on the CEM500K large-scale electron microscopy image dataset to obtain texture, boundary, and structural representations suitable for TEM images. The U-Net decoder fuses the multi-scale features output by the backbone encoder through progressive upsampling and skip connections to recover the spatial details of the target region. The decoupled output head generates pixel-level segmentation prediction results through 1×1 convolutions. During the offline training phase, a corresponding online adaptive interactive segmentation base model is trained for each dataset class. The model is built on the PyTorch framework, and the Adam optimization algorithm is used during training. The training batch size is 4, the learning rate is set to 0.00002, the number of training epochs is 1000, and the loss function uses binary cross-entropy loss as the global optimization objective.

[0041] Step 4: As Figure 2 As shown, based on the segmentation prediction results obtained in step 3, a short-term adaptive branch is established in response to positive or negative clicks input by the user. Local pseudo-label regions and interactive-aware weight masks are generated according to the click location, and the decoder and decoupled output head are updated online to obtain the corrected segmentation result for the current TEM image. During the testing phase, the model adopts a single-sample online update strategy. For the current TEM image, the system first generates an initial segmentation prediction result through step 3, and the user adds a positive or negative click based on this prediction result. After each click, the system performs a short-term adaptive update. During the short-term adaptive update process, the backbone encoder and decoder are frozen, and only the decoder and decoupled output head are updated. The optimization objectives of this update process include interactive-aware constrained cross-entropy loss, soft probability distillation loss, and L2 regularization of weight anchors, enabling the model to maintain stable predictions of non-interactive regions while quickly correcting boundary offsets, missed region classifications, and local missegmentation in the current image.

[0042] Step 5: Based on the corrected segmentation result of the current TEM image obtained in Step 4 or the final segmentation mask confirmed by the user, a long-term memory branch is established to consolidate the model online, and reliable structural information in the current TEM image is written into the backbone encoder representation. After the current TEM image interaction is completed, the system performs a long-term memory update. In this stage, the final segmentation mask of the current image confirmed by the user is used as a supervision signal. First, the decoder and decoupled output head are lightly consolidated to ensure that the model fully fits the reliable segmentation result of the current image. Then, error correction clicks are automatically generated based on the error regions between the predicted mask and the final confirmed mask, and local error regions are strengthened and corrected by combining interactively perceptual constraint cross-entropy loss. Finally, the decoder and decoupled output head are frozen, and the backbone encoder is updated only with a small learning rate. Through this process, reliable structural information in the current image is written into the backbone encoder representation, enabling the model to obtain better initial segmentation capabilities on subsequent TEM images from the same source.

[0043] Step 6: Use the updated model from Step 5 to process the next image in the TEM image sequence, generating the initial segmentation result for the next TEM image. Repeat steps 2 to 5 until all images in the TEM image sequence are segmented. The system processes images sequentially according to the TEM image sequence. After the current image completes short-term adaptive update and long-term memory update, the updated model is used for the initial segmentation of the next TEM image. For the next TEM image, the system re-receives user positive and negative clicks, generates corresponding interaction prior feature maps, and repeatedly performs interaction encoding, basic segmentation prediction, short-term adaptive update, and long-term memory update. This cycle continues, allowing the model to gradually inherit effective interaction experience from previous images in consecutive or batch TEM image sequences.

[0044] Step 7: Output the final segmentation mask for each TEM image, and save the segmentation results, click logs, and online updated model weights through an interactive segmentation system.

[0045] The invention was applied to specific datasets to demonstrate its segmentation performance. The results were showcased on two representative TEM image datasets. The invention was implemented on two datasets with ground truth annotations: a cellular structure TEM image dataset and a nanoparticle TEM image dataset. The actual interactive segmentation results for the cellular structure TEM dataset are shown below. Figure 3 As shown, the actual interactive segmentation effect of the nanoparticle TEM dataset is as follows: Figure 4 As shown.

[0046] It should be noted that this embodiment does not train the two datasets into a single model. Instead, it trains a separate base segmentation model for each dataset and performs online adaptive interactive segmentation validation on test sequences of that dataset. Before each test sequence, the model is initialized from the base model trained on the corresponding dataset, and the online updated parameters are not shared between different test sequences.

[0047] This embodiment compares the method of the present invention with the static interactive segmentation model Base, which does not perform online parameter updates. The Base model only uses user clicks as spatial cues for the current image and does not update model parameters during the testing phase. The method of the present invention introduces short-term adaptive branches and long-term memory branches during the testing phase, enabling user clicks to not only correct the current image but also influence the segmentation results of subsequent source images through online updates. The comparison metrics include the Intersection over Union (IoU) and the number of clicks (NoC) required to reach the target IoU. The values ​​in Table 1 are the average IoU results obtained from multiple tests, and the values ​​in Table 2 are the average number of clicks required to reach the target IoU.

[0048] Table 1. Comparison of IoU metrics on two datasets in this embodiment.

[0049] Table 2 Comparison of average clicks required to achieve the target IoU of 0.90 in this embodiment

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online adaptive interactive segmentation method for electron microscope image sequences, characterized in that, include: Obtain the electron microscope image dataset for training and testing, and preprocess the electron microscope images and their corresponding segmentation annotations; Construct an interaction encoder to generate interaction prior feature maps based on user interaction information; A basic segmentation network is constructed to fuse the original electron microscope image and the interactive prior feature map, and the segmentation prediction result of the current electron microscope image is obtained through the backbone encoder, decoder and decoupled output head; A short-time adaptive branch is established. When the user clicks to correct the current segmentation prediction result, the decoder and decoupled output head are updated online based on the click feedback to obtain the corrected segmentation result of the current image. A long-term memory branch is established. After the current image interaction is completed, the model is consolidated online using the final segmentation mask confirmed by the user, and the reliable structural information in the current image is written into the trunk encoder representation. Each image is processed sequentially according to the electron microscope image sequence, and the initial segmentation result is generated by the model updated by interaction with the previous images when processing the current image. After the current image interaction is completed, the final segmentation mask is output, and the segmentation results, click logs, and updated model weights are saved.

2. The online adaptive interactive segmentation method for electron microscope image sequences according to claim 1, characterized in that, The electron microscopy image dataset includes at least one of cell structure images and nanoparticle images; the electron microscopy images are subjected to size unification, grayscale normalization, and training / test segmentation to adapt to the input requirements of the network model.

3. The online adaptive interactive segmentation method for electron microscope image sequences according to claim 1, characterized in that, The user interaction information consists of positive and negative clicks, where positive clicks represent target regions and negative clicks represent background regions or mispredicted regions. A distance map is generated based on the click position, which describes the spatial distance relationship between each pixel in the image and the user's click point. The distance map is then sequentially processed through 1×1 convolution, LeakyReLU activation, 3×3 convolution, and Sigmoid mapping to obtain an interactive prior feature map with the same spatial size as the original electron microscope image.

4. The online adaptive interactive segmentation method for electron microscope image sequences according to claim 1, characterized in that, The basic segmentation network includes a backbone encoder, a decoder, and a decoupled output head. The backbone encoder uses a ResNet50 encoder, which concatenates the original electron microscope image with the interactive prior feature map in the channel dimension and then inputs it into the backbone encoder to form a joint representation of visual information and interactive prior. The decoder adopts a U-Net-style stepwise upsampling structure. Each decoding block includes an upsampling layer, a feature concatenation layer, a convolutional layer, a batch normalization layer, and a ReLU activation layer. Features of different scales from the backbone encoder are passed to the decoder through skip connections and concatenated with the features at the decoding end to supplement high-resolution spatial information and restore the boundary details of the target region. The decoupled output head uses an independent 1×1 convolutional structure to generate pixel-level segmentation prediction results.

5. The online adaptive interactive segmentation method for electron microscope image sequences according to claim 1, characterized in that, The short-time adaptive branch is used to handle real-time error correction during the interaction of a single electron microscope image; When a user adds a positive or negative click to the current segmentation prediction result, a local pseudo-label region is generated based on the click location, and an interactive perception weight mask is constructed. The short-term adaptive branch freezes the backbone encoder during the update process and only updates the decoder and decoupled output head, enabling the model to quickly correct boundary offsets, region omissions, and local missegmentation with low computational cost. The interactive perception weight mask considers both the spatial proximity of the click point and the local grayscale consistency. The spatial term uses Gaussian attenuation centered on the click point, and the grayscale term is suppressed based on the difference between the pixel grayscale and the median grayscale of the click point's neighborhood, so that the click feedback mainly acts on the local error area indicated by the user interaction. The short-time adaptive branch employs interactively-aware cross-entropy loss, weighted anchor L2 regularization, and softprobability distillation loss to jointly constrain the online update process. The interactively-aware cross-entropy loss ensures that the gradient is primarily concentrated in the local region corresponding to the user's click. The soft probability distillation loss maintains the original prediction distribution in non-interactive regions. The weighted anchor L2 regularization limits the parameters after online updates from deviating too far from the initial model, thus improving the stability of single-graph interactive correction.

6. The online adaptive interactive segmentation method for electron microscope image sequences according to claim 1, characterized in that, The long-term memory branch is used for experience accumulation between electron microscope images. After the current image interaction is completed, the long-term memory branch uses the segmentation mask of the current image confirmed by the user as a supervision signal to consolidate and update the model online. The long-term memory branch uses a mask-supervised decoder and a decoupled output head to enable the model to fit the reliable segmentation results of the current image; it automatically generates error correction clicks based on the error regions between the predicted mask and the confirmed mask, and strengthens and corrects local error regions by combining interactively-aware constrained cross-entropy loss; it freezes the decoder and the decoupled output head, updates the backbone encoder only with a small learning rate, and writes the interactively confirmed structural information in the current image into the long-term representation. The long-term memory branch employs a final segmentation mask supervision loss and a weight anchor L2 regularization term to jointly constrain the model update during the encoder memory update process. The final segmentation mask supervision loss is used to constrain the model to maintain its ability to represent the current image's reliable structure even in the absence of explicit click cues. The weight anchor L2 regularization term is used to limit the offset of encoder parameters relative to the initial parameters, reducing the risk of forgetting during continuous online adaptation.

7. The online adaptive interactive segmentation method for electron microscope image sequences according to claim 1, characterized in that, When processing each image sequentially according to the electron microscope image sequence, the electron microscope images to be processed are input into the model one by one according to the order of continuous slicing, the order of acquisition in the same batch, the order of file number, or the order of structural similarity; each image generates an initial segmentation result from the current model, and the user adds positive or negative clicks to correct it according to the segmentation result, and the system performs short-term adaptive updates; After the current image interaction is completed, the system performs a long-term memory update and uses the updated model for the next image.

8. An online adaptive interactive segmentation system for electron microscope image sequences, implemented based on the online adaptive interactive segmentation method for electron microscope image sequences according to any one of claims 1-7, characterized in that, include: The system includes a model loading module, an image sequence reading module, a positive / negative click interaction module, an interactive encoding module, a basic segmentation prediction module, a real-time mask display module, a short-term adaptive update module, a long-term memory update module, a result saving module, and a model weight saving module. The model loading module is used to load the trained basic segmentation model; The image sequence reading module is used to read the electron microscope image sequence to be processed; The positive and negative click interaction module is used to receive positive and negative clicks input by the user; The interactive encoding module is used to generate interactive prior feature maps; The basic segmentation prediction module is used to obtain the segmentation prediction result of the current electron microscope image; The real-time mask display module is used to display the original image, the user's click position, and the current predicted mask; The short-term adaptive update module is used to update the decoder and decoupled output head based on user click feedback; The long-term memory update module is used to update the backbone encoder representation after the current image interaction is completed; The result saving module is used to save the segmentation results and click logs; The model weight saving module is used to save the model weights after online updates.

Images