Microscope video frame interpolation method based on self-supervised learning

Abstract

The present application belongs to the technical field of video processing, and specifically relates to a microscope video frame interpolation method based on self-supervised learning. The method comprises the following steps: firstly, a deep learning model is trained by using a self-supervised training strategy to learn video frame interpolation from a noisy microscope video; secondly, a space-time feature alignment module based on a selective state space model is introduced to efficiently perform video frame interpolation; finally, a simulation data set is used to evaluate the microscope video frame interpolation task, and the data set contains different objects and noise levels. Experimental results show that the present model has significant advantages in video frame interpolation performance, and the processing memory consumption is only one fourth of that of the previous model, and the present model has high practical value.

Description

Technical Field

[0001] The invention belongs to the technical field of digital video processing, and in particular relates to a microscope video frame insertion method. Background Art

[0002] Video interpolation increases the total frame count of a video by creating new frames between existing frames. This is achieved by analyzing the content changes and motion trajectories between adjacent frames, generating seemingly natural intermediate frames, increasing the video's frame rate, and making video playback smoother. Early video interpolation technologies relied on simple linear interpolation methods, such as frame repetition or methods based on optical flow. Optical flow technology estimates the pixel positions of the intermediate frames by calculating the motion vectors of the pixels between two frames. While these methods can increase the video frame rate, they often cause blurring and artifacts when dealing with fast motion or complex scenes.

[0003] With the development of deep learning technology, video interpolation technology has also ushered in major breakthroughs. Deep learning models, especially convolutional neural networks (CNNs), can more accurately predict the content of intermediate frames. These models capture more complex spatiotemporal relationships and motion patterns by learning from large amounts of video data, significantly improving the quality of interpolation, especially when dealing with dynamic scenes and complex interactions. The latest interpolation technologies include methods based on transformer network models. These technologies can more comprehensively understand the dynamic changes in video content and provide more accurate frame interpolation. For example, by analyzing global motion and local detail changes, these models can generate high-quality frames with almost no perceptible differences. However, there are problems with the small receptive field of convolutional neural networks and the high computational cost of transformer networks.

[0004] Selective State-Space Model (SSM) [5] The Selective State Space Model (SSSM) is an efficient mathematical framework for sequential data processing that combines concepts from linear time-invariant systems with the advantages of deep learning. This model has been widely used in signal processing, system control, and, more recently, machine learning. The Selective State Space Model (SSSM) exhibits significant operational efficiency advantages over convolutional neural networks and transformer networks, particularly when processing large amounts of data and high-dimensional information. However, the SSSM has not yet been applied to video interpolation.

[0005] Video imaging technology is an important tool in biomedical research and diagnosis, especially for the dynamic observation of cells and microscopic biological structures. However, due to the limitations of microscope equipment and the complexity of the imaging environment, biological imaging videos are often interfered with by various noises, such as electronic noise, optical distortion, and motion blur. These noises can severely reduce the signal-to-noise ratio (SNR) of the video, thereby affecting the quality and usability of the video. Existing video interpolation algorithms often rely on a large amount of clean data for supervised learning, which is particularly difficult in the field of biological imaging because obtaining clean biological imaging videos is extremely expensive and it is difficult to ensure the consistency of samples. Microscope video interpolation requires a noise-robust training method to solve this problem. Summary of the Invention

[0006] The purpose of the present invention is to provide an efficient and robust microscope video interpolation method that can interpolate high frame rate and low noise video frames in biological imaging videos while solving the complexity and efficiency problems brought by previous video interpolation models.

[0007] The microscope video frame insertion method provided by the present invention is based on self-supervised learning technology and uses the selective state space model (SSM) for frame alignment. The specific steps are as follows:

[0008] (1) The video frames containing various noises and low frame rates generated during the biological imaging process are obtained through the video acquisition device, and the feature extraction module is used [1] Extract features from the acquired video frames and use the feature interpolation module [1] Interpolation is performed to obtain a video frame containing noise.

[0009] (2) Feature alignment based on selective state space: The present invention proposes a spatiotemporal feature alignment module (STAM) based on a selective state space model to perform feature alignment on the input noisy video frames. The spatiotemporal feature alignment module dynamically adjusts the areas and dimensions that the model focuses on, accurately processes the spatiotemporal features in the video, optimizes the integration of inter-frame information, and significantly improves the quality of the video frames.

[0010] (3) Using the frame synthesis module [1] After feature alignment and information integration, the video data is reconstructed into video frames, outputting clear videos with high frame rates and low noise. These video frames are visually close to ideal biological imaging results and are suitable for subsequent scientific analysis and research.

[0011] (4) A noise-robust loss function is used to train a video interpolation model (including a feature extraction module, a feature interpolation module, a spatiotemporal feature alignment module, and a frame synthesis module); wherein the temporal continuity and intrinsic correlation between adjacent frames in the video are utilized to automatically learn the transformation and interpolation details between video frames, thereby training a model capable of interpolating clear frames from noisy frames. The present invention does not rely on pre-labeled clean frames to train the video interpolation model.

[0012] (5) The performance of the interpolation task is evaluated using a data simulation method. The data contains biological objects with different motion patterns (such as cells, microtubules, etc.) and settings with different noise levels, which can be used to simulate the complex situations encountered in actual biological imaging.

[0013] Further:

[0014] In step (2), the spatiotemporal feature alignment module performs feature alignment processing on the input noisy video frames, and the steps are as follows: first, the features are divided into several spatiotemporal layers, where the spatiotemporal layer is defined as a slice of the feature with a shape of T×W (horizontal) or T×H (vertical); then, each spatiotemporal layer is processed according to VMamba [4] In this method, each feature in the spatiotemporal layer is regarded as a spatiotemporal token, and the token is expanded into a sequence along the rows and columns, and then a zigzag scan is performed along four different directions: from the upper left corner to the lower right corner, the lower right corner to the upper left corner, the upper right corner to the lower left corner, and the lower left corner to the upper right corner; the spatiotemporal token from the vertical direction is selected by the state space module [5] Processing; Finally, all spatiotemporal tokens of each layer are concatenated into vertical tokens and horizontal tokens, and connected using a convolution layer with a convolution kernel size of 1×1.

[0015] In step (4), the loss function is as follows: first, the Euclidean distance between pixel values ​​is used to constrain the reconstructed video frame and the real video frame:

[0016] L R =||f θ (φ ↓ (I N ))-I N ||2 , (1)

[0017] Among them, f θ is the restoration model with parameter θ, I N is the target noise frame, φ ↓ is a downsampler. However, learning directly from noisy frames will amplify the noise. Therefore, a noise-robust loss function is also adopted, which can be expressed as follows:

[0018]

[0019] Among them, φ↑ is the upsampling function, essentially bilinear interpolation. g1 and g2 are neighborhood subsamplers used to generate noisy image pairs. This method uses subsampling in denoising as a way to construct super-resolution pairs, thus avoiding the trade-off between denoising and resolution reduction.

[0020] The final loss function can be expressed as:

[0021] L=λ1·L R +λ2·L N , (3)

[0022] Among them, λ1 and λ2 are two hyperparameters at different training stages.

[0023] First, in a clean dataset (such as Vimeo [3] ) and let λ2 = 0 to learn motion modeling.

[0024] The model is then fine-tuned on noisy biological imaging data, and λ1 = 0 is set to learn enhanced features for noisy microscopy data.

[0025] In step (5), the data simulation method is as follows: We consider constructing data sets under different noise conditions and microscopy scenarios, which include three noise levels (SNR = 4, SNR = 3 and SNR = 7) and three microscopy scenarios (simulating moving vesicles, receptors and microtubules):

[0026] First, we consider constructing different noise conditions. In biological imaging, shot noise is the main noise and is proportional to the square root of the number of photoelectrons: where N is the number of photoelectrons. Therefore, the noise level in pixels containing fluorescent objects is higher than the background noise, even though the background noise may be more obvious visually. Failure to account for this relationship can lead to very large errors in estimating the signal-to-noise ratio. The signal-to-noise ratio (SNR) in biological imaging is usually calculated as object (I o ) and background (I b ) divided by the representative noise level (σ):

[0027]

[0028] The present invention selects three noise levels: SNR=4, SNR=3 and SNR=7. First, I is fixed b = 10 and calculate the corresponding I for the maximum signal value o , and then add Poisson noise with a mean of 10 to the series.

[0029] Secondly, we consider constructing datasets under different microscopy scenarios. We select three different scenarios to simulate real image data of moving vesicles, receptors and microtubules. Different particles have different motion modes: Brownian motion is similar to vesicle motion in the cytoplasm, directional motion is similar to microtubule transport, and the random switching between these two motion modes is similar to membrane receptors. For each case, we generate 500 frames of 100 density images at a resolution of 512×512 and split them in an 80 / 20 ratio for training and testing. After generating clean sequences annotated with motion labels, we use the motion labels to generate noisy frames to maintain the same motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a network flow chart of the present invention.

[0031] Figure 2 Schematic diagram of the loss function of the present invention.

[0032] Figure 3 Schematic diagram of the structure of the spatiotemporal feature alignment module (STAM) of the present invention.

[0033] Figure 4 This is a diagram of the processing results of the present invention.

[0034] Figure 5 Comparison of memory consumption between the present invention and other methods. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0036] The specific steps are:

[0037] (1) The network flow chart of the present invention is as follows Figure 1 As shown. The input frame number of the present invention is 4, and the output frame number is 7. The input video frame size is 128×128. The present invention follows the ZoomingSloMo [1] The feature extraction and feature interpolation network processes the input video frames. The dimension of the features in the network is set to 64.

[0038] (2) After feature extraction and feature interpolation network processing, the feature dimension accepted by the spatiotemporal feature alignment module is 7×128×128×64. The network structure is as follows Figure 3 As shown in the figure, the spatiotemporal layer size divided by the spatiotemporal feature alignment module is 7×128, and the feature dimension is 64. The feature dimension of the final output of the spatiotemporal feature alignment module is also 7×128×128×64.

[0039] (3) Final frame synthesis network [1]The aligned features are synthesized into the output video frame. The output video frame size is 128×128. The processing results are as follows Figure 4 shown.

[0040] (4) During training, a noise-robust loss function is used to train the video interpolation model (including feature extraction module, feature interpolation module, spatiotemporal feature alignment module and frame synthesis module), such as Figure 2 As shown. Using Adam [2] The optimizer uses cosine annealing to decay the learning rate from 4e-4 to 1e-7 in each batch.

[0041] (5) First on Vimeo [3] The training was performed on the dataset for 600,000 iterations with λ2 set to 0 in the loss function, and then on the synthetic microscope dataset for 4,000 iterations with λ1 set to 0 in the loss function.

[0042] (6) The synthetic microscopy data contains three noise levels (SNR = 4, SNR = 3, and SNR = 7) and three microscopy scenes (simulating moving vesicles, receptors, and microtubules). Each noise level and scene has 500 frames. We fine-tune the model using the synthetic microscopy dataset and evaluate its interpolation performance.

[0043] (7) The spatiotemporal feature alignment module of the present invention has significantly lower memory consumption. Figure 5 Compared with ViT [6] method and replace the selective state space model in our method with ViT [6] Our method significantly reduces memory consumption compared to other methods for high-resolution inputs, especially 1024×1024 and 1536×1536. At a resolution of 1536×1536, it only requires a quarter of the memory consumption of ViT-STAM.

[0044] References:

[0045] [1]Xiaoyu Xiang,Yapeng Tian,Yulun Zhang,Yun Fu,Jan P Allebach,andChenliangXu.2020.Zooming slow-mo:Fast and accurate one-stage space-time videosuper-resolution.In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.3370–3379.

[0046] [2]Kingma,Diederik P.,and Jimmy Ba."Adam:A method for stochasticoptimization."arXiv preprint arXiv:1412.6980(2014).

[0047] [3]Tianfan Xue,Baian Chen,Jiajun Wu,Donglai Wei,and William TFreeman.2019.Video enhancement with task-oriented flow.International Journalof Computer Vision 127(2019),1106–1125.

[0048] [4]Liu,Yue,et al."Vmamba:Visual state space model."arXiv preprintarXiv:2401.10166(2024).

[0049] [5]Gu,Albert,and Tri Dao."Mamba:Linear-time sequence modeling withselective state spaces."arXiv preprint arXiv:2312.00752(2023).

[0050] [6]Dosovitskiy,Alexey,et al."An image is worth 16x16 words:Transformers for image recognition at scale."arXiv preprint arXiv:2010.11929(2020)。

Claims

1. A microscope video interpolation method based on self-supervised learning, characterized in that: The specific steps are as follows: (1) acquiring video frames containing various noises and low frame rates generated during the biological imaging process through a video acquisition device, extracting features from the acquired video frames using a feature extraction module, and interpolating the features using a feature interpolation module to obtain video frames containing noise; (2) Using a spatiotemporal feature alignment module based on a selective state space model, the input video frames containing noise are subjected to feature alignment processing. This spatiotemporal feature alignment module accurately processes the spatiotemporal features in the video by dynamically adjusting the regions and dimensions that the model focuses on, optimizing the integration of inter-frame information and improving the quality of the video frames. (3) Use the frame synthesis module to reconstruct the video data after feature alignment and information integration into video frames, and output a clear video with high frame rate and low noise; (4) A noise-robust loss function is used to train a video interpolation model, which includes a feature extraction module, a feature interpolation module, a spatiotemporal feature alignment module, and a frame synthesis module. The model automatically learns the transformation and interpolation details between video frames by utilizing the temporal continuity and intrinsic correlation between adjacent frames in the video, thereby training a model that can interpolate clear frames from noisy frames. (5) Use data simulation methods to evaluate the performance of the interpolation task. The data contains biological objects with different motion patterns and different noise levels to simulate the complex situations encountered in actual biological imaging. The loss function described in step (4) is as follows: First, the Euclidean distance between pixel values ​​is used to constrain the reconstructed video frame and the real video frame: L R =||f θ (φ ↓ (I N ))-I N ||2 , (1) Among them, f θ is the restoration model with parameter θ, I N is the target noise frame, φ ↓ is a downsampler; a noise-robust loss function is also used, which is expressed as follows: Among them, φ ↑ is the upsampling function, which is bilinear interpolation, g1 and g2 are neighborhood subsamplers used to generate noise image pairs; The final loss function is expressed as: L=λ1·L R +λ2·L N , (3) Among them, λ1 and λ2 are two hyperparameters at different training stages; specifically: First, pre-train the model on a clean dataset and set λ2 = 0 to learn motion modeling; The model is then fine-tuned on noisy biological imaging data, and λ1 = 0 is set to learn enhanced features for noisy microscopy data.

2. The microscope video frame insertion method according to claim 1, characterized in that: The spatiotemporal feature alignment module described in step (2) performs feature alignment processing on the input noisy video frames. The specific process is as follows: first, the features are divided into several spatiotemporal layers. The definition of a spatiotemporal layer is a slice in the feature with a shape of horizontal T×W (length×width) or vertical T×H (length×height); then, each spatiotemporal layer is processed, and each feature in the spatiotemporal layer is regarded as a spatiotemporal token. The tokens are expanded into sequences along rows and columns, and then zigzag scanned along four different directions: from the upper left corner to the lower right corner, the lower right corner to the upper left corner, the upper right corner to the lower left corner, and the lower left corner to the upper right corner; the spatiotemporal tokens from the vertical direction are processed by the state space selection module; finally, all the spatiotemporal tokens of each layer are spliced ​​into vertical tokens and horizontal tokens, and connected using a convolution layer with a convolution kernel size of 1×1.

3. The microscope video frame insertion method according to claim 1, characterized in that: The data simulation method described in step (5) is as follows: In biological imaging, lens noise is the main noise and is proportional to the square root of the number of photoelectrons: N is the number of photoelectrons; the signal-to-noise ratio (SNR) in biological imaging is calculated as the average intensity of the object I o and background average intensity I b The difference between them is divided by the representative noise level σ: Specifically, three noise levels are selected: SNR=4, SNR=3, and SNR=7; first, I is fixed. b =10 and is the maximum signal value, calculate the corresponding I o , and then add Poisson noise with a mean of 10 to the series.

Images