Method for improving signal-to-noise ratio of image frame sequence and image processing device

Abstract

A method for improving the signal-to-noise ratio of image frames is disclosed. The method includes estimating a representative velocity of an optical flow in a sequence of image frames, determining an interpolation factor based on the representative velocity of the optical flow, and employing a trained artificial neural network to generate an extended sequence of image frames. The extended sequence of image frames includes a plurality of interpolated image frames, where each interpolated image frame is inserted between subsequent image frames of the sequence of image frames, where a number of interpolated image frames corresponds to the interpolation factor. A time-dependent combination of image frames from the extended sequence of image frames can be computed to generate an output sequence of image frames.

Description

Technical Field

[0001] This disclosure relates to systems and methods for improving the signal-to-noise ratio in a sequence of image frames. Background Technology

[0002] High-resolution imaging techniques such as magnetic resonance imaging or fluorescence microscopy allow for time-lapse imaging (also known as time-lapse or time-lapse) of objects such as biological samples or in vivo biological tissues. High-resolution time-lapse imaging of biological structures is challenging because it requires high frame rates to capture relatively fast movements while protecting the biological structures from damage caused by microscopy methods.

[0003] For example, time-delayed fluorescence microscopy of biological samples requires both a high frame rate to allow for rapid cell movement during imaging and a minimum excitation laser intensity to reduce cell damage. Furthermore, photobleaching leads to a loss of fluorescence imaging. This conflicting requirement often results in a very low signal-to-noise ratio (SNR) in the acquired images. Other microscopy methods face similar problems. The low SNR is particularly pronounced in confocal microscopy because the light excluded by the pinhole contributes nothing to the image. STED microscopy also suffers from these issues due to its inherently low photon budget caused by the depletion of most of the excitation at the focal point.

[0004] To improve the signal-to-noise ratio of an image frame sequence, a temporal correlation combination of image frames can be applied. For example, an exponentially weighted rolling average can be applied, such as according to the following formula.

[0005]

[0006] Where I is the image frame sequence, σ is the width, w is the window size, Θ is the Herveyd function, and * denotes convolution operation. However, because rolling average involves applying a blur kernel, fast-moving objects captured in the image frame sequence are blurred. Therefore, rolling average cannot be used to improve the signal-to-noise ratio of image frame sequences capturing fast-moving objects.

[0007] Another approach to improving the signal-to-noise ratio of image frames is to apply recent advancements in deep learning methods, which now readily provide solutions for image denoising. In such methods, a single image is taken as input, regardless of the movement of objects within the image. However, a significant problem with deep learning algorithms is the creation of insufficient detail from random patterns such as noise, known as illusions. This artifact becomes particularly noticeable in noisy time-lapse images.

[0008] Figure 1A illustrates the illusion problem in existing denoising methods. Column 12 of Figure 1A reproduces two consecutive fluorescence micrograph frames taken from a live biological specimen. Applying the existing denoising algorithm Nikon's denoise.ai to the image frames in column 12 produces the image frames in column 14. Clearly, the denoising algorithm infers the shape of a realistic-looking object from the very noisy image in column 12. However, a comparison of the denoised images in column 14 for consecutive image frames shows that the predicted shape and position of the object change significantly from the upper frame to the lower frame, raising strong doubts about the accuracy of the predicted object shape. Therefore, the application of denoising is largely dependent on the temporal realization of the noise, meaning that denoising cannot be reliably applied to noise delays.

[0009] Figure 1B illustrates the blurring problem that occurs when applying a weighted rolling average according to existing techniques. Panel 16 illustrates image frames of a simulated sequence of images of a fast-moving object. As indicated by the white arrows, the object moves at a speed of v = (2, -2) pixels per frame. Panel 18 reproduces the result of applying a weighted rolling average according to Equation (1) (where σ = 4 and w = 8). As indicated in Panel 18, the object's true shape is severely distorted due to blurring caused by adjacent image frames. Summary of the Invention

[0010] According to an embodiment, a method for improving the signal-to-noise ratio of image frames is disclosed. The method includes estimating a representative velocity of optical flow in a sequence of image frames, determining an interpolation factor from the representative velocity of optical flow, and employing a trained artificial neural network to generate an extended sequence of image frames. The extended sequence of image frames may include multiple interpolated image frames, each interpolated image frame being inserted between subsequent image frames in the sequence, wherein the number of interpolated image frames corresponds to the interpolation factor. The method further includes calculating a temporally correlated combination of image frames from the extended sequence of image frames to generate an output sequence of image frames.

[0011] According to another embodiment, estimating the representative velocity of optical flow includes calculating a histogram of optical flow between subsequent image frames in an image frame sequence and analyzing the histogram to determine the representative velocity.

[0012] According to one aspect, analyzing the histogram to determine the representative velocity involves using the histogram to determine the representative velocity as a quantile of a predetermined threshold. Calculating the histogram can be based on estimating pixel-wise optical flow, such as through methods based on... The method of the algorithm.

[0013] In a particular embodiment, the method for improving the signal-to-noise ratio in an image frame sequence can be applied to a microscopic image frame sequence.

[0014] According to one aspect, the trained artificial neural network used in the method for improving the signal-to-noise ratio in an image frame sequence involves a feature shaping operation with channel attention. In an embodiment, the feature shaping operation is a pixel shuffling operation.

[0015] In one embodiment, the interpolation factor is a power of two and the trained artificial neural network is configured to recursively generate interpolated image frames and add them to the image frame sequence, wherein the number of recursions corresponds to the power.

[0016] According to another aspect, calculating the temporal correlation combination of image frames includes applying a rolling average to an extended sequence of image frames. The rolling average can be a weighted rolling average, and applying a weighted rolling average includes determining parameters for the weighted rolling average from a representative velocity.

[0017] According to one aspect, the parameters used for the weighted rolling average are the window size that determines the set of image frames in the extended image frame sequence that contribute to the weighted rolling average, and the width that determines the weight of each image frame from that set that contributes to the weighted rolling average.

[0018] According to one approach, methods for improving the signal-to-noise ratio in an image frame sequence include applying a denoising algorithm to the output image frame sequence.

[0019] According to an embodiment, the artificial neural network has been trained by pre-training the artificial neural network with a non-domain-specific sequence of image frames and training the artificial neural network with domain-specific image frames.

[0020] According to yet another embodiment, an image processing apparatus for improving the signal-to-noise ratio of image frames is disclosed. The image processing apparatus includes a memory configured to at least temporarily store a sequence of image frames, processing circuitry configured to estimate a representative velocity of optical flow in the image frame sequence and to determine an interpolation factor from the representative velocity of optical flow, and a trained artificial neural network for generating an extended sequence of image frames, wherein the extended sequence of image frames includes a predetermined number of interpolated image frames, each interpolated image frame being inserted between subsequent image frames in the image frame sequence. This predetermined number may correspond to an interpolation factor, and the processing circuitry may also be configured to compute a temporally correlated combination of image frames from the extended sequence of image frames to generate an output sequence of image frames. Attached Figure Description

[0021] Figure 1A illustrates the application of denoising to an exemplary image frame according to a method of the prior art;

[0022] Figure 1B illustrates the application of a weighted rolling average to an exemplary image frame according to a method of the prior art;

[0023] Figure 2 The diagram illustrates a flowchart of a method for improving the signal-to-noise ratio in an image frame sequence;

[0024] Figure 3 The illustration shows a histogram of optical flow in an image frame sequence as defined in the embodiments;

[0025] Figure 4 The illustration shows an artificial neural network used according to an embodiment;

[0026] Figure 5 The illustration shows a flowchart of a method for training an artificial neural network according to an embodiment;

[0027] Figure 6 The illustration shows an image processing apparatus for improving the signal-to-noise ratio noise in an image frame sequence;

[0028] Figure 7 An exemplary image frame is reproduced, illustrating the improved signal-to-noise ratio achieved by employing the method of the embodiment; and

[0029] Figure 8 Another exemplary image frame is illustrated, demonstrating the improved signal-to-noise ratio achieved using the method of the embodiment. Detailed Implementation

[0030] Figure 2 The illustration depicts the steps of a method 200 for improving the signal-to-noise ratio in a sequence of image frames. The sequence of image frames corresponds to two-dimensional, three-dimensional, or even higher-dimensional image data that may include several color channels. In embodiments, the image frames are associated with microscopic time-lapse. For example, microscopic time-lapse can capture images of a live biological sample and its motion. In a particular embodiment, the image frames may be captured by a dedicated camera for fluorescence imaging integrated into a surgical microscope. In other embodiments, the image frames may be volumes of biological tissue captured by medical imaging such as magnetic resonance imaging.

[0031] Method 200 includes estimating 202 representative velocities v from an image frame sequence. repr The motion of an object captured in an image frame means the movement of the object's position from one image frame to the next. In the context of this application, the camera remains in a fixed position and the velocity is based on the inherent motion of the object being photographed, such as biological motion. The estimated representative velocity can be a velocity close to the maximum velocity of the object captured in the image frame.

[0032] In the embodiment, the representative velocity v is estimated. repr Step 202 may include calculating a histogram of optical flow between image frames in the image frame sequence. Calculating the histogram may involve determining the dense optical flow v between each pair of subsequent image frames in the image frame sequence. opt The value of . In an embodiment, dense optical flow can be achieved by employing Algorithms determine by pixels, such as "Two-Frame Motion Estimation Based on Polynomial Expansion," Scandinavian Conference on Image Analysis 2003, pp. 363-370. The histogram can be correlated with the absolute value of optical flow |v... opt The histogram corresponds to |.

[0033] Figure 3 An exemplary histogram of optical flow in an image frame sequence is shown. Typically, the histogram peaks from zero because the image background does not move with each frame. In this embodiment, a representative velocity v is... repr quantiles determined as a predetermined threshold

[0034] v repr =Quantile(|v opt |,p). (1)

[0035] For the purposes of this invention, quantiles are sufficient because this method minimizes the impact of outliers. A threshold of p = 0.9 is typically used so that all optical flow values ​​|v opt | 90% is less than the representative speed. In Figure 3 In the example, the estimated representative speed is 10 pixels per frame.

[0036] Refer again Figure 2 Method 200 further includes step 204, wherein the interpolation factor α is determined using the determined representative velocity. interp The interpolation factor corresponds to the expected increase in frame rate, such as two or four.

[0037] In the embodiment, the interpolation factor α can be determined. interp Make Pixels / frame. In this embodiment, the interpolation factor is determined as a power of 2 according to the following formula:

[0038]

[0039] Method 200 further includes generating 206 interpolated microframes using a trained artificial neural network. Generating 206 interpolated microframes involves generating a predetermined number of interpolated image frames, each interpolated image frame being inserted between subsequent image frames in an image frame sequence. Generating 206 interpolated microframes can produce an extended image frame sequence with a frame rate corresponding to the interpolation factor determined in step 204.

[0040] because The requirement is that the residual optical flow between consecutive image frames in the extended image frame sequence is less than one pixel per frame, and a rolling average can be used to improve the signal-to-noise ratio in the extended image frame sequence without producing the virtual effects discussed above.

[0041] Specifically, steps 202 to 206 of method 200 involve determining optical flow to determine an interpolation factor. However, in the described embodiments, determining the interpolated image frame corresponding to the interpolation factor does not depend on determining the optical flow. In embodiments, generating the 206 interpolated microscopic frames can employ an artificial neural network that does not depend on estimating the optical flow, as referenced below. Figure 4 The method described above, which does not rely on estimated optical flow to generate interpolated image frames, allows for a reduction in the latency of the step of generating interpolated image frames.

[0042] In this embodiment, generating 206 interpolated microscopic frames involves the recursive generation of interpolated image frames. In these embodiments, the interpolation factor is determined to be a power of 2. The input image frame sequence can be processed a first time to create an interpolated image frame between each consecutive image frame of the input image frame sequence, thereby producing a first extended microscopic frame sequence with a double frame rate. The first extended image frame sequence can be processed again by a trained artificial neural network to again generate interpolated image frames between each consecutive microscopic image frame of the first extended image frame sequence to produce a second extended image frame sequence with a frame rate four times that of the input image frame sequence. Therefore, the frame rate of the extended image frame sequence can be doubled until the interpolation factor α is reached. interp The expected multiple.

[0043] Method 200 further includes processing the extended image frame sequence 208 by applying a temporal correlation combination of image frames in the extended image frame sequence, such that the resulting enhanced image frame sequence has an improved signal-to-noise ratio. In one embodiment, the application of the temporal correlation combination of image frames may be based on a weighted rolling average. According to other embodiments, the application of the temporal correlation combination of image frames may be based on the application of a Kalman filter.

[0044] In another embodiment, processing the 208-extended image frame sequence may include applying a weighted scrolling according to equation (1). To avoid blurring of objects under equation (1), the velocity v of the moving objects in the image frame sequence is... obj It should be Pixels / frame. Furthermore, σ should be satisfied.

[0045] Specifically, the parameter of the weighted rolling average in equation (1) can be determined by v in step 204. repr Determined according to the following formula:

[0046]

[0047] Therefore, the image frame sequence is expanded with an appropriate number of interpolated image frames, and then a weighted rolling average as explained above is applied, corresponding to the motion-aware rolling average.

[0048] In an embodiment, method 200 may further include downsampling 210 of the enhanced image frame sequence generated from the step, such as downsampling the frame rate of the image frame sequence to correspond to the frame rate of the original image frame sequence. Therefore, downsampling the enhanced image frame sequence may include selecting each α-th step of the enhanced image frame sequence. interp Image frames. For example, when the original image frame sequence consists of [f1, f2, ..., f...] n The composition is determined in method step 204, and α is determined therein. interp When = 2, step 206 is executed to generate an extended image frame sequence [f1, f2, f3, f4, f5, f6, f7, f8, f9, f1, f1, f1, f2 ...3, f1, f2, f3, f1, f2, f3, f4, f5, f1, f6, f1 1.5 f2, f 2.5 , ..., f n ], and perform step 208 to generate an enhanced image frame sequence [r1, r 1.5 r2, r 2.5 ,...,r n From this, you can select [r1, r2, ... r]. n As an output image frame sequence with an improved signal-to-noise ratio.

[0049] In an embodiment, method 200 may further include applying a denoising algorithm 212 to the enhanced and optionally subsampled image frame sequence to further improve the signal-to-noise ratio. Prior art denoising algorithms consider only a single image frame without considering the motion of the depicted object. In an embodiment, applying the denoising algorithm may include applying a convolutional neural network. Because denoising is applied after the temporally correlated combination of the image frames, the illusion problem discussed above with reference to FIG1A is reduced. In particular, applying the denoising algorithm to the enhanced image frame sequence synergistically combines denoising methods and interpolation methods.

[0050] Figure 4 The diagram illustrates the architecture of a chain-like artificial neural network 400, configured to perform generative and alpha operations. interp Step 206 of the corresponding interpolation microframe.

[0051] The artificial neural network 400 can be selected as a lean artificial neural network. Specifically, the artificial neural network 400 may lack a dedicated submodule for estimating optical flow. Therefore, the artificial neural network 400 can provide low latency in performing step 206 of generating interpolated image frames. Thus, embodiments of this disclosure can relate to real-time processing of image frame sequences to improve the signal-to-noise ratio.

[0052] Because optical flow determination is highly sensitive to noise, and because microscope images are particularly susceptible to higher noise levels than ordinary photographic images, conventional frame interpolation methods for optical flow estimation fail. Furthermore, artificial neural networks used for optical flow-based frame interpolation rely on training with simulated videos, as described by Sun et al., “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. However, in the context of this application, large amounts of annotated data (e.g., for biological time delays) are unavailable.

[0053] Furthermore, an artificial neural network 400 can be selected to handle large shifts that imply temporal jitter and motion blur. In contrast, well-known methods, such as Baker et al., “A database and evaluation methodology for optical flow,” International Journal of Computer Vision 92.1, 2011, pp. 1–31, or Meyer et al., “Phase-based frame interpolation for video,” Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, require small object shifts between two frames.

[0054] In this embodiment, the artificial neural network 400 involves a feature shaping operation with channel attention, which replaces the optical flow calculation module of other methods. The feature shaping operation can correspond to a pixel shuffling operation. The artificial neural network can be configured to distribute information from the feature map of an image frame across multiple channels and extract motion information by participating in the channels for pixel-level synthesis of interpolated frames.

[0055] Specifically, the artificial neural network 400 can be configured to receive input image frames 402a and 402b, which are processed separately by a downwash operation 404. Downwash 404 is an operation that reorganizes image frames by pooling them with several switching variables to generate downsampled image frames, while upwash 416 corresponds to performing the inverse process. Downwash 404 separates image frames F∈R... HxW x C The spatial dimension is reduced by a factor s to obtain the image frame. After applying downwash 404, the downwashed image frames 406a and 406b are concatenated in the channel direction in box 407. The channel dimension is reduced by performing a convolution operation 408 followed by a ResGroups operation 410, which consists of 12 residual channel attention blocks. After another convolution 412, an intermediate image frame 414 is produced. Upwash 416 is applied to the intermediate image frame 414 to produce an interpolated microscopic image frame 418 inserted between image frames 402a and 402b. The artificial neural network can be configured as explained in Cho et al., “ChannelAttention Is All You Need for Video Frame Interpolation”, AAAI, 2020, 10663-10671, which is incorporated herein by reference in its entirety.

[0056] Figure 5 Methods are involved in training an artificial neural network (e.g., an artificial neural network configured as described above) for frame interpolation of image frames. In step 52 of the method, the artificial neural network is trained based on general, rather than domain-specific, image frames. For example, benchmark datasets such as Vimeo-90K (Xue et al., “Video enhancement with task-oriented flow International”, Journal of Computer Vision, 127, pp. 1106-1125, 2019) or SNU-FILM (Soomro et al., “UCF101: A dataset of 101 human actions classes from videos in the wild”, arXiv: 1212.0402, 2012) can be used. Step 52 involves training the first few layers of the artificial neural network based on simple image features (such as the edges and corners of blobs), which are general and not specific to any particular image domain. Moreover, the step of pre-training with non-microscopic videos with noise levels lower than those of the image frames in step 52 makes it easier for the network to learn these low-level features.

[0057] Method 50 further includes step 54, which involves fine-tuning a pre-trained artificial neural network by training it with domain-specific images, such as microscopic images or MRT images. The described transfer learning allows for the fine-tuning of an artificial neural network for a specific application, enabling the pre-trained network to be trained on the complex features of biological objects.

[0058] In this embodiment, the training data may include time-lapsed image frames acquired through different types of microscopes, such as bright-field microscopes, wide-field fluorescence microscopes, confocal microscopes, STED microscopes, and light-sheet microscopes. To train the artificial neural network 400 as described above for the field of microscope time-lapse, for example, 12,000 time-lapsed image frames may be used as training data.

[0059] Figure 6 An image processing apparatus is disclosed for improving the signal-to-noise ratio in a sequence of image frames. The image processing apparatus 60 includes a memory 62 for at least temporarily storing the sequence of image frames. The sequence of image frames can be received from a microscope into the memory 62 via a data communication link.

[0060] The image processing apparatus 60 also includes processing circuitry 64 configured to estimate a representative velocity of optical flow in the image frame sequence, as explained in further detail above. Processing circuitry 64 can be configured to determine an interpolation factor from the representative velocity of the optical flow, as explained above.

[0061] Components of the image processing apparatus 60 can also be configured to feed an image frame sequence and interpolation factors to a trained artificial neural network 400 for generating interpolated image frames based on the interpolation factors. The trained artificial neural network 400 can output an extended image frame sequence. The processing circuit 64 can also process the extended image frame sequence by applying a temporal correlation combination of the image frames to improve the signal-to-noise ratio. The processing circuit 64 can optionally be configured to apply a denoising algorithm to the enhanced image frame sequence. The enhanced image frame sequence can be output to a monitor in real time.

[0062] In a particular embodiment, the processing device 60 is provided in a surgical microscope equipped with a fluorescence microscope for imaging the emission of contrast agents from biological tissue. In this embodiment, the processing device 60 can operate continuously to provide a sequence of image frames with an improved signal-to-noise ratio, which can be output as a coverage map in the field of view of the surgical microscope.

[0063] Figure 7 An exemplary image frame is illustrated, which illustrates the improvement of this disclosure over the prior art in a simulation setting. Figure 7 The results of the simulated image frame in Figure 1B are illustrated. Plate 74 reproduces the results of the image frame in which the motion-aware rolling average is applied to plate 72 according to method 200. The parameters of the weighted rolling average applied as part of the motion-aware rolling average are the same as those used for plate 18 in Figure 1B, where σ = 4 and w = 8.

[0064] Figure 8The illustration shows the application of motion-aware rolling averages to consecutive fluorescence microscopy image frames in Figure 1A. The image frame in column 82 is part of a larger sequence of image frames on which the disclosed motion-aware rolling averages can be applied. Column 84 reproduces the image frames generated by applying the disclosed motion-aware rolling averages. (As shown from...) Figure 7 and 8 As is evident in the illustration, motion-aware rolling averages allow for an increase in signal-to-noise ratio without distorting the content of the image frame.

[0065] Therefore, the described method and system allow for a significant improvement in the signal-to-noise ratio (SNR) of noisy image frame sequences while avoiding artifacts arising from the inherent motion of the captured object. The described embodiments provide a low-latency solution for applying SNR improvements in real time.

[0066] Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method, where boxes or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of corresponding boxes, items, or features of the corresponding apparatus. Some or all of the method steps can be performed by (or using) hardware devices (e.g., processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, such devices can perform one or more of the most important method steps.

[0067] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. This implementation can be carried out using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, HDD, SSD, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, or flash memory) having electronically readable control signals stored thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding method to be executed. Therefore, the digital storage medium can be computer-readable.

[0068] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0069] Generally, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0070] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0071] In other words, therefore, an embodiment of the present invention is a computer program having program code that, when run on a computer, performs one of the methods described herein.

[0072] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) comprising a computer program stored thereon, which, when executed by a processor, is used to perform one of the methods described herein. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0073] Therefore, another embodiment of the invention represents a data stream or signal sequence for performing one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0074] Another embodiment includes a processing component, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0075] Another embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

[0076] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0077] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

Claims

1. A method for improving the signal-to-noise ratio of an image frame, the method comprising: Estimate the representative velocity of optical flow in an image frame sequence, wherein the image frame sequence includes the image frames, and the image frames are microscopic image frames; The interpolation factor is determined from the representative velocity of the optical flow; An extended image frame sequence is generated using a trained artificial neural network, wherein the extended image frame sequence includes multiple interpolated image frames, wherein each interpolated image frame is inserted between subsequent image frames in the image frame sequence, and wherein the number of interpolated image frames corresponds to the interpolation factor; and Calculate the temporal correlation combination of image frames from the extended image frame sequence to generate an output image frame sequence; The calculation of the time-related combination of image frames includes applying a weighted rolling average to the extended sequence of image frames; The application of the weighted rolling average includes determining the parameters for the weighted rolling average from the representative velocity.

2. The method of claim 1, wherein estimating the representative velocity of the optical flow comprises: Calculate the histogram of the optical flow between consecutive image frames in the image frame sequence; and Analyze the histogram to determine the representative velocity.

3. The method of claim 2, wherein analyzing the histogram to determine the representative velocity includes using the histogram to determine the representative velocity as a quantile of a predetermined threshold.

4. The method of claim 2, wherein the histogram is calculated based on estimated pixel-level optical flow.

5. The method of claim 4, wherein the estimation of the pixel-level optical flow is based on the Farnebäck algorithm.

6. The method of any one of claims 1-5, wherein the trained artificial neural network is configured to generate the interpolated image frame by applying a feature shaping operation with channel attention.

7. The method of claim 6, wherein the feature shaping operation includes a pixel shuffling operation.

8. The method of any one of claims 1-5, wherein the interpolation factor is a power of two and wherein the trained artificial neural network is configured to recursively generate interpolated image frames and add them to the image frame sequence, wherein the number of recursions corresponds to the power.

9. The method of any one of claims 1-5, wherein the parameters for the weighted rolling average are the window size for determining a set of image frames from the extended image frame sequence that contribute to the weighted rolling average, and the width for determining the weight of each image frame from that set that contributes to the weighted rolling average.

10. The method as described in any one of claims 1-5 further comprises applying a denoising algorithm to the output image frame sequence.

11. The method as described in any one of claims 1-5, wherein the artificial neural network has been trained in the following manner: The artificial neural network is pre-trained using domain-independent image frame sequences; and The artificial neural network is trained using domain-specific image frames.

12. An image processing apparatus for improving the signal-to-noise ratio of an image frame, performing the method of any one of the preceding claims, the image processing apparatus comprising: A memory configured to at least temporarily store a sequence of image frames, wherein the sequence of image frames includes the image frames, and the image frames are microscopic image frames; The processing circuit is configured to estimate the representative velocity of optical flow in the image frame sequence and to determine an interpolation factor from the representative velocity of optical flow. and A trained artificial neural network is used to generate an extended sequence of image frames, wherein the extended sequence of image frames includes a predetermined number of interpolated image frames, wherein each interpolated image frame is inserted between subsequent image frames in the sequence of image frames, and wherein the predetermined number corresponds to the interpolation factor. The processing circuitry is further configured to calculate a temporal correlation combination of image frames from the extended image frame sequence to generate an output image frame sequence; The processing circuitry is further configured to: A weighted rolling average is applied to the extended image frame sequence. The application of the weighted rolling average includes determining the parameters for the weighted rolling average from the representative velocity.

Images