Microscope video stream frame super-resolution method and system based on prediction assistance

Abstract

The invention provides a microscope video stream frame super-resolution method and system based on prediction assistance, and the method comprises the steps: carrying out the optical flow calculation of a historical frame, obtaining an initial optical flow, deducing the motion between a current frame and a future frame based on the initial optical flow, and obtaining a distorted optical flow and a propagation optical flow between the current frame and the future frame; calculating a motion vector field of pixels in the image of the current frame, and obtaining an estimated optical flow between the current frame and a future frame; obtaining a fusion optical flow field between the current frame and the future frame by combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources; and based on the fused optical flow field, propagating image information of the current frame to a future frame through back propagation, and generating a target super-resolution frame through frame synthesis. According to the method, the phenomena of loss, jamming or freezing of the microscope video frame caused by bandwidth limitation, network delay or reduction of image quality are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a prediction-assisted microscope video stream frame super-resolution method and system. Background Art

[0002] There are some technical challenges that we may encounter when using a microscope for video transmission. These challenges often include bandwidth limitations, which may result in insufficient data transmission speed, thus affecting the smoothness of the video. In addition, network latency is also a common problem, which refers to the time difference between the video source and the receiving end. This delay may cause the video to appear stuttered or incoherent when playing. In addition, the degradation of image quality is also a problem that needs attention. This may be due to the loss of details caused by the compression algorithm, or the loss of image frames due to poor network conditions. All of these factors combined may cause the loss, stuttering or freezing of video frames, which in turn affects the quality of microscope video transmission and user experience. Summary of the invention

[0003] In view of the defects in the prior art, the object of the present invention is to provide a prediction-assisted microscope video stream frame super-resolution method and system.

[0004] According to one aspect of the present invention, a prediction-assisted microscope video stream frame super-resolution method is provided, comprising:

[0005] Calculating the optical flow of the historical frames to obtain an initial optical flow, inferring the motion between the current frame and the future frame based on the initial optical flow, and obtaining the distorted optical flow and the propagated optical flow between the current frame and the future frame;

[0006] Calculate the motion vector field of pixels in the image of the current frame to obtain the estimated optical flow between the current frame and the future frame;

[0007] Combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field between a current frame and a future frame;

[0008] Based on the fused optical flow field, the image information of the current frame is propagated to the future frame through back propagation, and the target super-resolution frame is generated through frame synthesis.

[0009] Preferably, the optical flow calculation is performed on the historical frames to obtain the initial optical flow, and the motion between the current frame and the future frame is inferred based on the initial optical flow to obtain the distorted optical flow and the propagated optical flow between the current frame and the future frame, including:

[0010] Get n+1 frames of historical high-resolution images, and calculate the pixel intensity changes between adjacent frames to get the initial optical flow, including the initial optical flow from the current frame to the previous frame. And the initial optical flow from the previous frame to the current frame The two describe opposite directions, reflecting the relative motion of pixels between frames;

[0011] use right Perform reverse deformation to estimate the optical flow from time 0 to 1

[0012]

[0013] W represents the For input Perform reverse deformation;

[0014] Based on the fact that the motion between consecutive frames is uniform, the warped optical flow from 0 to t is derived

[0015]

[0016] Given two historical frames I -2 , I -1 and the current frame I0, respectively calculate two optical flow fields F 0→-2 、F 0→-1 , based on the fact that the motion between consecutive frames is the basis of uniform acceleration, we get:

[0017]

[0018] Where m∈[-1,-2], v0 and a represent velocity and acceleration respectively. By eliminating v0 and a, we can get the propagation optical flow field:

[0019]

[0020] Preferably, calculating the motion vector field of pixels in the image of the current frame to obtain the estimated optical flow between the current frame and the future frame includes:

[0021] Build a quality degradation model whose input is the high-resolution image I of the future frame t And scaling factor s, the output is a low-quality image of the current frame, specifically:

[0022]

[0023] Where D represents the degradation mapping function, s is the scaling factor, Represents the low-quality image of the current frame;

[0024] Use the optical flow estimation function E to estimate the estimated optical flow from time 0 to t:

[0025]

[0026] Among them, bic() represents the bicubic upsampling function and E represents the optical flow estimator.

[0027] Preferably, the estimated optical flow is tested by ablation experiments The optical flow effect under different scaling factors s∈{0,2,4,6,8,10,12,14} and the distorted optical flow and propagating optical flow The optical flow effect is compared with that of the original image, and the size of the scaling factor is inversely proportional to the image resolution and the amount of pixel information.

[0028] Preferably, combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field between the current frame and the future frame includes:

[0029] In the case of frame loss, the propagation optical flow and the distorted optical flow As a candidate optical flow;

[0030] The candidate optical flow is refined through U-Net to obtain the initial estimated optical flow field

[0031] U represents U-Net;

[0032] Learn the residual optical flow of U-Net and add it to the initial estimated optical flow field for refinement to obtain the final fused optical flow field

[0033]

[0034] Where σ(u) is the learned sampling offset of pixel u and r(u) is the learned residual optical flow.

[0035] Preferably, combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field comprises:

[0036] In the low-quality case, the propagated optical flow The distorted optical flow and estimate optical flow As a candidate optical flow;

[0037] The candidate optical flow is refined through U-Net to obtain the initial estimated optical flow field

[0038]

[0039] Learn the residual optical flow of U-Net and add it to the initial estimated optical flow field for refinement to obtain the final fused optical flow field

[0040]

[0041] Where σ(u) is the learned sampling offset of pixel u and r(u) is the learned residual optical flow.

[0042] Preferably, based on the fused optical flow field, propagating the image information of the current frame to the future frame by back propagation, and generating the target super-resolution frame by frame synthesis, comprises:

[0043] Align the current frame I0 to the future frame I t , get the initial frame of the target frame;

[0044] By back propagation, the image information is transferred from the current frame I0 to the initial frame of the target frame, and the optical flow field F from time t to 0 is obtained. t→0 (u):

[0045]

[0046] Where w(d) = e -d / σ is a weight function, N(u) represents the neighborhood of u, x and u represent the pixel positions in the current frame and the future frame respectively, and F 0→t (x) refers to the optical flow field in the process of propagating the pixel information of the current frame to the future frame through the optical flow field, F t→0 (u) refers to the optical flow field from time t to 0;

[0047] Based on the current frame, the optical flow field from time t to 0, and the low-resolution frame Use an image synthesis network with parameter set θ Synthesize the final target frame:

[0048]

[0049] Among them, W is based on the optical flow field F t→0 Function that unwarps the input I0.

[0050] According to a second aspect of the present invention, there is provided a prediction-assisted microscope video stream frame super-resolution system, comprising:

[0051] Optical flow prediction module: calculates the optical flow of historical frames to obtain the initial optical flow, infers the motion between the current frame and the future frame based on the initial optical flow, and obtains the distorted optical flow and propagation optical flow between the current frame and the future frame;

[0052] Optical flow estimation module: calculates the motion vector field of pixels in the image of the current frame and obtains the estimated optical flow between the current frame and the future frame;

[0053] Optical flow fusion module: combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field between the current frame and the future frame;

[0054] Frame synthesis module: Based on the fused optical flow field, the image information of the current frame is propagated to the future frame through back propagation, and the target super-resolution frame is generated through frame synthesis.

[0055] According to a third aspect of the present invention, there is provided a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the method described, or to run the system described, when executing the program.

[0056] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute the method described, or to run the system described.

[0057] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0058] In the embodiment of the present invention, the prediction-assisted microscope video stream frame super-resolution method, whose optical flow estimation improves the accuracy and robustness of future frame prediction: by introducing the optical flow propagation mechanism and combining the optical flow estimation of historical frames, complex non-uniform motion and acceleration changes are successfully modeled, thereby significantly improving the accuracy and robustness of future frame prediction, and overcoming the shortcomings of traditional methods in dynamic scenes.

[0059] The prediction-assisted microscope video stream frame super-resolution method in the embodiment of the present invention optimizes low-quality images by constructing a quality degradation model based on resolution degradation, effectively improves the accuracy of optical flow estimation in low-quality images, and significantly reduces the negative impact of noise and distortion on the optical flow estimation results, thereby achieving more accurate and robust optical flow estimation under low-quality image conditions.

[0060] In the embodiment of the present invention, the prediction-assisted microscope video stream frame super-resolution method, its multi-optical flow fusion improves the image synthesis accuracy: the designed multi-optical flow fusion, combining the propagation optical flow, the reverse warped optical flow and the estimated optical flow, and using the U-Net network optimization, significantly improves the accuracy and robustness of the optical flow estimation, effectively reduces the errors caused by motion blur and quality degradation, and ensures the generation of high-quality images in the case of low quality or lost frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0062] Figure 1 It is a flow chart of a prediction-assisted microscope video stream frame super-resolution method in one embodiment of the present invention;

[0063] Figure 2 It is a framework diagram of a prediction-assisted microscope video stream frame super-resolution method in one embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0065] like Figure 1 and Figure 2 As shown, an embodiment of a prediction-assisted microscope video stream frame super-resolution method is provided. Figure 1 As shown, the invented method includes four parts: optical flow prediction, optical flow estimation, optical flow fusion, and frame synthesis. Figure 1 The embodiment shown can be divided into the following steps:

[0066] The first step is optical flow prediction: optical flow calculation is performed on the historical frames to obtain the initial optical flow. Based on the initial optical flow, the motion between the current frame and the future frame is inferred to obtain the distorted optical flow and propagation optical flow between the current frame and the future frame.

[0067] The second step is optical flow estimation: calculate the motion vector field of the pixels in the image of the current frame to obtain the estimated optical flow between the current frame and the future frame;

[0068] The third step is optical flow fusion: combining the distorted optical flow, propagated optical flow and estimated optical flow from different sources to obtain the fused optical flow field between the current frame and the future frame;

[0069] The fourth step is frame synthesis: by fusing the optical flow field, the image information of the current frame is propagated to the future frame through back propagation to synthesize the target frame.

[0070] The above embodiment improves the accuracy and robustness of target frame prediction by introducing an optical flow propagation mechanism and combining the optical flow estimation of historical frames.

[0071] In a preferred embodiment of the present invention, the first step of optical flow prediction is implemented, the goal is to obtain the current frame I0 to the future frame I through optical flow propagation tThe specific steps of optical flow are as follows:

[0072] S1.1 Calculate the initial optical flow: Assuming that n+1 frames of historical high-resolution (HR) images have been received, the initial optical flow is obtained by calculating the pixel intensity changes between adjacent frame images.

[0073] The initial optical flow includes the initial optical flow from the current frame to the previous frame And the initial optical flow from the previous frame to the current frame The two describe opposite directions, reflecting the relative motion of pixels between frames;

[0074] S1.2 Optical flow deformation: Use right Perform the reverse warping to approximate the distorted optical flow from time 0 to t:

[0075]

[0076] The W in the formula represents the optical flow field For input Perform reverse deformation.

[0077] S1.3 Derive future optical flow: Derive the propagation optical flow from 0 to t by assuming that the motion between consecutive frames is uniform:

[0078]

[0079] Furthermore, in some preferred embodiments, higher-order information is used to perform more accurate motion modeling and improve optical flow prediction in complex scenes. It is assumed that there are acceleration changes in the motion between frames, so that complex non-uniform motion can be modeled. Through the optical flow propagation mechanism, the optical flow information of the previous frame is used to infer the motion trend from the current frame to the future frame, so as to obtain a more accurate optical flow prediction. This process can not only capture the changes in motion between frames, but also handle complex motion patterns in dynamic scenes, thereby improving the robustness of optical flow estimation. For example, for complex non-uniform motion, especially when the future frame It is lost, the linear assumption may not hold. Given two historical frames I -2 , I -1 and the current frame I0, respectively calculate two optical flow fields F 0→-2 , F 0→-1 , under the assumption of uniform acceleration, the following formula is used to model

[0080]

[0081] Where m∈[-1,-2], v0 and a represent velocity and acceleration respectively. By eliminating v0 and a, the optical flow field is propagated. It can be expressed as:

[0082]

[0083] The above embodiments describe that the optical flow can be propagated more accurately by considering the acceleration (the square term of time) and can capture complex non-uniform motions.

[0084] A large number of experiments have shown that the applicability of the optical flow propagation mechanism between high-quality and low-quality images has significant differences, such as clarity and noise: high-quality images have higher resolution and less noise, which makes the optical flow estimation more accurate and stable. Low-quality images have lower resolution and higher noise, resulting in loss of image details, and optical flow estimation will be more difficult, especially in areas with less texture or blurred edges. Detail preservation: high-quality images can retain more detail information, so motion information can be accurately captured during optical flow propagation. Low-quality images cause deviations in motion estimation due to information loss, especially in fast motion or complex backgrounds. Stability of motion estimation: high-quality images can provide more stable optical flow estimation because the motion patterns of their pixels are more consistent during the propagation process. On the contrary, low-quality images are easily disturbed by noise during the optical flow propagation process, resulting in unstable propagated optical flow and reduced estimation accuracy. Therefore, in a preferred embodiment, a quality degradation model is established when the optical flow is estimated in the second step of the embodiment. The core of this step is to calculate the motion vector field of pixels in the image to describe the motion of objects or scenes in the image sequence over time. In this embodiment, the impact of low-quality images on optical flow estimation is particularly considered, and an optimization method is proposed. Specifically, optical flow estimation can be performed using the following steps:

[0085] S2.1 Establish a quality degradation model whose input is the high-resolution original image I of the future frame t and scaling factor s, the output is a low-quality image of the current frame

[0086]

[0087] Where D represents the degradation mapping function and s is the scaling factor.

[0088] The role of the quality degradation model is to simulate the degradation process of image quality under conditions such as downsampling, blur, and noise. It is mainly used to study the generation and processing of low-quality images and provide a reference model for tasks such as image super-resolution and denoising. By simulating different quality degradations, it helps to evaluate the performance and robustness when processing images of different qualities.

[0089] S2.2 Estimate the optical flow field: Use the optical flow estimation function E to estimate the estimated optical flow from time 0 to t:

[0090]

[0091] Where bic(·) represents the bicubic upsampling function and E is the optical flow estimator (PWC-Net).

[0092] Furthermore, in a preferred embodiment, the accuracy of optical flow prediction and estimation is compared, and the optical flow prediction effect under different scaling factors s∈{0,2,4,6,8,10,12,14} is tested through ablation experiments.

[0093] In the above embodiment, a quality degradation model based on resolution degradation is introduced for low-quality images. This model studies the impact of image quality degradation on the accuracy of optical flow estimation by setting different scaling factors. Low-quality images usually lose more detail information. Different degradation models can be used to adjust the processing method of low-quality images, so that the optical flow propagation mechanism can more robustly cope with the challenges brought by quality degradation. In the above embodiment, adaptive propagation can be performed: when processing low-quality images, the propagation process can be adjusted adaptively. For example, when the image quality is low, the optical flow information from the historical frame can be automatically increased to compensate for the information loss of the current frame. In the above embodiment, estimation optimization is improved: by analyzing the impact of different scaling factors (representing different quality levels), it is possible to identify which areas in the low-quality image have a greater impact on the optical flow estimation (such as blurred edges, missing textures, etc.). In these areas, the accuracy of optical flow estimation can be enhanced, and specific optimization can be performed on areas with poor quality. In the above embodiment, multi-frame fusion and super-resolution processing can also be performed: for low-quality images, multi-frame fusion and super-resolution processing techniques can also be used to make up for the lack of image quality. For example, lost details can be supplemented by historical frame information to improve the accuracy of motion estimation in low-quality images.

[0094] In a preferred embodiment of the present invention, the third step of fusion is implemented to obtain a more accurate and stable optical flow field, including the following steps:

[0095] S3.1 Generate candidate optical flow:

[0096] In the case of frame loss, obtain the propagation optical flow and distorted optical flow

[0097] In the low-quality case, in addition to propagating optical flow and deformation optical flow In addition, the estimated optical flow is obtained

[0098] S3.2 Optical flow refinement:

[0099] The candidate optical flow is refined through U-Net. In the case of frame loss, the input is the propagation optical flow, the distortion optical flow and the image I0:

[0100]

[0101] The optical flow fusion process in low quality case is as follows:

[0102]

[0103] S3.3 Learning residual optical flow:

[0104] The residual optical flow is learned through U-Net and added to the initial estimated optical flow for refinement:

[0105]

[0106] Where σ(u) is the learned sampling offset of pixel u and r(u) is the learned residual optical flow.

[0107] The multi-optical flow fusion mechanism in the above embodiment combines the propagation optical flow, the reverse twisted optical flow and the estimated optical flow. Through the optimization of the U-Net network, the accuracy and robustness of the optical flow estimation are effectively improved. The design of multi-optical flow fusion can use the high-frequency detail information in the input image to refine the optical flow estimation result, thereby effectively reducing the error caused by motion blur or quality degradation.

[0108] In one embodiment of the present invention, the fourth step of frame synthesis is implemented to propagate the image information of the reference frame to the target frame through the estimated optical flow field. The optical flow information is used to generate the target frame at the current moment, and in the case of frame loss, the lost frame is synthesized. Alternatively, other required target frames are generated, such as improving the quality of the target frame.

[0109] S4.1 Reference frame alignment:

[0110] Align the input reference frame I0 to the future frame I t , get the initial frame of the future frame;

[0111] S4.2 reverse mapping, optical flow projection:

[0112] Through back propagation, the image information is transferred from the current frame I0 to the initial frame of the target frame, and the optical flow field F from time t to 0 is obtained. t→0 (u):

[0113]

[0114] Where w(d) = e -d / σ is a weight function, N(u) represents the neighborhood of u, x and u represent the pixel positions in the current frame and the future frame respectively, and F 0→t (x) refers to the optical flow field in the process of propagating the pixel information of the current frame to the future frame through the optical flow field, F t→0 (u) refers to the optical flow field from time t to 0. Note that Ft→0 It is used to estimate pixel movement and participate in the synthesis of future frames.

[0115] S4.3 Synthesized target frame:

[0116] Based on the current frame, the optical flow field from time t to 0, and the low-resolution frame Use a U-net based image synthesis network with a trainable parameter set θ Synthesize the final target frame:

[0117]

[0118] Among them, W is based on the optical flow field F t→0 Function that unwarps the input I0.

[0119] In the above frame synthesis stage, the back-projection technology is used to guide the image synthesis process through the optical flow information. This process can generate accurate target images in the case of low quality or missing frames, ensuring the restoration of image details. The back-projection technology can effectively use the optical flow information to reconstruct the target frame from multiple angles, ensuring the accuracy and robustness of the prediction results.

[0120] The aforementioned embodiments of the present invention can effectively solve the problem of restoring lost frames in a video. The main method is to combine the spatial and temporal information and image features of the historical frames to predict the content of the lost frames. At the same time, the previous high-resolution frames, packaging streams and propagation streams are fused to generate a more accurate future optical flow, thereby reducing the impact of lost frames on video continuity and ensuring smooth playback of the video stream. The above embodiments can also enhance low-quality frames based on received high-resolution microscope images, mainly by fusing the information of low-quality frames with received high-resolution frames, restoring and enhancing image details, thereby generating a clearer microscope image.

[0121] In summary, the above embodiments predict or restore future frames by making full use of previous high-resolution frame information in the case of lost frames or low-quality frames. By fusing estimated optical flow, propagated optical flow and predicted optical flow, the network can accurately estimate future optical flow and maintain high-quality video frame synthesis even when the current optical flow is lost or the frame resolution is low.

[0122] Based on the same inventive concept, other embodiments of the present invention further provide a prediction-assisted microscope video stream frame super-resolution system, comprising:

[0123] Optical flow prediction module: calculates the optical flow of historical frames to obtain the initial optical flow, infers the motion between the current frame and the future frame based on the initial optical flow, and obtains the distorted optical flow and propagation optical flow between the current frame and the future frame;

[0124] Optical flow estimation module: calculates the motion vector field of pixels in the image of the current frame and obtains the estimated optical flow between the current frame and the future frame;

[0125] Optical flow fusion module: combines the distorted optical flow, propagated optical flow and estimated optical flow from different sources to obtain the fused optical flow field between the current frame and the future frame;

[0126] Frame synthesis module: Based on the fused optical flow field, the image information of the current frame is propagated to the future frame through back propagation, and the target super-resolution frame is generated through frame synthesis.

[0127] The various modules / units in the above examples of the present invention may specifically refer to the implementation techniques of the corresponding steps of the prediction-assisted microscope video stream frame super-resolution method in the above embodiments, which will not be described in detail here.

[0128] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above-mentioned method or to run the above-mentioned system when executing the program.

[0129] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.

[0130] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0131] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0132] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method, or run the above system.

[0133] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A prediction-assisted microscopy video stream frame super-resolution method, characterized in that: include: Calculating the optical flow of the historical frames to obtain an initial optical flow, inferring the motion between the current frame and the future frame based on the initial optical flow, and obtaining the distorted optical flow and the propagated optical flow between the current frame and the future frame; Calculate the motion vector field of pixels in the image of the current frame to obtain the estimated optical flow between the current frame and the future frame; Combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field between a current frame and a future frame; Based on the fused optical flow field, the image information of the current frame is propagated to the future frame through back propagation, and the target super-resolution frame is generated through frame synthesis.

2. The prediction-assisted microscopy video stream frame super-resolution method according to claim 1, characterized in that: The optical flow calculation is performed on the historical frames to obtain the initial optical flow, and the motion between the current frame and the future frame is inferred based on the initial optical flow to obtain the distortion optical flow and the propagation optical flow between the current frame and the future frame, including: Get n+1 frames of historical high-resolution images, and calculate the pixel intensity changes between adjacent frames to get the initial optical flow, including the initial optical flow from the current frame to the previous frame. And the initial optical flow from the previous frame to the current frame The two describe opposite directions, reflecting the relative motion of pixels between frames; use right Perform reverse deformation to estimate the optical flow from time 0 to 1 W represents the For input Perform reverse deformation; Based on the fact that the motion between consecutive frames is uniform, the warped optical flow from 0 to t is derived Given two historical frames I -2 , I -1 and the current frame I0, respectively calculate two optical flow fields F 0→-2 、F 0→-1 , based on the fact that the motion between consecutive frames is the basis of uniform acceleration, we get: Where m∈[-1,-2], v0 and a represent velocity and acceleration respectively. By eliminating v0 and a, we can get the propagation optical flow field:

3. The prediction-assisted microscopy video stream frame super-resolution method according to claim 1, characterized in that: The step of calculating the motion vector field of pixels in the image of the current frame to obtain the estimated optical flow between the current frame and the future frame includes: Build a quality degradation model whose input is the high-resolution image I of the future frame t And scaling factor s, the output is a low-quality image of the current frame, specifically: Where D represents the degradation mapping function, s is the scaling factor, Represents the low-quality image of the current frame; Use the optical flow estimation function E to estimate the estimated optical flow from time 0 to t: Among them, bic() represents the bicubic upsampling function and E represents the optical flow estimator.

4. The prediction-assisted microscopy video stream frame super-resolution method according to claim 3, characterized in that: The estimated optical flow is tested by ablation experiments. The optical flow effect under different scaling factors s∈{0,2,4,6,8,10,12,14} and the distorted optical flow and propagating optical flow The optical flow effect is compared with that of the original image, and the size of the scaling factor is inversely proportional to the image resolution and the amount of pixel information.

5. The prediction-assisted microscopy video stream frame super-resolution method according to claim 4, characterized in that: The combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field between a current frame and a future frame includes: In the case of frame loss, the propagation optical flow and the distorted optical flow As a candidate optical flow; The candidate optical flow is refined through U-Net to obtain the initial estimated optical flow field U display U-Net; Learn the residual optical flow of U-Net and add it to the initial estimated optical flow field for refinement to obtain the final fused optical flow field Where σ(u) is the learned sampling offset of pixel u and r(u) is the learned residual optical flow.

6. The prediction-assisted microscopy video stream frame super-resolution method according to claim 5, characterized in that: The combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field includes: In the low-quality case, the propagated optical flow The distorted optical flow and estimate optical flow As a candidate optical flow; The candidate optical flow is refined through U-Net to obtain the initial estimated optical flow field Learn the residual optical flow of U-Net and add it to the initial estimated optical flow field for refinement to obtain the final fused optical flow field Where σ(u) is the learned sampling offset of pixel u and r(u) is the learned residual optical flow.

7. The prediction-assisted microscopy video stream frame super-resolution method according to claim 1, characterized in that: Based on the fused optical flow field, the image information of the current frame is propagated to the future frame by back propagation, and the target super-resolution frame is generated by frame synthesis, including: Align the current frame I0 to the future frame I t , get the initial frame of the target frame; By back propagation, the image information is transferred from the current frame I0 to the initial frame of the target frame, and the optical flow field F from time t to 0 is obtained. t→0 (u): Where w(d) = e -d / σ is a weight function, N(u) represents the neighborhood of u, x and u represent the pixel positions in the current frame and the future frame respectively, and F 0→t (x) refers to the optical flow field in the process of propagating the pixel information of the current frame to the future frame through the optical flow field, F t→0 (u) refers to the optical flow field from time t to 0; Based on the current frame, the optical flow field from time t to 0, and the low-resolution frame Use an image synthesis network with parameter set θ Synthesize the final target frame: Among them, W is based on the optical flow field F t→0 Function that unwarps the input I0.

8. A prediction-assisted microscopy video stream frame super-resolution system, characterized in that: include: Optical flow prediction module: calculates the optical flow of historical frames to obtain the initial optical flow, infers the motion between the current frame and the future frame based on the initial optical flow, and obtains the distorted optical flow and propagation optical flow between the current frame and the future frame; Optical flow estimation module: calculates the motion vector field of pixels in the image of the current frame and obtains the estimated optical flow between the current frame and the future frame; Optical flow fusion module: combining the distorted optical flow, the propagated optical flow and the estimated optical flow from different sources to obtain a fused optical flow field between the current frame and the future frame; Frame synthesis module: Based on the fused optical flow field, the image information of the current frame is propagated to the future frame through back propagation, and the target super-resolution frame is generated through frame synthesis.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images