Video frame interpolation method, device and equipment based on motion estimation, and storage medium

Abstract

This invention discloses a video frame interpolation method, apparatus, device, and storage medium based on motion estimation. The method generates a clear and accurate target intermediate frame image through three steps: texture feature extraction from the input image, motion feature extraction, and joint optimization of intermediate optical flow and intermediate frame image. Specifically, a pyramid encoder extracts multi-scale texture features from the image, ensuring that the extracted features possess both global information and local texture details. Motion feature extraction not only provides a clear correspondence between adjacent frames, but the long-distance motion optical flow can also capture the alternating constriction and dilation of coronary arteries. The encoder employs a joint optimization approach using intermediate optical flow and target value features to obtain the final interpolation result. Ultimately, it can generate clear, high-frame-rate sequences of coronary artery and other vascular motion that are coherent and accurate.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video frame interpolation method, apparatus, device, and storage medium based on nonlinear motion estimation. Background Technology

[0002] Coronary angiography is crucial for the diagnosis and treatment of cardiovascular diseases. The average adult heart rate is 50 to 100 beats per minute, and the cardiac cycle time of coronary artery motion is approximately 0.6 to 1.2 seconds. High frame rate coronary angiography sequences are essential for capturing various coronary artery morphologies under heartbeat. However, acquiring high frame rate sequences requires increased frequency and intensity of X-ray exposure, thereby increasing the associated health risks for both patients and physicians. To obtain high frame rate coronary angiography sequences with low X-ray radiation exposure, video interpolation is a critical step. Video interpolation algorithms synthesize target intermediate frames between adjacent input frames to produce smooth, continuous motion of objects in the sequence. Currently, video interpolation algorithms can be broadly categorized into two types: one based on motion estimation and the other based on interpolation kernels.

[0003] The first type of method first estimates the inter-frame motion of the input frame and then estimates the intermediate optical flow to spatially align the target intermediate frame and the input frame, thereby synthesizing the target interpolated frame. The second type of method estimates the interpolation kernel to weight and fuse the pixels of the input frame, thereby synthesizing the target interpolated frame. Kernel-based methods lack motion modeling and have relatively high computational costs, while motion-based methods exhibit superior performance.

[0004] Optical flow estimation networks have further accelerated the development of motion estimation-based video interpolation methods. Optical flow is a dense field of displacement vectors for each pixel, providing a clear correspondence between adjacent input frames. While motion between input frames can be obtained using mature optical flow networks, estimating the intermediate optical flow of the target intermediate frame—the image to be solved—presents a significant challenge for motion estimation-based video interpolation methods.

[0005] However, in existing motion-estimation-based video interpolation methods, the estimation of intermediate optical flow relies on the assumption that objects move in a straight line at a constant velocity, which is unsuitable for sequences characterized by complex nonlinear motion. The primary motion in coronary angiography sequences is the periodic reciprocating motion of the coronary arteries, which undergo alternating systolic and diastolic phases. During this alternating systolic and diastolic motion, the coronary artery location in the target frame is not located between the coronary artery locations in adjacent input frames.

[0006] Therefore, how to provide a video interpolation method that can interpolate low frame rate sequences to generate high frame rate sequences for nonlinear motion is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of the above problems, the present invention provides a video frame interpolation method, apparatus, device, and storage medium based on motion estimation to overcome or at least partially solve the above problems. This method uses a nonlinear motion estimation network for video frame interpolation, interpolating low frame rate sequences to generate high frame rate sequences without additional radiative exposure.

[0008] This invention provides the following solution:

[0009] A video frame interpolation method based on motion estimation includes:

[0010] Acquire input image pairs, which include two adjacent frames in a low frame rate angiography sequence;

[0011] A texture feature extraction module is used to extract multi-scale texture features from the input image. Combined with the multi-scale texture features, a motion extraction module is used to obtain inter-frame motion features. The motion features include bidirectional optical flow between adjacent frames and long-distance motion information obtained by estimation.

[0012] The multi-scale texture features and the motion features are concatenated by channels to obtain the feature extraction results of multiple frames of input images.

[0013] The intermediate optical flow is estimated using the feature extraction results, and the intermediate frame image is obtained by deformation using the intermediate optical flow. The intermediate frame features are obtained by convolution and image transformation operations on the intermediate frame image and the intermediate optical flow.

[0014] The target intermediate optical flow is obtained by jointly optimizing the intermediate optical flow and the intermediate frame features using an encoder, and the target intermediate optical flow is deformed to obtain the target intermediate frame image;

[0015] The target intermediate frame image is inserted between the input image pairs to perform a pin insertion operation.

[0016] Preferably, the texture feature extraction module includes a pyramid texture feature encoding module.

[0017] Preferably, the texture feature extraction module and the motion feature extraction module are represented by the following formula:

[0018]

[0019] In the formula, f encode () represents the pyramid texture feature encoding module, f motion () represents the motion extraction module, and k represents the pyramid level. Represents multi-scale texture features, (I -1 (I0, I1, I2) represents an image pair, [F0→1 ,F 1→0 [F] indicates bidirectional optical flow. -1→0 ,F 1→2 [] indicates long-distance movement information.

[0020] Preferably, the step of jointly optimizing the intermediate optical flow and the intermediate frame features using an encoder to obtain the target intermediate optical flow includes:

[0021] The underlying encoder estimates the initial intermediate optical flow based on the feature extraction results;

[0022] The initial intermediate optical flow shape is transformed into an initial intermediate frame image, and the initial intermediate frame features are generated by convolutional transformation of the initial intermediate optical flow and the initial intermediate frame image.

[0023] The initial intermediate frame features are input into the next layer encoder for optimization to obtain a new intermediate optical flow, until the top layer is optimized to obtain the target intermediate optical flow.

[0024] Preferably, the encoder joint optimization process is represented by the following formula:

[0025]

[0026] Among them, f inter () represents the intermediate optical flow estimation encoder, warp represents the image transformation operation based on optical flow, and Conv represents the convolution operation. Indicates the initial intermediate optical flow. This represents the new intermediate optical flow.

[0027] A video frame interpolation apparatus for performing the above-described motion estimation-based video frame interpolation method, the apparatus comprising:

[0028] An image pair acquisition unit is used to acquire input image pairs, the input image pairs including two adjacent frames in a low frame rate angiography sequence;

[0029] The feature extraction unit is used to extract multi-scale texture features from the input image using a texture feature extraction module, and combine the multi-scale texture features with a motion extraction module to obtain inter-frame motion features. The motion features include bidirectional optical flow and long-distance motion information between adjacent frames obtained by estimation.

[0030] The feature extraction result calculation unit is used to perform channel concatenation of the multi-scale texture features and the motion features to obtain the feature extraction results of the multi-frame input image;

[0031] The intermediate frame feature acquisition unit is used to estimate the intermediate optical flow using the feature extraction result, perform deformation using the intermediate optical flow to obtain the intermediate frame image, and obtain the intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow.

[0032] The joint optimization unit is used to jointly optimize the intermediate optical flow and the intermediate frame features using the encoder to obtain the target intermediate optical flow, and to deform the target intermediate optical flow to obtain the target intermediate frame image.

[0033] The pin insertion unit is used to insert the target intermediate frame image between the input image pairs to perform a pin insertion operation.

[0034] A video frame interpolation device, the device comprising a processor and a memory:

[0035] The memory is used to store program code and transmit the program code to the processor;

[0036] The processor is used to execute the above-described motion estimation-based video frame interpolation method according to the instructions in the program code.

[0037] A computer-readable storage medium for storing program code for performing the above-described motion estimation-based video frame interpolation method.

[0038] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0039] This application provides a video frame interpolation method, apparatus, device, and storage medium based on motion estimation. The method generates a clear and accurate target intermediate frame image through three steps: texture feature extraction from the input image, motion feature extraction, and joint optimization of intermediate optical flow and intermediate frame image. Specifically, a pyramid encoder extracts multi-scale texture features from the image, ensuring that the extracted features possess both global information and local texture details. Motion feature extraction not only provides a clear correspondence between adjacent frames, but the long-distance motion optical flow can also capture the alternating constriction and dilation of coronary arteries. The encoder employs a joint optimization method using intermediate optical flow and target value features to obtain the final interpolation result. Ultimately, it can generate a clear, high-frame-rate sequence of coronary artery and other vascular motion that is coherent and accurate.

[0040] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0042] Figure 1 This is a flowchart of a video frame interpolation method based on motion estimation provided in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the use of coronary angiography sequence interpolation as provided in this embodiment of the invention.

[0044] Figure 3 This is a flowchart of the image texture feature and motion feature extraction process provided in the embodiments of the present invention;

[0045] Figure 4 This is a flowchart of encoder joint optimization of intermediate optical flow and image generation provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of a video frame interpolation device provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of a video frame interpolation device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0049] See Figure 1 This invention provides a video frame interpolation method based on motion estimation, such as... Figure 1 As shown, the method may include:

[0050] S101: Obtain an input image pair, the input image pair including two adjacent frames in a low frame rate angiography sequence;

[0051] S102: A texture feature extraction module is used to extract multi-scale texture features from the input image. The motion extraction module is then used to obtain inter-frame motion features in combination with the multi-scale texture features. The motion features include bidirectional optical flow and long-distance motion information between adjacent frames obtained through estimation. In specific implementation, the texture feature extraction module in this application embodiment may include a pyramid texture feature encoding module.

[0052] The texture feature extraction module and the motion feature extraction module are represented by the following formula:

[0053]

[0054] In the formula, f encode () represents the pyramid texture feature encoding module, f motion () represents the motion extraction module, and k represents the pyramid level. Represents multi-scale texture features, (I -1 (I0, I1, I2) represents an image pair, [F 0→1 ,F 1→0 [F] indicates bidirectional optical flow. -1→0 ,F 1→2 [] indicates long-distance movement information.

[0055] S103: The multi-scale texture features and the motion features are concatenated by channels to obtain the feature extraction results of multiple frames of input images;

[0056] S104: The intermediate optical flow is estimated using the feature extraction result, the intermediate optical flow is used to perform deformation to obtain the intermediate frame image, and the intermediate frame features are obtained by convolution and image transformation operations on the intermediate frame image and the intermediate optical flow.

[0057] S105: The encoder is used to jointly optimize the intermediate optical flow and the intermediate frame features to obtain the target intermediate optical flow, and the target intermediate optical flow is deformed to obtain the target intermediate frame image; In specific implementation, the embodiments of this application can provide the underlying encoder to estimate the initial intermediate optical flow based on the feature extraction results;

[0058] The initial intermediate optical flow shape is transformed into an initial intermediate frame image, and the initial intermediate frame features are generated by convolutional transformation of the initial intermediate optical flow and the initial intermediate frame image.

[0059] The initial intermediate frame features are input into the next layer encoder for optimization to obtain a new intermediate optical flow, until the top layer is optimized to obtain the target intermediate optical flow.

[0060] The encoder joint optimization process is represented by the following equation:

[0061]

[0062] Among them, f inter () represents the intermediate optical flow estimation encoder, warp represents the image transformation operation based on optical flow, and Conv represents the convolution operation. Indicates the initial intermediate optical flow. This represents the new intermediate optical flow.

[0063] S106: Insert the target intermediate frame image between the input image pairs to perform a pin insertion operation.

[0064] The motion estimation-based video interpolation method provided in this application provides a method for obtaining high frame rate coronary angiography sequences under low X-ray radiation exposure conditions. Existing video interpolation algorithms typically assume that objects move in a straight line at a constant speed. Considering the periodicity and reciprocating nature of coronary artery motion, this application proposes a motion estimation-based video interpolation method based on a nonlinear motion estimation network to interpolate low frame rate sequences and generate high frame rate sequences without additional radiation exposure.

[0065] This method first extracts pyramid texture features from the input frames and then extracts motion features between the input frames to establish explicit correspondences. The features are then input into the decoder to estimate intermediate optical flow and generate the target intermediate frame image, employing a bottom-up joint optimization approach to obtain the final interpolation result. The entire network model is trained under joint supervision of a weighted self-supervised optical flow loss and a target frame synthesis loss.

[0066] The following section uses the application of coronary angiography sequence interpolation as an example to describe in detail the motion estimation-based video interpolation method provided in this application. The pyramid levels k = 0, 1, 2, 3.

[0067] The purpose of the method provided in this application is to acquire raw low-frame-rate coronary angiography sequences under low X-ray radiation exposure conditions, and to obtain high-frame-rate coronary angiography sequences through video frame interpolation algorithms, thereby reducing the frequency and intensity of X-ray use.

[0068] To generate accurate and clear intermediate frames, this application provides a nonlinear motion estimation network model, which is specifically implemented in three stages: texture feature extraction, motion feature extraction, and joint optimization of intermediate optical flow and intermediate frame image.

[0069] In the process of image texture feature extraction, the embodiments of this application use a feature pyramid encoder to extract multi-scale image texture features, which can effectively reduce the loss of texture information during downsampling and retain local texture details while capturing global image information.

[0070] During motion feature extraction, this embodiment not only estimates the bidirectional optical flow between adjacent frames but also estimates the long-distance motion optical flow to capture the alternating reciprocating motion of coronary artery contraction and relaxation. These motion features provide more accurate coronary artery locations for the generation of subsequent target intermediate frames.

[0071] In the joint optimization of intermediate optical flow and intermediate frame images, this embodiment designs a self-supervised optical flow loss to supervise the learning of motion representations between input frames, enhancing its relevance to the frame interpolation task. This embodiment employs a bottom-up joint optimization approach to obtain the final interpolation result. Joint optimization of intermediate optical flow and target frame features can further improve the accuracy of video frame interpolation.

[0072] As attached Figure 2 This is a flowchart of the motion estimation-based video interpolation method for coronary angiography sequences provided in this application. The model input is an image pair [I -1 The texture feature extraction module extracts multi-scale texture feature representations from the input image [I0, I1, I2]. The extracted features possess both global information and local texture details. The motion extraction module acquires inter-frame correlation features to establish explicit correspondences between images. The motion extraction module not only estimates the bidirectional optical flow [F] between adjacent frames... 0→1 ,F 1→0 It also estimates long-distance motion information [F] -1→0 ,F 1→2 This is used to capture the alternating systolic and diastolic motions of the coronary arteries. The encoder module jointly optimizes the intermediate optical flow estimation and the image features of the target intermediate frame to obtain the final intermediate frame interpolation result. By inputting a low frame rate angiography sequence into the entire video frame interpolation model, a high frame rate angiography sequence can be obtained.

[0073] Combined with appendix Figure 3 The implementation steps of feature extraction provided in this application embodiment are described in detail below. First, the texture features of each frame of the image are obtained through a pyramid feature encoding module. Then, the motion features between frames are obtained by estimating the bidirectional optical flow of adjacent frames. Furthermore, long-distance motion information is obtained by estimating the optical flow of interval frames to capture the alternating vasoconstriction and vasodilation of the coronary arteries. The formula is expressed as follows:

[0074]

[0075] Among them, f encode () represents the pyramid texture feature encoding module, f motion () represents the motion extraction module.

[0076] k = 0, 1, 2, 3 represents the pyramid level. Texture features With motion characteristics

[0077] [F 0→1 ,F 1→0 ,F -1→0 ,F 1→2 Channel concatenation is performed to obtain the feature extraction results of the final multi-frame input images.

[0078] Combined with appendix Figure 4 This paper describes in detail the implementation steps of the encoder jointly optimizing intermediate optical flow estimation and target image generation according to the present invention. The intermediate optical flow and the target intermediate frame image are conjugates. Jointly optimizing the generation of intermediate optical flow and target frame features in a bottom-up manner can significantly enhance the final intermediate frame interpolation result. The underlying encoder estimates the intermediate optical flow. This optical flow can be used to deform the input frame to obtain a coarse target frame, and convolution generates target frame features. The input to the next layer encoder is used to optimize and obtain a new intermediate optical flow. The bottom-up optimization process of the pyramid is expressed by the following formula:

[0079]

[0080] Among them, f inter () represents the intermediate optical flow estimation encoder, and warp represents the image operation based on optical flow transformation. The pyramid structure is optimized to the top layer to obtain the final intermediate optical flow [F]. 0.5→1 ,F 0.5→0 The deformation yields the final interpolation result.

[0081] In summary, the motion estimation-based video interpolation method provided in this application generates clear and accurate target intermediate frame images through three stages: texture feature extraction, motion feature extraction, and joint optimization of intermediate optical flow and intermediate frame images. Specifically, the pyramid encoder extracts multi-scale texture features from the image, ensuring that the extracted features possess both global information and local texture details. Motion feature extraction not only provides a clear correspondence between adjacent frames, but the long-distance motion optical flow can also capture the alternating constriction and dilation of coronary arteries. The encoder employs a joint optimization approach using intermediate optical flow and target value features to obtain the final interpolation result. Ultimately, this method generates clear, high-frame-rate sequences of coronary artery and other vascular motion that are coherent and accurate.

[0082] See Figure 5 This application embodiment can also provide a video frame interpolation device, such as... Figure 5 As shown, the device may include:

[0083] Image pair acquisition unit 501 is used to acquire input image pairs, the input image pairs including two adjacent frames of images in a low frame rate angiography sequence;

[0084] The feature extraction unit 502 is used to extract multi-scale texture features from the input image using a texture feature extraction module, and combine the multi-scale texture features with a motion extraction module to obtain inter-frame motion features. The motion features include bidirectional optical flow and long-distance motion information between adjacent frames obtained by estimation.

[0085] The feature extraction result calculation unit 503 is used to perform channel concatenation of the multi-scale texture features and the motion features to obtain the feature extraction results of the multi-frame input image.

[0086] The intermediate frame feature acquisition unit 504 is used to estimate the intermediate optical flow using the feature extraction result, use the intermediate optical flow to perform deformation to obtain the intermediate frame image, and obtain the intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow.

[0087] The joint optimization unit 505 is used to jointly optimize the intermediate optical flow and the intermediate frame features using the encoder to obtain the target intermediate optical flow, and to deform the target intermediate optical flow to obtain the target intermediate frame image.

[0088] The pin insertion operation unit 506 is used to insert the target intermediate frame image between the input image pairs to perform a pin insertion operation.

[0089] This application embodiment can also provide a video frame interpolation device, the device including a processor and a memory:

[0090] The memory is used to store program code and transmit the program code to the processor;

[0091] The processor is used to execute the steps of the motion estimation-based video frame interpolation method described above according to the instructions in the program code.

[0092] like Figure 6 As shown in the figure, a video frame interpolation device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0093] In the embodiments of this application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0094] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in embodiments of the motion estimation-based video frame interpolation method.

[0095] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0096] Acquire input image pairs, which include two adjacent frames in a low frame rate angiography sequence;

[0097] A texture feature extraction module is used to extract multi-scale texture features from the input image. Combined with the multi-scale texture features, a motion extraction module is used to obtain inter-frame motion features. The motion features include bidirectional optical flow between adjacent frames and long-distance motion information obtained by estimation.

[0098] The multi-scale texture features and the motion features are concatenated by channels to obtain the feature extraction results of multiple frames of input images.

[0099] The intermediate optical flow is estimated using the feature extraction results, and the intermediate frame image is obtained by deformation using the intermediate optical flow. The intermediate frame features are obtained by convolution and image transformation operations on the intermediate frame image and the intermediate optical flow.

[0100] The target intermediate optical flow is obtained by jointly optimizing the intermediate optical flow and the intermediate frame features using an encoder, and the target intermediate optical flow is deformed to obtain the target intermediate frame image;

[0101] The target intermediate frame image is inserted between the input image pairs to perform a pin insertion operation.

[0102] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0103] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0104] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0105] Of course, it should be noted that, Figure 6 The structure shown does not constitute a limitation on the video frame interpolation device in the embodiments of this application. In practical applications, the video frame interpolation device may include devices such as... Figure 6 More or fewer components as shown, or combinations of certain components.

[0106] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described motion estimation-based video frame interpolation method.

[0107] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0109] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A video frame interpolation method based on motion estimation, characterized in that, include: Acquire input image pairs, which include two adjacent frames in a low frame rate angiography sequence; A texture feature extraction module is used to extract multi-scale texture features from the input image. Combined with the multi-scale texture features, a motion extraction module is used to obtain inter-frame motion features. The motion features include bidirectional optical flow between adjacent frames and long-distance motion information obtained by estimation. The multi-scale texture features and the motion features are concatenated by channels to obtain the feature extraction results of multiple frames of input images. The intermediate optical flow is estimated using the feature extraction results, and the intermediate frame image is obtained by deformation using the intermediate optical flow. The intermediate frame features are obtained by convolution and image transformation operations on the intermediate frame image and the intermediate optical flow. The target intermediate optical flow is obtained by jointly optimizing the intermediate optical flow and the intermediate frame features using an encoder, and the target intermediate optical flow is deformed to obtain the target intermediate frame image; The target intermediate frame image is inserted between the input image pairs to perform a pin insertion operation.

2. The video frame interpolation method based on motion estimation according to claim 1, characterized in that, The texture feature extraction module includes a pyramid texture feature encoding module.

3. The video frame interpolation method based on motion estimation according to claim 2, characterized in that, The texture feature extraction module and the motion feature extraction module are represented by the following formula: In the formula, f encode () represents the pyramid texture feature encoding module, f motion () represents the motion extraction module, and k represents the pyramid level. Represents multi-scale texture features, (I -1 (I0, I1, I2) represents an image pair, [F 0→1 ,F 1→0 [F] indicates bidirectional optical flow. -1→0 ,F 1→2 [] indicates long-distance movement information.

4. The video frame interpolation method based on motion estimation according to claim 1, characterized in that, The step of jointly optimizing the intermediate optical flow and the intermediate frame features using an encoder to obtain the target intermediate optical flow includes: The underlying encoder estimates the initial intermediate optical flow based on the feature extraction results; The initial intermediate optical flow shape is transformed into an initial intermediate frame image, and the initial intermediate frame features are generated by convolutional transformation of the initial intermediate optical flow and the initial intermediate frame image. The initial intermediate frame features are input into the next layer encoder for optimization to obtain a new intermediate optical flow, until the top layer is optimized to obtain the target intermediate optical flow.

5. The video frame interpolation method based on motion estimation according to claim 4, characterized in that, The encoder joint optimization process is represented by the following equation: Among them, f inter () represents the intermediate optical flow estimation encoder, warp represents the image transformation operation based on optical flow, and Conv represents the convolution operation. Indicates the initial intermediate optical flow. This represents the new intermediate optical flow.

6. A video frame interpolation device, characterized in that, The apparatus for performing the motion estimation-based video frame interpolation method according to any one of claims 1-5, the apparatus comprising: An image pair acquisition unit is used to acquire input image pairs, the input image pairs including two adjacent frames in a low frame rate angiography sequence; The feature extraction unit is used to extract multi-scale texture features from the input image using a texture feature extraction module, and combine the multi-scale texture features with a motion extraction module to obtain inter-frame motion features. The motion features include bidirectional optical flow and long-distance motion information between adjacent frames obtained by estimation. The feature extraction result calculation unit is used to perform channel concatenation of the multi-scale texture features and the motion features to obtain the feature extraction results of the multi-frame input image; The intermediate frame feature acquisition unit is used to estimate the intermediate optical flow using the feature extraction result, perform deformation using the intermediate optical flow to obtain the intermediate frame image, and obtain the intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow. The joint optimization unit is used to jointly optimize the intermediate optical flow and the intermediate frame features using the encoder to obtain the target intermediate optical flow, and to deform the target intermediate optical flow to obtain the target intermediate frame image. The pin insertion unit is used to insert the target intermediate frame image between the input image pairs to perform a pin insertion operation.

7. A video frame interpolation device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the video frame interpolation method based on motion estimation as described in any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the video frame interpolation method based on motion estimation as described in any one of claims 1-5.

Images