Video frame insertion method and device based on motion estimation, equipment and storage medium

Abstract

The invention discloses a video frame insertion method, device and equipment based on motion estimation and a storage medium, and the method generates a clear and accurate target intermediate frame image through three links of texture feature extraction and motion feature extraction of an input image, and joint optimization of an intermediate optical flow and an intermediate frame image. Wherein the pyramid encoder extracts the multi-scale texture features of the image, so that the extracted features have global information and local texture details at the same time. Extraction of motion features not only provides a clear corresponding relation between adjacent frames, but also can capture alternate reciprocating motion conditions of coronary artery contraction and relaxation through long-distance motion optical flow. And the encoder adopts an intermediate optical flow and target value feature joint optimization mode to obtain a final interpolation result. And finally, a clear high-frame-rate sequence with coherent and accurate vascular movement such as coronary artery and the like can be generated.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video frame insertion method, device, equipment and storage medium based on nonlinear motion estimation. Background Art

[0002] Coronary angiography is key to the diagnosis and treatment of cardiovascular diseases. The heart rate of an adult is usually 50 to 100 beats per minute, and the cardiac cycle time of coronary artery movement is about 0.6 to 1.2 seconds. High frame rate coronary angiography sequences are essential for capturing various coronary artery morphologies under heart beating. However, the acquisition of high frame rate sequences requires increased frequency and intensity of X-ray exposure, thereby increasing the associated health risks for patients and doctors. In order to obtain high frame rate coronary angiography sequences with low X-ray radiation exposure, video interpolation is a crucial step. The video interpolation algorithm synthesizes the target intermediate frame between adjacent input frames to produce smooth and continuous motion of objects in the sequence. At present, video interpolation algorithms can be roughly classified into two categories: one is based on motion estimation, and the other is based on interpolation kernels.

[0003] The first type of method first estimates the motion between input frames, and further estimates the intermediate optical flow to spatially align the target intermediate frame and the input frame to synthesize the target interpolated frame. The second type of method estimates the interpolation kernel to weight the pixels of the input frame to synthesize the target interpolated frame. The interpolation kernel-based method lacks motion modeling and has a relatively high computational cost. The motion-based method shows better performance.

[0004] The optical flow estimation network further accelerates the development of motion estimation-based video interpolation methods. Optical flow is a dense field of displacement vectors for each pixel, which can provide a clear correspondence between adjacent input frames. The motion between input frames can be obtained through a mature optical flow network. Since the target intermediate frame is the image to be solved, the estimation of the intermediate optical flow is the difficulty and challenge faced by motion estimation-based video interpolation methods.

[0005] However, in existing video interpolation methods based on motion estimation, the estimation of intermediate optical flow relies on the assumption that the object moves in a straight line at a constant speed, which is not applicable to sequences characterized by complex nonlinear motion. The main motion in coronary angiography sequences is the periodic reciprocating coronary artery motion, which undergoes alternating systolic and diastolic motion. During the alternating systolic and diastolic motion, the coronary artery position of the target frame is not located between the coronary artery positions of adjacent input frames.

[0006] Therefore, how to provide a video interpolation method that can interpolate low frame rate sequences for non-linear motion and generate high frame rate sequences is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the invention

[0007] In view of the above problems, the present invention provides a video interpolation method, device, equipment and storage medium based on motion estimation for overcoming the above problems or at least partially solving the above problems. The method interpolates low frame rate sequences based on video interpolation of a nonlinear motion estimation network to generate high frame rate sequences without additional radiation exposure.

[0008] The present invention provides the following scheme:

[0009] A video frame insertion method based on motion estimation, comprising:

[0010] Acquire an input image pair, wherein the input image pair includes two adjacent frame images in a low frame rate angiography sequence;

[0011] A texture feature extraction module is used to extract multi-scale texture features from an input image, and a motion extraction module is used to obtain motion features between frames in combination with the multi-scale texture features, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information;

[0012] Channel-joining the multi-scale texture features and the motion features to obtain feature extraction results of multiple frames of input images;

[0013] The intermediate optical flow is estimated by using the feature extraction result, the intermediate optical flow is used to perform deformation to obtain an intermediate frame image, and the intermediate frame features are obtained by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow;

[0014] An encoder is used to jointly optimize the intermediate optical flow and the intermediate frame feature to obtain a target intermediate optical flow, and the target intermediate optical flow is deformed to obtain a target intermediate frame image;

[0015] Inserting the target intermediate frame image between the input image pairs implements a pin operation.

[0016] Preferably, the texture feature extraction module includes a pyramid texture feature encoding module.

[0017] Preferably, the texture feature extraction module and the motion feature extraction module are represented by the following formula:

[0018]

[0019] In the formula, f encode () is the pyramid texture feature encoding module, f motion () is the motion extraction module, k represents the pyramid level, Represents multi-scale texture features, (I -1 ,I0,I1,I2) represents an image pair, [F0→1 ,F 1→0 ] represents bidirectional optical flow, [F -1→0 ,F 1→2 ] indicates long-distance motion information.

[0020] Preferably, the step of using an encoder to jointly optimize the intermediate optical flow and the intermediate frame features to obtain a target intermediate optical flow comprises:

[0021] The bottom encoder estimates the initial intermediate optical flow based on the feature extraction result;

[0022] Transforming the initial intermediate optical manifold into an initial intermediate frame image, and performing a convolution transformation image operation on the initial intermediate optical flow and the initial intermediate frame image to generate the initial intermediate frame feature;

[0023] The initial intermediate frame features are input into the next layer encoder for optimization to obtain a new intermediate optical flow, until the top layer is optimized to obtain the target intermediate optical flow.

[0024] Preferably: the encoder joint optimization process is represented by the following formula:

[0025]

[0026] Among them, f inter () is the intermediate optical flow estimation encoder, warp is the image transformation operation based on the optical flow, Conv represents the convolution operation, represents the initial intermediate optical flow, Represents the new intermediate optical flow.

[0027] A video frame insertion device, used to execute the above-mentioned video frame insertion method based on motion estimation, the device comprising:

[0028] An image pair acquisition unit, configured to acquire an input image pair, wherein the input image pair comprises two adjacent frames of images in a low frame rate angiography sequence;

[0029] A feature extraction unit, configured to extract multi-scale texture features from an input image using a texture feature extraction module, and to obtain inter-frame motion features using a motion extraction module in combination with the multi-scale texture features, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information;

[0030] A feature extraction result calculation unit, used for performing channel splicing of the multi-scale texture features and the motion features to obtain feature extraction results of multiple frames of input images;

[0031] An intermediate frame feature acquisition unit, configured to estimate and obtain an intermediate optical flow using the feature extraction result, deform the intermediate optical flow to obtain an intermediate frame image, and obtain intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow;

[0032] A joint optimization unit, configured to jointly optimize the intermediate optical flow and the intermediate frame feature using an encoder to obtain a target intermediate optical flow, and deform the target intermediate optical flow to obtain a target intermediate frame image;

[0033] The pin operation unit is used to insert the target intermediate frame image between the input image pairs to implement the pin operation.

[0034] A video frame insertion device, the device comprising a processor and a memory:

[0035] The memory is used to store program code and transmit the program code to the processor;

[0036] The processor is used to execute the above-mentioned video frame insertion method based on motion estimation according to the instructions in the program code.

[0037] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned video frame insertion method based on motion estimation.

[0038] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0039] The embodiments of the present application provide a video interpolation method, device, equipment and storage medium based on motion estimation. The method generates a clear and accurate target intermediate frame image through three steps: texture feature extraction of the input image, motion feature extraction, and joint optimization of the intermediate optical flow and the intermediate frame image. Among them, the pyramid encoder extracts the multi-scale texture features of the image, so that the extracted features have both global information and local texture details. The extraction of motion features not only provides a clear correspondence between adjacent frames, but the long-distance motion optical flow can also capture the alternating reciprocating motion of coronary artery contraction and relaxation. The encoder adopts the method of joint optimization of the intermediate optical flow and the target value feature to obtain the final interpolation result. In the end, a clear high-frame rate sequence with coherent and accurate motion of blood vessels such as coronary arteries can be generated.

[0040] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flow chart of a video frame insertion method based on motion estimation provided by an embodiment of the present invention;

[0043] Figure 2 This is a flow chart of the use of frame insertion in a coronary angiography sequence provided by an embodiment of the present invention;

[0044] Figure 3 is a flowchart of extracting image texture features and motion features provided by an embodiment of the present invention;

[0045] Figure 4 is a flowchart of an encoder jointly optimizing intermediate optical flow and image generation provided by an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of a video frame insertion device provided by an embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of a video frame insertion device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0049] See also Figure 1 , a video frame insertion method based on motion estimation is provided in an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0050] S101: Acquire an input image pair, wherein the input image pair includes two adjacent frame images in a low frame rate angiography sequence;

[0051] S102: Use a texture feature extraction module to extract multi-scale texture features from the input image, and use a motion extraction module in combination with the multi-scale texture features to obtain motion features between frames, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information; in specific implementation, the embodiment of the present application can provide that the texture feature extraction module includes a pyramid texture feature encoding module.

[0052] The texture feature extraction module and the motion feature extraction module are represented by the following formula:

[0053]

[0054] In the formula, f encode () is the pyramid texture feature encoding module, f motion () is the motion extraction module, k represents the pyramid level, Represents multi-scale texture features, (I -1 ,I0,I1,I2) represents an image pair, [F 0→1 ,F 1→0 ] represents bidirectional optical flow, [F -1→0 ,F 1→2 ] indicates long-distance motion information.

[0055] S103: performing channel splicing on the multi-scale texture features and the motion features to obtain feature extraction results of multiple frames of input images;

[0056] S104: obtaining an intermediate optical flow by estimating the feature extraction result, deforming the intermediate optical flow to obtain an intermediate frame image, and obtaining intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow;

[0057] S105: using an encoder to jointly optimize the intermediate optical flow and the intermediate frame feature to obtain a target intermediate optical flow, and deforming the target intermediate optical flow to obtain a target intermediate frame image; in specific implementation, the embodiment of the present application may provide an underlying encoder to estimate an initial intermediate optical flow based on the feature extraction result;

[0058] Transforming the initial intermediate optical manifold into an initial intermediate frame image, and performing a convolution transformation image operation on the initial intermediate optical flow and the initial intermediate frame image to generate the initial intermediate frame feature;

[0059] The initial intermediate frame features are input into the next layer encoder for optimization to obtain a new intermediate optical flow, until the top layer is optimized to obtain the target intermediate optical flow.

[0060] The encoder joint optimization process is expressed as follows:

[0061]

[0062] Among them, f inter () is the intermediate optical flow estimation encoder, warp is the image transformation operation based on the optical flow, Conv represents the convolution operation, represents the initial intermediate optical flow, Represents the new intermediate optical flow.

[0063] S106: Insert the target intermediate frame image between the input image pairs to implement a pin operation.

[0064] The video interpolation method based on motion estimation provided in the embodiment of the present application is a method for obtaining a high frame rate coronary angiography sequence under low X-ray radiation exposure. Existing video interpolation algorithms usually assume that the object moves in a straight line at a constant speed. Considering the periodicity and reciprocating nature of coronary artery motion, the embodiment of the present application proposes a video interpolation method based on motion estimation based on a nonlinear motion estimation network, which interpolates low frame rate sequences and generates high frame rate sequences without additional radiation exposure.

[0065] The method first extracts pyramid texture features from the input frames and extracts motion features between the input frames to establish explicit correspondences. The features are then input into the decoder to estimate the intermediate optical flow and generate the target intermediate frame image, where a bottom-up joint optimization method is used to obtain the final interpolation result. The entire network model is trained under the joint supervision of the weighted self-supervised optical flow loss and the target frame synthesis loss.

[0066] The following uses the interpolation of coronary angiography sequence as an example to describe in detail the video interpolation method based on motion estimation provided by the embodiment of the present application, wherein the pyramid level k=0, 1, 2, 3.

[0067] The purpose of the method provided in the embodiment of the present application is to collect the original low frame rate coronary angiography sequence under low X-ray radiation exposure, and to obtain a high frame rate coronary angiography sequence through a video interpolation algorithm, thereby reducing the frequency and intensity of X-ray use.

[0068] In order to generate accurate and clear target intermediate frames, the embodiment of the present application provides a nonlinear motion estimation network model, which is specifically implemented in three steps: texture feature extraction, motion feature extraction, and joint optimization of intermediate optical flow and intermediate frame images.

[0069] In the process of image texture feature extraction, the embodiment of the present application uses a feature pyramid encoder to extract multi-scale image texture features, which can effectively reduce the loss of texture information during the downsampling process and retain local texture details while capturing the global information of the image.

[0070] In the process of motion feature extraction, the embodiment of the present application not only estimates the bidirectional optical flow between adjacent frames, but also estimates the long-distance motion optical flow to capture the alternating reciprocating motion of coronary artery contraction and relaxation. These motion features provide a more accurate coronary artery position for the image generation of subsequent target intermediate frames.

[0071] In the process of jointly optimizing the intermediate optical flow and the intermediate frame image, the embodiment of the present application designs a self-supervised optical flow loss to supervise the learning of the motion representation between the input frames, which enhances the relevance to the interpolation task. The embodiment of the present application adopts a bottom-up joint optimization method to obtain the final interpolation result. The joint optimization of the intermediate optical flow and the target frame features can further improve the accuracy of video interpolation.

[0072] As attached Figure 2 The present application provides a flow chart of a video interpolation method based on motion estimation in a coronary angiography sequence. The model input is an image pair [I -1 ,I0,I1,I2], the texture feature extraction module extracts multi-scale texture feature expressions from the input image The extracted features have both global information and local texture details. The motion extraction module obtains the correlation features between frames to establish an explicit correspondence between images. The motion extraction module not only estimates the bidirectional optical flow [F 0→1 ,F 1→0 ], and also estimates long-range motion information [F -1→0 ,F 1→2 ], which is used to capture the alternating reciprocating motion of coronary artery contraction and relaxation. The encoder module jointly optimizes the intermediate optical flow estimation and the image features of the target intermediate frame to obtain the final intermediate frame interpolation result By inputting the low frame rate angiography sequence into the entire video interpolation model, a high frame rate angiography sequence can be obtained.

[0073] Combined with Figure 3 , the implementation steps of feature extraction provided by the embodiment of the present application are described in detail. First, the texture features of each frame of the image are obtained through the pyramid feature encoding module. Then, the motion features between frames are obtained by estimating the bidirectional optical flow of adjacent frames. In addition, the long-distance motion information is obtained by estimating the optical flow of the interval frame to capture the alternating reciprocating motion of coronary contraction and relaxation. The formula is expressed as follows:

[0074]

[0075] Among them, f encode () is the pyramid texture feature encoding module, f motion () is the motion extraction module,

[0076] k=0,1,2,3 represents the pyramid level. With sports characteristics

[0077] [F 0→1 ,F 1→0 ,F -1→0 ,F 1→2 ] to perform channel splicing to obtain the feature extraction results of the final multi-frame input image

[0078] Combined with Figure 4 , the implementation steps of the encoder of the present invention jointly optimizing the intermediate optical flow estimation and the target image generation are described in detail. The intermediate optical flow and the target intermediate frame image are conjugate to each other. The bottom-up joint optimization of the intermediate optical flow and the generation of the target frame features can significantly enhance the final intermediate frame interpolation result. The bottom-level encoder estimates the intermediate optical flow The optical flow can be used to deform the input frame to obtain a rough target frame, and convolution generates the target frame features. Input the next layer encoder to optimize and obtain the new intermediate optical flow. The bottom-up optimization process of the pyramid is expressed as:

[0079]

[0080] Among them, f inter () is the intermediate optical flow estimation encoder, and warp is the image transformation operation based on the optical flow. The pyramid structure is optimized to the top layer to obtain the final intermediate optical flow [F 0.5→1 ,F 0.5→0 ], deformed to the final interpolation result

[0081] In summary, the video interpolation method based on motion estimation provided by the present application generates clear and accurate target intermediate frame images through three steps: texture feature extraction of the input image, motion feature extraction, and joint optimization of the intermediate optical flow and the intermediate frame image. Among them, the pyramid encoder extracts the multi-scale texture features of the image, so that the extracted features have both global information and local texture details. The extraction of motion features not only provides a clear correspondence between adjacent frames, but the long-distance motion optical flow can also capture the alternating reciprocating motion of coronary artery contraction and relaxation. The encoder adopts the method of joint optimization of the intermediate optical flow and the target value features to obtain the final interpolation result. In the end, a clear high-frame rate sequence with coherent and accurate motion of blood vessels such as coronary arteries can be generated.

[0082] See also Figure 5 , the embodiment of the present application can also provide a video frame insertion device, such as Figure 5 As shown, the device may include:

[0083] An image pair acquisition unit 501 is used to acquire an input image pair, wherein the input image pair includes two adjacent frame images in a low frame rate angiography sequence;

[0084] A feature extraction unit 502 is used to extract multi-scale texture features from the input image using a texture feature extraction module, and to obtain inter-frame motion features using a motion extraction module in combination with the multi-scale texture features, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information;

[0085] A feature extraction result calculation unit 503 is used to perform channel splicing on the multi-scale texture feature and the motion feature to obtain feature extraction results of multiple frames of input images;

[0086] An intermediate frame feature acquisition unit 504 is used to estimate and obtain an intermediate optical flow using the feature extraction result, deform the intermediate optical flow to obtain an intermediate frame image, and obtain intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow;

[0087] A joint optimization unit 505, configured to use an encoder to jointly optimize the intermediate optical flow and the intermediate frame feature to obtain a target intermediate optical flow, and deform the target intermediate optical flow to obtain a target intermediate frame image;

[0088] The pin operation unit 506 is used to insert the target intermediate frame image between the input image pairs to implement a pin operation.

[0089] The embodiment of the present application may also provide a video frame insertion device, the device comprising a processor and a memory:

[0090] The memory is used to store program code and transmit the program code to the processor;

[0091] The processor is used to execute the steps of the above-mentioned video frame insertion method based on motion estimation according to the instructions in the program code.

[0092] like Figure 6 As shown, a video frame insertion device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0093] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices.

[0094] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of a video frame interpolation method based on motion estimation.

[0095] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0096] Acquire an input image pair, wherein the input image pair includes two adjacent frame images in a low frame rate angiography sequence;

[0097] A texture feature extraction module is used to extract multi-scale texture features from an input image, and a motion extraction module is used to obtain motion features between frames in combination with the multi-scale texture features, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information;

[0098] Channel-joining the multi-scale texture features and the motion features to obtain feature extraction results of multiple frames of input images;

[0099] The intermediate optical flow is estimated by using the feature extraction result, the intermediate optical flow is used to perform deformation to obtain an intermediate frame image, and the intermediate frame features are obtained by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow;

[0100] An encoder is used to jointly optimize the intermediate optical flow and the intermediate frame feature to obtain a target intermediate optical flow, and the target intermediate optical flow is deformed to obtain a target intermediate frame image;

[0101] Inserting the target intermediate frame image between the input image pairs implements a pin operation.

[0102] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0103] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0104] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0105] Of course, it should be noted that Figure 6 The structure shown does not constitute a limitation on the video frame insertion device in the embodiment of the present application. In actual applications, the video frame insertion device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0106] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the steps of the above-mentioned video interpolation method based on motion estimation.

[0107] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0108] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0109] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video frame insertion method based on motion estimation, characterized in that: include: Acquire an input image pair, wherein the input image pair includes two adjacent frame images in a low frame rate angiography sequence; A texture feature extraction module is used to extract multi-scale texture features from an input image, and a motion extraction module is used to obtain motion features between frames in combination with the multi-scale texture features, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information; Channel-joining the multi-scale texture features and the motion features to obtain feature extraction results of multiple frames of input images; The intermediate optical flow is estimated by using the feature extraction result, the intermediate optical flow is used to perform deformation to obtain an intermediate frame image, and the intermediate frame features are obtained by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow; An encoder is used to jointly optimize the intermediate optical flow and the intermediate frame feature to obtain a target intermediate optical flow, and the target intermediate optical flow is deformed to obtain a target intermediate frame image; Inserting the target intermediate frame image between the input image pairs implements a pin operation.

2. The video frame insertion method based on motion estimation according to claim 1, characterized in that: The texture feature extraction module includes a pyramid texture feature encoding module.

3. The video frame insertion method based on motion estimation according to claim 2, characterized in that: The texture feature extraction module and the motion feature extraction module are represented by the following formula: In the formula, f encode () is the pyramid texture feature encoding module, f motion () is the motion extraction module, k represents the pyramid level, Represents multi-scale texture features, (I -1 ,I0,I1,I2) represents an image pair, [F 0→1 ,F 1→0 ] represents bidirectional optical flow, [F -1→0 ,F 1→2 ] indicates long-distance motion information.

4. The video frame insertion method based on motion estimation according to claim 1, characterized in that: The adopting an encoder to jointly optimize the intermediate optical flow and the intermediate frame features to obtain a target intermediate optical flow comprises: The bottom encoder estimates the initial intermediate optical flow based on the feature extraction result; Transforming the initial intermediate optical manifold into an initial intermediate frame image, and performing a convolution transformation image operation on the initial intermediate optical flow and the initial intermediate frame image to generate the initial intermediate frame feature; The initial intermediate frame features are input into the next layer encoder for optimization to obtain a new intermediate optical flow, until the top layer is optimized to obtain the target intermediate optical flow.

5. The video frame insertion method based on motion estimation according to claim 4, characterized in that: The encoder joint optimization process is expressed as follows: Among them, f inter () is the intermediate optical flow estimation encoder, warp is the image transformation operation based on the optical flow, Conv represents the convolution operation, represents the initial intermediate optical flow, Represents the new intermediate optical flow.

6. A video frame insertion device, characterized in that: The device is used to execute the video frame insertion method based on motion estimation according to any one of claims 1 to 5, comprising: An image pair acquisition unit, configured to acquire an input image pair, wherein the input image pair comprises two adjacent frames of images in a low frame rate angiography sequence; A feature extraction unit, configured to extract multi-scale texture features from an input image using a texture feature extraction module, and to obtain inter-frame motion features using a motion extraction module in combination with the multi-scale texture features, wherein the motion features include bidirectional optical flow between adjacent frames obtained by estimation and long-distance motion information; A feature extraction result calculation unit, used for performing channel splicing of the multi-scale texture features and the motion features to obtain feature extraction results of multiple frames of input images; An intermediate frame feature acquisition unit, configured to estimate and obtain an intermediate optical flow using the feature extraction result, deform the intermediate optical flow to obtain an intermediate frame image, and obtain intermediate frame features by performing convolution and image transformation operations on the intermediate frame image and the intermediate optical flow; A joint optimization unit, configured to jointly optimize the intermediate optical flow and the intermediate frame feature using an encoder to obtain a target intermediate optical flow, and deform the target intermediate optical flow to obtain a target intermediate frame image; The pin operation unit is used to insert the target intermediate frame image between the input image pairs to implement the pin operation.

7. A video frame insertion device, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the video frame insertion method based on motion estimation according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the video frame insertion method based on motion estimation according to any one of claims 1-5.

Images