High-resolution video frame insertion method and system based on texture mapping

Abstract

The invention relates to the technical field of video frame insertion, and provides a high-resolution video frame insertion method and system based on texture mapping. The method comprises the following steps: based on a first frame and a second frame of a video stream, adopting a motion estimator to obtain a first estimation stream and a second estimation stream; carrying out distortion deformation on the first frame and the second frame, and predicting a skeleton by adopting multilayer convolution based on the deformed first frame and second frame; extracting a first texture block and a second texture block from the first frame and the second frame; mapping the skeleton to a selected texture to obtain an interpolation result; and reconstructing the interpolation result, the first texture block and the second texture block to obtain a frame interpolation result. According to the method, the sampling precision on the optical flow can be effectively improved, the frame insertion texture continuity is enhanced, and the actual requirement of high-resolution video frame insertion is met.

Description

Technical Field

[0001] The present invention relates to the technical field of video frame insertion, and in particular to a high-resolution video frame insertion method and system based on texture mapping. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of image and video acquisition technology, the use of high-definition video has gradually become mainstream. However, due to the limitations of hardware performance and network transmission bandwidth, many high-definition videos usually show low frame rates during acquisition and transmission, which leads to motion blur and reduced visual perception in dynamic scenes. In order to increase the frame rate of the video and achieve a smoother and more coherent visual effect, video frame interpolation (VFI) technology has received widespread attention. Video frame interpolation improves the video frame rate and optimizes playback smoothness by synthesizing intermediate frames between adjacent frames.

[0004] Existing video interpolation techniques are mainly based on deep learning methods and can be divided into three categories: kernel-based interpolation methods, generation-based interpolation methods, and optical flow-based interpolation methods. Among them, kernel-based interpolation methods estimate inter-frame motion information through dynamic kernel weights or offsets, which is suitable for fine-grained motion capture; generation-based interpolation methods directly generate intermediate frames from the features of input frames, which are suitable for scenes with high requirements for lightweight and real-time performance; and optical flow-based interpolation methods describe inter-frame pixel motion by estimating optical flow, and use optical flow to align and fuse input frames, which is the current mainstream video interpolation technology. Optical flow-based interpolation methods usually include three core steps: first, generate low-resolution bidirectional optical flow by performing motion estimation on the low-resolution version of the input frame, thereby reducing computational complexity; second, map the low-resolution optical flow to high resolution through bilinear interpolation or adaptive optical flow upsampling (AFU) and other technologies; finally, use the interpolated optical flow to align the input frames, and generate the final interpolated frame through mask and residual fusion mechanism. However, such methods still face a number of technical challenges in high-resolution video scenes. First, the traditional bilinear interpolation method easily leads to blurred optical flow boundaries and misalignment with image boundaries, thus causing interpolation artifacts. Second, low-resolution motion estimation cannot capture the subtle movements of high-resolution pixels, resulting in discontinuous performance of interpolated frames in detail parts. In addition, the existing mask and residual fusion mechanism is based on dual-source pixel-level texture input. When errors occur in optical flow estimation, ghosting and blurring are prone to occur. Therefore, the existing technology cannot meet the actual needs of high-resolution video interpolation, and cannot effectively improve the optical flow upsampling accuracy and enhance interpolation texture continuity. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a high-resolution video interpolation method and system based on texture mapping, which can effectively improve the optical flow upsampling accuracy and enhance the interpolation texture continuity, thereby meeting the actual needs of high-resolution video interpolation.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a high-resolution video frame insertion method based on texture mapping.

[0008] A high-resolution video frame insertion method based on texture mapping, comprising:

[0009] Based on the first frame I0 and the second frame I1 of the video stream, a motion estimator is used to obtain a first estimated stream and the second estimation stream

[0010] The first frame I0 and the second frame I1 are distorted, and based on the deformed first frame I0 and the second frame I1, a skeleton is predicted by using multi-layer convolution; and the first texture block is extracted from the first frame I0 and the second frame I1 and the second texture block Map the skeleton to the selected texture and get the interpolation result;

[0011] For the interpolation result, the first texture block and the second texture block Reconstruct and obtain the interpolation result.

[0012] Furthermore, the process of using the motion estimator includes: based on the first frame I0 and the second frame I1 of the video stream, using a feature encoder to extract the first frame feature and the second frame feature; deforming the first frame feature and aligning it with the second frame feature to obtain a first low-resolution stream F 0→1 , deform the second frame features and align them with the first frame features to obtain the second low-resolution flow F 1→0 ; Use bilinear stream upsampling to upsample the first low-resolution stream F 0→1 and the second low-resolution stream F 1→0 Upsampling is performed, and the guidance information of the first frame I0 and the second frame I1 extracted by the convolution layer is used to correct the blurred details in the upsampled stream to obtain the first estimated stream and the second estimation stream

[0013] Furthermore, the skeleton is mapped onto the selected texture to obtain an interpolation result; the method comprises: mapping the first texture block Second texture block And the skeleton is divided into several third texture blocks Fourth texture block And the skeleton block According to the size of the divided blocks, the first estimated flow and the second estimation stream Downsample to get the first forward flow and the second forward flow The first forward flow and the second forward flow Convert to first backward flow and the second backward flow Based on the first backward flow and the second backward flow Use the Grid_Sample function to sample the third texture block Fourth texture block And the skeleton block Index and get the first index texture block B 0,t , the second index texture block B 1,t and skeleton texture blocks

[0014] Furthermore, the skeleton is mapped onto the selected texture to obtain an interpolation result; the method further comprises: mapping the first index texture block B 0,t , the second index texture block B 1,t And the skeleton texture block Processed as a low-resolution first index tensor Second index tensor and the third index tensor Separately, the first index tensor Second index tensor and the third index tensor Convert to first index vector Second index vector and the third index vector Q x,y , based on the third index vector Q x,y , in the first index vector and the second index vector Match the most relevant vector in the N×N neighborhood of x,y,(N) ; Evaluate the relevant vector set C x,y,(N) The largest element in the , get the position index of the element; according to the position index, all the skeleton texture blocks Mapped to the third index vector Q x,y On the top, restore the details of the interpolation result to obtain the interpolation result.

[0015] Furthermore, the Grid_Sample function is expressed by the following formula:

[0016]

[0017] Among them, B0 represents the texture block, and mode=Nearest represents the Nearest mode.

[0018] Furthermore, the interpolation result and the first texture block and the second texture block Reconstruction is performed to obtain an interpolation result; the method includes: respectively performing interpolation on the interpolation result and the first texture block and the second texture block Supervision is performed to transform the latent space into the image space and obtain the interpolation result.

[0019] A second aspect of the present invention provides a high-resolution video frame insertion system based on texture mapping.

[0020] A high-resolution video frame insertion system based on texture mapping, comprising:

[0021] The guide stream upsampling module is configured to: based on the first frame I0 and the second frame I1 of the video stream, use a motion estimator to obtain a first estimated stream and the second estimation stream

[0022] The texture mapping module is configured to: distort the first frame I0 and the second frame I1, and predict the skeleton based on the deformed first frame I0 and the second frame I1 by using multi-layer convolution; and extract the first texture block from the first frame I0 and the second frame I1 and the second texture block Map the skeleton to the selected texture and get the interpolation result;

[0023] A reconstruction module is configured to: perform interpolation on the interpolation result, the first texture block and the second texture block Reconstruct and obtain the interpolation result.

[0024] A third aspect of the present invention provides a computer-readable storage medium.

[0025] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the high-resolution video interpolation method based on texture mapping as described in the first aspect above.

[0026] A fourth aspect of the present invention provides a computer device.

[0027] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the high-resolution video interpolation method based on texture mapping as described in the first aspect above are implemented.

[0028] A fifth aspect of the present invention provides a computer program product or a computer program.

[0029] The present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the high-resolution video interpolation method based on texture mapping as described in the first aspect above.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention relates to a method for high-resolution video frame interpolation (VFI). In view of the problems of blurred boundaries, artifacts and texture discontinuity caused by insufficient optical flow upsampling and frame synthesis mechanism in existing high-resolution VFI methods, the present invention proposes a high-resolution video frame interpolation method and system based on texture mapping. The algorithm includes two core technologies: guided flow upsampling (Guided Flow Upsampling, Guided Flow Upsampling Module) and texture mapping. Guided flow upsampling uses input frames as guidance information to perform refined upsampling processing on task-oriented low-resolution optical flows, significantly reducing boundary artifacts and blurring in interpolation results, thereby improving the estimation accuracy of high-resolution optical flows. Texture mapping technology selects continuous high-quality regional texture blocks from the input frames and performs texture filling on the intermediate frames, avoiding the problems of detail loss and texture discontinuity caused by optical flow misalignment in traditional interpolation methods, and greatly improving the clarity and detail performance of interpolation results.

[0032] The present invention proposes a high-resolution video interpolation method and system based on texture mapping, which exhibits excellent performance in processing high-resolution video interpolation, especially significantly outperforming the prior art in detail restoration and boundary processing. Finally, by combining the guided flow upsampling and texture mapping modules, the present invention achieves a comprehensive improvement in the quality of high-resolution video interpolation, significantly reduces artifacts, blur and discontinuity, and reaches the advanced level in the field of interpolation. The present invention can be widely used in fields related to high-resolution video processing, such as video editing, augmented reality and virtual reality, and has important application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0034] Figure 1 is a framework diagram of a high-resolution video frame insertion method based on texture mapping shown in the present invention;

[0035] Figure 2 2 is a schematic diagram of a guide stream upsampling module according to the present invention. The upper side of the image introduces the difference between the guide stream upsampling module and the bilinear interpolation module, and the lower side shows the structure of the guide stream upsampling module. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0040] Embodiment 1

[0041] The present embodiment provides a high-resolution video interpolation method based on texture mapping. The present embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In the present embodiment, the method includes the following steps:

[0042] Based on the first frame I0 and the second frame I1 of the video stream, a motion estimator is used to obtain a first estimated stream and the second estimation stream

[0043] The first frame I0 and the second frame I1 are distorted, and based on the deformed first frame I0 and the second frame I1, a skeleton is predicted by using multi-layer convolution; and the first texture block is extracted from the first frame I0 and the second frame I1 and the second texture block Map the skeleton to the selected texture and get the interpolation result;

[0044] For the interpolation result, the first texture block and the second texture block Reconstruct and obtain the interpolation result.

[0045] The present embodiment is described in detail below. Figure 1 , Figure 2 As shown:

[0046] 1. Design of the guided flow upsampling module:

[0047] Thanks to its pyramid recursive structure, UPR-Net can be used as a lightweight motion estimator in the present invention. However, the bidirectional flow module of UPR-Net performs alignment in the interpolation time step t while refining the flow F 0→1 and F 1→0This difference may interfere with coordination and eventually lead to a decrease in the efficiency of flow estimation. In order to achieve more efficient alignment and more accurate flow estimation, the present invention refers to the structure of PWC-Net and redesigns the motion estimator. Specifically, the features of the previous frame are deformed and aligned with the features of the next frame, and these features are used to update the flow F' 0→1 And get the flow F 0→1 Convection F' 1→0 The same operation is also used to obtain more accurate motion estimation. The present invention trains the redesigned UPR-base according to the same training setting as UPR-Net. Using the redesigned UPR-base, the present invention obtains a low-resolution bidirectional flow F 0→1 and F 1→0 , and then upsampled by the guided stream upsampling module. The boundaries of the upsampled stream should be aligned with the boundaries of frame I0. To address the inherent challenges associated with stream upsampling, the present invention proposes a simple but effective guided stream upsampling module that integrates high-resolution image information to improve the quality of the upsampled stream.

[0048] First, the guided stream upsampling module uses bilinear stream upsampling to upsample the low-resolution stream to generate a preliminary upsampled stream, where motion edge blur may occur. Second, the guided information in the input frame is extracted through the convolution layer. Third, the guided stream upsampling module uses this guided information to correct the blurred details in the bilinear upsampled stream, thereby enhancing the clarity of the stream edges. After upsampling by the guided stream upsampling module, the stream and Deforming the previous frame and the next frame can be expressed as:

[0049]

[0050] in, and In a linear function, the estimated flow and Mapped to a specific time step t.

[0051] 2. Design of texture mapping module

[0052] The texture mapping module is used to generate an intermediate skeleton and map it to a selected high-quality texture. The present invention first predicts the intermediate skeleton and extracts texture blocks from two input frames, then performs block texture search through flow guidance, then performs local matching of block textures, and finally obtains the final interpolation result through the reconstruction module.

[0053] (1) Skeleton prediction and texture extraction

[0054] Based on the deformed front and back frames, the skeleton prediction module generates an intermediate skeleton

[0055]

[0056] Among them, C represents The number of channels, H and W represent the height and width of the input image size. The present invention uses multi-layer convolution for skeleton prediction. At the same time, the textures of two consecutive frames are extracted and divided into texture blocks of a specific size.

[0057] In order to Refine to interpolated frame The present invention uses the obtained skeleton To index the texture block and map it to the skeleton Based on frames I0 and I1, we get texture and

[0058]

[0059] The present invention uses a convolutional layer as an extractor. According to the block size s, the texture and Divided into many texture blocks and For example, It can be expressed as:

[0060]

[0061] in, Representing Texture Divided into s×s texture blocks, represents the texture block at position index (x,y). In addition, the skeleton is similarly divided into blocks for texture mapping, represented as

[0062]

[0063] (2) Flow-guided block texture search

[0064] By estimating the motion between given frames I0 and I1, the bidirectional flow and Since these streams contain direct motion information, they help in texture indexing. Therefore, to obtain preliminary texture block matches, the streams and Indexing, balancing computational efficiency and accuracy.

[0065] First, according to the block size s×s, and Downsampling, we get and In order to avoid texture blending that may be caused by bilinear forward or backward deformation, the forward stream is first Convert to backward flow

[0066]

[0067] Then, use the Grid_Sample (Nearest mode) function to index the texture block and get the indexed texture block B 0,t , which can be expressed as:

[0068]

[0069] According to the above formula, we get B 0,t and B 1,t , which are rearrangements of texture blocks B0 and B1.

[0070] (3) Local Matching of Block Textures

[0071] Due to the error in flow estimation, the skeleton block Usually with or In order to achieve a better match, the present invention uses multi-layer convolution to transform the skeleton into and texture mapping and Processed as a low-resolution index tensor

[0072]

[0073] Where C' represents The number of channels. and Do the same.

[0074] Inspired by the QKV mechanism, the present invention indexes the tensor and Convert to index vector Q x,y , and

[0075]

[0076] in, Then, based on Q x,y , the present invention matches the most relevant vectors in the N×N neighborhood of the two index vector groups K0 and K1. x,y,(N) Indicates that Q x,y The set of dependencies:

[0077]

[0078] Then, evaluate the set C x,y,(N) The largest element in And get the position index of the element (x+i,y+j,e). Note that this step ensures that the texture block comes from I0 or I1. Therefore, the texture block corresponding to the position (x,y) of skeleton Q is It will be mapped to Finally, follow the above steps to traverse all texture blocks

[0079] (4) Reconstruction module

[0080] To ensure that the texture blocks and The present invention not only improves the quality of the interpolation result With the true value I t Supervision, also texture and The reconstruction module is a UNet-like network that aims to transform the latent space into the image space. By using the shared weights of the reconstruction module, the present invention restores the texture and To get the result and And supervise them with input frames I0 and I1. These frames do not need to be reconstructed during inference.

[0081] 3. High-resolution video interpolation model training based on texture mapping

[0082] Model architecture: Use the pre-trained optical flow module UPR-Net to estimate the optical flow, and train the network on the public dataset of Vimeo-90K;

[0083] Training process: The features extracted from the previous and next frames of the video are reconstructed using a feature-sharing reconstruction module, and the same set of losses are used for supervision. The ratio of the reconstruction loss weight of the interpolated frame result and the reconstruction loss weight of the previous and next frames to the total loss is set to 8:1:1. During training, the learning rate is set to a cosine function transition from 0.0001 to 0.00001;

[0084] 4. Reconstruction of interpolation results

[0085] Input processing: The two frames before and after the high-resolution low-frame-rate video are used as network input to infer the intermediate results.

[0086] Output generation: The algorithm calculates the optical flow of the two input frames, extracts features, generates low-quality interpolation results, reorders the feature blocks, and outputs high-quality interpolation results.

[0087] 5. Experimental verification

[0088] Experimental settings: SNU-FILM, DAVIS, Xiph and other datasets are selected as test datasets to compare the interpolation results of different methods. Peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and perceptual similarity index (LPIPS) are used as evaluation indicators.

[0089] Experimental results: Compared with the SOTA interpolation methods (such as EDSC, FILM, RIFE, AMT-G, UPR-LLARGE, LDMVFI, EMA-VFI, SGM and PerVFI), the method of the present invention has significant improvements in PSNR, SSIM and LPIPS indicators. The interpolation results show excellent performance in terms of interpolation result quality and texture clarity, especially in the interpolation effect of high-resolution video.

[0090] Embodiment 2

[0091] This embodiment provides a high-resolution video frame insertion system based on texture mapping.

[0092] A high-resolution video frame insertion system based on texture mapping, comprising:

[0093] The guide stream upsampling module is configured to: based on the first frame I0 and the second frame I1 of the video stream, use a motion estimator to obtain a first estimated stream and the second estimation stream

[0094] The texture mapping module is configured to: distort the first frame I0 and the second frame I1, and predict the skeleton based on the deformed first frame I0 and the second frame I1 by using multi-layer convolution; and extract the first texture block from the first frame I0 and the second frame I1 and the second texture block Map the skeleton to the selected texture and get the interpolation result;

[0095] A reconstruction module is configured to: perform interpolation on the interpolation result, the first texture block and the second texture block Reconstruct and obtain the interpolation result.

[0096] In some embodiments, the guide stream upsampling module is further configured to: based on the first frame I0 and the second frame I1 of the video stream, use a feature encoder to extract the first frame feature and the second frame feature; deform the first frame feature and align it with the second frame feature to obtain a first low-resolution stream F 0→1 , deform the second frame features and align them with the first frame features to obtain the second low-resolution flow F 1→0 ; Use bilinear stream upsampling to upsample the first low-resolution stream F 0→1 and the second low-resolution stream F 1→0 Upsampling is performed, and the guidance information of the first frame I0 and the second frame I1 extracted by the convolution layer is used to correct the blurred details in the upsampled stream to obtain the first estimated stream and the second estimation stream

[0097] In some embodiments, the texture mapping module is further configured to: transform the first texture block Second texture block And the skeleton is divided into several third texture blocks Fourth texture block And the skeleton block According to the size of the divided blocks, the first estimated flow and the second estimation stream Downsample to get the first forward flow and the second forward flow The first forward flow and the second forward flow Convert to first backward flow and the second backward flow Based on the first backward flow and the second backward flow Use the Grid_Sample function to sample the third texture block Fourth texture block And the skeleton block Index and get the first index texture block B 0,t , the second index texture block B 1,t and skeleton texture blocks

[0098] In some embodiments, the texture mapping module is further configured to: map the first index texture block B 0,t , the second index texture block B 1,t And the skeleton texture block Processed as a low-resolution first index tensor Second index tensor and the third index tensor Separately, the first index tensor Second index tensor and the third index tensor Convert to first index vector Second index vector and the third index vector Q x,y , based on the third index vector Q x,y , in the first index vector and the second index vector Match the most relevant vector in the N×N neighborhood of x,y,(N) ; Evaluate the relevant vector set C x,y,(N) The largest element in the , get the position index of the element; according to the position index, all the skeleton texture blocks Mapped to the third index vector Q x,y On the top, restore the details of the interpolation result to obtain the interpolation result.

[0099] In some embodiments, the Grid_Sample function is represented by the following formula:

[0100]

[0101] Among them, B0 represents the texture block, and mode=Nearest represents the Nearest mode.

[0102] In some embodiments, the reconstruction module is further configured to: respectively perform the interpolation result and the first texture block and the second texture block Supervision is performed to transform the latent space into the image space and obtain the interpolation result.

[0103] Embodiment 3

[0104] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the high-resolution video interpolation method based on texture mapping as described in the first embodiment above are implemented.

[0105] Embodiment 4

[0106] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the high-resolution video interpolation method based on texture mapping as described in the first embodiment are implemented.

[0107] Embodiment 5

[0108] This embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the high-resolution video interpolation method based on texture mapping described in the first embodiment.

[0109] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A high-resolution video interpolation method based on texture mapping, characterized in that: include: Based on the first frame I0 and the second frame I1 of the video stream, a motion estimator is used to obtain a first estimated stream and the second estimation stream The first frame I0 and the second frame I1 are distorted, and based on the deformed first frame I0 and the second frame I1, a skeleton is predicted by using multi-layer convolution; and the first texture block is extracted from the first frame I0 and the second frame I1 and the second texture block Map the skeleton to the selected texture and get the interpolation result; For the interpolation result, the first texture block and the second texture block Reconstruct and obtain the interpolation result.

2. The high-resolution video frame insertion method based on texture mapping according to claim 1, characterized in that: The process of using the motion estimator includes: based on the first frame I0 and the second frame I1 of the video stream, using a feature encoder to extract the first frame feature and the second frame feature; deforming the first frame feature and aligning it with the second frame feature to obtain a first low-resolution stream F 0→1 , deform the second frame features and align them with the first frame features to obtain the second low-resolution flow F 1→0 ; Use bilinear stream upsampling to upsample the first low-resolution stream F 0→1 and the second low-resolution stream F 1→0 Upsampling is performed, and the guidance information of the first frame I0 and the second frame I1 extracted by the convolution layer is used to correct the blurred details in the upsampled stream to obtain the first estimated stream and the second estimation stream 3. The high-resolution video frame insertion method based on texture mapping according to claim 1, characterized in that: The skeleton is mapped onto the selected texture to obtain an interpolation result; the method comprises: mapping the first texture block Second texture block And the skeleton is divided into several third texture blocks Fourth texture block And the skeleton block According to the size of the divided blocks, the first estimated flow and the second estimation stream Downsample to get the first forward flow and the second forward flow The first forward flow and the second forward flow Convert to first backward flow and the second backward flow Based on the first backward flow and the second backward flow Use the Grid_Sample function to sample the third texture block Fourth texture block And the skeleton block Index and get the first index texture block B 0,t , the second index texture block B 1,t and skeleton texture blocks 4. The high-resolution video frame insertion method based on texture mapping according to claim 3, characterized in that: The skeleton is mapped onto the selected texture to obtain an interpolation result; the method further comprises: mapping the first index texture block B 0,t , the second index texture block B 1,t And the skeleton texture block Processed as a low-resolution first index tensor Second index tensor and the third index tensor Separately, the first index tensor Second index tensor and the third index tensor Convert to first index vector Second index vector and the third index vector Q x,y , based on the third index vector Q x,y , in the first index vector and the second index vector Match the most relevant vector in the N×N neighborhood of x,y,(N) ; Evaluate the relevant vector set C x,y,(N) The largest element in the , get the position index of the element; according to the position index, all the skeleton texture blocks Mapped to the third index vector Q x,y On the top, restore the details of the interpolation result to obtain the interpolation result.

5. The high-resolution video frame insertion method based on texture mapping according to claim 3, characterized in that: The Grid_Sample function is expressed by the following formula: Among them, B0 represents the texture block, and mode=Nearest represents the Nearest mode.

6. The high-resolution video frame insertion method based on texture mapping according to claim 1, characterized in that: The interpolation result, the first texture block and the second texture block Reconstruction is performed to obtain an interpolation result; the method includes: respectively performing interpolation on the interpolation result and the first texture block and the second texture block Supervision is performed to transform the latent space into the image space and obtain the interpolation result.

7. A high-resolution video interpolation system based on texture mapping, characterized in that: include: The guide stream upsampling module is configured to: based on the first frame I0 and the second frame I1 of the video stream, use a motion estimator to obtain a first estimated stream and the second estimation stream The texture mapping module is configured to: distort the first frame I0 and the second frame I1, and predict the skeleton based on the deformed first frame I0 and the second frame I1 by using multi-layer convolution; and extract the first texture block from the first frame I0 and the second frame I1 and the second texture block Map the skeleton to the selected texture and get the interpolation result; A reconstruction module is configured to: perform interpolation on the interpolation result, the first texture block and the second texture block Reconstruct and obtain the interpolation result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the high-resolution video interpolation method based on texture mapping as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the high-resolution video interpolation method based on texture mapping as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in the texture mapping-based high-resolution video interpolation method as claimed in any one of claims 1 to 6 are implemented.

Images