3DRS and pyramid model fused optical flow calculation method and system for real-time rendering frame insertion

Abstract

The invention discloses a 3D RS and pyramid model fused optical flow calculation method for real-time rendering frame insertion, and the method comprises the steps: 1, obtaining the information of each texture in a rendering frame, so as to construct a texture pyramid; step 2, selecting a grey-scale map corresponding to the lowest resolution to estimate an optical flow initial value; step 3, generating an optimal up-sampling optical flow according to the optical flow initial value, and updating the optimal up-sampling optical flow in a neighborhood to obtain an updated optical flow; 4, repeating the step 3 to obtain the up-sampling optical flow with the highest resolution; 5, performing fine adjustment on the up-sampling optical flow under the highest resolution; step 6, calculating motion vectors of target pixels in the previous rendering frame and the current rendering frame based on the rendering pipeline, and generating an optimal motion vector in combination with the up-sampling optical flow after fine tuning of the target pixel block; and 7, reconstructing a final intermediate frame image. The invention also provides an optical flow calculation system. The method provided by the invention can effectively solve the problem of wrong sampling of real-time rendering frame insertion.

Description

Technical Field

[0001] This invention belongs to the fields of computer graphics and real-time rendering technology, and particularly relates to an optical flow calculation method and system for real-time rendering frame interpolation that fuses 3DRS and pyramid models. Background Technology

[0002] In recent years, with the increasing demands on hardware computing power and power consumption of mobile real-time rendering programs, frame generation (frame interpolation) technology has gradually become a research hotspot in the field of real-time rendering. Thanks to the existence of the rendering pipeline, the frame interpolation process in the rendering field can utilize intermediate data generated by the rendering engine. For example, it can use depth maps to determine information related to unocclusion, use the pixel correspondence of motion vectors to predict the pixel position of intermediate frames, or use information in other G-Buffers to help the model predict information of the target frame.

[0003] In the industry, several relatively mature frameworks for real-time rendering frame interpolation have emerged. NVIDIA's DLSS 3 relies on hardware-accelerated optical flow estimation and utilizes neural networks to achieve high-quality frame interpolation effects on PCs. However, DLSS's frame interpolation scheme is designed for PCs, and the computational overhead involved in neural networks makes it difficult to achieve real-time effects on mobile devices. Furthermore, the hardware optical flow acceleration on PCs also introduces platform limitations, making it difficult to fully deploy on mobile devices.

[0004] The academic paper "Mob-FGSR: Frame Generation and Super Resolution for Mobile Real-Time Rendering" (In ACM SIGGRAPH 2024 Conference Papers, 2024) proposes Mob-FGSR, a novel lightweight super-resolution frame interpolation framework for real-time rendering on mobile devices, achieving excellent optimization results. However, in terms of frame interpolation, relying solely on motion vectors for image deformation, and the inability of motion vectors to represent pixel movement in areas such as shadows, reflections, and semi-transparent areas, leads to incorrect pixel sampling results in these regions.

[0005] AMD's FSR 3 employs a hierarchical optical flow estimation algorithm to address the issue that motion vectors cannot represent pixel motion. However, it uses a full search approach for optical flow estimation at each resolution level. While FSR 3 offers high real-time performance on PCs, the full search at high resolutions represents a significant overhead for mobile devices. Furthermore, FSR 3 relies on median filtering to constrain the spatial consistency of optical flow, but this operation results in a noticeable loss of accuracy in changing regions such as edges.

[0006] The academic paper Haan GD, Biezen PWAC, Huijgen H, et al. True Motion Estimation with 3-D Recursive Search Block Matching[J].IEEE Transactions on Circuits and Systems for Video Technology, 1993, 3(5):368-379, 388.DOI:10.1109 / 76.246088. discloses a scheme to obtain spatiotemporally consistent strong optical flow results by iteratively updating specific neighboring optical flows and historical optical flows. However, its drawback is that when the motion changes are slightly complex, the convergence speed of the iterative update becomes significantly slower, and the iterative process is inefficient for GPUs.

[0007] The academic paper Heinrich A, Bartels C, René J. van der Vleuten, et al. Optimization of Hierarchical 3DRS Motion Estimators for Picture Rate Conversion[J]. IEEE Journal of Selected Topics in Signal Processing, 2011, 5(2):262-274.DOI:10.1109 / JSTSP.2010.2063014. discloses a scheme that combines 3DRS with a hierarchical architecture. Although it gives 3DRS the ability to handle large-scale motion, it still cannot avoid multiple iterative updates because each layer still executes conventional 3DRS. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for optical flow calculation that fuses 3DRS and pyramid models for real-time rendering frame interpolation. This method can effectively solve the problem of incorrect sampling in areas such as shadows, reflections, and semi-transparencies in Mob-FGSR.

[0009] To achieve the first objective of this invention, the following technical solution is provided: a method for calculating optical flow by fusing 3DRS and pyramid models for real-time rendering frame interpolation, comprising the following steps: Step 1: Obtain the lighting-related texture information from the rendering pipeline and fuse the lighting-related texture information into multi-texture information in a single channel; The multi-texture information is downsampled sequentially based on resolutions set from high to low to output grayscale images at different resolutions and arrange each grayscale image according to the corresponding resolution to construct a texture pyramid corresponding to the current rendering frame and the previous rendering frame. Step 2: Select the grayscale image corresponding to the lowest resolution and use the full search block matching method to estimate the initial optical flow values ​​of the corresponding pixel blocks of the previous rendering frame and the current rendering frame. Step 3: Based on the initial value of optical flow and the grayscale image of the current resolution in the texture pyramid, construct a candidate optical flow set for the corresponding pixel block by the position of the target upsampled pixel block, and use similarity matching to select the optimal upsampled optical flow corresponding to the target pixel block; The optimal upsampled optical flow is updated within the neighborhood to output the corresponding updated upsampled optical flow; Step 4: Repeat step 3, sampling from the lowest resolution to the highest resolution, without updating at the highest resolution, in order to obtain the highest resolution upsampled optical flow. Step 5: Fine-tune the upsampling optical flow at the highest resolution using the 3DRS method; Step 6: Calculate the motion vectors of the target pixels in the previous rendering frame and the current rendering frame based on the rendering pipeline, and combine them with the upsampled optical flow after fine-tuning the target pixel block output in Step 5 to generate the corresponding optimal motion vectors. Step 7: Based on the optimal motion vector, use vector splashing to reconstruct the bidirectional motion vector of the intermediate frame; Based on the bidirectional motion vector, the images corresponding to the previous rendering frame and the current rendering frame are reverse-deformed to the intermediate frame, and the corresponding pixels are selected using depth to obtain the final intermediate frame image.

[0010] Specifically, the texture information includes one or more of the shadows, translucency, or reflections of the current rendered frame.

[0011] Specifically, the multi-texture information is obtained by converting illumination-related texture information into grayscale representation, taking the maximum value of the grayscale corresponding to the illumination-related texture information, and finally merging them to obtain a single-channel texture as the output multi-texture information.

[0012] Specifically, the construction process of the texture pyramid is as follows: Iterative downsampling is performed on the two consecutive grayscale images of the input, resulting in... , , , A grayscale image with a specific resolution.

[0013] Specifically, the steps of the full search block matching method are as follows: A full block search is performed on the textures of the previous and current rendering frames at the lowest resolution, with the search domain set to 24×24 and the pixel block size to 8×8.

[0014] Specifically, the construction process of the candidate optical flow set is as follows: For the four sub-optical flows corresponding to the low-resolution optical flow at high resolution, four candidate parent optical flows are first selected based on the relative position of the sub-optical flows in the parent optical flow. At the same time, based on the limited number of candidate optical flow values ​​at low resolution, a zero vector is introduced as the fifth candidate optical flow. The introduction of the zero vector takes into account the limited number of candidate optical flow values ​​at low resolution.

[0015] Specifically, the optimal upsampling optical flow is selected as follows: For each candidate optical flow, calculate the optical flow corresponding to the low-resolution value. The SAD similarity between the sub-block and the starting pixel block; The optical flow with the highest SAD similarity is selected as the optimal upsampling optical flow.

[0016] Specifically, the updated optical flow is obtained through an improved three-step search method, the steps of which are as follows: Calculate the similarity of four adjacent blocks at the position pointed to by the upsampled optical flow, and select the one with the highest matching degree as the first result; Calculate the similarity between the upsampled optical flow and the four adjacent blocks of the position pointed to by the first result, and select the one with the highest matching degree as the second result; Calculate the similarity between the upsampled optical flow and the four adjacent blocks and the center block at the position pointed to by the second result. Select the one with the highest matching degree as the third result and output the third result as the updated upsampled optical flow.

[0017] Specifically, the fine-tuning involves replacing the historical frame optical flow input of the 3DRS method with the optical flow upsampled to the highest resolution, and further simplifies the iterative update process of 3DRS. The optical flow with the highest similarity matching degree is calculated as the fine-tuning result of the 3DRS method.

[0018] Specifically, step 7 also includes special processing for moving objects and unobstructed areas, the special processing including: The region of moving objects is marked using dynamic object masking extracted from the pipeline. The unocclusion area is calculated using motion vectors and the depth of the preceding and following frames; In the areas of moving objects and unoccluded areas, select only the motion vector and not the optical flow.

[0019] To achieve the second objective of this invention, the following technical solution is provided: an optical flow calculation system for performing the steps of the above-described optical flow calculation method for real-time rendering frame interpolation that fuses 3DRS and pyramid models, comprising: The lighting separation unit is used to extract lighting-related texture information from the rendering pipeline and convert it into multi-texture information in a single channel. The optical flow estimation unit is used to perform motion estimation on the lighting effects of two consecutive frames to obtain the optical flow results corresponding to the multi-texture information. Motion selection unit, used to select motion vectors and / or optical flow results as intermediate frame motion vectors for the final pixel; The inserted frame reconstruction unit reconstructs the image of the inserted frame based on the motion vector of the intermediate frame.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By introducing optical flow estimation and selecting it in motion vector, this invention solves the problem of incorrect frame interpolation in the light-related effects of the original lightweight interpolation frame that only relied on motion vector; (2) Compared with most existing optical flow technologies (including FSR 3) that directly perform optical flow estimation on the grayscale image of the rendered image, this invention proposes to extract the texture buffers that need to be processed from the rendering pipeline, including effects such as shadows, semi-transparency, and reflections. (3) This invention designs an optical flow estimation framework for real-time rendering frame interpolation that is highly optimized for mobile devices. It fully leverages the advantages of full search, three-step search, and 3DRS, and improves them to suit the frame interpolation requirements of mobile devices. At the same time, through hierarchical design, it has a larger matching receptive field at low resolution, enabling it to capture a greater range of pixel motion; (4) This invention proposes to select motion vectors and optical flow on a block-by-block basis to solve the problem of incorrect selection caused by the lack of edge perception under the pixel-level selection strategy. At the same time, by introducing dynamic objects and removing occlusion area markings, motion vectors are used in key areas to obtain more stable frame interpolation results. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the optical flow calculation method for real-time rendering frame interpolation that fuses 3DRS and a pyramid model, as provided in this embodiment. Figure 2 This embodiment provides a flowchart framework for optical flow estimation. Figure 3 The frame interpolation process provided in this embodiment is the process following optical flow calculation; Figure 4 A comparison of the shadow effects of the present invention's solution provided in this embodiment with several other frame interpolation solutions; Figure 5 A comparison of the translucency and reflection effects of the present invention's solution provided in this embodiment with several other frame interpolation solutions. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown in this embodiment, a method for calculating optical flow by fusing 3DRS and pyramid models for real-time rendering frame interpolation is provided. The specific steps are as follows: Step 1: Obtain the lighting-related texture information from the rendering pipeline and fuse the lighting-related texture information into multi-texture information in a single channel; The multi-texture information is downsampled sequentially based on resolutions set from high to low to output grayscale images at different resolutions and arrange each grayscale image according to the corresponding resolution to construct a texture pyramid corresponding to the current rendering frame and the previous rendering frame. Step 2: Select the grayscale image corresponding to the lowest resolution and use the full search block matching method to estimate the initial optical flow values ​​of the corresponding pixel blocks of the previous rendering frame and the current rendering frame. Step 3: Based on the initial value of optical flow and the grayscale image of the current resolution in the texture pyramid, construct a candidate optical flow set for the corresponding pixel block by the position of the target upsampled pixel block, and use similarity matching to select the optimal upsampled optical flow corresponding to the target pixel block; The optimal upsampled optical flow is updated within the neighborhood to output the corresponding updated upsampled optical flow; Step 4: Repeat step 3, sampling from the lowest resolution to the highest resolution, without updating at the highest resolution, in order to obtain the highest resolution upsampled optical flow. Step 5: Fine-tune the upsampling optical flow at the highest resolution using the 3DRS method; Step 6: Calculate the motion vectors of the target pixels in the previous rendering frame and the current rendering frame based on the rendering pipeline, and combine them with the upsampled optical flow after fine-tuning the target pixel block output in Step 5 to generate the corresponding optimal motion vectors. Step 7: Based on the optimal motion vector, use vector splashing to reconstruct the bidirectional motion vector of the intermediate frame; Based on the bidirectional motion vector, the images corresponding to the previous rendering frame and the current rendering frame are reverse-deformed to the intermediate frame, and the corresponding pixels are selected using depth to obtain the final intermediate frame image.

[0024] Furthermore, in the embodiment, each texture information includes multi-channel textures such as reflection and semi-transparency. First, these textures are converted to grayscale representations. Then, the maximum value of multiple grayscale images, including shadow textures, is taken to obtain a single-channel multi-texture information after merging. Finally, the brightness value is cropped to between 0 and 1.

[0025] like Figure 2 As shown in (a), the processed texture is iteratively downsampled to obtain... , , , The resolution texture is used, and the pyramid texture obtained by downsampling two consecutive frames is used as the input for optical flow estimation. In the embodiment, the input is... Resolution (width and height are original) Perform a full search block match on the textures of the two frames before and after the image, with the search domain set to... The size of the search block is Pixel blocks.

[0026] The construction process of the candidate optical flow set is as follows: Figure 2 As shown in (b), for the four sub-optical flows corresponding to the low-resolution optical flow at high resolution, four candidate parent optical flows are first selected based on the relative position of the sub-optical flows in the parent optical flow. Considering that the candidate values ​​of optical flow are limited at low resolution, the 0 vector is introduced as the fifth candidate optical flow. For each candidate optical flow, calculate the optical flow corresponding to the low-resolution value. The SAD similarity between the sub-block and the starting pixel block is calculated as follows: in, Indicates the resolution level, with 0 representing the original resolution. Indicates optical flow. Indicates low resolution The position coordinates of the sub-pixel block, express The positions of all pixels in the pixel block. The coefficient representing the influence of optical flow magnitude; The optical flow with the highest similarity was selected as the upsampling result.

[0027] like Figure 2As shown in (c), the improved three-step search strategy is used to update the upsampled optical flow in the neighborhood. The process is as follows: 1) Calculate the similarity of four adjacent blocks (4-step interval) at the upsampled optical flow pointing position, and select the best matching result. ; 2) Calculate the upsampling optical flow and The similarity of the four adjacent blocks (spaced 4 steps apart) pointing to a location is used to select the best match. ; 3) Calculate the upsampling optical flow and The most matching result is selected based on the similarity between the four adjacent blocks (spaced 4 steps apart) pointing to the location and the center block. .

[0028] For streamer results where diagonal accuracy is critical, the above three-step search strategy can be repeated.

[0029] In this embodiment, targeted modifications were made to the traditional 3DRS algorithm by replacing the historical frame optical flow input of the 3DRS method with the optical flow upsampled to the highest resolution, and further simplifying the iterative update process of 3DRS. The revised optical flow candidate set is as follows: in Indicates the position of the target optical flow; The optical flow with the highest similarity match is calculated as the 3DRS correction result; Since the candidate optical flow is derived from the optical flow results of hierarchical calculation, it is not necessary to perform multiple iterations to convergence in 3DRS; only one optical flow update is required.

[0030] like Figure 3 As shown, this embodiment uses... The similarity matching results of motion vectors and optical flow are calculated separately for each pixel block. The result with higher similarity is selected as the selected motion vector. The specific selection method is as follows: in This represents the mask result. Represents pixel blocks The set of all pixel locations in the set. Represents the motion vector. Indicates optical flow; Masking of dynamic objects extracted from pipelines Mark the area of ​​moving objects: The unocclusion area is calculated using motion vectors and the depth of the preceding and following frames: in, It's a depth map. It is the threshold for judging the depth difference between the foreground and background; In the areas of moving objects and unoccluded areas, select only the motion vector and not the optical flow.

[0031] Simultaneously, the bidirectional motion vector of the intermediate frame is reconstructed using vector splashing. Then, a fine-line repair strategy is used to fill the gaps in the splashing result. After that, the color and depth of the two frames before and after are sampled using the reconstructed motion vector, and the corresponding pixels are selected to reconstruct the target intermediate frame image.

[0032] Furthermore, this embodiment also provides an optical flow computing system, including: The lighting separation unit is used to extract texture information from the rendering pipeline and convert it into a single-channel multimodal texture; The optical flow estimation unit is used to perform motion estimation on the lighting effects of the two consecutive frames separately to obtain the optical flow results corresponding to the multimodal texture; Motion selection unit, used to select motion vectors and / or optical flow results as intermediate frame motion vectors for the final pixel; The inserted frame reconstruction unit reconstructs the image of the inserted frame based on the motion vector of the intermediate frame.

[0033] By using the aforementioned system, and addressing the frame interpolation errors in shadow areas and other regions of existing lightweight frame interpolation solutions for mobile devices, a new optical flow estimation algorithm combining 3DRS and a pyramid model was designed for real-time rendering frame interpolation on mobile devices. This algorithm separates relevant buffers from the rendering pipeline for individual optical flow estimation. The optical flow results are then fused with motion vectors to fully leverage their respective advantages. This results in a final frame interpolation effect that corrects the original sampling errors while maintaining motion accuracy, thereby significantly improving the image quality of frame interpolation on mobile devices.

[0034] like Figure 4The image shows a comparison of the shadow effects of the scheme provided in the above embodiment with other frame interpolation methods. Here, M-FG refers to the result of Mob-FGSR, M-FG+Match refers to the result of adding FSR3 optical flow estimation to Mob-FGSR, and M-FG+SCN refers to applying the shading correction network from the reference (GFFE: G-buffer Free Frame Extrapolation for Low-latency Real-time Rendering, ACM Transactions on Graphics, 2024) to Mob-FGSR. As can be seen from the timing profile in the second row, the method of this invention significantly outperforms the Mob-FGSR method in terms of the timing results of shadow frame interpolation, and also achieves good frame interpolation results compared to other computationally expensive methods that employ neural networks.

[0035] like Figure 5 The figure shown is a comparison diagram between the solution provided in the above embodiment and Mob-FGSR in terms of semi-transparency and reflection effects; it can be seen that the method of the present invention is also superior to the method of Mob-FGSR in terms of timing results of frame interpolation for semi-transparency and reflection effects.

[0036] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0037] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0038] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optical flow computation of fusing 3DRS and pyramid model for real-time rendering of in-between frames, characterized in that, The method comprises the following steps: Step 1, obtaining illumination-related texture information in a rendering frame from a rendering pipeline, and fusing the illumination-related texture information into multi-texture information under a single channel; sequentially downsampling the multi-texture information based on resolutions set from high to low to output gray-scale maps under different resolutions and arranging the gray-scale maps at corresponding resolutions to construct a texture pyramid corresponding to the current rendering frame and the previous rendering frame; Step 2, selecting a gray-scale map corresponding to the lowest resolution, and estimating an initial value of an optical flow of a pixel block corresponding to the previous rendering frame and the current rendering frame by using a full search block matching method; Step 3, based on the initial value of the optical flow and the gray-scale map at the current resolution in the texture pyramid, constructing a candidate optical flow set of the corresponding pixel block by the position of a target up-sampling pixel block, and selecting an optimal up-sampling optical flow corresponding to the target pixel block by using a similarity matching method; updating the optimal up-sampling optical flow in a neighborhood to output a corresponding updated up-sampling optical flow; Step 4, repeating Step 3 from the lowest resolution to the highest resolution, and not updating at the highest resolution to obtain an up-sampling optical flow at the highest resolution; Step 5, fine-tuning the up-sampling optical flow at the highest resolution by using a 3DRS method; Step 6, calculating a motion vector of a target pixel in the previous rendering frame and the current rendering frame based on the rendering pipeline, and combining the fine-tuned up-sampling optical flow of the target pixel block output in Step 5 to generate a corresponding optimal motion vector; Step 7, reconstructing a bidirectional motion vector of an intermediate frame by using a vector splashing method based on the optimal motion vector; deforming the images corresponding to the previous rendering frame and the current rendering frame to the intermediate frame based on the bidirectional motion vector, and selecting corresponding pixels by using a depth to obtain a final intermediate frame image.

2. The optical flow computation method for real-time rendering of in- between frames fusing 3D RS and pyramid model according to claim 1, wherein, The multi-texture information is obtained by converting the illumination-related texture information into a gray-scale map representation, taking the maximum value of the gray-scale map corresponding to the illumination-related texture information, and finally merging a single-channel texture as output multi-texture information.

3. The optical flow computation method for real-time rendering of in- between frames fusing 3D RS and pyramid model according to claim 1, wherein, The construction process of the texture pyramid is as follows: The input two frames of gray scale images are iteratively down-sampled to obtain gray scale images with resolutions of , , , ​ 4. The optical flow computation method for real-time rendering of in- between frames fusing 3D RS and pyramid model according to claim 1, wherein, The steps of the full search block matching method are as follows: performing full search block matching on the textures of the previous rendering frame and the current rendering frame at the lowest resolution, setting the search domain as 24x24, and setting the size of the pixel block as 8x8.

5. The optical flow computation method for fusing 3DRS and pyramid model for real-time rendering of in-between frames according to claim 1, wherein, The construction process of the candidate optical flow set is as follows: for the four sub-optical flows corresponding to the low-resolution optical flow at the high resolution, first selecting four candidate parent optical flows according to the relative positions of the sub-optical flows in the parent optical flows, and introducing a 0 vector as the fifth candidate optical flow based on the limited candidate values of the optical flow at the low resolution.

6. The optical flow computation method for fusing 3DRS and pyramid model for real-time rendering of in-between frames according to claim 1, wherein, The selection method of the optimal up-sampling optical flow is as follows: For each candidate optical flow, the SAD similarity of the sub-blocks and the starting pixel block of the optical flow at low resolution is calculated of the sub-blocks and the starting pixel block of the optical flow at low resolution is calculated selecting the optical flow with the highest SAD similarity as the optimal up-sampling optical flow.

7. The optical flow computation method for fusing 3DRS and pyramid model for real-time rendering of in-between frames according to claim 1, wherein, The updated optical flow is obtained by updating the improved three-step search method, and the steps are as follows: calculating the similarity of the four adjacent blocks at the position pointed by the up-sampling optical flow, and selecting the highest matching degree as the first result; calculating the similarity of the four adjacent blocks at the position pointed by the up-sampling optical flow and the first result, and selecting the highest matching degree as the second result; The similarity of the up-sampled optical flow and four adjacent blocks and the center block of the second result pointing position is calculated, the highest matching degree is selected as the third result, and the third result is output as the updated up-sampled optical flow.

8. The optical flow computation method for fusing 3DRS and pyramid model for real-time rendering of in-between frames according to claim 1, wherein, The fine-tuning replaces the historical frame optical flow input of the 3DRS method with the optical flow up-sampled to the highest resolution, and further simplifies the iterative update process of the 3DRS; And the optical flow with the highest similarity matching degree is calculated as the fine-tuning result of the 3DRS method.

9. The optical flow computation method for fusing 3DRS and pyramid model for real-time rendering of in-between frames according to claim 1, wherein, In step 6, special processing is also included for moving objects and unoccluded areas, which includes: Marking the moving object area with the dynamic object mask extracted in the pipeline; Calculating the unoccluded area using the motion vector and the depth of the previous and next two frames; In the moving object area and the unoccluded area, only the motion vector is selected, not the optical flow.

10. An optical flow computation system, characterized by, The steps of the optical flow calculation method for real-time rendering of the fusion 3DRS and pyramid model for inserting frames, as claimed in any one of claims 1-9, include: An illumination separation unit for extracting illumination-related texture information in the rendering pipeline and converting it into multi-texture information under a single channel; An optical flow estimation unit for separately estimating the motion of the illumination effect of the previous and next two frames to obtain the corresponding optical flow result; A motion selection unit for selecting the motion vector or / and the optical flow result as the intermediate frame motion vector of the final pixel; An inserted frame reconstruction unit for reconstructing the picture of the inserted frame based on the intermediate frame motion vector.

Images