A block-based frame extrapolation method and program product

Abstract

The application discloses a block-based frame extrapolation method and a program product, and the method comprises the following steps: constructing a demodulation irradiance based on a rendered image frame at a reference time, and acquiring a geometry buffer, a motion template buffer and a shadow buffer at the reference time; generating a geometry mask and a shadow mask at a target time based on the motion template buffer and the shadow buffer; performing adaptive frame warping on the demodulation irradiance by using the geometry mask at the target time to obtain an adaptive warping result; dividing the adaptive warping result, the geometry buffer and the geometry mask into a plurality of blocks, and selecting blocks with a coverage ratio of the geometry mask greater than a threshold and storing the blocks in a corresponding selected block set; performing parallel inference on the selected block set to obtain a block-level residual result, and backfilling the residual result to obtain a repaired demodulation irradiance; performing re-shading on a shadow mask coverage area, and synthesizing a prediction frame by combining the re-shading result with the repaired demodulation irradiance. The application has high running efficiency and low resource consumption.

Description

Technical Field

[0001] This invention relates to computer graphics technology, and more particularly to a block-based frame extrapolation method and program product. Background Technology

[0002] With the development of computer graphics, using artificial intelligence for frame generation (frame interpolation or extrapolation) has become an important means to improve the frame rate of real-time rendering. Existing frame extrapolation techniques such as ExtraNet and ExtraSS are based on convolutional neural networks (CNNs) to process the entire frame image to predict and generate new intermediate or future frames from existing frames. However, these methods allocate computational resources to every pixel in the entire image equally. In real-world scenarios, there is a large amount of spatial-temporal similarity between adjacent frames, and most pixels overlap in consecutive frames after motion vector reprojection or remapping guided by a g-buffer. Only a few regions (such as occluded background areas, parallax-induced hidden areas, and areas with drastic changes in shadows and highlights caused by dynamic objects) show significant differences. Existing full-frame CNN methods do not optimize computation for this sparse difference, resulting in a large amount of computational power being wasted on areas that do not need repair, leading to high computational costs. At high resolution settings, the overhead of this full-frame processing makes real-time applications very difficult; the situation is even more severe on mobile devices, where performance and power consumption constraints force them to abandon CNN modules altogether in order to achieve real-time frame rates.

[0003] Therefore, existing technologies suffer from insufficient operating efficiency and high resource consumption in high-resolution and real-time frame generation on mobile devices, and there is an urgent need for a new frame extrapolation method that can adaptively allocate computing resources based on the sparsity of inter-frame variations. Summary of the Invention

[0004] To address the problems existing in the prior art, the purpose of this invention is to provide a block-based frame extrapolation method and program product with high operating efficiency and low resource consumption.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0006] A block-based frame extrapolation method includes the following steps:

[0007] (1) Construct the demodulation irradiance of the reference time based on the rendered image frame at the reference time, and obtain the geometry buffer, motion template buffer and shadow buffer at the reference time.

[0008] (2) Based on the motion template buffer and shadow buffer of the reference time, generate the geometric mask and shadow mask of the target time, wherein the geometric mask represents the geometrically unreliable region and the shadow mask represents the shadow temporally inconsistent region;

[0009] (3) Use the geometric mask of the target time to perform adaptive frame warping on the demodulation irradiance of the reference time to obtain the adaptive warping result of the target time;

[0010] (4) Divide the adaptive warping result, geometric buffer and geometric mask at the target time into a set of blocks to be finely repaired according to the preset block size, and store the blocks in which the coverage ratio of the selected geometric mask in each block is greater than the threshold.

[0011] (5) Perform parallel reasoning on the set of blocks to be finely repaired to obtain block-level residual results, and backfill the residual results to complete the fine repair of geometrically unreliable regions, and obtain the repaired demodulation irradiance;

[0012] (6) Perform recoloring on the shadow mask coverage area and synthesize the recoloring result with the repaired demodulated irradiance to obtain the final prediction frame.

[0013] Furthermore, the method for generating the geometric mask in step (2) includes:

[0014] (2.1.1) Reproject the motion template buffer, geometric buffer, and world coordinate position at the reference time to the target time to obtain the reprojected motion template buffer, reprojected geometric buffer, and reprojected world coordinate position;

[0015] (2.1.2) Construct a geometric mask based on the consistency of motion template, normal, and position:

[0016]

[0017] in, The geometric mask representing the target time. =1 indicates the pixel at the target time. This location belongs to a geometrically unreliable region. The motion template buffer represents the target time. This indicates a reprojection motion template buffer. and These represent the target time normal and the reprojection normal, respectively. and These represent the target world coordinate position and the reprojected world coordinate position at the target time, respectively. Represents the vector dot product. Describing the Euclidean norm, The threshold for normal similarity. This is the location distance threshold.

[0018] Furthermore, the method for generating the shadow mask in step (2) includes:

[0019] (2.2.1) Extract the binary shadow map of each point light source at each pixel from the shadow buffer at the reference time, and perform clipping based on the attenuation radius:

[0020]

[0021] in, , These represent the reference time point light sources, respectively. In pixels The clipped binary shadow image and the original binary shadow image at the location. For indicator functions; Pixels at reference time Corresponding world coordinates; Point light source for reference time World coordinates; Point light source attenuation radius, Represents a set of point light sources;

[0022] (2.2.2) Extract the binary shadow map of the upward light for each pixel from the shadow buffer at the reference time, and combine it with the clipped binary shadow maps of all point light sources to calculate the shadow buffer update value of that pixel at the reference time:

[0023]

[0024] in, , Representing the pixels at the reference time respectively The shadow buffer update value at the location, and the binary shadow map of the directional light. Represents a logical OR operation;

[0025] (2.2.3) Reproject the shadow buffer update value of each pixel at the reference time to the target time to obtain the reprojected shadow buffer, calculate the difference between the shadow buffer at the reference time, perform pixel dilation on the difference result, and obtain the shadow mask:

[0026]

[0027] in, Represents pixels Shadow mask at the location, Represents pixels There is a potential risk of shadow lag or shadow inconsistency, where n represents the boundary of the number of pixels dilated. Represents pixels The number of pixels that are dilated. , respectively represent the pixel at the reference time. Shadow buffer at the location, Represents the pixel at the target time. Reprojection shadow buffer at the location.

[0028] Furthermore, step (3) specifically includes:

[0029] (3.1) Calculate the reprojection sampling position for each pixel based on the motion vector from the target time to the reference time:

[0030]

[0031] in, Represents pixels The reprojection sampling position, Represents pixels The motion vector from the target time to the reference time;

[0032] (3.2) Rendered image frame at the reference time A candidate sample set is constructed around each pixel, and the candidate samples with confidence scores higher than a threshold are weighted and fused to obtain the geometrically buffered guided distorted irradiance.

[0033] (3.3) Adaptive selection of the distortion method based on geometric mask to obtain adaptive distortion result:

[0034]

[0035] in, Represents pixels The adaptive distortion result at the location, Represents pixels The geometric mask value at that location, Rendered image frame for reference time The effective pixel coordinate range Indicates to The distortion result obtained by bilinear inverse distortion Represents pixels The geometric buffer at the location guides the torsional irradiance.

[0036] Furthermore, step (3.2) specifically includes:

[0037] (3.2.1) A candidate sample set is generated around each pixel in the screen space using a hierarchical sampling mode:

[0038]

[0039] in, Represents pixels The candidate sampling set, Integer sampling index, For the sampling radius parameter, For layer step size, A hierarchical step size set;

[0040] (3.2.2) For each candidate pixel in the candidate sampling set, assign a fusion weight, which is obtained by multiplying the geometric buffer similarity term and the spatial attenuation term:

[0041]

[0042] in, Indicates candidate pixels The fusion weight, Represents the pixel at the target time. Geometric buffer, Represents candidate pixels at reference time Geometric buffer; It is a Gaussian function. Let represent the mean and variance of the Gaussian function, respectively. For spatial decay term, , This represents the 2-norm (Euclidean norm). Hyperparameters that define the effective range of linear decay;

[0043] (3.2.3) For each pixel A set is formed by selecting a preset number of candidate pixels with the largest fusion weight from its candidate sampling set. ; and for the candidate sampling set The fusion weights within the range are normalized to obtain normalized weights. ;

[0044] (3.2.4) The geometrically buffered distorted irradiance is obtained by weighted summation of the demodulation irradiance at the reference time using normalized weights:

[0045]

[0046] in, Represents candidate pixels at reference time Demodulated irradiance.

[0047] Furthermore, step (4) specifically includes:

[0048] (4.1) The geometric buffer, geometric mask, and adaptive warping results at the target time are arranged according to the side length. The non-overlapping square blocks are divided into several blocks; where the height of the geometric buffer, geometric mask, or adaptive warp result is... or width Cannot be During division, boundary padding is applied to the geometric buffer, geometric mask, or adaptive warp result at the target time. Then it was divided, among which and for Integer multiples of;

[0049] (4.2) For each geometric buffer block, geometric mask block, and adaptive warping result block at the target time, calculate the coverage ratio of the block covered by the geometric mask. When the coverage ratio exceeds the threshold, When that happens, add the block as a selection block to the corresponding selection block set.

[0050] Furthermore, step (5) specifically includes:

[0051] (5.1) From the selection block set of the geometric buffer and the selection block set of the geometric mask, find the geometric buffer selection block and the geometric mask selection block corresponding to each adaptive warp result selection block in the selection block set of the adaptive warp result. Input the adaptive warp result selection block and the corresponding geometric buffer selection block and geometric mask selection block into the trained block repair network to obtain the block-level residual. The block repair network includes a first encoder branch, a second encoder branch, and a decoder branch. The first encoder branch extracts features from the concatenated geometric mask selection block and adaptive warp result selection block. The second encoder branch extracts features from the geometric buffer selection block. The decoder branch fuses and decodes the features output by the first and second encoder branches, outputting a block-level residual. ;

[0052] (5.2) Obtain the position of each adaptive warp result selection block in screen space, and convert the corresponding block-level residuals. The residual is then backfilled to the corresponding position in screen space to obtain the full-resolution residual. ;

[0053] (5.3) Obtain the position of each geometric mask selection block in screen space, fill the geometric mask selection blocks into the corresponding positions in screen space, and obtain the full-resolution geometric mask. ;

[0054] (5.4) Apply repair to the geometric mask coverage area based on the full-resolution residual to obtain the repaired demodulated irradiance. :

[0055]

[0056] in, This indicates element-wise multiplication.

[0057] Furthermore, the loss function used during the training of the block repair network is:

[0058]

[0059]

[0060]

[0061] In the formula, Indicates the total loss. , Indicates the weighting coefficient. Indicates a void reconstruction item. Represents the residual regularization term. , Rendered image frame representing the reference time The effective pixel coordinate range Indicates the true irradiance, Describing the L1 norm, express In pixels The value at that location, express In pixels The value at that location, express In pixels The value at that location.

[0062] Furthermore, step (6) specifically includes:

[0063] (6.1) Shadow Mask Pixels marked as 1 are recolored, and the recoloring result is denoted as... ;

[0064] (6.2) Based on the albedo at the reference time The albedo at the target time is obtained by reprojection. ;

[0065] (6.3) Utilizing the albedo at the target time The repaired demodulated irradiance The color space is modulated back and combined with the recoloring result using a shadow mask to obtain the final predicted frame. :

[0066]

[0067] in, Indicates the shadow mask. This indicates element-wise multiplication.

[0068] A computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0069] Compared with the prior art, the beneficial effects of this invention are:

[0070] The block-based frame extrapolation method proposed in this invention has significant performance advantages and practical value compared to existing full-frame CNN methods. First, by performing neural network inference only on a select few image blocks, the number of pixels to be processed and the amount of convolutional computation are greatly reduced, achieving a frame generation speed several times faster than existing methods under the same hardware environment. Experimental tests show that at high resolutions such as 720p, this method generates intermediate frames significantly faster than existing algorithms, while maintaining comparable peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) image quality metrics with no significant visual loss. Second, this invention greatly reduces the demands on device computing power and memory bandwidth, making it particularly suitable for resource-constrained platforms such as mobile GPUs. Even on mobile devices, it can run high-quality frame extrapolation at near real-time frame rates, solving the problem of existing technologies being difficult to deploy on mobile devices. Furthermore, by introducing a shadow buffer-guided recoloring module, this method can handle shadow motion in dynamic lighting scenes, ensuring that generated frames maintain correct consistency even in complex scenes containing moving shadows, without the shadow lag artifacts common in traditional interpolation methods, further improving the realism and stability of the image. In summary, this invention significantly improves the efficiency and applicability of frame extrapolation while ensuring image quality, and can better meet the needs of application scenarios such as high resolution, real-time rendering and mobile terminals. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a block-based frame extrapolation method provided in an embodiment of the present invention;

[0072] Figure 2 This is a structural diagram of the block repair network TRNet provided in an embodiment of the present invention;

[0073] Figure 3 This is a comparison diagram of the frame generation effect of the method provided in this embodiment of the invention with other methods;

[0074] Figure 4 This is a time comparison diagram of the frame generation method provided in this embodiment of the invention and other methods. Detailed Implementation

[0075] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0076] Example 1

[0077] This invention provides a block-based frame extrapolation method. Before frame extrapolation, preprocessing is required, specifically including obtaining relevant data such as the geometry buffer (G-buffer) and rendering colors of the current and previous frames. Figure 1 As shown, the embodiments of the present invention include the following steps:

[0078] (1) Construct the demodulation irradiance of the reference time based on the rendered image frame at the reference time, and obtain the geometry buffer, motion template buffer and shadow buffer at the reference time.

[0079] This step specifically includes:

[0080] (1.1) At the reference time Get rendered frames and obtain with Corresponding albedo (or material color) information ;

[0081] (1.2) Based on and Demodulated irradiance at the reference time The demodulated irradiance is used to reduce the interference of material albedo on learning / repair, and can be represented, for example, as pixel-by-pixel division:

[0082]

[0083] in, This indicates element-wise division;

[0084] (1.3) At the target time Obtain the geometry buffer at the target time (a multi-channel buffer containing at least one of the following: normal, depth, roughness, metallicity, albedo, etc., denoted as ). The system includes a motion template buffer and a shadow buffer for shadow determination. The motion template buffer is a binary image that marks whether the object corresponding to each pixel is a moving object. The shadow buffer includes the binary shadow map of all point light sources at each pixel and the binary shadow map of the upward light at each pixel.

[0085] In this embodiment, the target frame that needs to be output at the target time is denoted as... Unless otherwise specified, the symbols below refer to the target time by default. .

[0086] (2) Generate the geometric mask and shadow mask of the target time based on the motion template buffer and shadow buffer of the reference time.

[0087] Among them, the geometric mask represents geometrically unreliable regions, specifically unreliable pixels in reprojection correspondence caused by motion, disocclusion, or moving boundaries; the shadow mask represents regions with inconsistent projection shadow timing caused by dynamic occlusions.

[0088] Specifically, the geometric mask The generation methods include:

[0089] (2.1.1) Reproject the motion template buffer, geometric buffer, and world coordinate position at the reference time to the target time to obtain the reprojected motion template buffer, reprojected geometric buffer, and reprojected world coordinate position;

[0090] (2.1.2) Construct a geometric mask based on the consistency of motion template, normal, and position:

[0091]

[0092] in, The geometric mask representing the target time. =1 indicates the pixel at the target time. This location belongs to a geometrically unreliable region. The motion template buffer represents the target time. This indicates a reprojection motion template buffer. and These represent the target time normal and the reprojection normal, respectively. and These represent the target world coordinate position and the reprojected world coordinate position at the target time, respectively. Represents the vector dot product. Describing the Euclidean norm, The threshold for normal similarity. This is a location distance threshold. Pixels that meet any of these conditions are designated as invalid pixels, meaning they are considered to be located in geometrically unreliable regions. (Use superscript) Indicates "from" Reprojection to (”).

[0093] Specifically, the method for generating the shadow mask includes:

[0094] (2.2.1) Extract the binary shadow map of each point light source at each pixel from the shadow buffer at the reference time, and perform clipping based on the attenuation radius:

[0095]

[0096] in, , These represent the reference time point light sources, respectively. In pixels The clipped binary shadow image and the original binary shadow image at the location. This is an indicator function; its value is 1 if the condition is true, and 0 otherwise. Pixels at reference time Corresponding world coordinates; Point light source for reference time World coordinates; Point light source attenuation radius, Represents a set of point light sources; clipping can reduce the invalid markers and subsequent recoloring costs introduced by point light sources outside of their finite attenuation range.

[0097] (2.2.2) Extract the binary shadow map of the upward light for each pixel from the shadow buffer at the reference time, and combine it with the clipped binary shadow maps of all point light sources to calculate the shadow buffer update value of that pixel at the reference time:

[0098]

[0099] in, , Representing the pixels at the reference time respectively The shadow buffer update value at the location, and the binary shadow map of the directional light. Represents a logical OR operation;

[0100] (2.2.3) Reproject the shadow buffer update value of each pixel at the reference time to the target time to obtain the reprojected shadow buffer, calculate the difference between the shadow buffer at the reference time, perform pixel dilation on the difference result, and obtain the shadow mask:

[0101]

[0102] in, Represents pixels Shadow mask at the location, Represents pixels There is a potential risk of shadow lag or inconsistency, requiring recoloring in subsequent steps. Here, 'n' represents the boundary of the number of pixels dilated. Represents pixels The number of pixels that are dilated. , respectively represent the pixel at the reference time. Shadow buffer at the location, Represents the pixel at the target time. Reprojected shadow buffering at the location. Dilation of the difference results can enhance robustness near soft shadow boundaries.

[0103] (3) Use the geometric mask of the target time to perform adaptive frame warping on the demodulation irradiance of the reference time to obtain the adaptive warping result of the target time.

[0104] This step specifically includes:

[0105] (3.1) Calculate the reprojection sampling position for each pixel based on the motion vector from the target time to the reference time:

[0106]

[0107] in, Represents pixels The reprojection sampling position, Represents pixels The motion vector from the target time to the reference time;

[0108] (3.2) Rendered image frame at the reference time A candidate sample set is constructed around each pixel, and the candidate samples with confidence scores higher than a threshold are weighted and fused to obtain the geometrically buffered guided distorted irradiance.

[0109] (3.3) Adaptive selection of the distortion method based on geometric mask to obtain adaptive distortion result:

[0110]

[0111] in, Represents pixels The adaptive distortion result at the location, Represents pixels The geometric mask value at that location, Rendered image frame for reference time The effective pixel coordinate range Indicates to The distortion results obtained by using traditional bilinear inverse distortion Represents pixels The geometric buffer at the location guides the torsional irradiance.

[0112] Step (3.2) specifically includes:

[0113] (3.2.1) Adopting a class The -trous hierarchical sampling mode generates a set of candidate samples around each pixel in screen space:

[0114]

[0115] in, Represents pixels The candidate sampling set, Integer sampling index, For the sampling radius parameter, For layer step size, For a hierarchical step size set, the acceptable values ​​at 720P resolution are... At other resolutions, the step size is scaled proportionally to the screen size to improve coverage of large areas where occlusion is removed.

[0116] (3.2.2) For each candidate pixel in the candidate sampling set, assign a fusion weight, which is obtained by multiplying the geometric buffer similarity term and the spatial attenuation term:

[0117]

[0118] in, Indicates candidate pixels The fusion weight, Represents the pixel at the target time. Geometric buffer, Represents candidate pixels at reference time Geometric buffer; It is a Gaussian function used to measure similarity in the geometric buffer feature space. Let represent the mean and variance of the Gaussian function, respectively. This is a spatial attenuation term used to suppress the influence of excessively distant candidate pixels on the fusion result. , This represents the 2-norm (Euclidean norm). Hyperparameters that define the effective range of linear decay;

[0119] (3.2.3) For each pixel From its candidate sampling set, select a set of candidate pixels with the largest fusion weights, a preset number (e.g., 4 candidate pixels). ; and for the candidate sampling set The fusion weights within the range are normalized to obtain normalized weights. ;

[0120] (3.2.4) The geometrically buffered distorted irradiance is obtained by weighted summation of the demodulation irradiance at the reference time using normalized weights:

[0121]

[0122] in, Represents candidate pixels at reference time Demodulated irradiance.

[0123] (4) Divide the adaptive distortion result, geometric buffer and geometric mask at the target time into a set of blocks to be finely repaired according to the preset block size, and store the blocks whose coverage ratio of the selected geometric mask in each block is greater than the threshold.

[0124] This step specifically includes:

[0125] (4.1) Geometric buffering of the target time Geometric mask and adaptive distortion results According to the side length The non-overlapping square blocks are divided to obtain several geometric buffer blocks, geometric mask blocks, and adaptive warping result blocks; among them, when the height of the geometric buffer, geometric mask, or adaptive warping result is... or width Cannot be During division, boundary padding is applied to the geometric buffer, geometric mask, or adaptive warp result at the target time. Then it was divided, among which and for Integer multiples of;

[0126] (4.2) For each geometric buffer block, geometric mask block, and adaptive warping result block at the target time, calculate the coverage ratio of the block covered by the geometric mask. When the coverage ratio exceeds the threshold, When this happens, the block is added as a selection block to the corresponding selection block set, as shown in the following formula:

[0127]

[0128]

[0129]

[0130] , , These are the selection blocks for the geometric buffer, geometric mask, and adaptive warping results, respectively. These represent the number of blocks in the selected block set for the geometric buffer, geometric mask, and adaptive warp result, respectively. This indicates a block selection operator used to return the corresponding set of selected blocks.

[0131] (5) Perform parallel reasoning on the selected block set of adaptive distortion results, geometric buffer and geometric mask to obtain block-level residual results, and backfill the residual results to complete the fine repair of geometrically unreliable regions and obtain the repaired demodulation irradiance.

[0132] This step specifically includes:

[0133] (5.1) From the selection block set of the geometric buffer and the selection block set of the geometric mask, find the geometric buffer selection block and the geometric mask selection block corresponding to each adaptive warp result selection block in the selection block set of the adaptive warp result. Input the adaptive warp result selection block and the corresponding geometric buffer selection block and geometric mask selection block into the trained block repair network to obtain the block-level residual. ;

[0134] Among them, such as Figure 2 As shown, the block repair network includes a first encoder branch, a second encoder branch, and a decoder branch. The first encoder branch is used to extract features after concatenating the geometric mask block and the adaptive warping result block. The second encoder branch is used to extract features from the geometric buffer block. The decoder branch is used to fuse and decode the features output by the first encoder branch and the second encoder branch, and output the block-level residual. The first encoder branch includes three cascaded encoders and two convolutional layers. The second encoder branch has the same structure as the first encoder branch, but a different size. The outputs of the encoders at the same level as the first encoder branch and the second encoder branch are concatenated to form a concatenated feature for decoding. The decoder branch includes a feature fusion layer, three cascaded decoders, and two convolutional layers.

[0135] (5.2) Obtain the position of each adaptive warp result selection block in screen space, and convert the corresponding block-level residuals. The residual is then backfilled to the corresponding position in screen space to obtain the full-resolution residual. ;

[0136] (5.3) Obtain the position of each geometric mask selection block in screen space, fill the geometric mask selection blocks into the corresponding positions in screen space, and obtain the full-resolution geometric mask. :

[0137] (5.4) Apply repair to the geometric mask coverage area based on the full-resolution residual to obtain the repaired demodulated irradiance. :

[0138]

[0139] in, This represents element-wise multiplication. This design ensures that the network does not introduce unnecessary changes in geometrically reliable regions, thereby improving temporal stability.

[0140] The loss function used during the training of the block repair network is a joint loss that includes a hole reconstruction term and a residual regularization term:

[0141]

[0142]

[0143]

[0144] In the formula, Indicates the total loss. , Indicates the weighting coefficient. Indicates a void reconstruction item. Represents the residual regularization term. , Rendered image frame representing the reference time The effective pixel coordinate range Indicates the true irradiance, Describing the L1 norm, express In pixels The value at that location, express In pixels The value at that location, express In pixels The value at that location.

[0145] (6) Perform recoloring on the shadow mask coverage area and synthesize the recoloring result with the repaired demodulated irradiance to obtain the final prediction frame.

[0146] This step specifically includes:

[0147] (6.1) Shadow Mask Pixels marked as 1 are recolored, and the recoloring result is denoted as... This is used to explicitly handle timing inconsistencies caused by shadow changes;

[0148] (6.2) Based on the albedo at the reference time The albedo at the target time is obtained by reprojection. ;

[0149] (6.3) Utilizing the albedo at the target time The repaired demodulated irradiance The color space is modulated back and combined with the recoloring result using a shadow mask to obtain the final predicted frame. :

[0150]

[0151] in, Indicates the shadow mask. This indicates element-wise multiplication. The selection mask represents the stable shadow region; the selection mask is used directly in the unstable shadow region. Replace the corresponding pixels in the network output to avoid brightness jumps and ghosting artifacts caused by shadow changes.

[0152] Finally, the invention was simulated and verified, specifically using an RTX 4090 graphics card and a 720P resolution for testing. Figure 3 The images show a comparison of rendering results using the present invention. The larger image is the rendering result of the present invention. The comparison images on the right, from left to right, represent the ExtraNet frame extrapolation method, ExtraSS frame extrapolation method, Mob-FGSR frame extrapolation method, the present invention's method, and a comparison of the actual rendered frame. It can be seen that the present invention does not suffer from significant visual loss. Figure 4 The comparison chart of frame generation time between this method and other methods shows that this method takes significantly less time and has a clear advantage in terms of time.

[0153] Example 2

[0154] This invention also provides a computer program product, such as an app on a mobile phone or tablet, or an installer on a computer. This product includes a computer program / instructions that, when executed by a processor, implement the method described in Embodiment 1. The code for the computer-executable program used to perform the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] It should be understood that the embodiments and descriptions above are only the principles, main features and advantages of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the invention, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A block-based frame extrapolation method, characterized in that, Includes the following steps: (1) Construct the demodulation irradiance of the reference time based on the rendered image frame at the reference time, and obtain the geometry buffer, motion template buffer and shadow buffer at the reference time. (2) Based on the motion template buffer and shadow buffer of the reference time, generate the geometric mask and shadow mask of the target time, wherein the geometric mask represents the geometrically unreliable region and the shadow mask represents the shadow temporally inconsistent region; (3) Use the geometric mask of the target time to perform adaptive frame warping on the demodulation irradiance of the reference time to obtain the adaptive warping result of the target time; (4) Divide the adaptive warping result, geometric buffer and geometric mask at the target time into several blocks according to the preset block size, and select the blocks whose coverage ratio of the geometric mask is greater than the threshold and store them in the corresponding selection block set; (5) Perform parallel reasoning on the selected block set of adaptive distortion results, geometric buffer and geometric mask to obtain block-level residual results, and backfill the residual results to complete the fine repair of geometrically unreliable regions and obtain the repaired demodulation irradiance; (6) Perform recoloring on the shadow mask coverage area and synthesize the recoloring result with the repaired demodulated irradiance to obtain the final prediction frame.

2. The block-based frame extrapolation method according to claim 1, characterized in that, The method for generating the geometric mask in step (2) includes: (2.1.1) Reproject the motion template buffer, geometric buffer, and world coordinate position at the reference time to the target time to obtain the reprojected motion template buffer, reprojected geometric buffer, and reprojected world coordinate position; (2.1.2) Construct a geometric mask based on the consistency of motion template, normal, and position: , in, The geometric mask representing the target time. =1 indicates the pixel at the target time. This location belongs to a geometrically unreliable region. The motion template buffer represents the target time. This indicates a reprojection motion template buffer. and These represent the target time normal and the reprojection normal, respectively. and These represent the target world coordinate position and the reprojected world coordinate position at the target time, respectively. Represents the vector dot product. Describing the Euclidean norm, The threshold for normal similarity. This is the location distance threshold.

3. The block-based frame extrapolation method according to claim 1, characterized in that, The method for generating the shadow mask in step (2) includes: (2.2.1) Extract the binary shadow map of each point light source at each pixel from the shadow buffer at the reference time, and perform clipping based on the attenuation radius: , in, , These represent the reference time point light sources, respectively. In pixels The clipped binary shadow image and the original binary shadow image at the location. For indicator functions; Pixels at reference time Corresponding world coordinates; Point light source for reference time World coordinates; Point light source attenuation radius, Represents a set of point light sources; (2.2.2) Extract the binary shadow map of the upward light for each pixel from the shadow buffer at the reference time, and combine it with the clipped binary shadow maps of all point light sources to calculate the shadow buffer update value of that pixel at the reference time: , in, , Representing the pixels at the reference time respectively The shadow buffer update value at the location, and the binary shadow map of the directional light. Represents a logical OR operation; (2.2.3) Reproject the shadow buffer update value of each pixel at the reference time to the target time to obtain the reprojected shadow buffer, calculate the difference between the shadow buffer at the reference time, perform pixel dilation on the difference result, and obtain the shadow mask: , in, Represents pixels Shadow mask at the location, Represents pixels There is a potential risk of shadow lag or shadow inconsistency, where n represents the boundary of the number of pixels dilated. Represents pixels The number of pixels that are dilated. , respectively represent the pixel at the reference time. Shadow buffer at the location, Represents the pixel at the target time. Reprojection shadow buffer at the location.

4. The block-based frame extrapolation method according to claim 1, characterized in that, Step (3) specifically includes: (3.1) Calculate the reprojection sampling position for each pixel based on the motion vector from the target time to the reference time: , in, Represents pixels The reprojection sampling position, Represents pixels The motion vector from the target time to the reference time; (3.2) Rendered image frame at the reference time A candidate sample set is constructed around each pixel, and the candidate samples with confidence scores higher than a threshold are weighted and fused to obtain the geometrically buffered guided distorted irradiance. (3.3) Adaptive selection of the distortion method based on geometric mask to obtain adaptive distortion result: , in, Represents pixels The adaptive distortion result at the location, Represents pixels The geometric mask value at that location, Rendered image frame at reference time The effective pixel coordinate range Indicates to The distortion result obtained by bilinear inverse distortion Represents pixels The geometric buffer at the location guides the torsional irradiance.

5. The block-based frame extrapolation method according to claim 4, characterized in that, Step (3.2) specifically includes: (3.2.1) A candidate sample set is generated around each pixel in the screen space using a hierarchical sampling mode: , in, Represents pixels The candidate sampling set, Integer sampling index, For the sampling radius parameter, For layer step size, A hierarchical step size set; (3.2.2) For each candidate pixel in the candidate sampling set, assign a fusion weight, which is obtained by multiplying the geometric buffer similarity term and the spatial attenuation term: , in, Indicates candidate pixels The fusion weight, Represents the pixel at the target time. Geometric buffer, Represents candidate pixels at reference time Geometric buffer; It is a Gaussian function. Let represent the mean and variance of the Gaussian function, respectively. For spatial decay term, , This represents the 2-norm (Euclidean norm). Hyperparameters that define the effective range of linear decay; (3.2.3) For each pixel A set is formed by selecting a preset number of candidate pixels with the largest fusion weight from its candidate sampling set. ; and for the candidate sampling set The fusion weights within the range are normalized to obtain normalized weights. ; (3.2.4) The geometrically buffered distorted irradiance is obtained by weighted summation of the demodulation irradiance at the reference time using normalized weights: , in, Represents candidate pixels at reference time Demodulated irradiance.

6. The block-based frame extrapolation method according to claim 1, characterized in that, Step (4) specifically includes: (4.1) The geometric buffer, geometric mask, and adaptive warping results at the target time are arranged according to the side length. The non-overlapping square blocks are divided into several blocks; where the height of the geometric buffer, geometric mask, or adaptive warp result is... or width Cannot be During division, boundary padding is applied to the geometric buffer, geometric mask, or adaptive warp result at the target time. Then it was divided, among which and for Integer multiples of; (4.2) For each geometric buffer block, geometric mask block, and adaptive warping result block at the target time, calculate the coverage ratio of the block covered by the geometric mask. When the coverage ratio exceeds the threshold, When that happens, add the block as a selection block to the corresponding selection block set.

7. The block-based frame extrapolation method according to claim 1, characterized in that, Step (5) specifically includes: (5.1) From the selection block set of the geometric buffer and the selection block set of the geometric mask, find the geometric buffer selection block and the geometric mask selection block corresponding to each adaptive warp result selection block in the selection block set of the adaptive warp result. Input the adaptive warp result selection block and the corresponding geometric buffer selection block and geometric mask selection block into the trained block repair network to obtain the block-level residual. The block repair network includes a first encoder branch, a second encoder branch and a decoder branch. The first encoder branch is used to extract features after concatenating the geometric mask selection block and the adaptive warp result selection block. The second encoder branch is used to extract features from the geometric buffer selection block. The decoder branch is used to fuse and decode the features output by the first encoder branch and the second encoder branch to output the block-level residual. ; (5.2) Obtain the position of each adaptive warp result selection block in screen space, and convert the corresponding block-level residuals. The residual is then backfilled to the corresponding position in screen space to obtain the full-resolution residual. ; (5.3) Obtain the position of each geometric mask selection block in screen space, fill the geometric mask selection blocks into the corresponding positions in screen space, and obtain the full-resolution geometric mask. ; (5.4) Apply repair to the geometric mask coverage area based on the full-resolution residual to obtain the repaired demodulated irradiance. : , in, This indicates element-wise multiplication.

8. The block-based frame extrapolation method according to claim 7, characterized in that, The loss function used during the training of the block repair network is: , , , In the formula, Indicates the total loss. , Indicates the weighting coefficient. Indicates a void reconstruction item. Represents the residual regularization term. , Rendered image frame representing the reference time The effective pixel coordinate range Indicates the true irradiance, Describing the L1 norm, express In pixels The value at that location, express In pixels The value at that location, express In pixels The value at that location.

9. The block-based frame extrapolation method according to claim 1, characterized in that, Step (6) specifically includes: (6.1) Shadow Mask Pixels marked as 1 are recolored, and the recoloring result is denoted as... ; (6.2) Based on the albedo at the reference time Reprojection yields the albedo at the target time. ; (6.3) Utilizing the albedo at the target time The repaired demodulated irradiance The color space is modulated back and combined with the recoloring result using a shadow mask to obtain the final predicted frame. : , in, Indicates the shadow mask. This indicates element-wise multiplication.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1-9.

Images