Real-time image super-resolution completion method and system

Abstract

The application provides a real-time image super-completion method and system, relates to the fields of high-fidelity rendering completion and super-completion, and specifically comprises the following steps: acquiring a sparse image, double motion vectors, a traditional motion vector, auxiliary features and a high-resolution completion result of a previous frame of a current frame; calculating warped re-projection of the previous frame based on the double motion vectors of the current frame and the high-resolution completion result of the previous frame; re-weighting historical frame features extracted from the warped re-projection of the previous frame based on the sparse image of the current frame; splicing the re-weighted historical frame features, the sparse image of the current frame and the auxiliary features into a tensor, inputting the tensor, the double motion vectors and the traditional motion vector into a space-time multi-scale super-completion network, and obtaining a high-resolution completion result of the current frame; and performing real-time super-completion on the obtained sparse low-resolution image to obtain a high-resolution, complete, artifact-free and high-quality completion image.

Description

Technical Field

[0001] This invention belongs to the field of highly realistic rendering completion and super-resolution technology, and particularly relates to a real-time image super-resolution completion method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Due to the increased display resolution and strict frame rate limitations, real-time realistic rendering in virtual reality headsets remains a significant challenge. Foveated rendering, through techniques such as adaptive resolution, geometric simplification, and shading simplification, adjusts the rendering quality of the image based on the gaze point, maintaining high-quality rendering in the gaze area while reducing image quality in the edge areas of the gaze. This technique significantly saves computation and improves rendering speed without a noticeable loss of visual quality.

[0004] In recent years, several image completion methods based on foveated rendering have been proposed. These methods perform sparse rendering of images based on foveated information, reconstructing reasonable peripheral regions from a small subset of pixels provided by each sparse image frame. Current image completion methods generally employ direct prediction, directly converting the last layer of the neural network into an image as the output. However, the unconstrained nature and complexity of the problem make optimization difficult. The variance of calculating stochastic gradients during training can be large, slowing down convergence and making it difficult to achieve good results. Furthermore, most super-resolution methods inevitably lead to inter-frame flicker when incorporating historical frame information for temporal reuse. This is because for motion-occluded regions, as moving objects move, the traditional motion vectors always point to the moving foreground object from the previous frame, resulting in non-reusable temporal information and severe ghosting. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a real-time image super-resolution completion method and system, which performs real-time super-resolution completion on the obtained sparse low-resolution image to obtain a high-resolution, complete, artifact-free, high-quality completed image.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a real-time image super-resolution completion method.

[0007] A real-time image super-resolution completion method includes: Obtain the sparse image, dual motion vectors, traditional motion vectors, auxiliary features, and high-resolution completion results of the previous frame for the current frame; Based on the dual motion vectors of the current frame and the high-resolution completion result of the previous frame, calculate the warped reprojection of the previous frame. Based on the sparse image of the current frame, the historical frame features extracted from the warped reprojection of the previous frame are reweighted. The reweighted historical frame features, the sparse image of the current frame, and auxiliary features are concatenated into a tensor, which is then combined with dual motion vectors and traditional motion vectors and input into a spatiotemporal multi-scale super-resolution completion network to obtain the high-resolution completion result of the current frame. The spatiotemporal multi-scale super-resolution completion network calculates pixel-wise kernel weights and hybrid weights at spatiotemporal multi-scales, applies the generated motion mask to the sparse image of the current frame and the distortion reprojection of the previous frame at different scales, and outputs a high-resolution, complete, and artifact-free completed image.

[0008] Furthermore, the dual motion vector is represented as ( ), and These are the offsets of the pixels in the current frame relative to the previous frame along the x and y axes, respectively. The auxiliary features include the albedo, depth, and normal of the current frame.

[0009] Furthermore, the specific method for calculating the warped reprojection of the previous frame is as follows: Create a two-dimensional tensor to store pixel coordinates and add it directly to the dual motion vectors to obtain the pixel coordinate mapping relationship between the current frame and the previous frame; Using the pixel coordinate mapping relationship, the high-resolution completion result of the previous frame is quickly aligned with the current frame to obtain the distortion reprojection of the previous frame.

[0010] Furthermore, the reweighting specifically refers to: Concatenate the features of historical frames with the depth information of the current frame into a tensor; Based on the tensor, the per-pixel weight of the distortion reprojection of the previous frame is calculated and applied to the historical frame features to obtain the reweighted historical frame features.

[0011] Furthermore, the spatiotemporal multi-scale super-resolution completion network includes a kernel prediction network, a motion mask generation module, and a spatiotemporal multi-scale hybrid network; The pixel-by-pixel kernel weight includes a pixel-by-pixel spatial kernel and a pixel-by-pixel temporal kernel; The hybrid weights include time-mixed weights and inter-layer hybrid weights.

[0012] Furthermore, the kernel prediction network calculates a pixel-wise spatial kernel, a pixel-wise temporal kernel, a temporal mixing weight, and an inter-layer mixing weight based on the input tensor. The motion mask generation module calculates the artifact region of the current frame based on the dual motion vector and the traditional motion vector of the current frame, and binarizes it to obtain the motion mask. The spatiotemporal multi-scale mixing network applies the pixel-wise spatial kernel to the sparse image of the current frame and uses the inter-layer mixing weights to perform a weighted summation to obtain a spatial multi-scale mixing result; it applies the pixel-wise temporal kernel to the warp reprojection of the previous frame to obtain a temporal filtering result; and it uses mask-based temporal mixing weights to perform a weighted summation of the spatial multi-scale mixing result and the temporal filtering result to obtain a spatiotemporal multi-scale mixing result, which is then used as the high-resolution completion result of the current frame.

[0013] Furthermore, the loss function of the spatiotemporal multi-scale super-resolution completion network consists of spatial loss, temporal loss, perceptual loss, and SSIM loss.

[0014] A second aspect of the present invention provides a real-time image super-resolution completion system.

[0015] A real-time image super-resolution completion system includes a data acquisition module, a distortion reprojection module, a feature weighting module, and a super-resolution completion module. The data acquisition module is configured to acquire the sparse image, dual motion vectors, traditional motion vectors, auxiliary features, and high-resolution completion results of the previous frame for the current frame. The warp reprojection module is configured to: calculate the warp reprojection of the previous frame based on the dual motion vectors of the current frame and the high-resolution completion result of the previous frame; The feature weighting module is configured to: reweight the historical frame features extracted from the warped reprojection of the previous frame based on the sparse image of the current frame; The super-resolution completion module is configured to: concatenate the reweighted historical frame features, the sparse image of the current frame, and auxiliary features into a tensor, and input the tensor, along with the dual motion vector and the traditional motion vector, into the spatiotemporal multi-scale super-resolution completion network to obtain the high-resolution completion result of the current frame. The spatiotemporal multi-scale super-resolution completion network calculates pixel-wise kernel weights and hybrid weights at spatiotemporal multi-scales, applies the generated motion mask to the sparse image of the current frame and the distortion reprojection of the previous frame at different scales, and outputs a high-resolution, complete, and artifact-free completed image.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a real-time image super-resolution completion method as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a real-time image super-resolution completion method as described in the first aspect of the present invention.

[0018] The above one or more technical solutions have the following beneficial effects: This invention provides a method for real-time sparse image completion and super-resolution, which can simultaneously perform real-time completion and super-resolution on input sparse images, significantly reducing the amount of rendering computation without a significant decrease in visual quality.

[0019] This invention employs a spatiotemporal multi-scale kernel prediction architecture, which applies a pixel-wise spatial kernel to a multi-scale incomplete image and uses inter-layer mixing weights for weighted summation to obtain a spatial multi-scale mixing result. Simultaneously, in the temporal reprojection stage, a dual motion vector method is used to obtain a reasonable previous frame distorted reprojection image, and a pixel-wise temporal kernel is applied to the previous frame distorted reprojection to obtain a temporal filtering result.

[0020] This invention introduces an effective motion mask that can fully utilize the semantic information of traditional motion vectors and dual motion vectors to calculate and generate motion masks for artifact regions, explicitly identify occlusion regions, effectively utilize historical information, and achieve better inter-frame stability.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the method in the first embodiment.

[0024] Figure 2 This is a schematic diagram of the conventional motion vector and dual motion vector in the first embodiment.

[0025] Figure 3 This is a schematic diagram illustrating the generation and application of motion mask in the first embodiment.

[0026] Figure 4 A comparison image showing the application of motion masking to the first embodiment.

[0027] Figure 5 This is a comparison chart of the results of the first embodiment and the integrated super-resolution method.

[0028] Figure 6 This is a comparison chart of the results of the first embodiment and the two-stage completion super-resolution method. Detailed Implementation

[0029] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Example 1 In one or more embodiments, a real-time image super-resolution completion method is disclosed, which consists of four main stages: data acquisition stage, distortion reprojection stage, feature weighting stage, and super-resolution completion stage, as shown in the flowchart below. Figure 1 As shown, the specific steps include: Step S1: Data acquisition stage, acquire the sparse image of the current frame, dual motion vectors, traditional motion vectors, auxiliary features, and high-resolution completion results of the previous frame.

[0032] The process involves acquiring the current frame image to be completed from a virtual reality headset. These original images are typically sparse and low-resolution, and a high-resolution, complete, and artifact-free high-quality completed image needs to be generated.

[0033] This invention utilizes the high-resolution completion result of the previous frame to calculate the warped reprojection for subsequent pixel-by-pixel weighted mixing. Therefore, the high-resolution completion result of the previous frame is obtained from the recurrent feedback loop of the spatiotemporal multi-scale super-resolution completion network. This refers to the high-resolution completed image of the previous frame.

[0034] Matrix operations are used to obtain the complete traditional motion vector and dual motion vector of the current frame without performing full rendering; the specific calculation method for the dual motion vector (DMV) is as follows: Obtain the low-resolution, two-dimensional, time-reliable dual motion vectors of the current frame. The two dimensions are the sparse image resolution of the current frame (192×256) and the value represents the difference between the pixel position of the object surface where the shading point is located in the current frame and the corresponding pixel position in the previous frame. and These are the offsets of the pixels in the current frame relative to the previous frame along the x and y axes, respectively; then, the motion vector is upsampled to the same resolution as the output image.

[0035] Auxiliary features include the albedo, depth, and normal of the current frame, which can be calculated in the rendering process based on the scene's geometry and material information.

[0036] Step S2: Warp reprojection stage, based on the dual motion vectors of the current frame ( (and the high-resolution completion result of the previous frame) Calculate the warp reprojection of the previous frame .

[0037] By using the final completion result of the previous frame as input, and performing pixel-wise mapping on this input using time-reliable dual motion vectors, a more reasonable distorted reprojection image of the previous frame aligned with the current frame is quickly generated. This image is then used as input to the spatiotemporal multi-scale super-resolution completion network and for subsequent pixel-wise weighted mixing, specifically: Create a two-dimensional tensor to store pixel coordinates and add it directly to the obtained dual motion vectors to obtain the pixel coordinate mapping relationship between the current frame and the previous frame. According to formula (1), use the grid_sample function and bilinear interpolation mode in the PyTorch framework, and use the pixel coordinate mapping obtained above to complete the high-resolution result of the previous frame. Quickly align with the current frame to obtain the warped reprojection of the previous frame. .

[0038] (1) in, , These are the horizontal and vertical coordinates of pixels, respectively.

[0039] Using dual motion vectors for reprojection will produce different results in occluded areas compared to using traditional motion vectors (SVGF motion vectors), such as... Figure 2 As shown, traditional motion vectors closely fit the edges of moving objects, while dual motion vectors have gaps in the opposite direction of the occluded object's movement. Therefore, in the motion occlusion area, the former (SVGF motion vectors) creates a repeated pasting pattern of moving objects, and their colors are often very different from the background. The latter (dual motion vectors) reuses background information to solve the problem of excessive color difference, further reducing the possibility of trailing caused by reusing historical frame results.

[0040] Step S3: Feature weighting stage. Based on the sparse image of the current frame, the historical frame features extracted from the warp reprojection of the previous frame are reweighted. The specific steps are as follows: (1) Downsample the distortion reprojection of the previous frame to the sparse image resolution size of the current frame (192×256). (2) Extract features from the downsampled warped reprojection to obtain historical frame features; (3) Concatenate the depth information in the historical frame features and auxiliary features into a tensor, calculate the weight of each pixel of the distortion reprojection of the previous frame, and multiply it with the historical frame features to remove invalid pixels and obtain the reweighted historical frame features.

[0041] Step S4: In the super-resolution completion stage, the reweighted historical frame features, the sparse image of the current frame, and auxiliary features are concatenated into a tensor. The tensor is then combined with dual motion vectors and traditional motion vectors and input into the spatiotemporal multi-scale super-resolution completion network to obtain the high-resolution completion result of the current frame.

[0042] A spatiotemporal multi-scale super-resolution completion network based on the PyTorch framework is used. The sparse image and auxiliary features of the current frame are combined with the reweighted historical frame features obtained in step S3 as input. The spatiotemporal multi-scale pixel-wise kernel weights and hybrid weights are trained through the spatiotemporal multi-scale super-resolution completion network. The generated motion mask is applied to the sparse image of the current frame and the distortion reprojection of the previous frame at different scales, and the high-resolution, complete, and artifact-free high-quality completed image is output.

[0043] The spatiotemporal multi-scale super-resolution completion network includes a kernel prediction network, a motion mask generation module, and a spatiotemporal multi-scale hybrid network. (1) Nuclear prediction network The sparse image of the current frame, auxiliary features (albedo, normal, depth), and the reweighted historical frame features obtained in step S3 are concatenated along the channel dimension to form a tensor of (B, N, 192, 256), which is then fed into the kernel prediction network to extract features from the input tensor. Once the resolution of the network feature layer reaches the input resolution, two more deformable convolutional super-division blocks are added to the traditional U-Net to achieve super-resolution. Compared to traditional convolution, the deformable convolutional layers used in the deformable convolutional super-division blocks can learn the offset of the original sampling position, thus incorporating more details. The formula for deformable convolution is as follows: (2) Where O is the offset set, S is the square sample region, and x is the current feature. and These are the learnable offset and modulation scalar for the nth sampling point, respectively.

[0044] Thus, in the last three layers of the kernel prediction network, pixel-wise kernel weights and hybrid weights (including pixel-wise spatial kernels, pixel-wise temporal kernels, temporal hybrid weights, and inter-layer hybrid weights) of corresponding image sizes (1×, 2×, 4×) will be output respectively. Figure 1 The details of the kernel prediction network are shown. This network has an encoder-decoder structure and, apart from two deformable convolutional super-blocks, uses 3×3 kernel-sized convolutional layers and a ReLU activation function. The first half of the network uses 2×2 max pooling for downsampling, while the second half uses bilinear interpolation for upsampling. Skip connections are used between layers of the same size, forming a symmetrical U-shaped network structure. The two super-blocks immediately following this U-shaped structure use 3×3 kernel-sized deformable convolutional layers, an ELU activation function, a bilinear upsampling layer, and a 1×1 convolutional layer for residual mapping.

[0045] The kernel prediction network consists of three output layers: the first output layer, the last output layer (768, 1024, 33×3), includes a 4×4 pixel spatial kernel, a 4×4 pixel temporal kernel, and one channel of temporal mixing weights. The second output layer, the penultimate output layer (384, 512, 17×3), includes a 4×4 pixel spatial kernel and one channel of inter-layer mixing weights. The third output layer, the third-to-last output layer (192, 256, 17×3), includes a 4×4 pixel spatial kernel and one channel of inter-layer mixing weights. Here, 3 represents the number of RGB channels.

[0046] (2) Motion Mask Generation Module The main function is to generate motion masks and mask artifact areas; for example... Figure 3 The unreliable artifact region is obtained by subtracting the dual motion vector from the traditional motion vector; after upsampling the unreliable artifact region to the target's high resolution, it is binarized with a threshold of 0.1 to obtain the motion mask; such as Figure 4 This motion mask can explicitly mask artifact areas to eliminate the effects of artifacts in historical frames.

[0047] (3) Spatiotemporal multi-scale hybrid network The main functions are spatial hierarchical mixing and temporal mixing. The spatiotemporal multi-scale mixing network applies the three-layer pixel-wise spatial kernel obtained by the kernel prediction network to sparse images of corresponding sizes (1×, 2×, 4×), and uses inter-layer mixing weights to weighted summation to obtain the spatial multi-scale mixing result. At the same time, the pixel-wise temporal kernel is applied to the warp reprojection of the previous frame to obtain the temporal filtering result. After applying the motion mask to the temporal mixing weights, the spatial multi-scale mixing result and the temporal filtering result are weighted summation using the temporal mixing weights to obtain the spatiotemporal multi-scale mixing result, which is then used as the high-resolution completion result of the current frame.

[0048] For spatial hierarchical mixing, as shown in formula (3), for each inter-layer mixing, c is the relatively coarse scale, f is the relatively fine scale, D is 2×2 max pooling downsampling, and U is bilinear upsampling. The result is the filtered image after applying the kernels. Three 4×4 pixel-wise spatial kernels obtained from the kernel prediction network are applied to sparse image inputs of different sizes: input size 1, input image size 2, and input image size 4, respectively, to obtain the filtered image. and use interlayer hybrid weights Weighted summation yields spatial multi-scale mixing results. .

[0049] Specifically, the third output layer and the second output layer are first blended. For this blending, the sparse graph of the second output layer (384, 512, 3) with a size of 2 × the input graph is a relatively fine scale, while the sparse graph of the third output layer (192, 256, 3) with a size of 1 × the input graph is a relatively coarse scale. The spatial kernel (384, 512, 16 × 3) obtained from the penultimate output layer of the kernel prediction network is applied to the sparse graph (384, 512, 3) of the second output layer to obtain... Similarly, applying the (192, 256, 16×3) obtained from the penultimate output layer of the kernel prediction network to the sparse graph (192, 256, 3) of the third output layer yields... Finally, the interlayer mixed weights (192, 256, 1) obtained from the penultimate output layer of the kernel prediction network are used. Blending is required (since the interlayer blending weight resolution is at a relatively coarse scale, it is necessary to perform blending). First, downsample and apply weights, then upsample; for After directly applying the weights, upsampling is required. Then, the mixture of the third and second output layers is blended with the first output layer in a similar hierarchical manner; the formula is as follows: (3) Where p is the pixel index at a relatively fine scale. The value at pixel p represents the result of filtering at a relatively fine scale. For filtering results at a relatively fine scale Obtained after downsampling Applying interlayer mixing weights with a relatively coarse scale Then, it undergoes upsampling. Restored to a fine scale The value of pixel p. Filtering results at a relatively coarse scale Interlayer mixed weights applying a relatively coarse system Then, upsampling is used to obtain a relatively fine-scale resolution. The value of pixel p. Spatial hierarchical blending stage yields spatial multi-scale blending results.

[0050] For temporal mixing, a 4×4 pixel-wise temporal kernel obtained from the kernel prediction network is applied to the warped reprojection obtained in step S2. Obtain the time filtering result The obtained motion mask is then applied to the temporal mixed weights obtained by the kernel prediction network. Using the masked temporal blending weights, the temporal filtering result is... Results of spatial multi-scale mixing Weighted summation yields spatiotemporal multi-scale mixing results. (That is, the high-resolution completion result of the current frame); the formula is as follows: (4) In this embodiment, the spatiotemporal multi-scale super-resolution completion network is constructed and trained using the PyTorch framework and the Adam optimizer, with an initial learning rate of 0.0001 and 100 generations per training iteration. The reference image, auxiliary feature caches (albedo, depth, and normal), and two motion vectors are obtained using the Unity framework.

[0051] The graphics processor used in this embodiment is an NVIDIA GeForce RTX 3090; the dataset is obtained from four scenes, each scene containing multiple sequences, each sequence consisting of 100 frames; since the obtained reference images are high-resolution (768×1024) complete images, they need to be preprocessed to obtain training and testing sets; after downsampling the data to a low resolution (192×256) using 4×4, the gaze mask probability function is generated using formula (5), where Let x be the probability that pixel p at position x is not occluded. For the focal point, it is randomly generated using a two-dimensional uniform distribution function; each pixel is uniformly sampled, and the obtained value is compared with the corresponding mask probability. Thresholding is performed to generate a gaze mask, which is then applied to the low-resolution full image to obtain a low-resolution sparse image.

[0052] (5) During training, loss functions are used to determine the performance of the network or model, thereby identifying directions for optimization and training. Using appropriate loss functions can improve the training performance of the network or model. In this embodiment, the loss functions of the spatiotemporal multi-scale super-resolution completion network are L1 spatial loss, L1 temporal loss, and perceptual loss. The sum of the SSIM loss and the specific formula is as follows: (6) (7) (8) (9) in, These represent the network output image and the reference image, respectively.

[0053] Based on the high-resolution completion result of the current frame, L1 spatial loss and L1 temporal loss are applied. = - , = - The perceptual loss and SSIM loss are calculated, backpropagated, and the high-resolution completion result of the current frame is stored as the high-resolution completion result of the previous frame in the next iteration. For use in step S1.

[0054] The completion and super-resolution method in this embodiment can be performed in real time, with an average completion and super-resolution time of 15.78 ms per frame, which can be subdivided into four time-consuming parts: warp reprojection (0.43 ms), feature weighting (0.17 ms), kernel prediction (14.84 ms), and spatiotemporal multi-scale mixing (0.34 ms).

[0055] Figure 5 and Figure 6 The results of this embodiment are compared with those of the integrated completion and super-resolution method and the two-stage completion and super-resolution method. Figure 5 and Figure 6 It can be seen that, compared with the prior art, the numerical values ​​and results of this embodiment are significantly improved, and the problems of excessive blurring, color distortion and trailing that exist in the prior art are significantly eliminated.

[0056] This invention applies a spatiotemporal multi-scale kernel prediction network to real-time sparse images to simultaneously perform real-time incomplete and super-resolution, obtaining high-quality, high-resolution incomplete results. It reconstructs and oversamples sparse edge regions through spatiotemporal multi-scale kernel prediction, significantly reducing rendering computation without noticeable degradation in visual quality. Multi-scale kernel prediction preserves detail information at different levels. An effective motion mask is introduced to explicitly identify motion-occluded and artifact regions, facilitating more efficient use of the previous frame's distorted reprojection image. This invention achieves better image quality and inter-frame stability than existing methods and can run in real-time.

[0057] Example 2 In one or more embodiments, a real-time image super-resolution completion system is disclosed, including a data acquisition module, a distortion reprojection module, a feature weighting module, and a super-resolution completion module: The data acquisition module is configured to acquire the sparse image, dual motion vectors, traditional motion vectors, auxiliary features, and high-resolution completion results of the previous frame for the current frame. The warp reprojection module is configured to: calculate the warp reprojection of the previous frame based on the dual motion vectors of the current frame and the high-resolution completion result of the previous frame; The feature weighting module is configured to: reweight the historical frame features extracted from the warped reprojection of the previous frame based on the sparse image of the current frame; The super-resolution completion module is configured to: concatenate the reweighted historical frame features, the sparse image of the current frame, and auxiliary features into a tensor, and input the tensor, along with the dual motion vector and the traditional motion vector, into the spatiotemporal multi-scale super-resolution completion network to obtain the high-resolution completion result of the current frame. The spatiotemporal multi-scale super-resolution completion network calculates pixel-wise kernel weights and hybrid weights at spatiotemporal multi-scales, applies the generated motion mask to the sparse image of the current frame and the distortion reprojection of the previous frame at different scales, and outputs a high-resolution, complete, and artifact-free completed image.

[0058] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0059] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a real-time image super-resolution completion method as described in Embodiment 1 of this disclosure.

[0060] Example 4 The purpose of this embodiment is to provide an electronic device.

[0061] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a real-time image super-resolution completion method as described in Embodiment 1 of this disclosure.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time image super-resolution completion method, characterized in that, include: Obtain the sparse image, dual motion vectors, traditional motion vectors, auxiliary features, and high-resolution completion results of the previous frame for the current frame; Based on the dual motion vectors of the current frame and the high-resolution completion result of the previous frame, calculate the warped reprojection of the previous frame. Based on the sparse image of the current frame, the historical frame features extracted from the warped reprojection of the previous frame are reweighted. The reweighted historical frame features, the sparse image of the current frame, and auxiliary features are concatenated into a tensor, which is then combined with dual motion vectors and traditional motion vectors and input into a spatiotemporal multi-scale super-resolution completion network to obtain the high-resolution completion result of the current frame. The spatiotemporal multi-scale super-resolution completion network includes a kernel prediction network, a motion mask generation module, and a spatiotemporal multi-scale hybrid network. The kernel prediction network calculates pixel-wise spatial kernels, pixel-wise temporal kernels, temporal hybrid weights, and inter-layer hybrid weights based on the input tensor. Two deformable convolutional super-resolution blocks are added to the traditional U-Net to achieve super-resolution. The pixel-wise kernel weights and hybrid weights of the spatiotemporal multi-scale network are calculated, and the generated motion mask is applied to the sparse image of the current frame and the warped reprojection of the previous frame at different scales to output a high-resolution, complete, and artifact-free completed image. The motion mask generation module calculates the artifact region of the current frame based on the dual motion vector and the traditional motion vector of the current frame, and binarizes it to obtain the motion mask. The spatiotemporal multi-scale mixing network applies the pixel-wise spatial kernel to the sparse image of the current frame and uses the inter-layer mixing weights to perform weighted summation to obtain the spatial multi-scale mixing result; it applies the pixel-wise temporal kernel to the warp reprojection of the previous frame to obtain the temporal filtering result; and it uses mask-based temporal mixing weights to perform weighted summation of the spatial multi-scale mixing result and the temporal filtering result to obtain the spatiotemporal multi-scale mixing result, which is then used as the high-resolution completion result of the current frame. The deformable convolutional superblock uses a 3×3 kernel-sized deformable convolutional layer, an ELU activation function, a bilinear upsampling layer, and a 1×1 convolutional layer for residual mapping. The formula for deformable convolution is as follows: ； Where O is the offset set, S is the square sample region, and x is the current feature. and These are the learnable offset and modulation scalar for the nth sampling point, respectively.

2. The real-time image super-resolution completion method as described in claim 1, characterized in that, The dual motion vector is represented as ( ), and These are the offsets of the pixels in the current frame relative to the previous frame along the x and y axes, respectively. The auxiliary features include the albedo, depth, and normal of the current frame.

3. The real-time image super-resolution completion method as described in claim 1, characterized in that, The specific method for calculating the warped reprojection of the previous frame is as follows: Create a two-dimensional tensor to store pixel coordinates and add it directly to the dual motion vectors to obtain the pixel coordinate mapping relationship between the current frame and the previous frame; Using the pixel coordinate mapping relationship, the high-resolution completion result of the previous frame is quickly aligned with the current frame to obtain the distortion reprojection of the previous frame.

4. The real-time image super-resolution completion method as described in claim 1, characterized in that, The reweighting is specifically as follows: Concatenate the features of historical frames with the depth information of the current frame into a tensor; Based on the tensor, the per-pixel weight of the distortion reprojection of the previous frame is calculated and applied to the historical frame features to obtain the reweighted historical frame features.

5. The real-time image super-resolution completion method as described in claim 1, characterized in that, The pixel-by-pixel kernel weight includes a pixel-by-pixel spatial kernel and a pixel-by-pixel temporal kernel; The hybrid weights include time-mixed weights and inter-layer hybrid weights.

6. The real-time image super-resolution completion method as described in claim 1, characterized in that, The loss function of the spatiotemporal multiscale super-resolution completion network consists of spatial loss, temporal loss, perceptual loss, and SSIM loss.

7. A real-time image super-resolution completion system, employing the real-time image super-resolution completion method as described in any one of claims 1-6, characterized in that, It includes a data acquisition module, a warp reprojection module, a feature weighting module, and a super-resolution completion module: The data acquisition module is configured to acquire the sparse image, dual motion vectors, traditional motion vectors, auxiliary features, and high-resolution completion results of the previous frame for the current frame. The warp reprojection module is configured to: calculate the warp reprojection of the previous frame based on the dual motion vectors of the current frame and the high-resolution completion result of the previous frame; The feature weighting module is configured to: reweight the historical frame features extracted from the warped reprojection of the previous frame based on the sparse image of the current frame; The super-resolution completion module is configured to: concatenate the reweighted historical frame features, the sparse image of the current frame, and auxiliary features into a tensor, and input the tensor, along with the dual motion vector and the traditional motion vector, into the spatiotemporal multi-scale super-resolution completion network to obtain the high-resolution completion result of the current frame. The spatiotemporal multi-scale super-resolution completion network calculates pixel-wise kernel weights and hybrid weights at spatiotemporal multi-scales, applies the generated motion mask to the sparse image of the current frame and the distortion reprojection of the previous frame at different scales, and outputs a high-resolution, complete, and artifact-free completed image.

8. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-6.

9. A storage medium characterized in that it non-transitory stores computer-readable instructions, wherein, When the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-6 are executed.

Images