Method, device and equipment for super-resolution of rendering content using auxiliary buffer

Abstract

The present invention discloses a method, apparatus, and device for super-resolution of rendered content using an auxiliary buffer. The method comprises: (1) establishing a multi-scale fusion network model, including an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch; (2) generating a plurality of image pairs including low-definition RGB images and corresponding high-definition G-buffer images as samples in a rendering scene, using the RGB images rendered at high resolution as sample labels, inputting the established multi-scale fusion network model, and performing network training; (3) inputting the low-definition RGB images to be super-resolved and the corresponding high-definition G-buffer images into the trained multi-scale fusion network model to obtain a super-resolved high-definition RGB image. The present invention has better effects.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a method, device and equipment for super-resolution of rendering content using an auxiliary buffer. Background Art

[0002] Image super-resolution is currently one of the most popular research areas in computer graphics and computer vision. Super-resolution for rendered scenes is generally included in temporal / spatial anti-aliasing and content reconstruction, focusing on balancing the appearance performance and computational cost associated with the sampling rate. Within the field of computer vision, there are branches such as single image super-resolution and video super-resolution. These fields primarily focus on photographic images and video streams, and their solutions are not fully applicable to rendered images. The essential difference between the two lies in the sampling point: rendered images sample points at different points in time and space, while photographic images use the area integral of each pixel.

[0003] In recent years, deep learning has made remarkable progress in rendering super-resolution, and many ideas have been applied to this field. The NSRR network, proposed by Facebook Reality Labs in 2020, selects multiple historical frames and their depth and motion vectors for network inference. However, this method, which aims to improve the quality of rendered image super-resolution results, suffers from difficulties in generalization and poor performance in real-world rendering scenarios. Summary of the Invention

[0004] Purpose of the invention: In order to solve the problems existing in the prior art, the present invention provides a method, apparatus, device and storage medium for super-resolution of rendering content using an auxiliary buffer with better effect.

[0005] Technical solution: The method for super-resolution of rendering content using an auxiliary buffer according to the present invention includes:

[0006] (1) Establishing a multi-scale fusion network model, the multi-scale fusion network model includes an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch, wherein the RGB image processing branch is used to extract feature information from a low-definition RGB image and perform zero upsampling, the G-buffer processing branch is used to extract feature information from a high-definition G-buffer image paired with the low-definition RGB image, and the multi-scale fusion prediction branch is used to encode and fuse the feature information extracted by the RGB image processing branch and the G-buffer processing branch at different scales, and then decode and predict the high-definition RGB image after super-resolution;

[0007] (2) Generate several image pairs including low-definition RGB images and corresponding high-definition G-buffer images as samples in the rendering scene, use the RGB images rendered at high resolution as sample labels, input the established multi-scale fusion network model, and perform network training;

[0008] (3) The low-resolution RGB image to be super-resolved and the corresponding high-resolution G-buffer image are input into the trained multi-scale fusion network model to obtain the high-resolution RGB image after super-resolvation.

[0009] Furthermore, the RGB image processing branch in step (1) specifically includes:

[0010] The feature extraction module consists of three convolutional layers connected in sequence;

[0011] The zero upsampling module is used to take the output of the feature extraction module as input and assign each pixel to any pixel position among the corresponding s pixels at high resolution with equal probability, while the remaining s-1 pixel values ​​will be set to 0, where s is the upsampling ratio.

[0012] Furthermore, the G-buffer processing branch in step (1) is specifically a feature extraction module, which includes three convolutional layers connected in sequence.

[0013] Furthermore, the multi-scale fusion prediction branch in step (1) specifically includes:

[0014] The RGB feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence, wherein the first convolution-downsampling module is connected to the RGB image processing branch and includes two connected convolution layers, the second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence, and the third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2;

[0015] A G-buffer feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence, wherein the first convolution-downsampling module is connected to the G-buffer processing branch and includes two connected convolution layers, the second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence, and the third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2;

[0016] The feature fusion part includes a first fusion module, a second fusion module and a third fusion module, wherein the first fusion module is respectively connected to the first convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, the second fusion module is respectively connected to the second convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, and the third fusion module is respectively connected to the third convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, and each fusion module includes a residual unit and a squeeze and excitation unit connected in sequence;

[0017] The predicted RGB image decoding part includes a first convolution module and a second convolution module connected in sequence. The first convolution module is connected to the third fusion module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the second fusion module. The second convolution module is connected to the first convolution module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the first fusion module.

[0018] Furthermore, the loss function used in step (2) for network training is:

[0019]

[0020]

[0021] Where L represents the total loss, y represents the high-definition RGB image predicted by the sample after passing through the multi-scale fusion network model, represents the sample label, Represents the structural similarity index measure between the predicted image and the sample label, μ y Represents the pixel average of the sample label and the predicted image, respectively, Represent the pixel variance of the sample label and the predicted image, respectively, Represents the pixel covariance of the sample label and the predicted image, c1=(k1P) 2 ,c2=(k2P) 2 ,P=255,k1=0.01,k2=0.03,w is the weight coefficient, Represents the learning-aware patch similarity between the predicted image and the sample label.

[0022] The rendering content super-resolution device using an auxiliary buffer according to the present invention comprises:

[0023] A network model establishment module is used to establish a multi-scale fusion network model, wherein the multi-scale fusion network model includes an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch. The RGB image processing branch is used to extract feature information from a low-definition RGB image and perform zero upsampling. The G-buffer processing branch is used to extract feature information from a high-definition G-buffer image paired with the low-definition RGB image. The multi-scale fusion prediction branch is used to encode and fuse the feature information extracted by the RGB image processing branch and the G-buffer processing branch at different scales, and then decode and predict the super-resolved high-definition RGB image.

[0024] The network training module is used to generate several image pairs including low-resolution RGB images and corresponding high-resolution G-buffer images as samples in the rendering scene. The RGB images rendered at high resolution are used as sample labels and input into the established multi-scale fusion network model for network training.

[0025] The image super-resolution module is used to input the low-resolution RGB image to be super-resolved and the corresponding high-resolution G-buffer image into the trained multi-scale fusion network model to obtain the super-resolved high-definition RGB image.

[0026] The rendering content super-resolution device using an auxiliary buffer according to the present invention comprises:

[0027] one or more processors;

[0028] a memory for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0030] The storage medium containing computer-executable instructions of the present invention is characterized in that the computer-executable instructions are used to perform the above method when executed by a computer processor.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0032] 1. The dual-branch design exploits the correlation between RGB images and the G-buffer. The rich features and detailed information obtained from the high-definition G-buffer assist in the super-resolution of low-definition RGB images.

[0033] 2. The method of gradual fusion of multi-scale high-dimensional features can gradually fill in details from low-resolution to high-resolution images, making the enhancement process more robust and less prone to problems such as blurred boundaries and content loss. It solves problems such as blurred outlines and missing details, and has better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 1 is a flow chart of a method for super-resolution of rendering content using an auxiliary buffer provided in a first embodiment of the present invention;

[0035] Figure 2 It is an architecture diagram of the multi-scale fusion network model provided by the present invention;

[0036] Figure 3 is the type of G-buffer used in the present invention;

[0037] Figure 4 1 is a schematic diagram of a structure of a content super-resolution device for rendering using an auxiliary buffer provided in a third embodiment of the present invention.

[0038] Figure 5 It is a comparison chart of the prediction details of the present invention and various methods;

[0039] Figure 6 is another comparison chart of the prediction details of the present invention and each method;

[0040] Figure 7 This is a comparison chart of the images predicted by the present invention and various methods. DETAILED DESCRIPTION

[0041] Example 1

[0042] This embodiment provides a method for super-resolution of rendering content using an auxiliary buffer, such as Figure 1 As shown, the following steps are included:

[0043] S1. Establish a multi-scale fusion network model.

[0044] like Figure 2 As shown, the established multi-scale fusion network model includes an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch. The RGB image processing branch is used to extract feature information from the low-resolution RGB image and perform zero upsampling. The G-buffer processing branch is used to extract feature information from the high-resolution G-buffer image paired with the low-resolution RGB image. The multi-scale fusion prediction branch is used to encode and fuse the feature information extracted by the RGB image processing branch and the G-buffer processing branch at different scales, and then decode and predict the super-resolved high-resolution RGB image. Each branch is described in detail below.

[0045] The RGB image processing branch specifically includes a feature extraction module and a zero upsampling module. The feature extraction module consists of three sequentially connected convolutional layers, taking as input a low-resolution RGB image. Compared to the high-resolution labeled image, low-resolution RGB images contain less high-frequency information, such as scene details, but more consistent low-frequency information. The zero upsampling module uses the output of the feature extraction module as input and assigns each pixel with equal probability to any of the s corresponding pixels at high resolution. The remaining s-1 pixel values ​​are set to 0, where s is the upsampling ratio. This method can provide the network with valid or invalid sample information, thereby enhancing the robustness of inference.

[0046] The G-buffer processing branch is specifically a feature extraction module, which includes three convolutional layers connected in sequence, and its input is 5 types of G-buffer images with a total of 9 channels, such as Figure 3 As shown. G-buffer stands for Geometry Buffer, which is a screen space representation of geometry and material information generated by the intermediate rendering process in the rendering pipeline. The five types of G-buffer images used are as follows: 1. Base Color, which has three channels and defines the overall color of the material. 2. Scene Depth, which has one channel and stores the depth value of each pixel in the scene. 3. Roughness, which has one channel and describes the surface roughness of each pixel, which determines the blur or clarity of the reflection. 4. Metallic, which has one channel and describes to what extent the surface of each pixel is "like metal". The corresponding value for non-metal is 0, and the corresponding value for metal is 1. 5. World Normal, which has three channels and stores the normal value of each pixel in the scene, affects the shading result after light reflection and refraction.

[0047] The multi-scale fusion prediction branch is similar to the U-net structure, specifically including an RGB feature encoding part, a G-buffer feature encoding part, a feature fusion part, and a predicted RGB image decoding part. The RGB feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence. The first convolution-downsampling module is connected to the RGB image processing branch and includes two connected convolution layers. The second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence. The third convolution-downsampling module has the same structure as the second convolution-downsampling module. The feature sizes of the three convolution-downsampling modules are reduced by 1 / 2 in sequence. The G-buffer feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence. The first convolution-downsampling module is connected to the G-buffer processing branch and includes two connected convolution layers. The second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence. The third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2. The present invention improves U-net by dividing the first half of U-net into two encoding parts. In this way, RGB features and G-buffer features can be extracted separately at different scales, obtaining rich content details and scene information in the G-buffer features and supplementing the RGB features to complete subsequent fusion operations. The feature fusion section includes a first fusion module, a second fusion module, and a third fusion module. The first fusion module is connected to the first convolution-downsampling module of the RGB feature encoding section and the first convolution-downsampling module of the G-buffer feature encoding section, respectively. The second fusion module is connected to the second convolution-downsampling module of the RGB feature encoding section and the second G-buffer feature encoding section, respectively. The third fusion module is connected to the third convolution-downsampling module of the RGB feature encoding section and the third G-buffer feature encoding section, respectively. Each fusion module includes a residual unit and a squeeze and excitation unit connected in sequence. The residual unit includes a 1×1 convolution layer and a 3×3 convolution layer. The residual result generated by the residual unit can be added to the G-buffer feature to supplement the RGB image features. Intuitively speaking, the high-resolution G-buffer is continuous in space, which helps the residual module locate invalid information in the RGB image features.Squeeze-and-excitation (SE) comes from the literature [Jie H, Li S, Gang S, et al. Squeeze-and-Excitation Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, PP(99).] It can adaptively adjust the feature response at the channel level to better mask the fusion. The predicted RGB image decoding part includes a first convolution module and a second convolution module connected in sequence. The first convolution module is connected to the third fusion module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the second fusion module. The second convolution module is connected to the first convolution module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the first fusion module.

[0048] The multi-scale fusion prediction branch mainly decodes the fusion information at different scales and predicts and reconstructs a high-resolution RGB image. Due to the presence of certain perspective distortion in the scene image, the visual sizes of different objects in the picture are different. However, the receptive field size of the convolutional layer is fixed. For visual elements with larger scales in the image, the receptive field can only cover part of its area, which easily leads to blurred boundaries; and for visual elements with smaller scales, the receptive field will include other visual elements, which easily causes the problem of unclear distinction of small objects. Therefore, the present invention adopts a method of gradual fusion of multi-scale high-dimensional features to gradually fill in details on image features of different resolutions. The feature dimension of the decoding part is gradually reduced, so as to achieve the effect of removing complex and useless features and retaining important features. At the same time, in order to adapt to the application of rendering scenes, the number of output channels of each convolutional layer is reduced on the basis of the original U-net structure to make the entire network structure lightweight.

[0049] S2. Generate several image pairs including low-definition RGB images and corresponding high-definition G-buffer images as samples in the rendering scene. Use the RGB images rendered at high resolution as sample labels and input them into the established multi-scale fusion network model for network training.

[0050] Among them, the loss function used in network training is:

[0051]

[0052]

[0053] Where L represents the total loss, y represents the high-definition RGB image predicted by the sample after passing through the multi-scale fusion network model, represents the sample label, Represents the structural similarity index measure (SSIM) between the predicted image and the sample label, μ y Represents the pixel average of the sample label and the predicted image, respectively, Represent the pixel variance of the sample label and the predicted image, respectively, Represents the pixel covariance of the sample label and the predicted image, c1=(k1P) 2 ,c2=(k2P) 2 ,P=255,k1=0.01,k2=0.03,w is the weight coefficient, Represents the learned perceptual image patch similarity (LPIPS) between the predicted image and the sample label. It comes from the perceptual loss function proposed in the literature [Zhang R, Isola P, Efros AA, et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric [J]. IEEE / CVF Conference on Computer Vision & Pattern Recognition, 2018.]. The pre-trained network model set in this embodiment is VGG-16. The perceptual loss module adds the feature map obtained by passing the predicted image through each layer of the convolutional neural network to the objective function to make the predicted image and the true value image more similar in high-level semantic features. Generally speaking, the LPIPS indicator can better reflect human perception of the similarity between two images than traditional indicators.

[0054] S3. Input the low-resolution RGB image to be super-resolved and the corresponding high-resolution G-buffer image into the trained multi-scale fusion network model to obtain the super-resolved high-resolution RGB image.

[0055] Example 2

[0056] This embodiment further provides a device for super-resolution of rendered content using an auxiliary buffer. The device may be implemented in software and / or hardware and may be configured in a terminal device, including:

[0057] A network model establishment module is used to establish a multi-scale fusion network model, wherein the multi-scale fusion network model includes an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch. The RGB image processing branch is used to extract feature information from a low-definition RGB image and perform zero upsampling. The G-buffer processing branch is used to extract feature information from a high-definition G-buffer image paired with the low-definition RGB image. The multi-scale fusion prediction branch is used to encode and fuse the feature information extracted by the RGB image processing branch and the G-buffer processing branch at different scales, and then decode and predict the super-resolved high-definition RGB image.

[0058] The network training module is used to generate several image pairs including low-resolution RGB images and corresponding high-resolution G-buffer images as samples in the rendering scene. The RGB images rendered at high resolution are used as sample labels and input into the established multi-scale fusion network model for network training.

[0059] The image super-resolution module is used to input the low-resolution RGB image to be super-resolved and the corresponding high-resolution G-buffer image into the trained multi-scale fusion network model to obtain the super-resolved high-definition RGB image.

[0060] The device provided in the embodiment of the present invention can be used to execute the method provided in the first embodiment, and has the corresponding functions and beneficial effects of executing the method.

[0061] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0062] Example 3

[0063] Figure 4 This is a schematic diagram of the structure of the device provided in the third embodiment of the present invention. This embodiment of the present invention provides services for the implementation of the method in the first embodiment, and can be configured with the device in the second embodiment. Figure 4 A block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The device 12 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present invention.

[0064] like Figure 4As shown, device 12 is implemented as a general-purpose computing device. Components of device 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components, including system memory 28 and processing unit 16.

[0065] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0066] Device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 12, including volatile and non-volatile media, removable and non-removable media.

[0067] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0068] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0069] The device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the device 12, and / or any device that enables the device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur through an input / output (I / O) interface 22. Furthermore, the device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) through a network adapter 20. Figure 4 As shown, network adapter 20 communicates with the other modules of device 12 via bus 18. It should be understood that although not shown, other hardware and / or software modules may be used in conjunction with device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0070] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28 , such as implementing the quotation method considering carbon costs provided in the first embodiment of the present invention.

[0071] Example 4

[0072] The fourth embodiment of the present invention further provides a storage medium containing computer-executable instructions, which are used to execute the method described in the first embodiment when executed by a computer processor.

[0073] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by an instruction execution system, device or device or used in combination with it.

[0074] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0075] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0076] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0077] Of course, the computer-executable instructions of a storage medium provided by an embodiment of the present invention are not limited to the above method operations, but can also execute related operations in the method provided by any embodiment of the present invention.

[0078] The present invention is simulated and verified below.

[0079] (1) Training details and parameter settings

[0080] This verification used UE4 as the rendering engine and introduced two scenes from the Unreal Engine Marketplace to construct a large-scale dataset. Each scene dataset consists of a 2500-frame training set and a 500-frame test set. The RGB images in the dataset are pre-tone-mapped images without AA processing. Furthermore, when generating the dataset, the "motion blur" option was disabled for each scene to reduce background blur caused by the movement of the subject and the scene. This verification selected 4×4 super-resolution for network training and testing. Therefore, the resolution of the labeled images and G-buffer in the dataset is 1920×1080 (1080P), and the resolution of the input images is 480×270. During training, the network divides the input images, labeled images, and G-buffer into 3×3 blocks.

[0081] The network was implemented and trained using the PyTorch framework. Mini-batch stochastic gradient descent (SGD) and the Adam optimizer were used for optimization. The training parameters were a batch size of 8, β1 = 0.9, β2 = 0.999 for the Adam optimizer, an initial learning rate of 1e-3, and a learning rate decay of 0.98 after each epoch. Training was performed for 100 epochs. The network was initialized using Xavier initialization.

[0082] (2) Comparison of experimental results

[0083] First, the prediction results of the present invention are compared with the reference image. Figure 7As shown, the present invention effectively supplements object contours and background detail information. Secondly, the present invention is compared with several recently published public models, using the evaluation metrics PSNR (peak signal-to-noise ratio) and SSIM (structural similarity). PSNR represents the ratio of the maximum possible signal power to the power of the destructive noise that affects the fidelity of its representation. In image reconstruction tasks, PSNR is often used as a metric to measure the quality of image or video reconstruction, while SSIM measures the structural similarity between the predicted image and the reference image from a holistic perspective. The advanced methods compared include: RCAN (a single image super-resolution method), NSRR (a rendered image super-resolution method), and RRN (a video super-resolution method). RRN's original approach is to downsample the sample image using a Gaussian kernel to obtain the input image. This is because RRN struggles to simultaneously obtain images at two resolutions when working with photographic images and video streams. In this comparison, the input image is adjusted to a low-resolution rendered scene image. Table 1 compares these metrics by calculating the average values ​​of each method across different scene test sets. From the numerical comparison, it can be seen that except for the PSNR index of the Bunker dataset, the RRN network is slightly better than the present invention, which is related to the use of the L1 loss function in the RRN network training. In terms of other indexes, the present invention is significantly better than other methods. Figure 5 、 6 From the local examples of the two scenarios, it can be seen that the super-resolution prediction results of the present invention perform better in terms of material surface effects, and are more realistic in visual perception than other methods.

[0084] Table 1. Comparison of test set performance of various advanced methods.

[0085] RF and BK represent the Redwood Forest and Bunker scene datasets, respectively.

[0086]

[0087] At the same time, this verification performed single-frame 4×4 super-resolution inference on a network model in the onnx format under the TensorRT framework at different resolutions of 1080P and 720P, and at different inference precisions of half-length float (FP-16) and float (FP-32). The average of 10,000 runs was taken, and the runtime statistics are shown in Table 2. It can be seen that the present invention can meet the latency requirements of real-time rendering at FP-16 inference precision.

[0088] Table 2 Single-frame inference time of 4×4 super-resolution performed by the present invention at different resolutions and inference accuracies.

[0089] The running time unit is milliseconds (ms).

[0090] Inference accuracy\resolution 1080P 720P FP-16 23.158 10.570 FP-32 45.572 20.548

[0091] The above disclosure is only a preferred embodiment of the present invention and cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for super-resolution of rendering content using an auxiliary buffer, characterized in that The method includes: (1) Establishing a multi-scale fusion network model, the multi-scale fusion network model includes an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch, wherein the RGB image processing branch is used to extract feature information from a low-definition RGB image and perform zero upsampling, the G-buffer processing branch is used to extract feature information from a high-definition G-buffer image paired with the low-definition RGB image, and the multi-scale fusion prediction branch is used to encode and fuse the feature information extracted by the RGB image processing branch and the G-buffer processing branch at different scales, and then decode and predict the high-definition RGB image after super-resolution; (2) Generate several image pairs including low-definition RGB images and corresponding high-definition G-buffer images as samples in the rendering scene, use the RGB images rendered at high resolution as sample labels, input the established multi-scale fusion network model, and perform network training; (3) Input the low-resolution RGB image to be super-resolved and the corresponding high-resolution G-buffer image into the trained multi-scale fusion network model to obtain the high-resolution RGB image after super-resolvation; The multi-scale fusion prediction branch specifically includes: The RGB feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence, wherein the first convolution-downsampling module is connected to the RGB image processing branch and includes two connected convolution layers, the second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence, and the third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2; A G-buffer feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence, wherein the first convolution-downsampling module is connected to the G-buffer processing branch and includes two connected convolution layers, the second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence, and the third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2; The feature fusion part includes a first fusion module, a second fusion module and a third fusion module, wherein the first fusion module is respectively connected to the first convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, the second fusion module is respectively connected to the second convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, and the third fusion module is respectively connected to the third convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, and each fusion module includes a residual unit and a squeeze and excitation unit connected in sequence; The predicted RGB image decoding part includes a first convolution module and a second convolution module connected in sequence. The first convolution module is connected to the third fusion module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the second fusion module. The second convolution module is connected to the first convolution module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the first fusion module.

2. The method for super-resolution of rendering content using an auxiliary buffer according to claim 1, wherein: The RGB image processing branch in step (1) specifically includes: The feature extraction module consists of three convolutional layers connected in sequence; The zero upsampling module is used to take the output of the feature extraction module as input and assign each pixel to any pixel position among the corresponding s pixels at high resolution with equal probability, while the remaining s-1 pixel values ​​will be set to 0, where s is the upsampling ratio.

3. The method for super-resolution of rendering content using an auxiliary buffer according to claim 1, wherein: The G-buffer processing branch in step (1) is specifically a feature extraction module, which includes three convolutional layers connected in sequence.

4. The method for super-resolution of rendering content using an auxiliary buffer according to claim 1, wherein: The loss function used in step (2) for network training is: Where L represents the total loss, y represents the high-definition RGB image predicted by the sample after passing through the multi-scale fusion network model, represents the sample label, Represents the structural similarity index measure between the predicted image and the sample label, Represents the pixel average of the sample label and the predicted image, respectively, Represent the pixel variance of the sample label and the predicted image, respectively, Represents the pixel covariance of the sample label and the predicted image, c1=(k1P) 2 ,c2=(k2P) 2 ,P=255,k1=0.01,k2=0.03,w is the weight coefficient, Represents the learning-aware patch similarity between the predicted image and the sample label.

5. A rendering content super-resolution device using an auxiliary buffer, characterized in that include: A network model establishment module is used to establish a multi-scale fusion network model, wherein the multi-scale fusion network model includes an RGB image processing branch, a G-buffer processing branch, and a multi-scale fusion prediction branch. The RGB image processing branch is used to extract feature information from a low-definition RGB image and perform zero upsampling. The G-buffer processing branch is used to extract feature information from a high-definition G-buffer image paired with the low-definition RGB image. The multi-scale fusion prediction branch is used to encode and fuse the feature information extracted by the RGB image processing branch and the G-buffer processing branch at different scales, and then decode and predict the super-resolved high-definition RGB image. The network training module is used to generate several image pairs including low-resolution RGB images and corresponding high-resolution G-buffer images as samples in the rendering scene. The RGB images rendered at high resolution are used as sample labels and input into the established multi-scale fusion network model for network training. The image super-resolution module is used to input the low-resolution RGB image to be super-resolved and the corresponding high-resolution G-buffer image into the trained multi-scale fusion network model to obtain the super-resolved high-definition RGB image; The multi-scale fusion prediction branch specifically includes: The RGB feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence, wherein the first convolution-downsampling module is connected to the RGB image processing branch and includes two connected convolution layers, the second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence, and the third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2; A G-buffer feature encoding part includes a first convolution-downsampling module, a second convolution-downsampling module, and a third convolution-downsampling module connected in sequence, wherein the first convolution-downsampling module is connected to the G-buffer processing branch and includes two connected convolution layers, the second convolution-downsampling module includes a maximum pooling layer and two convolution layers connected in sequence, and the third convolution-downsampling module has the same structure as the second convolution-downsampling module, and the feature sizes of the three convolution-downsampling modules are sequentially reduced by 1 / 2; The feature fusion part includes a first fusion module, a second fusion module and a third fusion module, wherein the first fusion module is respectively connected to the first convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, the second fusion module is respectively connected to the second convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, and the third fusion module is respectively connected to the third convolution-downsampling module of the RGB feature encoding part and the G-buffer feature encoding part, and each fusion module includes a residual unit and a squeeze and excitation unit connected in sequence; The predicted RGB image decoding part includes a first convolution module and a second convolution module connected in sequence. The first convolution module is connected to the third fusion module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the second fusion module. The second convolution module is connected to the first convolution module, including a transposed convolution layer, a connection operation and two convolution layers connected in sequence. The connection operation also connects the first fusion module.

6. A device for super-resolution of rendering content using an auxiliary buffer, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A storage medium containing computer-executable instructions, characterized in that: The computer executable instructions are used to perform the method according to any one of claims 1 to 4 when executed by a computer processor.

Images