A Game Super-Resolution Method Based on Recurrent Neural Networks
By using the recursive structure of recurrent neural networks and the mapping of motion vector information, the real-time performance and latency issues of game super-resolution models are solved, achieving efficient high-resolution image output and improving game image quality and speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-03-28
- Publication Date
- 2026-07-17
AI Technical Summary
Existing game super-resolution models are insufficient in terms of processing speed and latency, unable to output high-resolution images in real time, and the complex alignment operation leads to increased memory usage and latency, affecting the gaming experience.
A recursive structure based on a recurrent neural network is adopted. The hidden layer is used to pass information from past frames and it is mapped to the current frame through motion vector information. The image correlation is judged by combining depth information to construct the input structure and realize the real-time output of high-resolution images.
It improves the processing speed and image quality of the game's super-resolution model, meets real-time and latency requirements, improves the clarity and detail of the game screen, and enhances the gaming experience.
Smart Images
Figure CN116523743B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of super-resolution technology, and specifically relates to a game super-resolution method based on recurrent neural networks. Background Technology
[0002] Game super-resolution is an image processing technique designed to improve the visual quality and detail of a game by processing low-resolution images of the game scene to generate high-resolution images. This technique is based on deep learning and machine learning, using a high-resolution training image set to train a model, and then leveraging these models to convert low-resolution game images into high-resolution images. Game super-resolution can improve the clarity and detail of game visuals, enhancing the gaming experience.
[0003] Game super-resolution technology has extremely high real-time requirements because it typically needs to process game visuals in real-time or near-real-time. Real-time requirements manifest in two aspects: processing speed and latency. Processing speed refers to the algorithm's ability to process low-resolution images quickly enough and output high-resolution images in real-time or near-real-time. Latency refers to the time required from inputting a low-resolution image to outputting a high-resolution image; super-resolution algorithms should avoid using information from future frames as much as possible.
[0004] Compared to game super-resolution models, general video super-resolution models often require complex alignment operations to improve the accuracy and effect of super-resolution. This presents two problems for game super-resolution: first, the alignment operations slow down the processing speed of general super-resolution networks, typically requiring a certain amount of time to process the entire image; second, these alignment operations often require support from multiple frames before and after, preventing the network from outputting a high-resolution image from a low-resolution input frame. This not only consumes more GPU memory but also increases latency, impacting the gaming experience. Therefore, it is unsuitable as a game super-resolution model. Summary of the Invention
[0005] To address the aforementioned technical issues, this invention discloses a game super-resolution method based on recurrent neural networks. It employs the recursive structure of recurrent neural networks and uses hidden layers to transmit information from past frames, enabling the output image to simultaneously absorb information from both past and current frames. This allows for real-time input of a low-resolution image and output of a high-resolution image, making the network suitable for game application scenarios.
[0006] The technical solution adopted by this invention to solve its technical problem is as follows:
[0007] A game super-resolution method based on recurrent neural networks includes:
[0008] S1, obtain the previous low-resolution image, the previous high-resolution image estimation result, and the previous hidden layer information passed from the recurrent neural network;
[0009] S2, obtain the motion vector information of the low-resolution image of the current frame, and use the motion vector information of the current frame to map the hidden layer information from the previous time domain to the current time domain;
[0010] S3, obtain the depth information of the low-resolution images of the previous frame and the current frame, determine whether the two frames are related, and construct the input structure of the previous frame based on the association result;
[0011] S4, take the input structure of the previous frame and the low-resolution image of the current frame as the input of the recurrent neural network, and separate the high-resolution estimation information and the hidden layer information of the current frame from the output of the recurrent neural network.
[0012] S5. Based on the high-resolution estimation information of the current frame and the low-resolution image of the current frame, the high-resolution image estimation result of the current frame is reconstructed.
[0013] S6. Repeat steps S1 to S5 to output the high-resolution image estimation result of the low-resolution image of the current frame in real time.
[0014] Furthermore, methods for determining whether two consecutive images are related include:
[0015] 3.1) Obtain two known related low-resolution images from consecutive frames and calculate the depth difference image D:
[0016] D = D1 - D2
[0017] Where D1 is the depth information of the previous low-resolution image, and D2 is the depth information of the current low-resolution image.
[0018] 3.2) Sort the pixel values of the depth difference image D obtained in step 3.1) by size, calculate the frequency of each pixel value, and select the pixel value with the highest frequency as the threshold;
[0019] 3.3) For the two consecutive low-resolution images to be calculated, first calculate the depth difference image using the formula in step 3.1). If the highest frequency value of the depth difference image is greater than twice the threshold, it means that the two images are not related; otherwise, the two images are related.
[0020] Furthermore, if the two frames are related, the concatenation result of the previous low-resolution image, the downsampled previous high-resolution image estimation result, and the hidden layer information mapped to the current time domain is used as the input structure of the previous frame; if they are not related, the previous low-resolution image is replaced by the current low-resolution image, and the previous high-resolution image estimation result and the hidden layer information passed from the previous recurrent neural network are both cleared to zero to construct the previous frame input structure.
[0021] Furthermore, the estimated result of the previous high-resolution image after downsampling has the same size as the previous low-resolution image.
[0022] Furthermore, the recurrent neural network includes a 3x3 input convolutional layer, several depth separation convolutional modules, and a 3x3 output convolutional layer connected in sequence. The 3x3 output convolutional layer restores the output result of the last depth separation convolutional module to obtain an output result containing the high-resolution estimation information of the current frame and new hidden layer information.
[0023] Furthermore, the depthwise separable convolution module consists of a depthwise separable convolutional layer, a 1x1 convolutional layer, and an activation function following each layer.
[0024] Furthermore, the output of the 3x3 output convolutional layer has 28 channels. The first 12 channels of the output are used as the high-resolution estimation information of the current frame, and the last 16 channels are used as the hidden layer information of the current frame.
[0025] Furthermore, in step S5, the high-resolution estimation information of the current frame and the low-resolution image of the current frame are superimposed, and the superimposed result is upsampled using a sub-pixel convolutional layer to reconstruct the high-resolution image estimation result of the current frame.
[0026] The beneficial effects of this invention are:
[0027] This invention designs a game super-resolution method based on recurrent neural networks. It fully utilizes the recursive structure of recurrent networks, uses hidden layers to transmit information from past frames, and uses motion vector information to ensure that the information in the hidden layers can be aligned with the information in the current time domain each time. This further improves the effectiveness of the hidden layers in transmitting information from past frames. Thus, while achieving real-time input of a low-resolution image and output of a high-resolution image, it absorbs information from past and current frames, making the network's estimation of high-resolution images more accurate and meeting the requirements of game super-resolution in terms of quality, real-time performance, and latency. Attached Figure Description
[0028] Figure 1 This is a framework diagram illustrating a game super-resolution method based on a recurrent neural network, as shown in an embodiment of the present invention.
[0029] Figure 2 This is a schematic diagram illustrating the alignment of the hidden layer in this invention.
[0030] Figure 3 This is a schematic diagram of the previous frame input structure construction process in the implementation of this invention.
[0031] Figure 4 This is a schematic diagram illustrating the determination of the correlation between two consecutive frames in this invention.
[0032] Figure 5 This is a schematic diagram of the neural network structure during the implementation of the present invention.
[0033] Figure 6 This is a flowchart illustrating the process of implementing this invention.
[0034] Figure 7 This is an overall flowchart illustrating the game super-resolution method based on recurrent neural networks in an embodiment of the present invention. Detailed Implementation
[0035] The method of the present invention will be further described below with reference to the accompanying drawings.
[0036] like Figure 1 As shown, this invention provides a game super-resolution method based on recurrent neural networks. It utilizes the hidden layers of the recurrent neural network to continuously update and pass information from past frames, and achieves real-time output of a high-resolution image after inputting a low-resolution image in a single super-resolution re-loop manner.
[0037] The implementation process of this invention can be referred to Figure 6 and Figure 7 This includes the following steps:
[0038] Step 1: Obtain the data information from the previous frame.
[0039] In this invention, the estimation results of the previous frame's low-resolution image and high-resolution image, as well as the hidden layer information transmitted by the previous frame's recurrent neural network, are acquired and processed. The hidden layer information transmitted by the recurrent neural network contains potential feature information of the image. Through continuous updating by the recurrent network, the hidden layer information is mapped from the previous time domain to the current time domain.
[0040] like Figure 2 As shown, motion vector information is used to map hidden layer information from the previous time domain to the current time domain:
[0041] I1(x,y)=I2(x+u,y+v)
[0042] Where x and y represent the height and width of the low-resolution image, respectively, and I1 and I2 represent the low-resolution image of the previous frame and the low-resolution image of the current frame, respectively.
[0043] In this embodiment, the method of mapping the hidden layer information to the current time domain is to use the motion vector information of the low-resolution image. The motion vector information is the offset of the height and width from the previous time domain to the current time domain, which can be directly collected by the game engine. In this embodiment, u and v correspond to the first and second channels in the motion vector information, respectively, representing the offset of the height and width from the previous time domain to the current time domain.
[0044] Furthermore, the input size of the previous frame for the current frame must be consistent with the size of the low-resolution image of the current frame. Therefore, the estimation result of the high-resolution image of the previous frame needs to be downsampled from C×rH×rW to r. 2 Given C×H×W, ensure that the input dimensions are all H×W, where r is the scaling factor, which is 2 here.
[0045] Step 2: Determine the reference value of the data information in the previous frame.
[0046] Based on the data information from the previous frame obtained in step 1, it is also necessary to determine whether it has any reference value. For example... Figure 4 As shown, calculate the depth difference between two consecutive frames:
[0047] D = D1 - D2
[0048] Where D1 is the depth information of the previous low-resolution image, and D2 is the depth information of the current low-resolution image.
[0049] For two related consecutive frames, sort the pixel values of the depth difference image D by size, calculate the frequency of each pixel value, and then select the pixel value with the highest frequency as the threshold.
[0050] For two consecutive low-resolution images to be calculated, the depth difference is first calculated using the formula mentioned above. If the most frequent value of the depth difference image is greater than twice the threshold, it indicates that the two images are not related. If the two images are not related, they have no reference value, and the hidden layer information needs to be reset. The specific method is as follows: designate the previous low-resolution image as the current low-resolution image, and clear the estimation result of the previous high-resolution image and the hidden layer information passed by the previous recurrent neural network. Use the superposition result of the previous low-resolution image, the cleared previous high-resolution image estimation result, and the hidden layer information as the input structure of the previous frame.
[0051] If the most frequent value of the depth difference image is no greater than twice the threshold, it indicates that the two frames are correlated. After mapping the hidden layer information from the previous time domain to the current time domain, as follows... Figure 3As shown, the low-resolution image of the previous frame, the estimation result of the high-resolution image of the previous frame, and the hidden layer information mapped to the current time domain are used as the input structure of the previous frame, that is, as the input of the neural network.
[0052] Step 3: Combine the input structure of the previous frame with the low-resolution image of the current frame and input it into the neural network to obtain an output result that includes high-resolution estimation information of the current frame and new hidden layer information.
[0053] In one specific embodiment of the present invention, depthwise separable convolution is used to improve the inference speed of the model. The number of parameters to be learned in depthwise separable convolution is M×k×k+M×N, which saves a lot of computation compared to the general convolution M×N×k×k. Here, M and N refer to the number of layers of the input feature map and the number of layers of the output feature map, respectively, and k refers to the size of the convolution kernel, which is 3 here.
[0054] In this embodiment, the neural network structure diagram is as follows: Figure 5 As shown, it includes a 3x3 convolutional layer, several depthwise segregating convolutional modules consisting of depthwise segregating convolutional layers, 1x1 convolutional layers, and activation functions following each layer, and a 3x3 convolutional layer. Specifically, after the last depthwise segregating convolutional module, a 3x3 convolution is performed to restore the output of the last depthwise segregating convolutional module, resulting in an output containing high-resolution estimation information for the current frame and new hidden layer information.
[0055] Step 4: Separate the high-resolution estimation information and hidden layer information of the current frame from the output of the neural network structure, and reconstruct the high-resolution image estimation result of the current frame based on the high-resolution estimation information of the current frame.
[0056] like Figure 6 As shown, it is necessary to separate the current frame high-resolution estimation information and the new hidden layer information from the output result. The current frame high-resolution image estimation result is the result of transforming the current frame high-resolution estimation information through upsampling. In this embodiment, for RGB images, the first 12 channels of the output result of the last 3x3 convolutional layer in the neural network structure diagram are used as the current frame high-resolution estimation information, and the last 16 channels are used as the current frame hidden layer information. The generated new hidden layer information will be passed to the next frame to start the next loop.
[0057] In this embodiment, subpixel convolutional layers are used to achieve upsampling. The high-resolution estimation information of the first 12 channels of the current frame and the low-resolution image of the current frame from the output of the last 3x3 convolutional layer are passed through a subpixel convolutional layer, an implicit convolutional layer with a stride of 1 / r that does not require additional computation, to stack the result r. 2 The C×H×W conversion is the final high-resolution frame result of C×rH×rW, where r is the scaling factor, which is 2 here.
[0058] Complete the current loop and pass the output to the next loop.
[0059] The above description is merely a specific embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A game super-resolution method based on recurrent neural networks, characterized in that, include: S1, obtain the previous low-resolution image, the previous high-resolution image estimation result, and the previous hidden layer information passed from the recurrent neural network; S2, obtain the motion vector information of the low-resolution image of the current frame, and use the motion vector information of the current frame to map the hidden layer information from the previous time domain to the current time domain; S3, obtain the depth information of the low-resolution images of the previous frame and the current frame, determine whether the two frames are related, and construct the input structure of the previous frame based on the association result; S4, take the input structure of the previous frame and the low-resolution image of the current frame as the input of the recurrent neural network, and separate the high-resolution estimation information and the hidden layer information of the current frame from the output of the recurrent neural network. S5. Based on the high-resolution estimation information of the current frame and the low-resolution image of the current frame, the high-resolution image estimation result of the current frame is reconstructed. S6. Repeat steps S1 to S5 to output the high-resolution image estimation result of the low-resolution image of the current frame in real time.
2. The game super-resolution method based on recurrent neural networks according to claim 1, characterized in that, Methods for determining whether two consecutive frames are related include: 3.1) Obtain two known related low-resolution images from consecutive frames and calculate the depth difference image D: D = |D1 - D2| Where D1 is the depth information of the previous low-resolution image, and D2 is the depth information of the current low-resolution image. 3.2) Sort the pixel values of the depth difference image D obtained in step 3.1) by size, calculate the frequency of each pixel value, and select the pixel value with the highest frequency as the threshold; 3.3) For the two consecutive low-resolution images to be calculated, first calculate the depth difference image using the formula in step 3.1). If the highest frequency value of the depth difference image is greater than twice the threshold, it means that the two images are not related; otherwise, the two images are related.
3. The game super-resolution method based on recurrent neural networks according to claim 1, characterized in that, If two consecutive frames are related, the concatenation result of the previous low-resolution image, the estimation result of the previous high-resolution image after downsampling, and the hidden layer information mapped to the current time domain is used as the input structure of the previous frame. If they are not related, the low-resolution image of the current frame is used to replace the low-resolution image of the previous frame, and the estimation result of the high-resolution image of the previous frame and the hidden layer information passed from the recurrent neural network of the previous frame are both cleared to zero, so as to construct the input structure of the previous frame.
4. The game super-resolution method based on recurrent neural networks according to claim 3, characterized in that, The estimated size of the previous high-resolution image after downsampling is consistent with that of the previous low-resolution image.
5. The game super-resolution method based on recurrent neural networks according to claim 1, characterized in that, The recurrent neural network includes a 3x3 input convolutional layer, several depth separation convolutional modules, and a 3x3 output convolutional layer connected in sequence. The 3x3 output convolutional layer restores the output of the last depth separation convolutional module to obtain an output containing high-resolution estimation information of the current frame and new hidden layer information.
6. The game super-resolution method based on recurrent neural networks according to claim 5, characterized in that, The aforementioned depthwise separable convolutional module consists of a depthwise separable convolutional layer, a 1x1 convolutional layer, and an activation function following each layer.
7. A game super-resolution method based on a recurrent neural network according to claim 5, characterized in that, The output of the 3x3 output convolutional layer has 28 channels. The first 12 channels of the output are used as the high-resolution estimation information of the current frame, and the last 16 channels are used as the hidden layer information of the current frame.
8. The game super-resolution method based on recurrent neural networks according to claim 1, characterized in that, In step S5, the high-resolution estimation information of the current frame and the low-resolution image of the current frame are superimposed, and the superimposed result is upsampled by a subpixel convolutional layer to reconstruct the high-resolution image estimation result of the current frame.