Cloud game video transmission method based on video enhancement and bitrate adaptation algorithm

Abstract

The application discloses a cloud game video transmission method based on a video enhancement and bit rate adaptive algorithm, which combines edge computing, a bit rate adaptive algorithm based on reinforcement learning and a video enhancement technology.The application decides the transmission bit rate through reinforcement learning when the network fluctuates, reduces the video data amount to be transmitted by reducing the frame rate and resolution, avoids lag, and then improves the game video quality through super resolution and frame interpolation.The video enhancement is unloaded from the terminal device of a game player to an edge cloud server by using an edge computing architecture, so that the problem of insufficient terminal computing power is avoided.The application can guarantee high-quality and stable cloud game service in the case of network fluctuation, and improve the game experience of cloud game users.

Description

Technical Field

[0001] This invention belongs to the fields of cloud gaming, network optimization, and video enhancement, and involves bitrate adaptive algorithms, reinforcement learning, video enhancement, and edge computing techniques. Background Technology

[0002] Cloud gaming has developed rapidly in recent years, and its market share is expected to reach $8 billion by 2025. Unlike traditional games, which use a centralized client-server architecture that requires players to have high-performance devices for real-time rendering, cloud gaming utilizes the computing power of cloud servers for rendering. After receiving player control events, the cloud gaming server renders and encodes the game scene and transmits it to the end user as a video stream. In this way, players can access the game anytime without being limited by the computing power and image processing capabilities of their terminal devices. Game content providers do not need to be compatible with different underlying hardware architectures, thus reducing development costs and accelerating development speed. However, cloud gaming service providers need to transmit high-resolution, high-frame-rate, and low-latency game video streams to players, meaning that changes in network bandwidth, latency, and packet loss can all affect the gaming experience.

[0003] Cloud gaming requires continuous and stable bandwidth to support high-quality game video streams. However, in the highly heterogeneous modern internet, factors such as user numbers, mobility, network congestion, and channel fading can cause fluctuations in the player's network connection. When the sending rate at the sending end does not match the available bandwidth, network congestion may occur, leading to increased transmission latency or frame drops, thus rapidly degrading the player's gaming experience. On the other hand, the degradation in real-time game video quality due to network limitations necessitates effective video enhancement to provide gamers with a stable gaming service.

[0004] To address the degraded gaming experience caused by network fluctuations, technologies such as congestion control, adaptive frame rate, forward error correction coding, and split video coding offer solutions. For example, the Google team proposed a congestion control algorithm, SQP, based on rate and latency, for low-latency interactive video streaming, aiming to provide real-time bandwidth estimation. It relies on QUIC for reliable video frame transmission and provides packet timestamps to estimate network bandwidth. By adjusting the rate of each frame, it uses gradient-based updates to smoothly change the bandwidth estimate and video bitrate to respond to congestion. Alhilal et al. combined sender-side rate control with forward error correction coding to propose an end-to-end cloud gaming framework, providing fault tolerance and achieving optimal video quality. The packet loss recovery scheme Prism employs split video coding technology to meet the needs of ultra-low latency video streaming applications. Prism performs deep learning-based packet loss prediction, splitting the video stream into two sub-streams upon identifying potential packet loss: a low-latency, unreliable IDR frame stream and a high-quality, reliable P-frame stream. IDR frames allow applications to continue displaying frames with lower latency. When lost P-frame data is retransmitted, the application quickly switches back to a higher-quality P-frame stream. However, while simple congestion control algorithms and split-stream video coding techniques can alleviate network congestion and ensure smooth game video, they can reduce the quality of transmitted video during network congestion, thus affecting the resolution or frame rate of the game screen and resulting in a poor gaming experience. While combining adaptive frame rate and forward error correction coding can improve the robustness and reliability of the network system through redundant coding, it has high computational complexity, and the redundant coding introduces significant additional overhead, increasing latency while reducing available bandwidth.

[0005] To address the issue of degraded real-time game video quality due to network limitations, many studies have proposed video enhancement through super-resolution and frame interpolation, which can restore high-resolution, high-frame-rate video even in poorly formatted conditions. However, existing super-resolution and frame interpolation methods cannot meet the high frame-rate and high-resolution requirements of cloud gaming in terms of both inference time and inference performance. For example, SRResNet and VDSR super-resolution models cannot meet the speed requirements for super-resolution inference. Bicubic and SRCNN methods can meet the inference speed requirements, but their super-resolution performance is unsatisfactory. While EGVSR and RDN super-resolution models can meet the 30fps inference speed requirement when scaled up by 4x, the scaling effect is poor; the inference performance is better when scaled up by 2x, but it cannot meet the time requirements for real-time inference. The inference time of frame interpolation models such as IFRNet, phase-based frame interpolation methods, and Sepconv-slomo models also cannot meet the inference requirements for high frame rates, with each frame inference time exceeding 60ms.

[0006] Therefore, proposing a transmission optimization scheme to enhance the quality of cloud gaming is a technical problem that urgently needs to be solved by those skilled in the art, so as to ensure a high-quality experience for interactive real-time video applications with high frame rates and high resolutions, such as cloud gaming. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and solve the problem of unstable cloud gaming video quality caused by network fluctuations such as bandwidth, latency, and packet loss. This invention can effectively guarantee high resolution, high frame rate, and low latency for cloud gaming videos even under network fluctuations, avoiding stuttering, frame drops, and blurry images during gameplay, thereby improving the player's gaming experience.

[0008] Technical solution of the present invention

[0009] A cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithms, the method is as follows:

[0010] S1. A method combining edge cloud architecture, bitrate adaptive algorithm, and video enhancement technology is proposed for cloud gaming video transmission and enhancement. The edge cloud architecture offloads video enhancement from the player's terminal device to the edge cloud server, solving the problem of insufficient terminal computing power. The bitrate adaptive algorithm is combined with video enhancement technology to perform frame skipping encoding and resolution reduction during network fluctuations, reducing the amount of data transmitted over the backbone network. Video enhancement is then performed on the edge cloud server, improving the gaming experience.

[0011] S2. Constructing a reinforcement learning-based bitrate adaptive algorithm. Addressing the characteristics of high frame rate, high bit rate, and low latency in cloud gaming, as well as the volatility of modern network environments, a bitrate adaptive algorithm tailored to these characteristics is proposed. Through reinforcement learning, frame skipping, and resolution reduction, the algorithm effectively matches the bit rate decision with the coding rate, alleviating network congestion.

[0012] S3. Constructing super-resolution and frame interpolation methods for cloud gaming. Super-resolution and frame interpolation models are proposed for high-frame-rate, high-resolution interactive real-time video applications, such as cloud gaming, effectively improving video quality and gaming experience on cloud gaming terminals through video enhancement.

[0013] The edge cloud architecture consists of a three-layer network: a core cloud server, edge cloud servers, and game players. The core cloud server trains super-resolution and frame interpolation models for different game scenarios, and also trains a reinforcement learning-based bitrate adaptive algorithm to render game visuals based on player actions. Distributed edge cloud servers forward game videos to players in their vicinity, providing real-time online enhancement for low-quality video. Players' devices decode and merge the video streams before playback, and then send the player's latest actions back to the core cloud server.

[0014] The steps of the method include:

[0015] 1) When a game player registers or logs in to the core cloud server and starts the game, they are assigned to the nearest edge cloud server under the core cloud deployment. The core cloud server updates the latest game super-resolution model and frame interpolation model to the edge cloud server. Afterwards, the player's game video stream is forwarded through this edge cloud server.

[0016] 2) After the game starts, the player sends game commands to the core cloud server. The core cloud server renders the game screen in real time based on the player's commands and transmits the game screen to the edge cloud server in real time based on the frame rate and resolution decisions made by the reinforcement learning-based bitrate adaptive method. If the backbone network conditions between the core cloud server and the edge cloud server are good, the bitrate adaptive method will decide to send high frame rate and high bit rate game video to the edge cloud server. The edge cloud server will then forward the game video directly to the player's terminal device. Otherwise, it will send lower frame rate and lower bit rate game video to the edge cloud server. The edge cloud server will then perform super-resolution and frame interpolation inference on the game video, encode the enhanced video stream, and send it to the player's terminal device along with the original video stream.

[0017] 3) The player's terminal device will decode and merge the received game video stream and play it, and send the player's next action to the core cloud server.

[0018] The method combines a bitrate adaptive algorithm with video enhancement technology for cloud gaming video transmission and enhancement. The bitrate adaptive algorithm based on reinforcement learning determines the transmission bitrate, and the number of frame skips and video resolution during encoding are determined based on the transmission bitrate. Then, the corresponding game video encoding is performed, thereby reducing transmission latency, reducing backbone network transmission pressure, and avoiding congestion and stuttering.

[0019] Real-time perception of network state is the basis for reasonable decision-making bit rate. Formula (1) is the state formula for reinforcement learning in this invention, and loss tThe packet loss rate is obtained through the FractionLost field in the RTCP packet, rtt. t The latency is calculated using the LSR and DLSR fields in the RTCP packet, delta. t The difference between the arrival interval at the receiver and the corresponding departure interval at the sender for two consecutive RTP packets is given by packets. t The deviation is the number of RTP packets received by the receiver within the DRL algorithm update interval. t Let t be the difference between the actual transmission bit rate at time t and the bit rate determined by the DRL algorithm.

[0020] A reasonable reward function is the basis for a reinforcement learning agent to continuously correct its action decisions. Formula (2) is the reward formula for the bit rate adaptive algorithm, where packets t The bitrate is the total number of data packets received by the receiver within the interval. t Let be the bit rate of the DRL algorithm decision at time t, and let loss be... t The packet loss rate within the time interval, RTT t Deviation is the round-trip delay within the time interval. t The bit rate execution deviation is represented by a, b, c, d, and e, which are the weights of the corresponding indicators.

[0021] To ensure that the bit rate decided by the bit rate adaptive algorithm matches the coding rate in a timely manner, and to enable the encoder to better execute the algorithm decision, this invention proposes to perform frame skipping coding and resolution reduction when the bit rate decided at the current moment is lower than that at the previous moment. Formula (3) is the frame skipping decision formula, r t Let be the bit rate at time t. If the bit rate at time t is less than that at the previous time, no frame skipping is performed. If the bit rate at time t is greater than that at the previous time, the number of frame skips is the difference between the bit rates at the two times divided by the average bit rate per frame (bpf) at the previous time. t-1 Formula (4) is the formula for calculating bpf, where fps t Let t be the frame rate at time t.

[0022] S t ={loss t ,rtt t delta t ,packets t ,deviation t} (1)

[0023] R t =a*packets t +b*bitrate t +c*loss t+d*rtt t +e*deviation t (2)

[0024]

[0025]

[0026] To improve the resolution of cloud gaming videos, this invention proposes a real-time super-resolution model for cloud gaming with high frame rates, high resolution, and low latency. The model first divides the input video frame into 10×10 pixel blocks. For each pixel block, two convolutional layers are used to compute a 3×3 convolutional kernel with 3 input channels and 16 output channels. This kernel is then applied to the corresponding pixel block. Finally, the segmented video frames are reassembled, and three convolutional layers and a pixel reconstruction layer are used to improve the resolution of the video frames.

[0027] This invention proposes a real-time frame interpolation model for cloud gaming. The model is trained on a core cloud server before the game starts and inferred on an edge cloud server after the game begins. Each encoder in the model consists of two 3×3 convolutional layers, each activated by a PReLU layer. Each input frame undergoes four encoding operations, passing through a total of eight convolutional layers. By progressively reducing the spatial size, the number of feature channels is gradually increased to 12, 18, 27, and 36, generating feature E for input frame I0 and frame I1, respectively. 0j E 1j (j = 1, 2, 3, 4); Each decoder in the frame interpolation model consists of a 3×3 convolutional layer, a PReLU activation layer, a residual block, and a 4×4 deconvolution layer, where the residual block consists of five 3×3 convolutional layers and four PReLU activation layers; after frames I0 and I1 pass through four encoders in sequence, feature E is obtained. 04 E 14 Feature E 04 E 14 The flow field D is obtained by gradually refining the data through a decoder. 0j D 1j (j = 1, 2, 3, 4) and intermediate features T of higher-level reconstruction i (j = 1, 2, 3); the flow field D and the corresponding feature E are reverse-distorted and then enter the next layer of decoder together with the original flow field and intermediate features; when passing through the decoder for the last time, the output flow field D is output. 01 D 11 The single-channel mask M and the three-channel image residual R for detail compensation are used to finally calculate the synthesized intermediate frame I. t .

[0028] The cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithms meets the characteristics and practical needs of cloud gaming, namely high frame rate, high bit rate, and low latency, and can improve the player's gaming experience.

[0029] The present invention has the following advantages over the prior art:

[0030] This invention utilizes an edge computing architecture to propose a reinforcement learning-based bitrate adaptive algorithm and introduces a video enhancement model. Compared with other existing technologies, the bitrate adaptive algorithm can make reasonable transmission bitrate decisions in a timely manner. Frame skipping coding and resolution reduction schemes can quickly match the coding rate with the bitrate decision, while effectively reducing the transmission pressure on the backbone network. The video enhancement technology can restore high-quality game videos on edge cloud servers, thereby achieving a good and stable cloud gaming service. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention;

[0032] Figure 2 This is the overall design architecture diagram of the present invention;

[0033] Figure 3 This is a diagram of the bit rate adaptive model based on reinforcement learning of the present invention;

[0034] Figure 4 This is a super-resolution model diagram for cloud gaming based on the present invention;

[0035] Figure 5 This is a frame interpolation model diagram for cloud gaming based on the present invention. Detailed Implementation

[0036] To more clearly illustrate the technical solution of the present invention, a detailed description will be provided below in conjunction with the accompanying drawings and embodiments.

[0037] This invention builds Figure 1 The network architecture shown uses one NVIDIA GeForce RTX 3090 as the core cloud server for cloud gaming, multiple NVIDIA GeForce RTX 3080s as edge cloud servers, and multiple Intel(R) Core(TM) i7-9700s as terminals simulating cloud game players. The entire system is implemented in a WebRTC architecture. TCP is used for the initial connection establishment. After exchanging SDP information, the TCP connection is closed, and RTP is used for real-time video transmission. Network status information is obtained through RTCP packets.

[0038] Game players register or log in to the core cloud server and start the game. Under the deployment of the core cloud, they join the corresponding edge cloud server nearby. The core cloud server needs to transmit the rendered high frame rate, high resolution game video to the edge cloud server according to the player's operation. Therefore, this invention designs a bit rate adaptive decision based on reinforcement learning algorithm between the core cloud server and the edge cloud server, thereby alleviating the congestion problem of the backbone network. Figure 2 This paper demonstrates a reinforcement learning model for a bitrate adaptive algorithm, including state, action, and reward. The reinforcement learning agent continuously observes the instantaneous network state and, at a certain moment, decides on an action to adjust the transmission bitrate, anticipating a match with the available bandwidth of the current network. The bitrate action determined by the agent is then used to encode the game video on a video codec, and the encoded video is sent to an edge server. After the data passes through the network, a new state is generated, initiating a new round of reinforcement learning bitrate prediction.

[0039] Experiments revealed that the video encoder could not quickly encode the video stream using the bitrate determined by the bitrate adaptive algorithm. Increasing the encoding bitrate from 5 Mbps to 10 Mbps took approximately 10 seconds, meaning each 1 Mbps increase required 2 seconds. To ensure the video encoder's encoding bitrate changes according to the bitrate adaptive algorithm's decisions, this invention sets the reinforcement learning decision frequency to the second level. Simultaneously, it proposes frame skipping coding and reducing the resolution of video frames to guarantee that the bitrate determined by the bitrate adaptive algorithm matches the encoding rate in a timely manner.

[0040] Cloud gaming systems should avoid affecting the perceived quality of game videos due to network fluctuations or jumps in transmission bit rate decisions. Therefore, this invention incorporates two real-time video enhancement models, super-resolution and frame interpolation, into the edge cloud server to reduce the video quality degradation caused by changes in the decisions of the reinforcement learning agent. Figure 3 For super-resolution models, Figure 4 This is a frame interpolation model. For different cloud gaming scenarios, the video enhancement model is trained on the core cloud server and periodically deployed to edge cloud servers. When the edge cloud server receives a game video with a low frame rate or low resolution, it first performs super-resolution inference, and then performs frame interpolation inference. To ensure the effect of video enhancement, this invention only performs super-resolution inference with a magnification of 2x.

[0041] To achieve the low latency requirements of cloud gaming, this invention does not re-encode all video frames on the edge server. Instead, it encodes the enhanced frames separately as video streams and sends the original low-frame-rate, low-resolution video stream received from the core cloud server together with the stream to the player's terminal. The player's terminal decodes both video streams in parallel and plays them, then sends the player's next action to the core cloud server.

[0042] To demonstrate the effectiveness of the bit rate adaptive algorithm of this invention, it was compared with two classic real-time communication congestion control algorithms, GCC and ONRL, with random bit rate as the baseline method. The bandwidth fluctuation was simulated using a 300-second bandwidth segment from the FCC18 dataset in the PiTree dataset. The bit rate adaptive algorithm and the ONRL algorithm of this invention were pre-trained for 24 hours on 30 randomly selected bandwidths from the FCC18 dataset with bandwidth fluctuations of approximately 20 Mbps. The random bit rate range was [5 Mbps, 21 Mbps].

[0043] Table 1 presents the experimental results, demonstrating improvements in various performance aspects. Regarding bandwidth utilization, the GCC algorithm achieves only 60%, while this invention achieves higher bandwidth utilization than the ONRL algorithm. The packet loss rate of this invention is reduced by 8.7% compared to ONRL, and the latency is reduced by 5.9% compared to ONRL, similar to the GCC algorithm. Since the GCC algorithm employs conservative bandwidth estimation, bit rate execution deviation is not a significant issue, therefore, GCC execution deviation was not statistically analyzed. Due to the frame rate and resolution adaptation mechanisms of this invention, the bit rate execution deviation is significantly reduced, decreasing by approximately 35% compared to the ONRL algorithm, resulting in effective algorithm execution.

[0044] Table 1 Performance Comparison of Bit Rate Adaptive Algorithms

[0045] GCC 60.42 1.8004 6.3825 / Random bit rate 71.84 2.9569 5.2603 394028 ONRL 74.28 2.5417 6.7529 301536 This invention 80.57 2.3240 6.3622 191745

[0046] To demonstrate the effectiveness of the video enhancement model of this invention, the super-resolution model of this invention is compared with four super-resolution methods: RDN, SRCNN, EGVSR, and Bicubic. The frame interpolation model of this invention is also compared with three frame interpolation models: large, medium, and small, within IFRNet. All models were trained on the same game video for 200 rounds before inference. Table 2 shows the performance comparison of the super-resolution models, revealing that the super-resolution model of this invention has the best performance, with an inference time of less than 33ms, which meets the inference requirements for 30fps videos. Table 3 shows the performance comparison of the frame interpolation models, where L represents the large model and S represents the small model. The small and large models of this invention show slightly lower performance than IFRNet, with PSNR reduced by approximately 2%, SSIM by approximately 0.2%, MS-SSIM by approximately 0.8%, and VMAF by approximately 1%. However, the frame interpolation model of this invention significantly reduces inference time, by 62% and 51% respectively compared to the IFRNet model, both within 33ms, making the frame interpolation model applicable to high frame rate, high bit rate, and low latency cloud games.

[0047] Table 2 Performance Comparison of Super-Resolution Models

[0048] This invention 28.9451 40.2455 0.9784 100 2.7MB RDN 71.0374 39.9572 0.9772 95.3107 4.1MB SRCNN 13.0923 37.0092 0.9702 84.9009 83KB EGVSR 69.4792 20.1910 0.8534 19.2685 10MB BICUBIC 0.2725 32.6132 0.9521 86.1225 /

[0049] Table 3 Performance Comparison of Frame Interpolation Models

[0050] IFRNet_L 196.5254 34.1151 0.9870 93.4535 75MB IFRNet 66.0976 34.4065 0.9876 93.3089 19MB IFRNet_S 52.9963 34.2306 0.9871 93.1573 11MB This invention_L 32.4752 33.7603 0.9863 92.5475 5MB This invention_S 25.0486 33.4210 0.9856 92.3359 3MB

[0051] It should be clarified that the above embodiments are only used to further illustrate the technical solution of the present invention, and are not intended to limit the scope of protection of the present invention. Any obvious improvements made on the basis of the technical concept of the present invention should fall within the scope of protection of the present invention.

Claims

1. A cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithms, the method comprising: Step 1: The game player registers and logs in to the core cloud server and starts the game. Under the deployment of the core cloud, the player joins the corresponding edge cloud server. The core cloud server updates the latest game super-resolution model and real-time frame interpolation model to the edge cloud server. After that, the game player's game video stream is forwarded through this edge cloud server. Step 2: After the game starts, the player sends the game operation to the core cloud server. The core cloud server renders the game screen in real time according to the player's operation, and makes a transmission bit rate decision based on the reinforcement learning-based bit rate adaptive algorithm. The game screen is then encoded in real time and transmitted to the edge cloud server. For cloud gaming video transmission and enhancement, a bitrate adaptive algorithm is combined with video enhancement technology. The bitrate adaptive algorithm based on reinforcement learning determines the transmission bitrate, and the number of frame skips and video resolution during encoding are determined based on the transmission bitrate. Then, the corresponding game video encoding is performed, thereby reducing transmission latency, reducing backbone network transmission pressure, and avoiding congestion and stuttering. Formula (1) is the state formula for reinforcement learning. (1) loss t For packet loss rate, RTT t For delay, delta t The difference between the arrival interval at the receiver and the corresponding departure interval at the sender for two consecutive RTP packets is given by packets. t The deviation is the number of RTP packets received by the receiver within the DRL algorithm update interval. t Let t be the difference between the actual transmitted bit rate at time t and the bit rate determined by the DRL algorithm. Formula (2) is the reward formula for the bit rate adaptive algorithm. (2) packets t The bitrate is the total number of data packets received by the receiver within the interval. t Let be the bit rate of the DRL algorithm decision at time t, and let loss be... t The packet loss rate within the time interval, RTT t Deviation is the round-trip delay within the time interval. t For bit rate execution deviation, a, b, c, d, and e are the weights of the corresponding indicators; Formula (3) is the frame skipping decision formula. (3) r t Let be the bit rate at time t. If the bit rate at time t is less than that at the previous time, no frame skipping is performed. If the bit rate at time t is greater than that at the previous time, the number of frame skips is the difference between the bit rates at the two times divided by the average bit rate per frame (bpf) at the previous time. t-1 Formula (4) is the formula for calculating bpf. (4) fps t Let be the frame rate transmitted at time t; Step 2.1: If the transmission bit rate determined by the bit rate adaptive algorithm is high, then a high frame rate and high bit rate game video is sent to the edge cloud server. After receiving the game video, the edge cloud server directly forwards it to the player's terminal device. Step 2.2: If the transmission bit rate determined by the bit rate adaptive algorithm is low, a game video with a lower frame rate and lower bit rate is sent to the edge cloud server. The edge cloud server performs video enhancement on the low-quality video, and then encodes the enhanced video stream and sends it together with the original video stream to the player's terminal device. Step 3: The player's terminal device decodes and merges the received game video stream and plays it, and sends the player's next action to the core cloud server.

2. The cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithm according to claim 1, characterized in that, The bit rate adaptive algorithm based on reinforcement learning is: The core cloud server continuously observes the instantaneous network state based on reinforcement learning algorithms, learns the changes in the network state at different times, and determines a transmission bit rate that matches the available bandwidth of the current network. If the determined transmission bit rate is high, high-quality game video is transmitted directly; otherwise, decisions are made to skip frames and reduce the video frame resolution, and the corresponding frame rate and resolution are encoded on the video codec, and the low-quality game video stream is sent to the edge server.

3. The cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithm according to claim 1, characterized in that, The video enhancement includes: The core cloud server trains personalized super-resolution and frame interpolation models for different game scenarios and periodically deploys them to the edge cloud server. The edge cloud server first performs real-time super-resolution based on deep learning to obtain low frame rate, high resolution video from the received low frame rate, low resolution video, and then performs real-time frame interpolation based on deep learning to finally obtain high-quality video.

4. The cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithm according to claim 1, characterized in that, The encoding and decoding of the enhanced video stream and the original video stream are as follows: The edge cloud server forwards the received video stream directly to the game player, decodes and enhances the original video stream, and then encodes the enhanced video frames before sending them to the game player. The game player receives the original video stream and the enhanced video stream, decodes them in parallel, and uses the enhanced video frames to replace or insert into the original video stream for playback.

5. The cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithm according to claim 1, characterized in that, The game super-resolution model first divides the input video frame into small pixel blocks of 10×10 pixels. For each small pixel block, two convolutional layers are used to compute a 3×3 convolutional kernel with 3 input channels and 16 output channels. The convolutional kernel is then applied to the corresponding small pixel block. Finally, the segmented video frame is reassembled, and the resolution of the video frame is improved using 3 convolutional layers and a pixel reassembly layer.

6. The cloud gaming video transmission method based on video enhancement and bitrate adaptive algorithm according to claim 1, characterized in that, Each encoder in the real-time frame interpolation model consists of two 3×3 convolutional layers, each activated by a PReLU layer. It performs four encoding operations on each input frame, passing through a total of eight convolutional layers. By progressively reducing the spatial size, the number of feature channels is gradually increased to 12, 18, 27, and 36, generating feature E for input frame I0 and frame I1, respectively. 0j E 1j j=1, 2, 3, 4; each decoder in the frame interpolation model consists of a 3×3 convolutional layer, a PReLU activation layer, a residual block, and a 4×4 deconvolution layer, where the residual block consists of five 3×3 convolutional layers and four PReLU activation layers; feature E is obtained after frames I0 and I1 pass through four encoders sequentially. 04 E 14 Feature E 04 E 14 The flow field D is obtained by gradually refining the data through a decoder. 0j D 1j The intermediate features T of j=1, 2, 3, 4 and higher-level reconstructions i j=1, 2, 3; the flow field D and the corresponding feature E are reverse-distorted and then enter the next layer of decoder together with the original flow field and intermediate features; when passing through the decoder for the last time, the output flow field D is output. 01 D 11 The single-channel mask M and the three-channel image residual R for detail compensation are used to finally calculate the synthesized intermediate frame I. t .

Images