Cloud game picture processing method and apparatus, and electronic device and storage medium

Abstract

The present application relates to the technical field of image processing. Provided are a cloud game picture processing method and apparatus, and an electronic device and a storage medium. The method comprises: a client sending a display refresh rate to a server; on the basis of the display refresh rate, the server determining a rendering frame rate of the server; on the basis of the rendering frame rate of the server, the server performing rendering and encoding processing on cloud game picture data, so as to generate encoded data and transmit same to the client; the client receiving the encoded data and performing decoding and rendering processing on the encoded data to obtain decoded data, wherein a rendering frame rate of the decoded data is less than or equal to the display refresh rate; and the client adjusting the display refresh rate to be consistent with the rendering frame rate of the decoded data. By adjusting the rendering frame rate of the server on the basis of the display refresh rate of the client, the present application provides an effective upward redundancy space for a frame rate of data transmission, thereby effectively eliminating the problem of picture tearing in a cloud game by adjusting the display refresh rate.

Description

A cloud gaming screen processing method, apparatus, electronic device, and storage medium

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411696313.5, filed on November 25, 2024, entitled "A cloud gaming screen processing method, apparatus, electronic device and storage medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and more specifically, to a cloud gaming screen processing method, apparatus, electronic device, and storage medium. Background Technology

[0004] The digital wave is sweeping the globe, and cloud computing has become the core of the IT (Information Technology) industry. Cloud gaming, as an emerging technological force, shifts game rendering from player devices to centralized servers, revolutionizing the gaming experience. However, high frame rate cloud gaming faces the challenge of screen tearing. Vertical synchronization (VPS) and G-Sync / FreeSync technologies can respectively solve the problem of screen tearing caused by the graphics card's frame rate being higher or lower than the monitor's refresh rate. The former leads to latency and stuttering, while the latter is only suitable for local gaming scenarios.

[0005] Cloud gaming is more complex due to the processes involved, including server rendering, encoding, network transmission, client decoding, and display. G-Sync / FreeSync technologies require communication between the graphics card and the monitor to synchronize frame rates, but in cloud gaming, the graphics card is on the server and the monitor is on the client, making synchronization impossible. Furthermore, due to network transmission, even if the server-side frame rate and the client's monitor refresh rate are kept constant, the frame generation time on the client side becomes unstable after encoding / decoding and network transmission delays, resulting in screen tearing. Therefore, eliminating screen tearing in cloud gaming is a pressing issue that needs to be addressed. Summary of the Invention

[0006] This application provides a cloud gaming screen processing method, apparatus, electronic device, and storage medium, aiming to solve the problem of screen tearing in cloud gaming.

[0007] The first aspect of this application provides a cloud gaming screen processing method, applied to a system including a server and a client, the cloud gaming screen processing method including:

[0008] The client sends the display refresh rate to the server;

[0009] The server determines its rendering frame rate based on the monitor's refresh rate, and the upper limit of the server's rendering frame rate is less than or equal to the monitor's refresh rate.

[0010] The server renders and encodes the cloud gaming screen data based on the server's rendering frame rate, and then transmits the encoded data to the client.

[0011] The client receives encoded data and decodes and renders it to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate.

[0012] The client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

[0013] In some embodiments of this application, the server determines the server's rendering frame rate based on the display refresh rate, including:

[0014] Based on the decoded data obtained in the previous second, the client generates the frame rate adjustment ratio for the current second and sends it to the server.

[0015] The server receives the frame rate adjustment ratio for the current second from the client, and adjusts the display refresh rate downwards based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

[0016] In some embodiments of this application, the client generates the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second, including:

[0017] Based on the decoded data obtained in the previous second, the client obtains the latency difference ratio of all frames in the previous second;

[0018] Based on the latency difference ratio of all frames in the previous second, the first perturbation frame in the previous second is determined. The first perturbation frame is the frame whose latency difference ratio is greater than or equal to the first threshold. The first threshold is equal to the frame rate adjustment ratio corresponding to the previous second.

[0019] If the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to the first preset proportion, the frame rate adjustment proportion corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment proportion for the current second.

[0020] In some embodiments of this application, after obtaining the latency difference ratio of all frames in the previous second, the method further includes:

[0021] Based on the latency difference ratio of all frames in the previous second, the second perturbation frame in the previous second is determined. The second perturbation frame is a frame whose latency difference ratio is less than or equal to the second threshold. The second threshold is the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value.

[0022] If the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion, the frame rate adjustment proportion corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment proportion for the current second.

[0023] In some embodiments of this application, the cloud gaming screen processing method further includes:

[0024] If the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset proportion, and / or if the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset proportion, the frame rate adjustment proportion corresponding to the previous second shall be used as the frame rate adjustment proportion for the current second.

[0025] In some embodiments of this application, the latency difference ratio of each frame in the decoded data is obtained in the following manner:

[0026] The client obtains the latency of each frame in the decoded data;

[0027] The client obtains the delays corresponding to a preset number of frames before the target frame, and determines the average delay based on the delays corresponding to the preset number of frames. The target frame is any frame in the decoded data.

[0028] The client determines the latency difference ratio of the target frame based on the latency of the target frame and the average latency.

[0029] In some embodiments of this application, the client obtains the latency of each frame in the decoded data, including:

[0030] When the server encodes the cloud gaming screen data, it generates the first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client.

[0031] When the client decodes the encoded data, the client generates a second timestamp for each frame in the decoded data.

[0032] The client uses the difference between the first and second timestamps of each frame as the delay for each frame in the decoded data.

[0033] A second aspect of this application provides a cloud gaming screen processing apparatus, the apparatus comprising:

[0034] The first sending module is used by the client to send the display refresh rate to the server.

[0035] The frame rate determination module is used by the server to determine the server's rendering frame rate based on the monitor refresh rate. The upper limit of the server's rendering frame rate is less than or equal to the monitor refresh rate.

[0036] The encoding module is used by the server to render and encode cloud gaming screen data based on the server's rendering frame rate, and generate encoded data to be transmitted to the client.

[0037] The decoding module is used by the client to receive encoded data and decode and render the encoded data to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate.

[0038] The adaptive synchronization module is used by the client to adjust the display refresh rate to match the rendering frame rate of the decoded data.

[0039] The third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in the cloud gaming screen processing method of any one of the first aspects of this application.

[0040] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the cloud gaming screen processing method of any one of the first aspects of this application. Beneficial effects:

[0041] This application provides a cloud gaming screen processing method, apparatus, electronic device, and storage medium, applied to a system including a server and a client. The method includes: the client sending its display refresh rate to the server; the server determining its rendering frame rate based on the display refresh rate, wherein the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate; the server rendering and encoding cloud gaming screen data based on the server's rendering frame rate to generate encoded data which is then transmitted to the client; the client receiving the encoded data and decoding and rendering it to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate; and the client adjusting its display refresh rate to match the rendering frame rate of the decoded data. This application, by adjusting the server's rendering frame rate according to the client's display refresh rate, provides effective upward redundancy for the data transmission frame rate, ensuring that the client's decoded frame rate remains less than or equal to the display refresh rate, thereby effectively eliminating screen tearing issues in cloud gaming by adjusting the display refresh rate. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 is a flowchart of a cloud gaming screen processing method according to an embodiment of this application;

[0044] Figure 2 is a flowchart of example steps of a cloud gaming screen processing method according to an embodiment of this application;

[0045] Figure 3 is a schematic diagram of the logical interaction of various modules in an existing cloud gaming screen processing method;

[0046] Figure 4 is a schematic diagram of the logical interaction of each module in a cloud gaming screen processing method proposed in an embodiment of this application;

[0047] Figure 5 is a schematic diagram of the structure of a cloud gaming screen processing device according to an embodiment of this application;

[0048] Figure 6 is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] In the accompanying drawings, the size of constituent elements, the thickness of layers, or areas may sometimes be exaggerated for clarity. Therefore, any implementation of this application is not necessarily limited to the dimensions shown in the drawings, and the shapes and sizes of components in the drawings do not reflect true proportions. Furthermore, the drawings schematically illustrate ideal examples, and any implementation of this application is not limited to the shapes or values ​​shown in the drawings.

[0051] The digital wave is sweeping the globe, and cloud computing has become the core of the IT industry's development. Cloud gaming, as an emerging technological force, shifts game rendering from player devices to centralized servers, revolutionizing the gaming experience. However, high frame rate cloud gaming faces the challenge of screen tearing.

[0052] Screen tearing is a display issue that can occur in video games. It typically happens in fast-moving scenes or when frame rates are out of sync (the graphics card's output frame rate is higher or lower than the monitor's refresh rate). This problem causes a mismatch between two adjacent frames on the screen, making the image appear "torn" into two parts, with the upper and lower halves not fully synchronized, severely impacting the user and player experience. Screen tearing has plagued players since the birth of gaming. In traditional native games, screen tearing occurs because the monitor's refresh rate and the graphics card's frame rate are out of sync. When the monitor refreshes the screen, if the graphics card simultaneously generates the next frame, the registers in the video memory that store the rendered image are switching between the two frames. When the monitor reads the registers, the upper and lower halves of the image will come from different frames, causing screen tearing. Therefore, screen tearing will occur when the graphics card's output frame rate is higher or lower than the monitor's refresh rate.

[0053] To address screen tearing, the industry commonly uses Vertical Sync (V-Sync) technology. This technology effectively reduces screen tearing by lowering the graphics card's frame rate to match the monitor's refresh rate. However, V-Sync is not a perfect solution. It causes the graphics card to wait for the monitor to refresh before rendering the image, resulting in input lag. Furthermore, this function uses a fixed monitor refresh rate. When the graphics card's frame rate is lower than the monitor's refresh rate, V-Sync will limit the graphics card's frame rate to half or a quarter of the monitor's refresh rate (multiples will not cause screen tearing), resulting in a significant drop in frame rate and affecting game smoothness. Therefore, this technology is mainly used in scenarios where the graphics card's frame rate is higher than the monitor's refresh rate. Weighing the pros and cons, some gamers choose to disable V-Sync to pursue higher gaming performance.

[0054] To address the shortcomings of vertical synchronization, graphics card manufacturers introduced G-Sync and FreeSync technologies. These technologies work by dynamically adjusting the monitor's refresh rate to match the graphics card's frame rate. When the graphics card's frame rate is between its lower limit (G-Sync supports ultra-lower limits of 4-10Hz, while G-Sync Compatible and FreeSync are mostly 42Hz) and the monitor's maximum refresh rate, the graphics card does not need to reduce its frame rate. This effectively solves the problem of screen tearing while ensuring smooth gameplay and low latency. Therefore, this technology is mainly used in scenarios where the graphics card's frame rate is lower than the monitor's maximum refresh rate, but it requires monitor compatibility.

[0055] Vertical sync and G-Sync / FreeSync technologies can solve the screen tearing problem caused by the graphics card's frame rate being higher or lower than the monitor's refresh rate, respectively. Vertical sync causes latency and stuttering, while G-Sync / FreeSync is only suitable for local gaming scenarios. Cloud gaming is more complex due to the server-side rendering, encoding, network transmission, client-side decoding, and display processes. G-Sync / FreeSync requires communication between the graphics card and the monitor to achieve frame rate synchronization, but in cloud gaming, the graphics card is on the server and the monitor is on the client, making communication synchronization impossible. Furthermore, due to the network transmission involved, even if the server-side frame rate and the client's monitor refresh rate are fixed, the frame generation time on the client side becomes unstable after encoding / decoding and network transmission delays. For example, even if the server stably generates 60 FPS (Frames Per Second), the frame rate received by the client often fluctuates between 50 and 70 FPS after encoding / decoding and network transmission. Therefore, even if the monitor is set to a 60 FPS refresh rate, screen tearing will still occur on the client side.

[0056] In view of this, this application proposes a cloud gaming screen processing method. This method is applied to a system including a server and a client. The server is the core part of the cloud gaming service and typically resides on a virtual server (vMonitor) provided by the cloud gaming provider. The virtual server uses software methods to simulate the function of a physical monitor. The screen rendered by the graphics card is transmitted to the virtual monitor for cloud gaming screen capture. Compared to a physical monitor, the virtual monitor has more flexible settings to support a large number of players playing online simultaneously and provide a smooth gaming experience. A complete game program runs on the cloud gaming server. When a player connects to the server through a client, the server renders the game screen in real time, processes the game logic, and sends the results to the client as a video stream. Simultaneously, the server receives player input (such as key presses, mouse movements, etc.) from the client and updates the game status accordingly. The client typically refers to the device or interface used by the player to access the cloud gaming service. These devices can be smartphones, tablets, smart TVs, personal computers (PCs), etc.

[0057] In this embodiment of the application, Figure 1 shows a flowchart of a cloud gaming screen processing method according to an embodiment of the application. As shown in Figure 1, the method includes the following steps:

[0058] S101, The client sends the display refresh rate to the server.

[0059] In specific implementation step S101, the client first obtains the refresh rate of its display. Since screen tearing is caused by the asynchrony between the client's display refresh rate and the graphics card frame rate, when the display refresh rate is higher than the graphics card frame rate, the client's display can adjust its refresh rate through adaptive synchronization to keep it consistent with the graphics card frame rate, avoiding screen tearing caused by some frames having a lower frame rate than the display refresh rate. However, in cloud gaming scenarios, when the server transmits data to the client, the frame rate received by the client will fluctuate further, causing more frames to have a higher frame rate than the display refresh rate. This reduces the number of frames that the display refresh rate can cover, increasing the screen tearing problem. Therefore, in this embodiment, the client's display refresh rate is sent to the server so that the server can adjust its rendering frame rate based on the display refresh rate. This ensures that the frame rate of more or even all frames in the data transmitted to the client can be covered by the display refresh rate, thereby using adaptive synchronization to keep the display refresh rate consistent with the frame rate of all frames, effectively eliminating the screen tearing problem.

[0060] In some embodiments of this application, the server in this application includes a virtual display, which has the function of supporting dynamic modification of the server's rendering frame rate, so that the server can dynamically adjust the rendering frame rate of the virtual display based on the display refresh rate.

[0061] S102. The server determines the server's rendering frame rate based on the monitor's refresh rate. The upper limit of the server's rendering frame rate is less than or equal to the monitor's refresh rate.

[0062] In specific implementation step S102, in the cloud gaming scenario, after the cloud gaming screen data from the server is transmitted to the client, the client's rendering frame rate will fluctuate further compared to the server's rendering frame rate due to factors such as encoding / decoding latency and network latency. This results in a greater difference from the monitor refresh rate. For example, if the monitor refresh rate is 60 FPS, and the server's rendering frame rate is set to the same 60 FPS as the monitor refresh rate, the client's rendering frame rate will often fluctuate between 50 FPS and 70 FPS after the data is transmitted to the client. This will cause half of the frames to have a rendering frame rate greater than the monitor refresh rate, resulting in screen tearing in these frames.

[0063] Therefore, to reduce or even eliminate the problem of the client's rendering frame rate exceeding the monitor's refresh rate, the server's rendering frame rate needs to be adjusted based on the monitor's refresh rate. This ensures that the upper limit of the server's rendering frame rate is less than or equal to the monitor's refresh rate. The portion of the server's rendering frame rate that is less than the monitor's refresh rate serves as upward redundancy space for unstable frame generation time (frame rate fluctuations) caused by latency after data frames arrive at the client. This minimizes screen tearing caused by the client's decoded frame rate exceeding the monitor's refresh rate. In some embodiments of this application, the server's adaptive vertical synchronization function can be enabled via the graphics card API (Application Programming Interface) to control the server's rendering frame rate between 0 FPS and the upper limit of the server's rendering frame rate.

[0064] For example, if the monitor refresh rate is 60 FPS, the client obtains the monitor refresh rate and sends it to the server. The server uses the monitor refresh rate of 60 FPS as a benchmark and lowers its rendering frame rate to a maximum of 50 FPS. When the cloud gaming screen data is transmitted to the client after encoding, decoding and network transmission based on a rendering frame rate of 50 FPS, the client's rendering frame rate is within the range of 40 FPS to 60 FPS due to latency disturbances. That is, it is still less than or equal to the monitor refresh rate. The client can adjust the monitor refresh rate and the client's rendering frame rate to be consistent through adaptive synchronization, effectively eliminating screen tearing.

[0065] In some embodiments of this application, since the frame rate fluctuation caused by the transmission delay between the server and the client changes at different time points, in order to better ensure that the client's rendering frame rate is effectively covered by the display refresh rate, the server's rendering frame rate is dynamically adjusted during transmission based on the delay of the previous second. That is, if the frame rate fluctuation caused by the transmission delay of the previous second is large, in order to ensure that the client's rendering frame rate in the next second is less than or equal to the display refresh rate, the adjustment value of the server's rendering frame rate is increased, thereby increasing the upward redundancy range of the frame rate fluctuation. Conversely, if the frame rate fluctuation caused by the transmission delay of the previous second is small, in order to ensure that the client's rendering frame rate in the next second is less than or equal to the display refresh rate, the adjustment value of the server's rendering frame rate is decreased, thereby decreasing the upward redundancy range of the frame rate fluctuation.

[0066] Specifically, the server-side rendering frame rate is determined as follows: First, the client generates the current second's frame rate adjustment ratio (n%) based on the decoded data obtained in the previous second and sends it to the server. The frame rate adjustment ratio is configured to determine the fluctuation of all frames in the decoded data obtained in the previous second. The frame rate adjustment ratio is determined by the client based on the timestamp information sent from the server to the client in the previous second. Subsequently, the server receives the current second's frame rate adjustment ratio sent by the client and adjusts the display refresh rate downward based on the current second's frame rate adjustment ratio to obtain the server-side rendering frame rate for the current second.

[0067] In some embodiments of this application, the server's rendering frame rate for the current second is obtained by reducing the frame rate adjustment ratio for the current second based on the display refresh rate. That is, the frame rate adjustment ratio for the current second forms an upward redundancy space range with the server's rendering frame rate for the current second. After the server encodes and transmits the cloud game screen data to the client based on the server's rendering frame rate for the current second, the frame rate of the decoded data formed by the client after decoding processing fluctuates upward within the upward redundancy space, but is still less than or equal to the display refresh rate, thereby ensuring that the display refresh rate can be kept consistent with the client's rendering frame rate through adaptive synchronization.

[0068] For example, if the display refresh rate is 240Hz, the client generates a frame rate adjustment ratio (n%) of 10% for the current second based on the decoded data obtained in the previous second. The client then sends this frame rate adjustment ratio to the server. Subsequently, the server receives the frame rate adjustment ratio sent by the client and adjusts the display refresh rate by 10% based on this ratio, resulting in a maximum rendering frame rate of 216Hz for the current second. Since the server's rendering frame rate is within the range of 0 to 216Hz, the rendering frame rate of the data frames after client decoding remains within the range of 240Hz.

[0069] S103. The server renders and encodes the cloud gaming screen data based on the server's rendering frame rate, and generates encoded data which is then transmitted to the client.

[0070] In specific implementation step S103, after the server determines the server's rendering frame rate, the server renders the cloud gaming screen data according to the server's rendering frame rate, then performs encoding processing to generate encoded data, and sends the encoded data to the client.

[0071] In some embodiments of this application, since the client needs to obtain the frame rate adjustment ratio for the current second based on the decoded data of the previous second, and the frame rate adjustment ratio for the current second needs to be determined based on the latency between the server and the client, the server needs to provide the client with the encoding time node when generating the encoded data. Specifically, after the streaming starts, after the server's graphics card finishes rendering a frame, when the server performs encoding processing on the cloud gaming screen data, the server generates the first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client. The first timestamp information is used to characterize the encoding time node of each frame in the cloud gaming screen data.

[0072] S104. The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain the decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate.

[0073] In specific implementation step S104, the client receives the encoded data sent by the server and sequentially decodes and renders the encoded data to obtain decoded data. The decoded data is then stored in the client's frame buffer, and the cloud gaming screen is displayed on the client's display interface using the decoded data in the frame buffer. Simultaneously, the client also generates the frame rate adjustment ratio for the current second, dynamically adjusting the frame rate adjustment ratio every second through negative feedback, ensuring that the client's rendering frame rate is overridden by the display refresh rate every second.

[0074] Figure 3 illustrates the logical interaction of various modules in an existing cloud gaming screen processing method. As shown in Figure 3, in the existing cloud gaming screen processing method, the server-side encoding module encodes the rendered cloud gaming screen data and sends it to the client. Since the server-side rendering frame rate is directly synchronized with the monitor refresh rate, when the client-side decoding module decodes the data and renders it, the rendering frame rate of the decoded data will be disturbed compared to the server-side rendering frame rate due to the transmission delay between the server and the client. This results in the rendering frame rate of some frames being higher than the monitor refresh rate, making it impossible for the monitor refresh rate of these frames to be adjusted to match the rendering frame rate through adaptive vertical synchronization, causing screen tearing in the cloud gaming screen.

[0075] In this embodiment of the application, Figure 4 shows a schematic diagram of the logical interaction of each module in a cloud gaming screen processing method proposed in an embodiment of the application. As shown in Figure 4, the server includes a rendering module, an encoding module, and a virtual display. The rendering module performs rendering processing on cloud gaming screen data based on a vertical synchronization adaptive mode. At the same time, it adjusts the rendering frame rate of the server by adjusting the frame rate ratio n%. The refresh rate of the virtual display is (100-n)%A, so that the refresh rate of the virtual display (the rendering frame rate of the server) is less than or equal to the display refresh rate A. This provides sufficient upward redundancy space for the rendering frame rate, so that when the decoding processing generated by the client's decoding module is rendered on the client, the rendering frame rate of the decoded data is still less than or equal to the display refresh rate A after being disturbed by the transmission delay between the server and the client. Meanwhile, in order to further improve the effective coverage of the display refresh rate with the rendering frame rate of the decoded data every second, the client also generates the frame rate adjustment ratio for the current second through the decoding data of the previous second, so as to dynamically adjust the frame rate adjustment ratio for each second through negative feedback, thereby dynamically adjusting the upper limit of the server's rendering frame rate for each second.

[0076] In some embodiments of this application, firstly, the client obtains the latency difference ratio of all frames within the previous second based on the decoded data obtained in the previous second. The latency difference ratio is used to characterize the frame rate disturbance caused by transmission and latency after each frame of data is transmitted from the server to the client. In this embodiment, the adjustment direction of the frame rate adjustment ratio for the current second is determined by the latency difference ratio of all frames within the previous second. Specifically, the latency difference ratio of each frame in the decoded data is obtained as follows: the client obtains the latency of each frame in the decoded data, where latency characterizes the time delay of each frame's transmission from the server to the client; the client obtains the latency corresponding to a preset number of frames before the target frame, and determines the average latency based on the latency corresponding to the preset number of frames, where the target frame is any frame in the decoded data. For example, the preset number is 10 frames, the client obtains the latency corresponding to the 10 frames before the target frame, and uses the average of the latency corresponding to the 10 frames before the target frame as the average latency corresponding to the target frame.

[0077] Subsequently, the client obtains the difference between the delay of the target frame and the average delay based on the delay of the target frame and the average delay, as the delay difference; and obtains the proportion of the delay difference to the average delay of the target frame, which is determined as the delay difference proportion of the target frame.

[0078] In some embodiments of this application, a first perturbation frame is determined based on the latency difference ratio of all frames in the previous second. The first perturbation frame is a frame whose latency difference ratio is greater than or equal to a first threshold, which is equal to the frame rate adjustment ratio corresponding to the previous second. The first perturbation frame is used to characterize frames with a large degree of perturbation due to transmission latency in the previous second. If the proportion of the first perturbation frame to all frames in the previous second is large, it indicates that the adjustment range of the server's rendering frame rate according to the frame rate adjustment ratio of the previous frame is small. This may cause the client's rendering frame rate to exceed the display refresh rate when the data is transmitted in the current second, resulting in screen tearing in some frames. Therefore, compared to the frame rate adjustment ratio of the previous second, the server data in the current second requires a larger frame rate adjustment ratio during rendering.

[0079] Specifically, if the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to a first preset proportion, the frame rate adjustment proportion corresponding to the previous second is increased by the first preset value to obtain the frame rate adjustment proportion for the current second. For example, the frame rate adjustment proportion corresponding to the previous second is n%, the first preset proportion is 10%, the first preset value is 10, and the first threshold is n%. If the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to the first preset proportion of 10%, the frame rate adjustment proportion n% corresponding to the previous second is increased by 10 to obtain the frame rate adjustment proportion for the current second (n+10)%.

[0080] In some embodiments of this application, a second perturbation frame is determined based on the latency difference ratio of all frames in the previous second. The second perturbation frame is a frame whose latency difference ratio is less than or equal to a second threshold, which is the difference between the frame rate adjustment ratio corresponding to the previous second and a first preset value. The second perturbation frame is used to characterize frames with a smaller degree of perturbation due to transmission latency in the previous second. If the proportion of the second perturbation frame to all frames in the previous second is large, it indicates that the adjustment range of the server's rendering frame rate according to the frame rate adjustment ratio of the previous frame is large. Under the premise of avoiding screen tearing, the rendering frame rate can be increased by reducing the frame rate adjustment ratio to ensure the display effect on the client. Therefore, compared with the frame rate adjustment ratio of the previous second, the server data in the current second needs a smaller frame rate adjustment ratio during the rendering process.

[0081] Specifically, if the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to a second preset proportion, the frame rate adjustment proportion corresponding to the previous second is reduced by a first preset value to obtain the frame rate adjustment proportion for the current second. For example, if the frame rate adjustment proportion corresponding to the previous second is n%, the second preset proportion is 90%, and the first preset value is 10, then the second threshold (n-10)% is: if the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion of 90%, the frame rate adjustment proportion n% corresponding to the previous second is reduced by 10 to obtain the frame rate adjustment proportion for the current second (n-10)%. Since the frame rate adjustment proportion for each second is dynamically adjusted based on the frame rate adjustment proportion for the previous second, the frame rate adjustment proportion for each second is greater than or equal to 0.

[0082] In some embodiments of this application, if the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than a first preset proportion, and / or the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than a second preset proportion, it indicates that adjusting the server's rendering frame rate according to the frame rate adjustment ratio of the previous frame is appropriate. Adjusting the server's rendering frame rate according to the frame rate adjustment ratio of the previous second can ensure the display effect on the client while avoiding screen tearing. Therefore, in this case, the frame rate adjustment ratio corresponding to the previous second is directly used as the frame rate adjustment ratio of the current second.

[0083] In some embodiments of this application, the delay of each frame in the decoded data is the duration between the time node when the cloud game screen data is encoded on the server and the time node when the cloud game screen data is decoded on the client. Specifically, the delay of each frame in the decoded data is obtained according to the following steps: In this embodiment of the application, referring to the content in step S103 above, when the server encodes the cloud game screen data, the server generates first timestamp information corresponding to each frame in the cloud game screen data and sends it to the client; subsequently, the client receives the first timestamp information corresponding to each frame in the cloud game screen data; when the client decodes the encoded data, the client generates second timestamp information corresponding to each frame in the decoded data, and the second timestamp information is used to characterize the decoding time node of each frame in the cloud game screen data; the client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0084] S105. The client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

[0085] In specific implementation step S105, since the server's rendering frame rate is adjusted based on the monitor refresh rate, the rendering frame rate of the decoded data obtained after decoding and rendering on the client is less than or equal to the monitor refresh rate. The rendering frame rate of each frame in the decoded data is overridden by the monitor refresh rate. In this embodiment, the server uses adaptive vertical synchronization technology to adjust the monitor refresh rate to match the rendering frame rate of the decoded data, thereby achieving real-time synchronization between the rendering frame rate of each frame on the client and the monitor refresh rate, effectively eliminating screen tearing issues in cloud gaming on the client side.

[0086] In some embodiments of this application, the client enables the G-Sync / FreeSync function. The G-Sync / FreeSync function is used to adjust the display refresh rate in real time to adapt to the changes in the rendered frame rate after decoding, based on the client rendering frame rate which is lower than the display refresh rate, when generating the decoded data of each frame. This reduces the display refresh rate of each frame to be the same as the rendered frame rate of each frame, thereby effectively eliminating the problem of screen tearing in cloud gaming.

[0087] To enable those skilled in the art to more clearly understand this application, the cloud gaming screen processing method of this application will now be described in detail through the following embodiments.

[0088] Figure 2 shows an example flowchart of a cloud gaming screen processing method according to an embodiment of this application. As shown in Figure 2, the client first obtains the refresh rate of its display and sends it to the server so that the server can adjust its rendering frame rate based on the display refresh rate. The server determines its rendering frame rate based on the display refresh rate, then renders screen data within the server's virtual display refresh rate (the server's rendering frame rate range), and then performs encoding processing to generate encoded data. Simultaneously, during encoding processing, the server generates a first timestamp information corresponding to each frame in the cloud gaming screen data. The server then sends the encoded data and the corresponding first timestamp information to the client.

[0089] The client is equipped with G-Sync / FreeSync enabled. After the streaming starts, it receives the encoded data and first timestamp information of each frame, decodes and renders the encoded data to obtain decoded data, and then stores the decoded data in the client's frame buffer. At the same time, during the decoding process, it generates the second timestamp information corresponding to each frame in the cloud gaming screen data. Based on the first and second timestamp information, it determines the delay of each frame and dynamically adjusts the frame rate adjustment ratio for the next second based on the delay of each frame. This is then sent to the server so that the server can modify the upper limit of the server's rendering frame rate based on the dynamically adjusted frame rate adjustment ratio in the next second. This ensures that after all frames are transmitted within each second, the client's rendering frame rate is lower than the display refresh rate.

[0090] Finally, the client enables G-Sync / FreeSync. G-Sync / FreeSync adjusts the monitor's refresh rate in real-time to adapt to changes in the rendered frame rate after decoding, using a client-side rendering frame rate lower than the monitor's refresh rate as a baseline when generating decoded data for each frame. This ensures that the monitor's refresh rate for each frame drops to match the rendered frame rate. Furthermore, the solution provided in this embodiment reduces latency and avoids a significant drop in frame rate because it eliminates the need for the graphics card to wait for the monitor to refresh.

[0091] This application provides a cloud gaming screen processing method, apparatus, electronic device, and storage medium, applied to a system including a server and a client. The method includes: the client sending its display refresh rate to the server; the server determining its rendering frame rate based on the display refresh rate, wherein the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate; the server rendering and encoding cloud gaming screen data based on the server's rendering frame rate to generate encoded data which is then transmitted to the client; the client receiving the encoded data and decoding and rendering it to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate; and the client adjusting its display refresh rate to match the rendering frame rate of the decoded data. This application, by adjusting the server's rendering frame rate according to the client's display refresh rate, provides effective upward redundancy for the data transmission frame rate, ensuring that the client's decoded frame rate remains less than or equal to the display refresh rate, thereby effectively eliminating screen tearing issues in cloud gaming by adjusting the display refresh rate.

[0092] Based on the same inventive concept, this application discloses a cloud gaming screen processing system. The cloud gaming screen processing system is used to execute the cloud gaming screen processing method in this application embodiment. The system includes a server and a client. The client is used to send the display refresh rate to the server. The server is used to determine its rendering frame rate based on the display refresh rate, wherein the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate. The server then renders and encodes the cloud gaming screen data based on the server's rendering frame rate, generating encoded data which is then transmitted to the client. The client is also used to receive the encoded data and decode and render it to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate. Finally, the client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

[0093] Based on the same inventive concept, this application discloses a cloud gaming screen processing device. Figure 5 shows a schematic diagram of the structure of a cloud gaming screen processing device according to an embodiment of this application. As shown in Figure 5, the cloud gaming screen processing device includes:

[0094] The first sending module is used by the client to send the display refresh rate to the server.

[0095] The frame rate determination module is used by the server to determine the server's rendering frame rate based on the monitor refresh rate. The upper limit of the server's rendering frame rate is less than or equal to the monitor refresh rate.

[0096] The encoding module is used by the server to render and encode cloud gaming screen data based on the server's rendering frame rate, and generate encoded data to be transmitted to the client.

[0097] The decoding module is used by the client to receive encoded data and decode and render the encoded data to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate.

[0098] The adaptive synchronization module is used by the client to adjust the display refresh rate to match the rendering frame rate of the decoded data.

[0099] In some embodiments of this application, the frame rate determination module includes:

[0100] The frame rate adjustment ratio submodule is used by the client to generate the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and send it to the server.

[0101] The frame rate determination submodule is used by the server to receive the frame rate adjustment ratio for the current second sent by the client, and to adjust the display refresh rate downward based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

[0102] In some embodiments of this application, the frame rate determination submodule includes:

[0103] The latency difference ratio determination submodule is used by the client to obtain the latency difference ratio of all frames in the previous second based on the decoded data obtained in the previous second.

[0104] The first perturbation frame submodule is used to determine the first perturbation frame in the previous second based on the delay difference ratio of all frames in the previous second. The first perturbation frame is a frame whose delay difference ratio is greater than or equal to a first threshold. The first threshold is equal to the frame rate adjustment ratio corresponding to the previous second.

[0105] The first frame rate adjustment submodule is used to increase the frame rate adjustment ratio corresponding to the previous second according to the first preset value when the proportion of the first disturbance frame in the previous second to all frames in the previous second is greater than or equal to the first preset proportion, so as to obtain the frame rate adjustment ratio of the current second.

[0106] In some embodiments of this application, the frame rate determination submodule further includes:

[0107] The second perturbation frame submodule is used to determine the second perturbation frame in the previous second based on the delay difference ratio of all frames in the previous second. The second perturbation frame is a frame whose delay difference ratio is less than or equal to a second threshold. The second threshold is the difference between the frame rate adjustment ratio corresponding to the previous second and a first preset value.

[0108] The second frame rate adjustment submodule is used to reduce the frame rate adjustment ratio corresponding to the previous second according to the first preset value when the proportion of the second disturbance frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion, so as to obtain the frame rate adjustment ratio of the current second.

[0109] In some embodiments of this application, the frame rate determination submodule further includes:

[0110] The third frame rate adjustment submodule is used to take the frame rate adjustment ratio corresponding to the previous second as the frame rate adjustment ratio of the current second when the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset ratio, and / or the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset ratio.

[0111] In some embodiments of this application, the delay difference ratio determination submodule includes:

[0112] The delay unit is used by the client to obtain the delay of each frame in the decoded data;

[0113] The average delay unit is used by the client to obtain the delays of a preset number of frames before the target frame, and to determine the average delay based on the delays of the preset number of frames. The target frame is any frame in the decoded data.

[0114] The latency difference ratio unit is used by the client to determine the latency difference ratio of the target frame based on the latency of the target frame and the average latency.

[0115] In some embodiments of this application, the delay unit includes:

[0116] The first timestamp information subunit is used to generate the first timestamp information corresponding to each frame in the cloud game screen data and send it to the client when the server encodes the cloud game screen data.

[0117] The second timestamp information subunit is used by the client to generate the second timestamp information corresponding to each frame in the decoded data when the client decodes the encoded data.

[0118] The delay subunit is used by the client to use the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

[0119] Based on the same inventive concept, this application discloses an electronic device. Figure 6 shows a schematic diagram of an electronic device proposed in this application. As shown in Figure 6, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus. The memory 110 stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set can be loaded and run on the processor 120 to implement the steps in the cloud gaming screen processing method disclosed in this application.

[0120] Based on the same inventive concept, embodiments of this application disclose a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set thereon. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the steps in the cloud gaming screen processing method disclosed in embodiments of this application.

[0121] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0122] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal equipment to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal equipment, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0125] Although some embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including some embodiments of the present application as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0126] Finally, throughout this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0127] The above provides a detailed description of a cloud gaming screen processing method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A cloud game picture processing method, characterized in that, Applied to systems including both server and client components, the method includes: The client sends the display refresh rate to the server. The server determines its rendering frame rate based on the display refresh rate, and the upper limit of the server's rendering frame rate is less than or equal to the display refresh rate. The server renders and encodes the cloud gaming screen data based on the server's rendering frame rate, and generates encoded data which is then transmitted to the client. The client receives the encoded data and performs decoding and rendering processing on the encoded data to obtain decoded data. The rendering frame rate of the decoded data is less than or equal to the display refresh rate. The client adjusts the display refresh rate to match the rendering frame rate of the decoded data.

2. The cloud game picture processing method of claim 1, wherein, The server determines its rendering frame rate based on the display refresh rate, including: The client generates the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second and sends it to the server. The server receives the frame rate adjustment ratio for the current second from the client, and adjusts the display refresh rate downward based on the frame rate adjustment ratio for the current second to obtain the server's rendering frame rate for the current second.

3. The cloud game picture processing method of claim 2, characterized in that, The client generates the frame rate adjustment ratio for the current second based on the decoded data obtained in the previous second, including: The client obtains the latency difference ratio of all frames in the previous second based on the decoded data obtained in the previous second. Based on the latency difference ratio of all frames in the previous second, the first perturbation frame in the previous second is determined. The first perturbation frame is a frame whose latency difference ratio is greater than or equal to a first threshold. The first threshold is equal to the frame rate adjustment ratio corresponding to the previous second. If the proportion of the first perturbation frame in the previous second to all frames in the previous second is greater than or equal to a first preset proportion, the frame rate adjustment proportion corresponding to the previous second is increased according to the first preset value to obtain the frame rate adjustment proportion for the current second.

4. The cloud game picture processing method of claim 3, characterized in that, After obtaining the latency difference ratio of all frames in the previous second, the method further includes: Based on the latency difference ratio of all frames in the previous second, a second perturbation frame in the previous second is determined. The second perturbation frame is a frame whose latency difference ratio is less than or equal to a second threshold. The second threshold is the difference between the frame rate adjustment ratio corresponding to the previous second and the first preset value. If the proportion of the second perturbation frame in the previous second to all frames in the previous second is greater than or equal to the second preset proportion, the frame rate adjustment proportion corresponding to the previous second is reduced according to the first preset value to obtain the frame rate adjustment proportion of the current second.

5. The cloud game picture processing method of claim 4, characterized in that, The method further includes: If the proportion of the first perturbation frame in the previous second to all frames in the previous second is less than the first preset proportion, and / or if the proportion of the second perturbation frame in the previous second to all frames in the previous second is less than the second preset proportion, the frame rate adjustment proportion corresponding to the previous second shall be used as the frame rate adjustment proportion for the current second.

6. The cloud game picture processing method of claim 3, wherein, The latency difference ratio of each frame in the decoded data is obtained as follows: The client obtains the delay of each frame in the decoded data; The client obtains the delays corresponding to a preset number of frames before the target frame, and determines the average delay based on the delays corresponding to the preset number of frames. The target frame is any frame in the decoded data. The client determines the latency difference ratio of the target frame based on the latency of the target frame and the average latency.

7. The cloud game picture processing method of claim 6, characterized in that, The client obtains the delay of each frame in the decoded data, including: When the server encodes the cloud gaming screen data, the server generates a first timestamp information corresponding to each frame in the cloud gaming screen data and sends it to the client. When the client decodes the encoded data, the client generates a second timestamp information corresponding to each frame in the decoded data; The client uses the difference between the first timestamp information and the second timestamp information corresponding to each frame as the delay of each frame in the decoded data.

8. A cloud game picture processing apparatus, characterized by comprising: The device includes: The first sending module is used for the client to send the display refresh rate to the server; A frame rate determination module is used for the server to determine the rendering frame rate of the server based on the display refresh rate, wherein the upper limit of the rendering frame rate of the server is less than or equal to the display refresh rate. The encoding module is used by the server to render and encode the cloud gaming screen data based on the server's rendering frame rate, and generate encoded data to be transmitted to the client. A decoding module is used for the client to receive the encoded data and perform decoding and rendering processing on the encoded data to obtain decoded data, wherein the rendering frame rate of the decoded data is less than or equal to the display refresh rate; An adaptive synchronization module is used by the client to adjust the display refresh rate to match the rendering frame rate of the decoded data.

9. An electronic device, comprising: The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the cloud gaming screen processing method according to any one of claims 1-7.

10. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the cloud gaming screen processing method according to any one of claims 1-7.

Images